Rubric evaluation of pediatric emergency medicine fellows

Who Trains the Trainers?: Development of a Faculty Bootcamp for Pediatric Emergency Medicine Resuscitation Procedures

OBJECTIVES

Attending physicians in pediatric emergency medicine (PEM) must be able to perform lifesaving procedures, yet guidelines for maintaining procedural competency do not exist. We implemented a biannual 2-hour "bootcamp" designed to help PEM faculty maintain procedural competency.

METHODS

A survey-based needs assessment was used to create a set of goals and objectives for the session and determine which procedural skills to include. Sessions of 4 simulated skills were held twice a year and limited to 12 faculty. Post-bootcamp evaluations were administered at the 1-year and 6-year marks to evaluate the usefulness of the training.

RESULTS

Twenty-eight of our 55 current faculty members (50%) responded to the 6-year follow-up evaluation. Overall, the bootcamp was felt to be beneficial, with 64% of faculty rating it "great" (5) or "highly useful" (6) on a 6-point Likert scale. The majority of participants also rated the airway, vascular access, and cardiopulmonary resuscitation/defibrillator training favorably. Faculty who later had the opportunity to perform specific resuscitation procedures clinically felt that the circulation (cardiopulmonary resuscitation/defibrillator) and airway stations contributed to the success of their procedure performance.

CONCLUSIONS

The clinical setting alone may be insufficient in maintaining procedural competency in lifesaving skills in PEM. Giving faculty the opportunity to practice these skills is feasible and can be effective in increasing confidence. Future training sessions should aim toward practicing to a defined mastery level.

The Arthroscopic Surgical Skill Evaluation Tool (ASSET)

BACKGROUND

Surgeries employing arthroscopic techniques are among the most commonly performed in orthopaedic clinical practice; however, valid and reliable methods of assessing the arthroscopic skill of orthopaedic surgeons are lacking.

HYPOTHESIS

The Arthroscopic Surgery Skill Evaluation Tool (ASSET) will demonstrate content validity, concurrent criterion-oriented validity, and reliability when used to assess the technical ability of surgeons performing diagnostic knee arthroscopic surgery on cadaveric specimens.

STUDY DESIGN

Cross-sectional study; Level of evidence, 3.

METHODS

Content validity was determined by a group of 7 experts using the Delphi method. Intra-articular performance of a right and left diagnostic knee arthroscopic procedure was recorded for 28 residents and 2 sports medicine fellowship-trained attending surgeons. Surgeon performance was assessed by 2 blinded raters using the ASSET. Concurrent criterion-oriented validity, interrater reliability, and test-retest reliability were evaluated.

RESULTS

Content validity: The content development group identified 8 arthroscopic skill domains to evaluate using the ASSET. Concurrent criterion-oriented validity: Significant differences in the total ASSET score (P < .05) between novice, intermediate, and advanced experience groups were identified. Interrater reliability: The ASSET scores assigned by each rater were strongly correlated (r = 0.91, P < .01), and the intraclass correlation coefficient between raters for the total ASSET score was 0.90. Test-retest reliability: There was a significant correlation between ASSET scores for both procedures attempted by each surgeon (r = 0.79, P < .01).

CONCLUSION

The ASSET appears to be a useful, valid, and reliable method for assessing surgeon performance of diagnostic knee arthroscopic surgery in cadaveric specimens. Studies are ongoing to determine its generalizability to other procedures as well as to the live operating room and other simulated environments.