Objective structured assessment of technical skill in upper extremity surgery

Orthopaedic Hand Surgical Simulation Training: A Review

In recent years, new orthopaedic surgical simulation and virtual reality (VR) training models have emerged to provide unlimited education medium to an unlimited number of trainees with no time limit, especially in response to trainee work-hour restrictions. Surgical simulators range from simple wooden boxes to animal and cadaver models to three-dimensional-printed and VR simulators. The coronavirus disease 2019 pandemic further highlighted the need for at-home learning tools for orthopaedic surgical trainees. Advancement in simulating shoulder and knee arthroscopies using VR simulators surpasses the other fields in orthopaedic surgery. Despite the high degree of precision needed to operate at a microscopic level involving vessels, nerves, and the small bones of the hand, the simulation tools have limited advancement in the field of orthopaedic hand surgery. This narrative review summarizes the status of surgical simulation and training techniques available to orthopaedic hand surgical trainees, factors affecting their application, and areas in hand surgery that still lag behind their surgical subspecialty counterparts.

Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2  = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2  = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2  = 0.83) and behavior tally (R2  = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2  = 0.40), expert consensus (R2  = 0.76), and performer experience (R2  = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2  = 0.11).

Validity Evidence for Procedure-specific Competence Assessment Tools in Orthopaedic Surgery: A Scoping Review

INTRODUCTION

Competency-based training requires frequent assessment of residents' skills to determine clinical competence. This study reviews existing literature on procedure-specific competence assessment tools in orthopaedic surgery.

METHODS

A systematic search of eight databases up to May 2023 was conducted. Two reviewers independently assessed validity evidence and educational utility of each assessment tool and evaluated studies' methodological quality.

RESULTS

Database searching identified 2,556 unique studies for title and abstract screening. Full texts of 290 studies were reviewed; 17 studies met the inclusion criteria. Bibliography review identified another five studies, totaling 22 studies examining 24 assessment tools included in the analysis. These tools assessed various orthopaedic surgery procedures within trauma, sports medicine, spine, and upper extremity. Overall validity evidence was low across all studies, and was lowest for consequences and highest for content. Methodological quality of studies was moderate. Educational utility assessment was not explicitly done for most tools.

DISCUSSION

The paucity of current procedure-specific assessment tools in orthopaedic surgery lacks the validity evidence required to be used reliably in high-stake summative assessments. Study strengths include robust methodology and use of an evidence-based validity evidence framework. Poor-quality existing evidence is a limitation and highlights the need for evidence-based tools across more subspecialties.

Assessment of Technical Competence in Distal Radius Fracture Fixation by a Volar Locking Plate: A Global Delphi Consensus Study

PURPOSE

Volar locking plate fixation of distal radius fractures is a common orthopedic procedure and should be mastered by graduating orthopedic residents. Surgical education is transitioning from a traditional time-based approach to competency-based medical education. Valid and objective assessment is essential for successful transition. The purpose of this study was to develop a comprehensive, procedure-specific assessment tool to evaluate technical competence in volar locking plate osteosynthesis of a distal radius fracture.

METHODS

International orthopedic/trauma experts involved in resident education participated as panelists in a four-round online Delphi process to reach consensus on the content of the assessment tool. Round 1 was an item-generating round, in which the panelists identified potential assessment parameters. In round 2, the panelists rated the importance of each suggested assessment parameter and reached consensus on which to include in the assessment tool. Round 3 yielded specific assessment score intervals for specific bone and fracture models and is not reported in this study. In round 4, the panelists assigned weights to the assessment parameters on a 1-10 scale to determine how each parameter should have an impact on the overall results.

RESULTS

Eighty-seven surgeons, representing 42 countries, participated in the study. Round 1 resulted in 45 assessment parameters, grouped into five procedural steps. After round 2, the number of parameters was reduced to 39. After the final round, an additional parameter was removed and weights were assigned to the remaining parameters.

CONCLUSIONS

Using a systematic methodology, a preliminary assessment tool to evaluate technical competence in distal radius fracture fixation was developed. A consensus of international experts supports the content validity of the assessment tool.

CLINICAL RELEVANCE

This assessment tool represents the first step in the evidence-based assessment essential for competency-based medical education. Before implementation, further studies exploring validity of variations of the assessment tool in different educational contexts are required.

Utilising 3D-printed ex vivo biomimetics to improve open reduction and internal fixation (ORIF) simulation training for hand fractures

Background

Surgery for hand trauma accounts for a significant proportion of the plastic surgery training curriculum. The aim of this article is to create a standardised simulation training module for hand fracture fixation on open reduction and internal fixation (ORIF) techniques for residents in order to create a standardised hand-training framework that universally hones their skill and prepares them for their first encounter in a clinical setting.

Methods

A step-ladder approach training using three-dimensional (3D)-printed ex vivo hand biomimetics was employed on a cohort of 15 plastic surgery residents (n = 15). Assessment of skills using a score system (global rating scale) was performed in the beginning and the end of the module by hand experts in our unit.

Results

The overall average score of the cohort pre- and post-assessment were 22.08/50 (44.16%) and 41.54/50 (83.08%) respectively. Significant (p < 0.01) difference of improvement of skills was noted on all trainees. All trainees confirmed that the simulated models provided in this module were akin to the patient scenario and noted that it helped them improve their skills with regards to ORIF techniques including improvement of their understanding of the 3D bone topography.

Conclusion

We demonstrate a standardised simulation training framework that employs 3D-printed ex vivo hand biomimetics proven to improve the skills of residents and which paves the way to more universal, standardised and validated training across hand surgery. This is, to our knowledge, the first standardised method of simulated training on such hand-surgical cases.Level of Evidence: Not ratable.

Contemporary Issues in the Acquisition of Orthopaedic Surgical Skills During Residency: Competency-Based Medical Education and Simulation

➤

Orthopaedic education should produce surgeons who are competent to function independently and can obtain and maintain board certification.

➤

Contemporary orthopaedic training programs exist within a fixed 5-year time frame, which may not be a perfect match for each trainee.

➤

Most modern orthopaedic residencies have not yet fully adopted objective, proficiency-based, surgical skill training methods despite nearly 2 decades of evidence supporting the use of this methodology.

➤

Competency-based medical education backed by surgical simulation rooted in proficiency-based progression has the potential to address surgical skill acquisition challenges in orthopaedic surgery.

Minimally Trained Analysts Can Perform Fast, Objective Assessment of Orthopedic Technical Skill from Fluoroscopic Images

Skill assessment in orthopedics has traditionally relied on subjective impressions from a supervising surgeon. The feedback derived from these tools may be limited by bias and other practical issues. Objective analysis of intraoperative fluoroscopic images offers an inexpensive, repeatable, and precise assessment strategy without bias. Assessors generally refrain from using the scores of images obtained throughout the operation to evaluate skill for practical reasons. A new system was designed to facilitate rapid analysis of this fluoroscopy via minimally trained analysts. Four expert and four novice analysts independently measured one objective metric for skill using both a custom analysis software and a commercial alternative. Analysts were able to measure the objective metric three times faster when using the custom software, and without a practical difference in accuracy in comparison to the expert analysts using the commercial software. These results suggest that a well-designed fluoroscopy analysis system can facilitate inexpensive, reliable, and objective assessment of surgical skills.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Surgical Skill Can be Objectively Measured From Fluoroscopic Images Using a Novel Image-based Decision Error Analysis (IDEA) Score

BACKGROUND

To advance orthopaedic surgical skills training and assessment, more rigorous and objective performance measures are needed. In hip fracture repair, the tip-apex distance is a commonly used summative performance metric with clear clinical relevance, but it does not capture the skill exercised during the process of achieving the final implant position. This study introduces and evaluates a novel Image-based Decision Error Analysis (IDEA) score that better captures performance during fluoroscopically-assisted wire navigation.

QUESTIONS/PURPOSES

(1) Can wire navigation skill be objectively measured from a sequence of fluoroscopic images? (2) Are skill behaviors observed in a simulated environment also exhibited in the operating room? Additionally, we sought to define an objective skill metric that demonstrates improvement associated with accumulated surgical experience.

METHODS

Performance was evaluated both on a hip fracture wire navigation simulator and in the operating room during actual fracture surgery. After examining fluoroscopic image sequences from 176 consecutive simulator trials (performed by 58 first-year orthopaedic residents) and 21 consecutive surgical procedures (performed by 19 different orthopaedic residents and one attending orthopaedic surgeon), three main categories of erroneous skill behavior were identified: off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling. Skill behaviors were measured by comparing wire adjustments made between consecutive images against the goal of targeting the apex of the femoral head as part of our new IDEA scoring methodology. Decision error metrics (frequency, magnitude) were correlated with other measures (image count and tip-apex distance) to characterize factors related to surgical performance on both the simulator and in the operating room. An IDEA composite score integrating decision errors (off-target wire adjustments, out-of-plane wire adjustments, and off-target drilling) and the final tip-apex distance to produce a single metric of overall performance was created and compared with the number of hip wire navigation cases previously completed (such as surgeon experience levels).

RESULTS

The IDEA methodology objectively analyzed 37,000 images from the simulator and 688 images from the operating room. The number of decision errors (7 ± 5 in the operating room and 4 ± 3 on the simulator) correlated with fluoroscopic image count (33 ± 14 in the operating room and 20 ± 11 on the simulator) in both the simulator and operating room environments (R2 = 0.76; p < 0.001 and R2 = 0.71; p < 0.001, respectively). Decision error counts did not correlate with the tip-apex distance (16 ± 4 mm in the operating room and 12 ± 5 mm on the simulator) for either the simulator or the operating room (R2 = 0.08; p = 0.15 and R2 = 0.03; p = 0.47, respectively), indicating that the tip-apex distance is independent of decision errors. The IDEA composite score correlated with surgical experience (R2 = 0.66; p < 0.001).

CONCLUSION

The fluoroscopic images obtained in the course of placing a guide wire contain a rich amount of information related to surgical skill. This points the way to an objective measure of skill that also has potential as an educational tool for residents. Future studies should expand this analysis to the wide variety of procedures that rely on fluoroscopic images.

CLINICAL RELEVANCE

This study has shown how resident skill development can be objectively assessed from fluoroscopic image sequences. The IDEA scoring provides a basis for evaluating the competence of a resident. The score can be used to assess skill at key timepoints throughout residency, such as when rotating onto/off of a new surgical service and before performing certain procedures in the operating room, or as a tool for debriefing/providing feedback after a procedure is completed.

Utilization of Flexible-Wearable Sensors to Describe the Kinematics of Surgical Proficiency

BACKGROUND

Traditional assessment (e.g., checklists, videotaping) for surgical proficiency may lead to subjectivity and does not predict performance in the clinical setting. Hand motion analysis is evolving as an objective tool for grading technical dexterity; however, most devices accompany with technical limitations or discomfort. We purpose the use of flexible wearable sensors to evaluate the kinematics of surgical proficiency.

METHODS

Surgeons were recruited and performed a vascular anastomosis task in a single institution. A modified objective structured assessment of technical skills (mOSATS) was used for technical qualification. Flexible wearable sensors (BioStamp RCTM, mc10 Inc., Lexington, MA) were placed on the dorsum of the dominant hand (DH) and nondominant hand (nDH) to measure kinematic parameters: path length (T_path), mean (V_mean) and peak (V_peak) velocity, number of hand movements (N_move), ratio of DH to nDH movements (rMov), and time of task (tTask) and further compared with the mOSATS score.

RESULTS

Participants were categorized as experts (n = 12) and novices (n = 8) based on a cutoff mean mOSATS score. Significant differences for tTask (P = 0.02), rMov (P = 0.07), DH T_path (P = 0.04), V_mean (P = 0.07), V_peak (P = 0.04), and nDH N_move (P = 0.02) were in favor of the experts. Overall, mOSATS had significant correlation with tTask (r = -0.69, P = 0.001), N_move of DH (r = -0.44, P = 0.047) and nDH (r = -0.66, P = 0.001), and rMov (r = 0.52, P = 0.017).

CONCLUSIONS

Hand motion analysis evaluated by flexible wearable sensors is feasible and informative. Experts utilize coordinated two-handed motion, whereas novices perform one-handed tasks in a hastily jerky manner. These tendencies create opportunity for improvement in surgical proficiency among trainees.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Surgical Games: A Simulation-Based Structured Assessment of Orthopedic Surgery Resident Technical Skill

BACKGROUND

Simulation-based education and objective evaluation of surgical skill have been incorporated into many surgical training programs. We describe the development and implementation of a timed, multitask, station-based Surgical Games to evaluate orthopedic resident surgical skills.

METHODS

Participants in the study were postgraduate-year 2 to 5 orthopedic surgery residents from a single institution. Residents completed 4-timed simulated tasks: cadaveric carpal tunnel release (CTR), Sawbones model of total knee arthroplasty (TKA), Sawbones model of ankle fracture open reduction internal fixation (ORIF), and knee arthroscopy simulator (KAS) of removal of loose body. Evaluations were performed using standardized score sheets by attending surgeons. Resident performance was analyzed by postgraduate-year and number of weeks of prior task-related residency training.

RESULTS

A total of 32 residents were assessed at the 4 stations. Total scores were significantly different for CTR (p = 0.006), TKA (p = 0.05), and the KAS (p = 0.004) by year of training, but not for the ankle ORIF task. Residents with more task-specific experience performed significantly better on the KAS (p < 0.001), TKA (p = 0.002), and CTR (p = 0.02) tasks but not on the ankle ORIF task (p = 0.1). Overall, residents rated the exercise valuable to their education with mean scores of 3.9 ± 0.54 on a 5-point Likert Scale.

CONCLUSIONS

This Surgical Games exercise provided an objective evaluation of surgical skill that correlated with year-in-training and prior experience in skill-specific rotations for the KAS, TKA, and CTR tasks. This surgical skills assessment provided an opportunity for effective structured feedback and identification of areas for improvement.

The Oral and Maxillofacial Objective Structured Assessment of Technical Skills (OMOSATS) examination: a pilot study

The aim of this study was to develop a simulated technical skills examination specific to oral and maxillofacial surgery, assessing the validity of the test and its psychometric properties and obtaining feedback on the examination from both resident trainees and expert evaluators. The Oral and Maxillofacial Objective Structured Assessment of Technical Skills (OMOSATS) examination is a simulated procedure-based eight-station technical skills examination. Fourteen oral and maxillofacial surgery (OMFS) residents of various levels participated in 12-minute surgical tasks and were evaluated by blinded expert OMFS surgeons. The tasks included suturing, tracheostomy, arthroscopy, dental implant placement, plate bending and lag screw application, microvascular anastomosis, and skin grafting. These tasks reflect a broad range of technical milestones necessary for the practice of OMFS. Tasks were evaluated using a validated global rating scale and task-specific checklists. The results of the pilot study indicated strong face and construct validity for seven of the eight OMOSATS stations in discerning the level of training based on technical skill demonstrated at each of the eight surgical stations. Additionally, through one-way ANOVA, P-values were observed to confirm the level of discernment. The OMOSATS examination is a novel way to assess the technical skills of OMFS residents. This type of performance-based assessment has been used in other surgical domains for many years. There is strong evidence to support the use of technical skills examinations to help evaluate, teach, and provide surgeon educators with training outcomes for our OMFS trainees.

Orthopaedic Education During the COVID-19 Pandemic

The COVID-19 global pandemic presents a challenge to orthopaedic education. Around the world, including in the United States, elective surgeries are being deferred and orthopaedic residents and fellows are being asked to make drastic changes to their daily routines. In the midst of these changes are unique opportunities for resident/fellow growth and development. Educational tools in the form of web-based learning, surgical simulators, and basic competency tests may serve an important role. Challenges are inevitable, but appropriate preparation may help programs ensure continued resident growth, development, and well-being while maintaining high-quality patient care.

Improved Complex Skill Acquisition by Immersive Virtual Reality Training: A Randomized Controlled Trial

BACKGROUND

There has been limited literature on immersive virtual reality (VR) simulation in orthopaedic education. The purpose of this multicenter, blinded, randomized controlled trial was to determine the validity and efficacy of immersive VR training in orthopaedic resident education.

METHODS

Nineteen senior orthopaedic residents (resident group) and 7 consultant shoulder arthroplasty surgeons (expert group) participated in the trial comparing immersive VR with traditional learning using a technical journal article as a control. The examined task focused on achieving optimal glenoid exposure. Participants completed demographic questionnaires, knowledge tests, and a glenoid exposure on fresh-frozen cadavers while being examined by blinded shoulder arthroplasty surgeons. Training superiority was determined by the outcome measures of the Objective Structured Assessment of Technical Skills (OSATS) score, a developed laboratory metric, verbal answers, and time to task completion.

RESULTS

Immersive VR had greater realism and was superior in teaching glenoid exposure than the control (p = 0.01). The expert group outperformed the resident group on knowledge testing (p = 0.04). The immersive VR group completed the learning activity and knowledge tests significantly faster (p < 0.001) at a mean time (and standard deviation) of 11 ± 3 minutes than the control group at 20 ± 4 minutes, performing 3 to 5 VR repeats for a reduction in learning time of 570%. The immersive VR group completed the glenoid exposure significantly faster (p = 0.04) at a mean time of 14 ± 7 minutes than the control group at 21 ± 6 minutes, with superior OSATS instrument handling scores (p = 0.03). The immersive VR group scored equivalently in surprise verbal scores (p = 0.85) and written knowledge scores (p = 1.0).

CONCLUSIONS

Immersive VR demonstrated substantially improved translational technical and nontechnical skills acquisition over traditional learning in senior orthopaedic residents. Additionally, the results demonstrate the face, content, construct, and transfer validity for immersive VR.

CLINICAL RELEVANCE

This adequately powered, randomized controlled trial demonstrated how an immersive VR system can efficiently (570%) teach a complex surgical procedure and also demonstrate improved translational skill and knowledge acquisition when compared with a traditional learning method.

Development of a certification examination for orthopedic sports medicine fellows

Background

The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship.

Methods

Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows’ performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale.

Results

Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair.

Conclusion

There was a disparity between fellows’ performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows’ competence.

Did a Minimum Case Requirement Improve Resident Surgical Volume for Closed Wrist and Forearm Fracture Treatment in Orthopedic Surgery?

OBJECTIVE

The purpose of this study is to determine whether the 2013 implementation of ACGME minimum case requirements was associated with increased documented case volume of closed manipulation of forearm and wrist fractures (CMFWF) for graduating orthopedic surgery residents.

DESIGN

We reviewed ACGME case log data for CMFWF among graduating orthopedic surgery residents from 2007 to 2016. Annual national mean, and median number of CMFWF performed by residents in the 10th, 30th, 50th, and 90th case volume percentile were evaluated. Preminimum (2007-2010) data was compared to postminimum (2013-2016) values to assess the impact of ACGME minimum requirements on resident case volume.

SETTING

Review of publically available ACMGE Orthopedic Surgery Residency Program case log data.

PARTICIPANTS

ACGME case log data for orthopedic surgery residents graduating between 2007 and 2016.

RESULTS

National mean number of CMFWF increased significantly pre- to postminimum requirement (30.0 ± 2.84 to 45.0 ± 3.36, p < 0.001). Between 2010 and 2016 there was a 1100%, 300%, 83%, and 9% increase in the median number of CMFWF within the 10th, 30th, 50th, and 90th percentiles, respectively.

CONCLUSIONS

ACGME's 2013 case minimum requirement corresponded to an increase in case counts for CMFWF; the greatest increase occurred in residents below the 50th percentile of case volume. Implementation of case minimum requirements may allow for more accurate depiction of resident experience and program strengths with regards to procedural exposure. However, the current case log system measures only case quantity, which may inaccurately depict mastery of given procedures. Future work should focus not only on improving case counts in underperforming residents and training sites, but also on refining metrics that ensure accurate assessment of resident skill for essential orthopedic procedures prior to graduation.

Resident Surgical Skills Web-Based Evaluation: A Comparison of 2 Assessment Tools

BACKGROUND

Evaluation of surgical skill competency is necessary as graduate medical education moves toward a competency-based curriculum. This study by the American Board of Orthopaedic Surgery (ABOS) and the Council of Orthopaedic Residency Directors (CORD) compares 2 web-based evaluation tools that assess the level of autonomy that is demonstrated by residents during surgical procedures in the operating room as measured by faculty.

METHODS

Two hundred and ninety-four residents from 16 orthopaedic surgery residency programs were evaluated by 370 faculty using 2 web-based evaluation tools in a crossover design in which residents requested faculty review of their surgical skills before starting a case. One thousand, one hundred and fifty Ottawa Surgical Competency Operating Room Evaluation (O-Score) assessments, which included a 9-question evaluation of 8 steps of the surgical procedure, were compared with 1,186 P-score evaluations, which included a single-question summative evaluation. Twenty-five different surgical procedures were evaluated.

RESULTS

There were no significant differences in rates of resident requests or faculty completion of the 2 scores. The most common surgical procedures that were assessed were total knee arthroplasty (n = 254, 11%), carpal tunnel release (n = 191, 8%), open reduction and internal fixation (ORIF) of stable hip fractures (n = 170, 7%), ORIF of simple ankle fractures (n = 169, 7%), and total hip arthroplasty (n = 166, 7%). Both instruments disclosed significant differences in competency among entry, intermediate, and advanced-level residents. The findings support the construct validity of the evaluation method. The survey results indicated that >70% of the faculty were confident that use of either the P-score or the O-score allowed them to distinguish a resident who can perform the surgery independently from one who needs additional training.

CONCLUSIONS

This research has led to the modification of the O-score and the P-score into a combined OP-score instrument. The ABOS envisions that the OP-score instrument can be used with an expanded number of surgical procedures as a required element of residency training in the near future.

CLINICAL RELEVANCE

This study allows the profession of orthopaedic surgery education to take a leadership role in the measurement of competence for surgical skills for orthopaedic surgeons in residency training, an important clinically relevant topic to the practice of orthopaedic surgery.

Establishing Validity Evidence for an Operative Performance Rating System for Plastic Surgery Residents

OBJECTIVE

The aim of this study was to describe an operative performance rating system for plastic surgery residents and provide validity evidence for the instrument.

METHODS

Three plastic surgery residents (PGY levels 1, 5, and 6) from Southern Illinois University School of Medicine (SIUSOM) performed a carpal tunnel release with audio video recording. The 3 videos were reviewed by 8 expert hand surgeons and 3 SIUSOM faculty using the operative performance rating system instrument to assess resident operative performance. Validity evidence including content, internal structure, and relationship to other variables was collected.

RESULTS

Inter-rater reliability was consistently fair to moderate (weighted Cohen's Kappa 0.44-0.84 for experts, 0.24-0.55 for SIUSOM raters), and all assessment items were highly correlated (Cronbach's alpha of 0.9867). Local SIUSOM faculty routinely demonstrated higher overall scores for PGY 1 and PGY 6 residents compared to expert raters.

CONCLUSIONS

Although limited by small numbers, this pilot study suggests that potential bias based upon PGY year, identity, and performance history may exist and independent assessment by unbiased raters or comparison to national operative norms may be valuable. Our study provides baseline validity evidence for a resident operative performance assessment tool that can be integrated into practice in plastic surgery training programs.

Examination of Skill Acquisition and Grader Bias in a Distal Radius Fracture Fixation Model

OBJECTIVES

Primary: Assess the ability of faculty graders to predict the objectively measured strength of distal radius fracture fixation. Secondary: Compare resident skill variation and retention related to other knowable training data.

DESIGN

Residents were allowed 60 minutes to stabilize a standardized distal radius fracture using an assigned fixed-angle volar plate. Faculty observed and subjectively graded the residents without providing real-time feedback. Objective biomechanical evaluation (construct strength and stiffness) was compared to subjective grades. Resident-specific characteristics (sex, PGY, and ACGME case log) were also used to compare the objective data.

SETTING

A simulated operating room in our laboratory.

PARTICIPANTS

Post-graduate year 2, 3, 4, and 5 orthopedic residents.

RESULTS

Primary: Faculty were not successful at predicting objectively measured fixation, and their subjective scoring suggests confirmation bias as PGY increased. Secondary: Resident year-in-training alone did not predict objective measures (p = 0.53), but was predictive of subjective scores (p < 0.001). Skills learned were not always retained, as 29% of residents objectively failed subsequent to passing. Notably, resident-reported case-specific experience alone was inversely correlated with objective fixation strength.

CONCLUSIONS

This testing model enabled the collection of objective and subjective resident skill scores. Faculty graders did not routinely predict objective measures, and their subjective assessment appears biased related to PGY. Also, in vivo case volume alone does not predict objective results. Familiar faculty teaching consistency, and resident grading by external faculty unfamiliar with tested residents, might alter these results.

Psychomotor Testing for Orthopedic Residency Applicants: A Pilot Study

OBJECTIVE

The purpose of this study was to develop an objective motor skills testing system to aid in the evaluation of potential orthopedic residents.

DESIGN

Participants attempted a battery of 5 motor skills tests (4 novel tests and the Grooved Pegboard [GPT] Test) in one 10-minute session. A percentile-based scoring system was created for each test based on raw scores. One-way analysis of variance was used to compare testing scores among 3 cohorts. Each novel test and overall scores were compared with GPT scores as a relative measure of validity.

SETTING

The 2015 orthopedic surgery residency interview season at an academic institution.

PARTICIPANTS

Thirty orthopedic residents and 72 nonresidents (15 community volunteers and 57 orthopedic surgery residency applicants).

RESULTS

Overall, residents performed better than nonresidents (p < 0.0001) and applicants performed worse than residents or volunteers (p < 0.0001). There were positive correlations between the GPT score and overall battery score (r = 0.63), screw and nut test (r = 0.40), and mimic a structure test (r = 0.26). The fracture reduction test and drilling test scores did not correlate to performance on the GPT.

CONCLUSIONS

Psychomotor testing for surgical applicants is an area in need of study. This investigation successfully piloted a novel battery of tests, which is easily reproducible and thus may be feasible for use in the orthopedic surgery residency interview setting. Longitudinal evaluation is required to explore correlation with future operative skill.

Reliability and Validity of 3 Methods of Assessing Orthopedic Resident Skill in Shoulder Surgery

OBJECTIVE

Traditional measures for evaluating resident surgical technical skills (e.g., case logs) assess operative volume but not level of surgical proficiency. Our goal was to compare the reliability and validity of 3 tools for measuring surgical skill among orthopedic residents when performing 3 open surgical approaches to the shoulder.

METHODS

A total of 23 residents at different stages of their surgical training were tested for technical skill pertaining to 3 shoulder surgical approaches using the following measures: Objective Structured Assessment of Technical Skills (OSATS) checklists, the Global Rating Scale (GRS), and a final pass/fail assessment determined by 3 upper extremity surgeons. Adverse events were recorded. The Cronbach α coefficient was used to assess reliability of the OSATS checklists and GRS scores. Interrater reliability was calculated with intraclass correlation coefficients. Correlations among OSATS checklist scores, GRS scores, and pass/fail assessment were calculated with Spearman ρ. Validity of OSATS checklists was determined using analysis of variance with postgraduate year (PGY) as a between-subjects factor. Significance was set at p < 0.05 for all tests.

RESULTS

Criterion validity was shown between the OSATS checklists and GRS for the 3 open shoulder approaches. Checklist scores showed superior interrater reliability compared with GRS and subjective pass/fail measurements. GRS scores were positively correlated across training years. The incidence of adverse events was significantly higher among PGY-1 and PGY-2 residents compared with more experienced residents.

CONCLUSION

OSATS checklists are a valid and reliable assessment of technical skills across 3 surgical shoulder approaches. However, checklist scores do not measure quality of technique. Documenting adverse events is necessary to assess quality of technique and ultimate pass/fail status. Multiple methods of assessing surgical skill should be considered when evaluating orthopedic resident surgical performance.

What are the important manoeuvres for beginners to minimize surgical time in primary total knee arthroplasty?

PURPOSE

It was hypothesized that surgical time of beginners would be much longer than that of experts. Our purpose was to investigate and clarify the important manoeuvres for beginners to minimize surgical time in primary total knee arthroplasty (TKA) as a multicentre study.

METHODS

A total of 300 knees in 248 patients (averaged 74.6 years) were enrolled. All TKAs were done using the same instruments and the same measured resection technique at 14 facilities by 25 orthopaedic surgeons. Surgeons were divided into three surgeon groups (four experts, nine medium-volume surgeons and 12 beginners). The surgical technique was divided into five phases. Detailed surgical time and ratio of the time in each phase to overall surgical time were recorded and compared among the groups in each phase. A total of 62, 119, and 119 TKAs were done by beginners, medium-volume surgeons, and experts, respectively.

RESULTS

Significant differences in surgical time among the groups were seen in each phase. Concerning the ratio of the time, experts and medium-volume surgeons seemed cautious in fixation of the permanent component compared to other phases. Interestingly, even in ratio, beginners and medium-volume surgeons took more time in exposure of soft tissue compared to experts. (0.14 in beginners, 0.13 in medium-volume surgeons, 0.11 in experts, P < 0.05).

CONCLUSION

Beginners took more time in exposure and closure of soft tissue compared to experts. Improvement in basic technique is essential to minimize surgical time among beginners. First of all, surgical instructors should teach basic techniques in primary TKA for beginners.

LEVEL OF EVIDENCE

Therapeutic studies, Level IV.

Objective Structured Assessments of Technical Skills (OSATS) Does Not Assess the Quality of the Surgical Result Effectively

BACKGROUND

Performance assessment in skills training is ideally based on objective, reliable, and clinically relevant indicators of success. The Objective Structured Assessment of Technical Skill (OSATS) is a reliable and valid tool that has been increasingly used in orthopaedic skills training. It uses a global rating approach to structure expert evaluation of technical skills with the experts working from a list of operative competencies that are each rated on a 5-point Likert scale anchored by behavioral descriptors. Given the observational nature of its scoring, the OSATS might not effectively assess the quality of surgical results.

QUESTIONS/PURPOSES

(1) Does OSATS scoring in an intraarticular fracture reduction training exercise correlate with the quality of the reduction? (2) Does OSATS scoring in a cadaveric extraarticular fracture fixation exercise correlate with the mechanical integrity of the fixation?

METHODS

Orthopaedic residents at the University of Iowa (six postgraduate year [PGY]-1s) and at the University of Minnesota (seven PGY-1s and eight PGY-2s) undertook a skills training exercise that involved reducing a simulated intraarticular fracture under fluoroscopic guidance. Iowa residents participated three times during 1 month, and Minnesota residents participated twice with 1 month between their two sessions. A fellowship-trained orthopaedic traumatologist rated each performance using a modified OSATS scoring scheme. The quality of the articular reduction obtained was then directly measured. Regression analysis was performed between OSATS scores and two metrics of articular reduction quality: articular surface deviation and estimated contact stress. Another skills training exercise involved fixing a simulated distal radius fracture in a cadaveric specimen. Thirty residents, distributed across four PGY classes (PGY-2 and PGY-3, n = 8 each; PGY-4 and PGY-5, n = 7 each), simultaneously completed the exercise at individual stations. One of three faculty hand surgeons independently scored each performance using a validated OSATS scoring system. The mechanical integrity of each fixation construct was then assessed in a materials testing machine. Regression analysis was performed between OSATS scores and two metrics of fixation integrity: stiffness and failure load.

RESULTS

In the intraarticular fracture model, OSATS scores did not correlate with articular reduction quality (maximum surface deviations: R = 0.17, p = 0.25; maximum contact stress: R = 0.22, p = 0.13). Similarly in the cadaveric extraarticular fracture model, OSATS scores did not correlate with the integrity of the mechanical fixation (stiffness: R = 0.10, p = 0.60; failure load: R = 0.30, p = 0.10).

CONCLUSIONS

OSATS scoring methods do not effectively assess the quality of the surgical result. Efforts must be made to incorporate assessment metrics that reflect the quality of the surgical result.

CLINICAL RELEVANCE

New objective, reliable, and clinically relevant measures of the quality of the surgical result obtained by a trainee are urgently needed. For intraarticular fracture reduction and extraarticular fracture fixation, direct physical measurement of reduction quality and of mechanical integrity of fixation, respectively, meet this need.

Trends in Arthroscopic Procedures Performed During Orthopaedic Residency: An Analysis of Accreditation Council for Graduate Medical Education Case Log Data

PURPOSE

To analyze orthopaedic resident case log data to report temporal trends in performing arthroscopic procedures and to assess variability in arthroscopic case volume among residents.

METHODS

Accreditation Council for Graduate Medical Education orthopaedic surgery resident case logs were reviewed from 2007 to 2013. The mean number of wrist, elbow, shoulder, knee, and ankle arthroscopic procedures performed by graduating residents was analyzed. The median number of arthroscopic procedures reported by the 70th and 30th percentiles of graduating residents (by arthroscopic case volume) was also recorded. Temporal trends were assessed using a linear regression model.

RESULTS

From 2007 to 2013, there were significant increases in the mean number of wrist (5.8 to 6.3; P = .038), elbow (2.5 to 3.2; P < .001), shoulder (93.5 to 133.8; P < .001), knee (170.6 to 185.1; P = .011), and ankle (5.8 to 7.4; P < .001) arthroscopies performed per resident. Residents in the 70th percentile of caseload performed significantly more wrist (7 v 2; P < .001), elbow (3.3 v 1; P < .001), shoulder (134.1 v 70.6; P < .001), knee (205 v 128.7; P < .001), and ankle (7.9 v 2.9; P < .001) arthroscopies than residents in the 30th percentile.

CONCLUSIONS

Our findings indicate that arthroscopic caseload is increasing among orthopaedic residents. However, resident experience performing arthroscopic procedures is substantially disparate. Although the educational implications of this disparity are not well understood, our findings may aid in efforts to optimize arthroscopic training during orthopaedic residency.

National Trends in Carpal Tunnel Release and Hand Fracture Procedures Performed During Orthopaedic Residency: An Analysis of ACGME Case Logs

BACKGROUND

Mastery in performing carpal tunnel release (CTR) and hand fracture procedures is an essential component of orthopaedic residency training.

OBJECTIVE

To assess orthopaedic resident case log data for temporal trends in CTR and hand fracture cases and to determine the degree of variability in case volume among residents.

METHODS

Accreditation Council for Graduate Medical Education orthopaedic surgery resident case logs were reviewed for graduation years 2007 through 2014. Annual data regarding the mean number of CTR and hand fracture/dislocation procedures were recorded, as well as the median number of procedures reported by the top and bottom 10% of residents (by case volume). Temporal trends were assessed using linear regression modeling.

RESULTS

There was no change in the mean number of CTRs performed per resident. Over the 8-year period, the top 10% of residents performed a significantly greater number of CTRs than the bottom 10% (62.1 versus 9.3, P < .001). Similarly, no change was noted in the mean number of total hand fracture/dislocation cases performed, with the top 10% of residents performing significantly more hand fracture cases than the bottom 10% (47.1 versus 9.3, P < .001).

CONCLUSIONS

Our results indicate no change in CTR and hand fracture caseload for orthopaedic residents. However, as resident experience performing both procedures varies significantly, this variability likely has important educational implications.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment

CONTEXT

The relative advantages and disadvantages of checklists and global rating scales (GRSs) have long been debated. To compare the merits of these scale types, we conducted a systematic review of the validity evidence for checklists and GRSs in the context of simulation-based assessment of health professionals.

METHODS

We conducted a systematic review of multiple databases including MEDLINE, EMBASE and Scopus to February 2013. We selected studies that used both a GRS and checklist in the simulation-based assessment of health professionals. Reviewers working in duplicate evaluated five domains of validity evidence, including correlation between scales and reliability. We collected information about raters, instrument characteristics, assessment context, and task. We pooled reliability and correlation coefficients using random-effects meta-analysis.

RESULTS

We found 45 studies that used a checklist and GRS in simulation-based assessment. All studies included physicians or physicians in training; one study also included nurse anaesthetists. Topics of assessment included open and laparoscopic surgery (n = 22), endoscopy (n = 8), resuscitation (n = 7) and anaesthesiology (n = 4). The pooled GRS-checklist correlation was 0.76 (95% confidence interval [CI] 0.69-0.81, n = 16 studies). Inter-rater reliability was similar between scales (GRS 0.78, 95% CI 0.71-0.83, n = 23; checklist 0.81, 95% CI 0.75-0.85, n = 21), whereas GRS inter-item reliabilities (0.92, 95% CI 0.84-0.95, n = 6) and inter-station reliabilities (0.80, 95% CI 0.73-0.85, n = 10) were higher than those for checklists (0.66, 95% CI 0-0.84, n = 4 and 0.69, 95% CI 0.56-0.77, n = 10, respectively). Content evidence for GRSs usually referenced previously reported instruments (n = 33), whereas content evidence for checklists usually described expert consensus (n = 26). Checklists and GRSs usually had similar evidence for relations to other variables.

CONCLUSIONS

Checklist inter-rater reliability and trainee discrimination were more favourable than suggested in earlier work, but each task requires a separate checklist. Compared with the checklist, the GRS has higher average inter-item and inter-station reliability, can be used across multiple tasks, and may better capture nuanced elements of expertise.

Filling the void: defining invasive bedside procedural competency for internal medicine residents

BACKGROUND

Residents perform invasive bedside procedures in most training programs. To date, there is no universal approach for determining competency and ensuring quality and safety of care.

OBJECTIVE

We developed and implemented an assessment of central venous catheter insertion competency for internal medicine and internal medicine-pediatrics residents, using measurements for knowledge, skill, and attitude and linking them to procedural outcomes.

METHODS

We conducted a cohort study of a 4-week, resident-run procedure service from July 2007 through June 2011 at a large academic medical center. Knowledge was assessed by using a written test, technical skill by using a checklist, and attitude by self- and supervisor assessments of residents' confidence and capability. Competence was defined as (1) a minimum written test score (70%); (2) a perfect checklist score; (3) a resident's self-assessed confidence and capability scores of 4 or 5 of 5; and (4) faculty rating of the resident's confidence and capability as 5 of 5. A composite success rate was based on procedural outcomes (eg, completed procedures, less than 3 forward needle passes, and complication rate) and was compared to the checklist scores.

RESULTS

A total of 148 internal medicine and medicine-pediatrics residents inserted 639 catheters, and 53 (36%) achieved competence by the end of 4 weeks. Residents judged to be competent by checklist scores had a higher composite success rate than those deemed not competent.

CONCLUSIONS

Our multi-factorial criteria used to define central venous catheter insertion competency effectively discriminated between residents judged to be competent and those judged not competent, using data from procedural outcomes.

Is there a valid and reliable assessment of diagnostic knee arthroscopy skill?

BACKGROUND

The Basic Arthroscopic Knee Skill Scoring System (BAKSSS) has construct validity as an objective measure of arthroscopic proficiency when used to assess the task of performing arthroscopic meniscectomies on cadaver knees. The reliability of this instrument is unknown.

QUESTIONS/PURPOSE

We asked whether (1) a simple modification of the BAKSSS would show construct validity similar to that in the initial BAKSSS study, (2) this assessment would be reliable, and (3) this assessment could be used as a high-stakes pass or fail test.

METHODS

Twenty-three orthopaedic residents performed diagnostic knee arthroscopies on cadaveric knee specimens. Their competency was assessed by three live raters using the modified BAKSSS. Interrater reliability was assessed by comparing the scores given by each rater to each subject.

RESULTS

The modified BAKSSS showed construct validity with junior residents achieving lower scores (mean score, 20) than senior residents (mean score, 33). The modified BAKSSS had an interrater reliability of kappa = 0.685-0.852. The modified BAKSSS had a kappa = 0.543 when used as a proficiency test for diagnostic arthroscopy.

CONCLUSIONS

The modified BAKSSS is useful for assessing diagnostic knee arthroscopy proficiency. Future scoring systems should be designed to be generalizable so they can be applied to multiple procedures without the need for modification, allow for video-based assessment, and must be rigorously tested for reliability and other types of validity (eg, face validity, content validity, and criterion-related validity).

CLINICAL RELEVANCE

Having a valid and reliable assessment of basic arthroscopic procedures may allow educators to more adequately evaluate individual residents and the effectiveness of various training modalities.

Essential hand surgery procedures for mastery by graduating orthopedic surgery residents: a survey of program directors

PURPOSE

To establish the essential hand surgery procedures that should be mastered by graduating orthopedic surgery residents. This framework can then be used as a guideline for developing an Objective Structured Assessment of Technical Skill to teach and document technical skill in hand surgery.

METHODS

A select group of 10 expert hand surgeons was surveyed regarding the essential hand surgery procedures that should be mastered by graduating orthopedic surgery residents. The top 10 procedures from this survey were then used to survey all 155 American Council of Graduate Medical Education-approved orthopedic surgery program directors regarding the essential procedures that should be mastered by graduating orthopedic surgery residents.

RESULTS

We had a 39% response rate to the program director survey. The top 8 hand surgery procedures as determined by the orthopedic surgery program directors included open carpal tunnel release, open A1 pulley release, open reduction internal fixation of distal radius fracture, flexor tendon sheath steroid injection, excision of dorsal or volar ganglion, closed reduction and percutaneous pinning of metacarpal fracture, open cubital tunnel release, and incision and drainage of flexor tendon sheath for flexor tenosynovitis.

CONCLUSIONS

Surgical educators need to develop objective methods to teach and document technical skill. The Objective Structured Assessment of Technical Skill is a valid method to accomplish this task. However, there has been no consensus regarding which hand surgery procedures should be mastered by graduating orthopedic surgery residents. We have identified 8 procedures that were overwhelmingly supported by orthopedic surgery program directors. These 8 procedures can be used as a guideline for developing an Objective Structured Assessment of Technical Skill to teach and document technical skill in hand surgery.

CLINICAL RELEVANCE

This study addresses the future of orthopedic surgery education as it pertains to hand surgery.