Using the arthroscopic surgery skill evaluation tool as a pass-fail examination

Simulation training to retool practicing orthopedic surgeons is rare

Purpose

Modern surgical practice is continuously changing as technology develops. New techniques are often implemented after a surgeon has made the transition to independent clinical practice. There is therefore a need to 'retool' technical skills. Additionally, practicing surgeons must maintain and develop skills such as leadership, communication, critical thinking, teaching, and mentoring. Our aim was to perform a scoping review to assess the current status of simulation education for practicing Orthopedic Surgeons (OS).

Methods

A 10 year search of PubMed, ERIC, and Web of Science was performed with a medical librarian. Controlled vocabulary Medical Subject Headings terms and natural language were developed with subject matter experts describing simulation, training and OS. Two trained reviewers evaluated all abstracts for inclusion. Exclusion criteria were articles that did not assess simulation education involving practicing OS. Data were extracted from the included full text articles by two reviewers: details of study design, type of participants, type of simulation and role of OS in the educational event.

Results

Initial search identified 1824 articles of which 443 were duplicates, and 1381 articles were further screened. Of these, 1155 were excluded, 226 full text articles were assessed for eligibility and 80 included in analysis. Most were published in the last 6 years and from the United States. The majority (99%) described technical skill simulations (arthroscopy 56%, screw placement 23%, ligament reconstruction 19%). OS were rarely the only learners with 91% studies also having residents participate. OS were the targeted learner in 6% studies. OS provided content validity for 15 (19%) and construct validity in 59 (74%) studies.

Conclusions

Simulation training to educate practicing OS is rare. OS are often used to validate work rather than being the center of an educational endeavor. A refocusing is needed to provide adequate training for practicing surgeons to retool skills as new techniques become available.

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BACKGROUND

Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.

METHODS

A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird's-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.

RESULTS AND DISCUSSION

The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio-video quality, and questionable feasibility for assessing readiness for practice.

CONCLUSION

Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio-video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

Learning and Short-Term Retention of Simulation-Based Arthroscopic Skills

OBJECTIVE

The study aimed to examine the learning curve and short-term retention of arthroscopic skills acquired on a simulator.

DESIGN

Cohort study.

SETTING

Clinical Skills Training Center of Zhujiang Hospital of Southern Medical University PARTICIPANT AND METHODS: Orthopaedic residents (n = 14) without previous arthroscopy experience were included. After basic information was collected and an initial arthroscopy knowledge level test was administered, the subjects received standardised training on the simulator (day 1); then, they completed tasks on the simulator, including guided diagnostics (4 times), triangulation (5 times) and loose body removal (7 times). A learning curve for each skill was generated based on the total scores. The score of the last repetition of each task was the training level.

RESULTS

A total of 14 orthopedic residents were enrolled. All participants completed the training and testing. There was a learning curve over the course of training for all 3 arthroscopic skills (p < 0.001). On day 8 after the training, the mean score for guided diagnostics decreased from 49.9 to 48.9 (p = 0.001), and the retention rate was 97.8%. For triangulation, the mean total score decreased from 58.9 to 53.6 (p < 0.001), and the retention rate was 90.8%. For loose body removal, the mean total score decreased from 87.1 to 80.7 (p < 0.001), and the retention rate was 92.7%.

CONCLUSIONS

Orthopaedic residents' arthroscopic skills learned through simulator training declined significantly in 1 week after the training, especially more difficult skills.

The frequency of assessment tools in arthroscopic training: a systematic review

BACKGROUND

Multiple assessment tools are used in arthroscopic training and play an important role in feedback. However, it is not fully recognized as to the standard way to apply these tools. Our study aimed to investigate the use of assessment tools in arthroscopic training and determine whether there is an optimal way to apply various assessment tools in arthroscopic training.

METHODS

A search was performed using PubMed, Embase and Cochrane Library electronic databases for articles published in English from January 2000 to July 2021. Eligible for inclusion were primary research articles related to using assessment tools for the evaluation of arthroscopic skills and training environments. Studies that focussed only on therapeutic cases, did not report outcome measures of technical skills, or did not mention arthroscopic skills training were excluded.

RESULTS

A total of 28 studies were included for review. Multiple assessment tools were used in arthroscopic training. The most common objective metric was completion time, reported in 21 studies. Technical parameters based on simulator or external equipment, such as instrument path length, hand movement, visual parameters and injury, were also widely used. Subjective assessment tools included checklists and global rating scales (GRS). Among these, the most commonly used GRS was the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Most of the studies combined objective metrics and subjective assessment scales in the evaluation of arthroscopic skill training.

CONCLUSIONS

Overall, both subjective and objective assessment tools can be used as feedback for basic arthroscopic skill training, but there are still differences in the frequency of application in different contexts. Despite this, combined use of subjective and objective assessment tools can be applied to more situations and skills and can be the optimal way for assessment.

LEVEL OF EVIDENCE

Level III, systematic review of level I to III studies. Key messagesBoth subjective and objective assessment tools can be used as feedback for basic arthroscopic skill training.Combined use of subjective and objective assessment tools can be applied to more situations and skills and can be the optimal way for assessment.

Is the Virtual Reality Fundamentals of Arthroscopic Surgery Training Program a Valid Platform for Resident Arthroscopy Training?

BACKGROUND

Training in arthroscopy is associated with a steep learning curve for trainees and bears risks for patients. Virtual reality (VR) arthroscopy simulation platforms seek to overcome this and to provide a safe environment for surgical learners. The Fundamentals of Arthroscopic Surgery Training (FAST) program is one such platform. It is currently not known whether the VR FAST program can be employed as a useful teaching or examination tool to assess the basic arthroscopic skills of surgical trainees.

QUESTIONS/PURPOSES

(1) Does the VR FAST program differentiate among novice, intermediate, and expert arthroscopists? (2) Does ambidextrous performance in the VR FAST program correlate with arthroscopic experience?

METHODS

We prospectively recruited orthopaedic interns (novices), residents (intermediates), and fellows and attendings (experts) to complete the VR FAST program over a 1-year period from four major orthopaedic training programs on a voluntary basis. Sixty-six of 156 invited orthopaedic surgeons participated: 26 of 50 novices (16 men and 10 women), 27 of 65 intermediates (20 men and seven women), and 13 of 41 experts (10 men and three women). Surgeons of any arthroscopic experience were included, with only those with prior experience on the VR FAST program being excluded. The program consists of eight modules: three basic camera modules (Image Centering, Horizon Control, and Telescoping), three advanced camera modules (Periscoping, Trace the Line, and Trace the Curve), and two instrumented bimanual-dexterity modules (Probe Triangulation and Gather the Stars). Time taken to complete each task and measures of economy of movement (camera and instrument path length, camera alignment) were used as measures of arthroscopic experience. Every participant completed the modules using their dominant and nondominant hands. Equality in proficiency in completing the tasks using the dominant and nondominant hands were determined to be measures of arthroscopic experience. Due to the large number of outcome variables, only p values < 0.01 were considered to be statistically significant.

RESULTS

Six of eight VR FAST modules did not discriminate among novice, intermediate, and expert arthroscopy participants. However, two did, and the ones that were most effective at distinguishing participants by level of experience were the Periscoping and Gather the Stars modules. For the Periscoping module using the dominant hand, novices required longer to complete the task with a median time of 231 seconds (IQR 149 to 358) and longer camera path length median of 191 cm (IQR 128 to 273) compared with intermediates who needed 127 seconds (IQR 106 to 233) and 125 cm (IQR 92 to 159) and experts who needed 121 seconds (IQR 93 to 157) and 119 cm (IQR 90 to 134) (p = 0.001 and p = 0.003, respectively). When using the nondominant hand, novices took longer to complete the task with a median time of 231 seconds (IQR 170 to 350) and longer camera path length 204 cm (IQR 169 to 273) compared with intermediates who required 132 seconds (IQR 97 to 162) and 111 cm (IQR 88 to 143) and experts who needed 119 seconds (IQR 104 to 183) and 120 cm (IQR 108 to 166) (p < 0.001 and p < 0.001, respectively). For the Gather the Stars module using the nondominant hand, only the novices needed longer to complete the task at a median of 131 seconds (IQR 112 to 157) and needed a longer grasper path length of 290 cm (IQR 254 to 332) compared with intermediates who needed 84 seconds (IQR 72 to 119) and 232 cm (IQR 195 to 254) and experts who needed 98 seconds (IQR 87 to 107) and 244 cm (IQR 215 to 287) (p < 0.001 and p = 0.001, respectively).

CONCLUSION

Six of eight VR FAST modules did not demonstrate construct validity, and we found no correlation between arthroscopic experience and ambidextrous performance. Two modules demonstrated construct validity; however, refinement and expansion of the modules is needed with further validation in large prospective trials so that pass-fail thresholds can be set for use in high-stakes examinations.

CLINICAL RELEVANCE

Most VR FAST modules were not discriminatory; however, they can form essential conceptual and procedural building blocks in an arthroscopic curriculum that are beneficial for novices when developing key psychomotor skills. In their present format, however, they are unsuitable for assessing arthroscopic proficiency.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

The Arthroscopic Surgery Skill Evaluation Tool Global Rating Scale is a Valid and Reliable Adjunct Measure of Performance on a Virtual Reality Simulator for Hip Arthroscopy

PURPOSE

The purpose of this study is to further evaluate the construct validity and interobserver reliability of a hip arthroscopy virtual simulator using the Arthroscopic Surgery Skill Evaluation Tool (ASSET) global rating scale.

METHODS

Thirty participants (23 male/7 female) completed a diagnostic arthroscopy and a loose body retrieval simulation on the VirtaMed Arthros Hip Simulator (Zurich, Switzerland) twice at a minimum of 1 week apart. Subjects consisted of 12 novices (medical students, postgraduate year [PGY] 1-2), 5 intermediate trainees (PGY3-4), 9 senior trainees (PGY5 and fellows), and 4 attending faculty. Simulator metrics were recorded and then compiled to generate a total simulator score (TSS). The loose body retrieval was graded using the ASSET scoring tool. Inter-rater and intrarater reliability for the ASSET for 2 blinded raters and construct validity of the ASSET and the TSS were calculated. Correlation between the TSS, ASSET and individual simulator metrics was determined.

RESULTS

Prior simulation experience (P ≤ 0.01) correlated with higher TSS and higher ASSET, while video game experience correlated with higher TSS on the diagnostic module only (P = 0.004). There was a significant difference in ASSET score among all experience groups (P < 0.04). Novices had the lowest mean ASSET whereas experts had the highest mean ASSET with a difference of 17.4 points. Overall performance on the surgical module significantly correlated with the ASSET score (r = 0.444, P = 0.016). There was a significant positive correlation among higher ASSET and number of loose bodies retrieved, operation time, camera path and grasper path length, and percentage of cartilage injury. ASSET demonstrated excellent intrarater reliability and showed substantial or better inter-reliability in 8 of 9 domains.

CONCLUSION

The VirtaMed hip arthroscopy simulator demonstrated good construct validity and excellent reliability for simulator-based metrics and ASSET score. Use of both simulator metrics and ASSET offers a more comprehensive performance assessment on hip arthroscopy simulation than either measure alone.

CLINICAL RELEVANCE

As virtual reality simulation for arthroscopy becomes more commonplace in orthopaedic training, evaluation of the most effective objective and subjective measures of performance is necessary to optimize simulation training.

Technical Obstacles for Low-Volume Surgeons in Primary Anterior Cruciate Ligament Reconstruction

A longer surgical time will lead to postoperative complications in orthopaedic surgery. According to previous reports, surgeon volume affects a surgical time in anterior cruciate ligament (ACL) reconstruction. However, little attention has been paid to difference of a surgical time between high- and low-volume surgeons in ACL reconstruction. The purpose of the present study was to investigate the surgical time for both high- and low-volume surgeons and to clarify technical obstacles as well as the important technique to reduce the surgical time in ACL reconstruction. A total of 103 knees in 103 patients with ACL tear were enrolled. All the procedures were performed by 13 different surgeons. The surgeons were divided into two surgeon groups (high- and low-volume groups) based on the number of annual cases. The operative technique was divided into the same five steps to perform comparisons of step-by-step surgical time between the surgeon groups of different volumes. The statistical comparison was done between high- and low-volume groups in terms of surgical time in each step. Total operation times were 87.8 ± 17.6 and 129.9 ± 42.9 minutes in high- and low-volume group, respectively. Low-volume surgeons took much time in tendon harvesting and bone tunnel creation using arthroscopy. Low-volume surgeons took more time for primary ACL reconstruction, especially in tendon harvesting, compared with high-volume surgeons. Tendon harvesting was extremely important, as well as bone tunnel creation, for low-volume surgeons to reduce surgical time in primary ACL reconstruction.

The Effect of Triangulation Simulator Training on Arthroscopy Skills: A Prospective Randomized Controlled Trial

Purpose

To prospectively evaluate the transferability of skills acquired on a low-cost, at-home, nonanatomic triangulation simulation system to cadaveric models.

Methods

We randomized 28 medical students into either a simulator-training group (n = 14) or group with no training (control, n = 14). All subjects were pretested using a standardized checklist of arthroscopic skills on cadaveric knees and shoulders. Training-group subjects practiced on the triangulation simulator for 90 minutes per week for 4 consecutive weeks. Control subjects received no training. All subjects completed a post-test checklist of arthroscopic skills on cadaveric knees and shoulders, as well as 4 training tasks on the simulator. A blinded orthopaedic surgeon evaluated the arthroscopic videos using the Arthroscopic Surgical Skill Evaluation Tool (ASSET) score.

Results

Training-group knee and shoulder ASSET scores increased from 12.2 ± 1.85 to 14.6 ± 2.76 (P = .02) and from 14.6 ± 3.5 to 17.9 ± 4.5 (P = .29), respectively. In the control group, knee and shoulder ASSET scores increased from 14.3 ± 3.12 to 14.25 ± 4.67 (P = .99) and from 14.2 ± 2.7 to 17.07 ± 6.7 (P = .58), respectively. There were no significant differences in the mean post-test ASSET scores between the training group and control group for either knee or shoulder arthroscopy. The post-test ASSET safety subscore during knee arthroscopy was significantly higher in the training group (P = .03). The training group was able to complete significantly more simulator tasks compared with controls (P = .003) at post-testing. A significant positive correlation was found between knee arthroscopy performance and the number of tasks completed during simulation post-testing (P = .043). There was no significant correlation between shoulder arthroscopy performance and simulation performance (P = .532).

Conclusions

Basic triangulation skills may be acquired by training on a low-cost, at-home, nonanatomic triangulation simulation system, although the degree of transferability and universal joint applicability, as well as the existence of an early ceiling effect in skill development, could not be shown.

Level of Evidence

Level II, randomized controlled trial.

Global Rating Scales for the Assessment of Arthroscopic Surgical Skills: A Systematic Review

PURPOSE

To evaluate whether sufficient validity and reliability evidence exists to support the use of global rating scales (GRS) as evaluation tools in both formative assessment and competency assessment of arthroscopic procedures.

METHODS

A search of PubMed, Embase, and Scopus was conducted for articles published between 1990 and 2018. Studies reporting measures of validity and reliability of GRS relating to arthroscopic skills were included. Procedural checklists and other assessment tools were excluded.

RESULTS

A total of 39 articles met the inclusion criteria. In total, 7 de novo GRS specific for arthroscopic education and 3 pre-existing GRS repurposed 4 times for arthroscopic education were identified in the literature. The 11 GRS were used to assess 1175 surgeons 3890 times. Three GRS tools explicitly defined an arbitrary minimum competency threshold, 6 of 11 tools demonstrated construct validity-the ability to significantly discriminate between groups of differing experience-and 5 of 11 tools assessed inter-rater reliability, but only the Arthroscopic Surgical Skills Evaluation Tool demonstrated excellent inter-rater reliability. The Arthroscopic Surgical Skills Evaluation Tool was validated by 16 articles for a total of 537 surgeons for hip, knee, shoulder, and ankle arthroscopy in both simulated and clinical environments but was found to be invalid in wrist arthroscopy. The Basic Arthroscopic Knee Skill Scoring System was validated by 15 articles for a total of 497 surgeons for knee, hip, and shoulder in both clinical and simulated environments. The remaining 9 GRS were validated by 2 or fewer studies.

CONCLUSIONS

Overall, GRS have contributed to training, feedback, and formative assessment practices. The GRS reviewed demonstrate both construct and concurrent validity as well as reliability in multiple arthroscopic procedures in multiple joints. Currently, there is sufficient evidence to use GRS as a feedback tool. However, there is insufficient evidence for its use in high-stakes examinations or as a minimum competency assessment.

LEVEL OF EVIDENCE

Level III, systematic review of level I to III studies.

Arthroscopic Rotator Cuff Repair Metrics: Establishing Face, Content, and Construct Validity in a Cadaveric Model

PURPOSE

To create and determine face validity and content validity of arthroscopic rotator cuff repair (ARCR) performance metrics, to confirm construct validity of the metrics coupled with a cadaveric shoulder, and to establish a performance benchmark for the procedure on a cadaveric shoulder.

METHODS

Five experienced arthroscopic shoulder surgeons created step, error, and sentinel error metrics for an ARCR. Fourteen shoulder arthroscopy faculty members from the Arthroscopy Association of North America formed the modified Delphi panel to assess face and content validity. Eight Arthroscopy Association of North America shoulder arthroscopy faculty members (experienced group) were compared with 9 postgraduate year 4 or 5 orthopaedic residents (novice group) in their ability to perform an ARCR. Instructions were given to perform a diagnostic arthroscopy and a 2-anchor, 4-simple suture repair of a 2-cm supraspinatus tear. The procedure was videotaped in its entirety and independently scored in blinded fashion by trained, paired reviewers.

RESULTS

Delphi panel consensus for 42 steps and 66 potential errors was obtained. Overall performance assessment showed a mean inter-rater reliability of 0.93. Novice surgeons completed 17% fewer steps (32.1 vs 37.5, P = .001) and enacted 2.5 times more errors than the experienced group (6.21 vs 2.5, P = .012). Fifty percent of the experienced group members and none of the novice group members achieved the proficiency benchmark of a minimum of 37 steps completed with 3 or fewer errors.

CONCLUSIONS

Face validity and content validity for the ARCR metrics, along with construct validity for the metrics and cadaveric shoulder, were verified. A proficiency benchmark was established based on the mean performance of an experienced group of arthroscopic shoulder surgeons.

CLINICAL RELEVANCE

Validated procedural metrics combined with the use of a cadaveric shoulder can be used to accurately assess the performance of an ARCR.

Reliability of Observational Assessment Methods for Outcome-based Assessment of Surgical Skill: Systematic Review and Meta-analyses

BACKGROUND

Reliable performance assessment is a necessary prerequisite for outcome-based assessment of surgical technical skill. Numerous observational instruments for technical skill assessment have been developed in recent years. However, methodological shortcomings of reported studies might negatively impinge on the interpretation of inter-rater reliability.

OBJECTIVE

To synthesize the evidence about the inter-rater reliability of observational instruments for technical skill assessment for high-stakes decisions.

DESIGN

A systematic review and meta-analysis were performed. We searched Scopus (including MEDLINE) and Pubmed, and key publications through December, 2016. This included original studies that evaluated reliability of instruments for the observational assessment of technical skills. Two reviewers independently extracted information on the primary outcome (the reliability statistic), secondary outcomes, and general information. We calculated pooled estimates using multilevel random effects meta-analyses where appropriate.

RESULTS

A total of 247 documents met our inclusion criteria and provided 491 inter-rater reliability estimates. Inappropriate inter-rater reliability indices were reported for 40% of the checklists estimates, 50% of the rating scales estimates and 41% of the other types of assessment instruments estimates. Only 14 documents provided sufficient information to be included in the meta-analyses. The pooled Cohen's kappa was .78 (95% CI 0.69-0.89, p < 0.001) and pooled proportion agreement was 0.84 (95% CI 0.71-0.96, p < 0.001). A moderator analysis was performed to explore the influence of type of assessment instrument as a possible source of heterogeneity.

CONCLUSIONS AND RELEVANCE

For high-stakes decisions, there was often insufficient information available on which to base conclusions. The use of suboptimal statistical methods and incomplete reporting of reliability estimates does not support the use of observational assessment instruments for technical skill for high-stakes decisions. Interpretations of inter-rater reliability should consider the reliability index and assessment instrument used. Reporting of inter-rater reliability needs to be improved by detailed descriptions of the assessment process.

Editorial Commentary: Harnessing Surgical Simulation-Saddle up for a Long and Rigorous Ride!

Arthroscopic simulation tools will assist in the training of skills that are transferrable to the operating room. Success in that endeavor requires the following: relevant outcome metrics rather than surrogate measures of performance, evaluation tools that enable scoring of step and error metrics rather than global rating scales, construct validation of simulators and metrics based on actual performance, development of simulators that not only assess performance but also train additional skills and provide feedback, and a focus first on effective training as opposed to high-fidelity realism.

Wrist Arthroscopy: Can We Gain Proficiency Through Knee Arthroscopy Simulation?

OBJECTIVE

Wrist arthroscopy is a challenging discipline with limited training exposure during residency. The purpose of this study was to evaluate the effectiveness of virtual knee arthroscopy simulation training for gaining proficiency in wrist arthroscopy.

DESIGN

Participants were recorded performing a cadaveric wrist arthroscopy simulation. The residents then practiced knee arthroscopy on a virtual reality simulator and repeated the wrist arthroscopy simulation. All videos were blinded prior to assessment. Proficiency was graded using the Arthroscopic Surgery Skill Evaluation Tool global rating scale. In addition, participants were asked to complete a survey assessing the value of the virtual reality knee arthroscopy simulator for wrist arthroscopy.

SETTING

Orthopaedic Surgery Residency Program, Carolinas Medical Center, a large, public, nonprofit hospital located in Charlotte, North Carolina.

PARTICIPANTS

Orthopaedic residents at our center were asked to participate in the simulation training. Participation was voluntary and nonincentivized. All orthopaedic residents at our institution (N = 27) agreed to participate. In total, there were 10 Intern (PGY-0 and PGY-1), 10 Junior (PGY-2 and PGY-3), and 7 Senior (PGY-4 and PGY-5) residents. In addition, a fellowship-trained hand surgeon was recruited to participate in the study, performing the wrist arthoscopy simulation. Two additional fellowship-trained hand surgeons, for a total of 3, assessed the blinded videos.

RESULTS

There was a trend toward better wrist Arthroscopic Surgery Skill Evaluation Tool scores by training level, although the difference was not statistically significant. Interns improved by an average of 1.8 points between baseline and postknee simulation tests. Junior and senior residents decreased by 1.6 and 5.0 points, respectively.

CONCLUSIONS

Knee arthroscopy simulation training did not objectively improve wrist arthroscopy proficiency among residents. A wrist-specific arthroscopy simulation program is needed if measurable competence through simulation is desired.

Efficacy of standardized training on a virtual reality simulator to advance knee and shoulder arthroscopic motor skills

Background

Most studies demonstrated, that training on a virtual reality based arthroscopy simulator leads to an improvement of technical skills in orthopaedic surgery. However, how long and what kind of training is optimal for young residents is unknown. In this study we tested the efficacy of a standardized, competency based training protocol on a validated virtual reality based knee- and shoulder arthroscopy simulator.

Methods

Twenty residents and five experts in arthroscopy were included. All participants performed a test including knee -and shoulder arthroscopy tasks on a virtual reality knee- and shoulder arthroscopy simulator. The residents had to complete a competency based training program. Thereafter, the previously completed test was retaken. We evaluated the metric data of the simulator using a z-score and the Arthroscopic Surgery Skill Evaluation Tool (ASSET) to assess training effects in residents and performance levels in experts.

Results

The residents significantly improved from pre- to post training in the overall z-score: − 9.82 (range, − 20.35 to − 1.64) to − 2.61 (range, − 6.25 to 1.5); p < 0.001. The overall ASSET score improved from 55 (27 to 84) percent to 75 (48 to 92) percent; p < 0.001. The experts, however, achieved a significantly higher z-score in the shoulder tasks (p < 0.001 and a statistically insignificantly higher z-score in the knee tasks with a p = 0.921. The experts mean overall ASSET score (knee and shoulder) was significantly higher in the therapeutic tasks (p < 0.001) compared to the residents post training result.

Conclusions

The use of a competency based simulator training with this specific device for 3-5 h is an effective tool to advance basic arthroscopic skills of resident in training from 0 to 5 years based on simulator measures and simulator based ASSET testing. Therefore, we conclude that this sort of training method appears useful to learn the handling of the camera, basic anatomy and the triangulation with instruments.

Psychomotor Testing for Orthopedic Residency Applicants: A Pilot Study

OBJECTIVE

The purpose of this study was to develop an objective motor skills testing system to aid in the evaluation of potential orthopedic residents.

DESIGN

Participants attempted a battery of 5 motor skills tests (4 novel tests and the Grooved Pegboard [GPT] Test) in one 10-minute session. A percentile-based scoring system was created for each test based on raw scores. One-way analysis of variance was used to compare testing scores among 3 cohorts. Each novel test and overall scores were compared with GPT scores as a relative measure of validity.

SETTING

The 2015 orthopedic surgery residency interview season at an academic institution.

PARTICIPANTS

Thirty orthopedic residents and 72 nonresidents (15 community volunteers and 57 orthopedic surgery residency applicants).

RESULTS

Overall, residents performed better than nonresidents (p < 0.0001) and applicants performed worse than residents or volunteers (p < 0.0001). There were positive correlations between the GPT score and overall battery score (r = 0.63), screw and nut test (r = 0.40), and mimic a structure test (r = 0.26). The fracture reduction test and drilling test scores did not correlate to performance on the GPT.

CONCLUSIONS

Psychomotor testing for surgical applicants is an area in need of study. This investigation successfully piloted a novel battery of tests, which is easily reproducible and thus may be feasible for use in the orthopedic surgery residency interview setting. Longitudinal evaluation is required to explore correlation with future operative skill.

Validation of an Arthroscopic Training Device

PURPOSE

To investigate the usefulness and conduct validation of a simulated arthroscopy training device to train basic arthroscopy skills.

METHODS

Forty-six participants including 12 novices, 12 intermediates, and 22 experts completed a questionnaire regarding demographics, previous arthroscopic experience, training potential, and statements about the device. Furthermore, participants performed a single task on the arthroscopic training device using the 0° camera and a probe. The task consisted of an attempt to carry a rubber ring across a helix inside a box as fast as possible. Construct validity was evaluated by comparing total task time and portal replacements of the camera and probe between all groups (median values [interquartile range]; Kruskal-Wallis test).

RESULTS

The median age was 35 (29-44) years. There were 4 female and 42 male participants. A total of 89% of the participants graded the overall training capacity ≥5 (35% graded it as 5, 39% as 6, and 15% as 7), and 83% believed that it is useful to improve any kind of arthroscopy. Ninety-three percent of the participants would recommend the arthroscopic training device to their colleagues. Sixty-one percent of the participants stated that there are certain disadvantages. The median time to complete the task was 108 (58-236) seconds. Novices (259 [123-435] seconds) performed tasks significantly slower than intermediates (169 [67-257] seconds) and experts (75 [49-132] seconds) (P = .005). Furthermore, portal changes were significantly more common in novices and intermediates than experts (P = .019).

CONCLUSIONS

High scores in training potential were achieved with this arthroscopy simulator box, and most study participants believed that practice with the arthroscopic training device is useful for any kind of arthroscopy. Construct validity was established since novices, intermediates, and experts in real arthroscopy were discriminated with the arthroscopic training device in terms of time to successful completion of a task. However, 61% of the participants stated that there were certain disadvantages.

CLINICAL RELEVANCE

In every training tool using simulation, it is crucial to pass the first steps in the validation cascade. This study provides this step for further evaluation of this arthroscopic training device.

A Checklist Intervention to Assess Resident Diagnostic Knee and Shoulder Arthroscopic Efficiency

PURPOSE

The purpose of this investigation was to apply an arthroscopic shoulder and knee checklist in the evaluation of orthopedic resident arthroscopic skill efficiency and to demonstrate the use of a surgical checklist for assessing resident surgical efficiency over the course of a surgical rotation.

METHODS

Orthopedic surgery residents rotating on the sports medicine service at our institution between 2011 and 2015 were enrolled in this study. Residents were administered a shoulder and knee arthroscopy assessment tool at the beginning and end of their 6-week rotation. The assessment tools consisted of checklist items for knee and shoulder arthroscopy skills. Residents were timed while performing these checklist tasks. The primary outcome measure was resident improvement as a function of time to completion for the checklist items, and the intervention was participation in a 6-week resident rotation with weekly arthroscopy didactics, cadaver simulator work, and operating room experience. A paired t test was used to compare means.

RESULTS

Mean time to checklist completion during week 1 among study participants for the knee checklist was 787.4 seconds for the knee checklist and 484.4 seconds at the end of the rotation. Mean time to checklist completion during week 1 among study participants for the shoulder checklist was 1655.3 seconds and 832.7 seconds for the shoulder checklist at the end of the rotation. Mean improvement in time to completion was 303 seconds (p = 0.0006, SD = 209s) and 822.6 seconds (p = 0.00008, SD = 525.2s) for the arthroscopic knee and shoulder assessments, respectively.

CONCLUSIONS

An arthroscopic checklist is 1 method to evaluate and assess resident efficiency and improvement during surgical training. Among residents participating in this study, we found statistically significant improvements in time for arthroscopic task completion.

LEVEL OF EVIDENCE

II.

What are the important manoeuvres for beginners to minimize surgical time in primary total knee arthroplasty?

PURPOSE

It was hypothesized that surgical time of beginners would be much longer than that of experts. Our purpose was to investigate and clarify the important manoeuvres for beginners to minimize surgical time in primary total knee arthroplasty (TKA) as a multicentre study.

METHODS

A total of 300 knees in 248 patients (averaged 74.6 years) were enrolled. All TKAs were done using the same instruments and the same measured resection technique at 14 facilities by 25 orthopaedic surgeons. Surgeons were divided into three surgeon groups (four experts, nine medium-volume surgeons and 12 beginners). The surgical technique was divided into five phases. Detailed surgical time and ratio of the time in each phase to overall surgical time were recorded and compared among the groups in each phase. A total of 62, 119, and 119 TKAs were done by beginners, medium-volume surgeons, and experts, respectively.

RESULTS

Significant differences in surgical time among the groups were seen in each phase. Concerning the ratio of the time, experts and medium-volume surgeons seemed cautious in fixation of the permanent component compared to other phases. Interestingly, even in ratio, beginners and medium-volume surgeons took more time in exposure of soft tissue compared to experts. (0.14 in beginners, 0.13 in medium-volume surgeons, 0.11 in experts, P < 0.05).

CONCLUSION

Beginners took more time in exposure and closure of soft tissue compared to experts. Improvement in basic technique is essential to minimize surgical time among beginners. First of all, surgical instructors should teach basic techniques in primary TKA for beginners.

LEVEL OF EVIDENCE

Therapeutic studies, Level IV.

Trends in Arthroscopic Procedures Performed During Orthopaedic Residency: An Analysis of Accreditation Council for Graduate Medical Education Case Log Data

PURPOSE

To analyze orthopaedic resident case log data to report temporal trends in performing arthroscopic procedures and to assess variability in arthroscopic case volume among residents.

METHODS

Accreditation Council for Graduate Medical Education orthopaedic surgery resident case logs were reviewed from 2007 to 2013. The mean number of wrist, elbow, shoulder, knee, and ankle arthroscopic procedures performed by graduating residents was analyzed. The median number of arthroscopic procedures reported by the 70th and 30th percentiles of graduating residents (by arthroscopic case volume) was also recorded. Temporal trends were assessed using a linear regression model.

RESULTS

From 2007 to 2013, there were significant increases in the mean number of wrist (5.8 to 6.3; P = .038), elbow (2.5 to 3.2; P < .001), shoulder (93.5 to 133.8; P < .001), knee (170.6 to 185.1; P = .011), and ankle (5.8 to 7.4; P < .001) arthroscopies performed per resident. Residents in the 70th percentile of caseload performed significantly more wrist (7 v 2; P < .001), elbow (3.3 v 1; P < .001), shoulder (134.1 v 70.6; P < .001), knee (205 v 128.7; P < .001), and ankle (7.9 v 2.9; P < .001) arthroscopies than residents in the 30th percentile.

CONCLUSIONS

Our findings indicate that arthroscopic caseload is increasing among orthopaedic residents. However, resident experience performing arthroscopic procedures is substantially disparate. Although the educational implications of this disparity are not well understood, our findings may aid in efforts to optimize arthroscopic training during orthopaedic residency.

Performance of medical students on a virtual reality simulator for knee arthroscopy: an analysis of learning curves and predictors of performance

Background

Ethical concerns for surgical training on patients, limited working hours with fewer cases per trainee and the potential to better select talented persons for arthroscopic surgery raise the interest in simulator training for arthroscopic surgery. It was the purpose of this study to analyze learning curves of novices using a knee arthroscopy simulator and to correlate their performance with potentially predictive factors.

Methods

Twenty medical students completed visuospatial tests and were then subjected to a simulator training program of eight 30 min sessions. Their test results were quantitatively correlated with their simulator performance at initiation, during and at the end of the program.

Results

The mean arthroscopic performance score (z-score in points) at the eight test sessions were 1. -35 (range, -126 to -5) points, 2. -16 (range, -30 to -2), 3. -11 (range, -35 to 4), 4. -3 (range, -16 to 5), 5. -2 (range, -28 to 7), 6. 1 (range, -18 to 8), 7. 2 (range, -9 to 8), 8. 2 (range, -4 to 7). Scores improved significantly from sessions 1 to 2 (p = 0.001), 2 to 3 (p = 0.052) and 3 to 4 (p = 0.001) but not thereafter. None of the investigated parameters predicted performance or development of arthroscopic performance.

Conclusion

Novices improve significantly within four 30 min test virtual arthroscopy knee simulator training but not thereafter within the setting studied. No factors, predicting talent or speed and magnitude of improvement of skills could be identified.

Assessing Diagnostic Arthroscopy Performance in the Operating Room Using the Arthroscopic Surgery Skill Evaluation Tool (ASSET)

PURPOSE

To determine the validity and reliability of using the Arthroscopic Surgery Skill Evaluation Tool (ASSET) to assess arthroscopic skill in the operating room.

METHODS

Eight orthopaedic residents, 1 sports medicine fellow, and 3 sports medicine faculty members recorded the diagnostic portion of 3 shoulder and 3 knee procedures in the operating room. Two blinded raters used the ASSET to assess each recorded procedure video. Criterion for a passing score on a procedure was attaining a 3 or greater in all 8 domains assessed.

RESULTS

In total 70 videos (36 shoulder, 34 knee) were evaluated by each rater. The attending/fellow group was assigned significantly higher mean ASSET scores compared with resident groups for both procedures (P = .01). The attending/fellow group also had the highest passing percentage (95.8%). Raters were in agreement for total ASSET scores assigned to both diagnostic arthroscopy of the shoulder (intraclass correlation coefficient [ICC] = 0.84) and knee (ICC = 0.81). Agreement on individual ASSET domains was moderate (ICC = 0.61 to 0.80) for all domains except safety and difficulty of procedure. Raters concurred on the pass-fail evaluation in 62 of 70 (88.6%) of procedures evaluated.

CONCLUSIONS

Using the ASSET to assess surgical skills in the operating room is feasible, reliable, and valid.

CLINICAL RELEVANCE

When combined with previously published results using the ASSET, the ASSET may provide a validated and reliable method for evaluating arthroscopic surgical skills in the surgical simulation lab and operating room.

Ankle Arthroscopy Simulation Improves Basic Skills, Anatomic Recognition, and Proficiency During Diagnostic Examination of Residents in Training

BACKGROUND

The purpose of this study was to determine whether low-fidelity arthroscopic simulation training improves basic ankle arthroscopy performance and efficiency among orthopedic trainees.

METHODS

Twenty-nine orthopedic surgery trainees with varying levels of experience in ankle arthroscopy were randomized into either simulation or standard practice groups. At baseline testing, all participants performed simulator-based testing and a cadaveric diagnostic ankle arthroscopy with video recording. The simulation group subsequently received 4 one-on-one, 15-minute simulation training sessions over a 4-month period, while the standard practice group received no additional simulation training or exposure. After intervention, both groups were reevaluated with simulator testing and a second recorded cadaveric diagnostic ankle arthroscopy. Two blinded, independent experts evaluated each randomized arthroscopic performance using the 15-point checklist, Arthroscopic Surgery Skill Evaluation Tool (ASSET), and total elapsed time, and all outcome measures were compared within and between groups.

RESULTS

Baseline arthroscopic experience, simulator task performance measures, and ASSET scores were equivalent between the simulation and standard practice groups. After completion of training, the simulation group outscored the control group in total ASSET score (34.9 vs 19.6; P < .001) and checklist score (14.5 vs 8.4; P < .001) and achieved nearly expert ASSET Safety scores (4.7 vs 2.9; P < .001) on the simulator model. Cadaver testing also demonstrated significant improvements in total ASSET score (28.8 vs 16.8; P < .001), checklist score (12.6 vs 7.1; P < .001), and ASSET Safety score (3.9 vs 2.6; P < .001).

CONCLUSION

These results demonstrate that low-fidelity ankle arthroscopy simulation training can improve basic surgical skills, efficiency of movement, and anatomic recognition. The results suggest greater patient safety during ankle arthroscopy following simulation training.

LEVEL OF EVIDENCE

Level I, prospective comparative study.

The role of simulation in developing surgical skills

Surgical training has followed the master-apprentice model for centuries but is currently undergoing a paradigm shift. The traditional model is inefficient with no guarantee of case mix, quality, or quantity. There is a growing focus on competency-based medical education in response to restrictions on doctors' working hours and the traditional mantra of "see one, do one, teach one" is being increasingly questioned. The medical profession is subject to more scrutiny than ever before and is facing mounting financial, clinical, and political pressures. Simulation may be a means of addressing these challenges. It provides a way for trainees to practice technical tasks in a protected environment without putting patients at risk and helps to shorten the learning curve. The evidence for simulation-based training in orthopedic surgery using synthetic models, cadavers, and virtual reality simulators is constantly developing, though further work is needed to ensure the transfer of skills to the operating theatre.

The knee arthroscopy learning curve: quantitative assessment of surgical skills

PURPOSE

To assess orthopaedic trainees performing diagnostic knee arthroscopies and evaluate procedural competence using a novel statistical method, the Cumulative Summation Test for Learning Curve (LC-CUSUM).

METHODS

Twenty orthopaedic trainees in postgraduate year (PGY) 1 through 5 performed diagnostic knee arthroscopy and were evaluated intraoperatively with a validated 10-point knee task-specific checklist (TSCL) and 50-point global rating scale (GRS). A score of 40 points or greater (of 50 points) for the GRS and 8 points or greater (of 10 points) for the knee TSCL was considered to indicate a successful procedure. For the LC-CUSUM analysis, adequate performance was defined as a 10% failure rate, inadequate performance was defined as a 30% failure rate, and an acceptable deviation from adequate performance was defined as 10%. A limit h equal to 1.6 was selected to give a true-discovery rate of 90% and a false-discovery rate of 10% over 50 procedures.

RESULTS

A total of 340 consecutive procedures were performed by 20 trainees during the 12-month study period. The cumulative number of arthroscopic procedures performed by trainees before study start increased with increasing PGY. The median number of arthroscopic procedures performed per trainee was 16.5 (interquartile range, 14 to 21.75). Competency in knee arthroscopy for the TSCL was achieved by 8 trainees (40%), after a median of 16 procedures (interquartile range, 13 to 20), and for the GRS by only a single trainee (5%), after 14 procedures. Threshold-adjusted curves stratified by PGY level enabled multiple trainees to achieve competency for both the TSCL and GRS.

CONCLUSIONS

The LC-CUSUM can be successfully applied to knee arthroscopy to provide an individualized assessment of performance and quantitatively demonstrate competency for basic arthroscopic tasks.

CLINICAL RELEVANCE

The LC-CUSUM is an effective method to evaluate procedure competence in arthroscopic training and can provide objective feedback and benchmarks in the learning phase.