Diagnostic knee arthroscopy: a pilot study to evaluate surgical skills

Comparing Skill Acquisition and Validity of Immersive Virtual Reality with Cadaver Laboratory Sessions in Training for Reverse Total Shoulder Arthroplasty

Immersive virtual reality (iVR) allows surgical trainees to practice skills without risking harm to patients or the need for cadaveric training resources. However, iVR has never been directly compared with cadaver training, the longtime gold standard for surgical skill training. We aimed to compare skill acquisition using cadaver laboratory and iVR training methods for augmented baseplate implantation during reverse total shoulder arthroplasty (rTSA).

Methods

In a randomized controlled trial, junior orthopaedic surgery residents were assigned to a 1-hour training with either iVR or a cadaveric laboratory session with shoulder specimens. Before training, all participants viewed an overview lecture and technique video demonstrating key steps of augmented baseplate implantation for rTSA. Participants were assessed by a blinded evaluator using validated competency checklists during cadaveric glenoid baseplate implantation. Continuous and categorial variables were analyzed using the 2-sample t test and Fisher exact test.

Results

Fourteen junior residents (3 incoming matched postgraduate year [PGY1], 6 PGY1s, 1 PGY2, and 4 PGY3s) were randomized to training with either iVR (n = 6) or cadaver laboratory (n = 8). There were no significant differences in demographic data, previous experience with rTSA, or previous use of iVR (p > 0.05). There were no significant difference in total Objective Structured Assessment of Technical Skill score (91.2% [15.2] vs. 93.25% [6.32], -0.1406 to 0.1823, p = 0.763), Global Rating Scale score (4.708 [0.459] vs. 4.609 [0.465], -0.647 to 0.450, p = 0.699), or time to completion (546 seconds [158] vs. 591 seconds [192], -176.3 to 266.8, p = 0.655) in cadaveric glenoid baseplate implantation. Average cost of iVR hardware and a 1-year software license was $4,900, and average cost of a single cadaver laboratory was $1,268.20 per resident.

Conclusions

Among junior orthopaedic residents, there is similar skill acquisition when training with either cadaver laboratory or iVR. Although additional research into this field is needed, iVR may provide an important and cost-effective tool in surgical education.

Clinical Relevance

Emerging simulation and iVR technology simulation in surgical training programs can increase access to effective and high-level surgical training across the globe and improve quality of care.

The Diagnostic Arthroscopy Skill Score (DASS): a reliable and suitable assessment tool for arthroscopic skill training

PURPOSE

To develop and validate a novel score to more objectively assess the performance of diagnostic knee arthroscopy using a simulator.

METHODS

A Diagnostic Arthroscopy Skill Score (DASS) was developed by ten AGA (AGA-Society for Arthroscopy and Joint-Surgery) instructors for the assessment of arthroscopic skills. DASS consists of two parts: the evaluation of standardized diagnostic knee arthroscopy (DASS_part1) and the evaluation of manual dexterity, including ambidexterity and triangulation, using objective measurement parameters (DASS_part2). Content validity was determined by the Delphi method. One hundred and eleven videos of diagnostic knee arthroscopies were recorded during simulator training courses and evaluated by six specially trained instructors using DASS. Construct validity, measurement error calculated by the minimum detectable change (MDC), internal consistency using Cronbach's alpha and interrater and intrarater reliability were assessed. The Bland-Altman method was used to calculate the intrarater agreement.

RESULTS

Six skill domains were identified and evaluated for each knee compartment. DASS, DASS_part1, and DASS_part2 showed construct validity, with experts achieving significantly higher scores than competents and novices. MDC was 4.5 ± 1.7 points for DASS_part1. There was high internal consistency for all domains in each compartment from 0.78 to 0.86. The interrater reliability showed high agreement between the six raters (ICC = 0.94). The evaluation of intrarater reliability demonstrated good and excellent agreement for five raters (ICC > 0.80) and moderate agreement for one rater (ICC = 0.68). The Bland-Altman comparison showed no difference between the first and second evaluations in five out of six raters. Precision, estimated by the regression analysis and comparison with the method of Bland and Altman, was excellent for four raters and moderate for two raters.

CONCLUSIONS

The results of this study indicate good validity and reliability of DASS for the assessment of the surgical performance of diagnostic knee arthroscopy during simulator training. Standardized training is recommended before arthroscopy surgery is considered in patients.

LEVEL OF EVIDENCE

II.

Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training: A Systematic Review

BACKGROUND

Robust assessment of skills acquisition and surgical performance during training is vital to ensuring operative competence among orthopaedic surgeons. A move to competency-based surgical training requires the use of tools that can assess surgical skills objectively and systematically. The aim of this systematic review was to describe the evidence for the utility of assessment tools used in evaluating operative performance in trauma and orthopaedic surgical training.

METHODS

We performed a comprehensive literature search of MEDLINE, Embase, and Google Scholar databases to June 2019. From eligible studies we abstracted data on study aim, assessment format (live theater or simulated setting), skills assessed, and tools or metrics used to assess surgical performance. The strengths, limitations, and psychometric properties of the assessment tools are reported on the basis of previously defined utility criteria.

RESULTS

One hundred and five studies published between 1990 and 2019 were included. Forty-two studies involved open orthopaedic surgical procedures, and 63 involved arthroscopy. The majority (85%) were used in the simulated environment. There was wide variation in the type of assessment tools in used, the strengths and weaknesses of which are assessor and setting-dependent.

CONCLUSIONS

Current technical skills-assessment tools in trauma and orthopaedic surgery are largely procedure-specific and limited to research use in the simulated environment. An objective technical skills-assessment tool that is suitable for use in the live operative theater requires development and validation, to ensure proper competency-based assessment of surgical performance and readiness for unsupervised clinical practice.

CLINICAL RELEVANCE

Trainers and trainees can gain further insight into the technical skills assessment tools that they use in practice through the utility evidence provided.

Impact of cadaveric practices on arthroscopic knee training for Orthopaedic Surgery and Traumatology residents

BACKGROUND AND OBJECTIVES

There currently is a considerable variability in the content of training programs and courses for orthopaedic surgeons in training. Besides this, there is still no consensus regarding which is the most effective methodology for improving learning of the technique or for its qualitative evaluation. A study was carried out to assess the influence of carrying out a cadaveric practical course on the acquisition of arthroscopic technical skills in fifth-year residents.

MATERIAL AND METHODS

A knee arthroscopy cadaveric course was performed in which 48 students participated. Different technical aspects during a knee arthroscopy, as well as the residents skill and the execution time were evaluated before and after the course using a specific questionnaire.

RESULTS

There was a significant statistic improvement between the initial and final evaluation, increasing from 11,7 till 17,1 (maximum 20) points. Also, a significant improvement in all the parameters analysed was observed; the identification of anatomic structures (p = 0.001), the palpation of specific structures (p = 0.032), and the time necessary for the arthroscopic knee check (p = 0.015).

CONCLUSIONS

A knee arthroscopy cadaveric course improves significantly the technical skills of surgeons in training.

Three days of training with a low-fidelity arthroscopy triangulation simulator box improves task performance in a virtual reality high-fidelity virtual knee arthroscopy simulator

PURPOSE

The aim of this work is to evaluate the effectiveness of training with the low-fidelity ArthroBox^® regarding performance of different basic arthroscopy tasks using a validated high-fidelity virtual reality simulator of the knee.

METHODS

Nineteen volunteers (14 females and 5 males) without any previous experience in arthroscopy were randomly assigned either to the ArthroBox^® training group (n =10) or the non-training group (n =9). The training group underwent a supervised ArthroBox ^® training consisting of a daily 60-min session for three consecutive days. Both groups completed the basic and the final assessment using a validated virtual reality-based passive haptic knee arthroscopy simulator (ArthroS, VirtaMed™). The following three factors were measured in different exercises (explained in "Materials and methods"): amount of time to finish the task, length of camera and scope path within the joint. Furthermore, the volunteers' demographics (age, sex, dexterity, video game experience, sport activities and profession) was assessed but showed no differences between the groups.

RESULTS

There were no significant differences between the training and non-training group regarding the above-mentioned demographic factors. However, the training group showed significant improvement from baseline to follow-up in most activities (e.g. task performance time in seconds, intra-articular camera and grasp distance in centimetres; see Table 1) in comparison to the non-training group.

CONCLUSIONS

The results from this study demonstrate that training for three consecutive days using a portable and versatile low-fidelity simulator significantly improves arthroscopy performance when using a validated high-fidelity virtual knee simulator. Arthroscopic triangulation training outside the operating theatre with a portable, low-cost simulator has proven to be a valuable educational tool to improve the arthroscopic skills of trainee surgeons.

LEVEL OF EVIDENCE

Diagnostic study, Level II.

Proficiency-Based Training Using Simulator-Based Tools Could be Validated for Certification of Surgical Procedural Proficiency

Airline pilots can learn how to fly without ever leaving the ground, and pilots are required to pass proficiency checks using simulator-based tools to ensure they are competent, and remain competent, to fly specific types of planes under specific circumstances. In contrast, generally, surgical training is based on an apprenticeship model, and surgical certification does not require demonstration of hands-on skills in performing basic, advanced, or subspecialized procedures. Proficiency-based training, or PBT, also known as proficiency-based progression or PBP, is a tool that can be used to train surgeons to safely and efficiently perform surgical procedures. Moreover, PBT, simulators, and similar tools can be used to evaluate a surgeon's proficiency. Ultimately, in the interest of quality, value, and patient safety, objective evaluation of hands-on surgical skill using simulators should be implemented. In the interest of fairness to surgeons, certification must be developed with a goal of not only testing but supporting development of procedural proficiency. As we follow our peers in the airline industry, pathways chosen to validate surgical competence must be evidence-based, fair, and valid.

Cigar Box Arthroscopy: A Randomized Controlled Trial Validates Nonanatomic Simulation Training of Novice Arthroscopy Skills

PURPOSE

The goal of this study was to validate the cigar box arthroscopy trainer (CBAT) as a training tool and then compare its effectiveness to didactic training and to another previously validated low-fidelity but anatomic model, the anatomic knee arthroscopy trainer (AKAT).

METHODS

A nonanatomic knee arthroscopy training module was developed at our institution. Twenty-four medical students with no prior arthroscopic or laparoscopic experience were enrolled as subjects. Eight subjects served as controls. The remaining 16 subjects were randomized to participate in 4 hours of either the CBAT or a previously validated AKAT. Subjects' skills were assessed by 1 of 2 faculty members through repeated attempts at performing a diagnostic knee arthroscopy on a cadaveric specimen. Objective scores were given using a minimally adapted version of the Basic Arthroscopic Knee Skill Scoring System. Total cost differences were calculated.

RESULTS

Seventy-five percent of subjects in the CBAT and AKAT groups succeeded in reaching minimum proficiency in the allotted time compared with 25% in the control group (P < .05). There was no significant difference in the number of attempts to reach proficiency between the CBAT and AKAT groups. The cost to build the CBAT was $44.12, whereas the cost was $324.33 for the AKAT.

CONCLUSIONS

This pilot study suggests the CBAT is an effective knee arthroscopy trainer that may decrease the learning curve of residents without significant cost to a residency program. This study demonstrates the need for an agreed-upon objective scoring system to properly evaluate residents and compare the effectiveness of different training tools.

Performance Assessment of Arthroscopic Rotator Cuff Repair and Labral Repair in a Dry Shoulder Simulator

PURPOSE

To evaluate the use of dry models to assess performance of arthroscopic rotator cuff repair (RCR) and labral repair (LR).

METHODS

Residents, fellows, and sports medicine staff performed an arthroscopic RCR and LR on a dry model. Any prior RCR and LR experience was noted. Staff surgeons assessed participants by use of task-specific checklists, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), and a final overall global rating. All procedures were video recorded and were scored by a fellow blinded to the year of training of each participant.

RESULTS

A total of 51 participants and 46 participants performed arthroscopic RCR and LR, respectively, on dry models. The internal consistency or reliability (Cronbach α) using the total ASSET score for the RCR and LR was high (>0.9). One-way analysis of variance for the total ASSET score showed a difference between participants based on year of training (P < .001) for both procedures. The inter-rater reliability for the ASSET score was excellent (>0.9) for both procedures. A good correlation was seen between the ASSET score and the year of training, as well as the previous number of sports rotations.

CONCLUSIONS

The results of this study show evidence of construct validity when using dry models to assess performance of arthroscopic RCR and LR by residents.

CLINICAL RELEVANCE

The results of this study support the use of arthroscopic simulation in the training of residents and fellows learning arthroscopic shoulder surgery.

Validation of a Dry Model for Assessing the Performance of Arthroscopic Hip Labral Repair

BACKGROUND

Arthroscopic hip labral repair is a technically challenging and demanding surgical technique with a steep learning curve. Arthroscopic simulation allows trainees to develop these skills in a safe environment.

PURPOSE

The purpose of this study was to evaluate the use of a combination of assessment ratings for the performance of arthroscopic hip labral repair on a dry model.

STUDY DESIGN

Cross-sectional study; Level of evidence, 3.

METHODS

A total of 47 participants including orthopaedic surgery residents (n = 37), sports medicine fellows (n = 5), and staff surgeons (n = 5) performed arthroscopic hip labral repair on a dry model. Prior arthroscopic experience was noted. Participants were evaluated by 2 orthopaedic surgeons using a task-specific checklist, the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task completion time, and a final global rating scale. All procedures were video-recorded and scored by an orthopaedic fellow blinded to the level of training of each participant.

RESULTS

The internal consistency/reliability (Cronbach alpha) using the total ASSET score for the procedure was high (intraclass correlation coefficient > 0.9). One-way analysis of variance for the total ASSET score demonstrated a difference between participants based on the level of training ( F_3,43 = 27.8, P < .001). A good correlation was seen between the ASSET score and previous exposure to arthroscopic procedures ( r = 0.52-0.73, P < .001). The interrater reliability for the ASSET score was excellent (>0.9).

CONCLUSION

The results of this study demonstrate that the use of dry models to assess the performance of arthroscopic hip labral repair by trainees is both valid and reliable. Further research will be required to demonstrate a correlation with performance on cadaveric specimens or in the operating room.

Virtual Reality Compared with Bench-Top Simulation in the Acquisition of Arthroscopic Skill: A Randomized Controlled Trial

BACKGROUND

Work-hour restrictions as set forth by the Accreditation Council for Graduate Medical Education (ACGME) and other governing bodies have forced training programs to seek out new learning tools to accelerate acquisition of both medical skills and knowledge. As a result, competency-based training has become an important part of residency training. The purpose of this study was to directly compare arthroscopic skill acquisition in both high-fidelity and low-fidelity simulator models and to assess skill transfer from either modality to a cadaveric specimen, simulating intraoperative conditions.

METHODS

Forty surgical novices (pre-clerkship-level medical students) voluntarily participated in this trial. Baseline demographic data, as well as data on arthroscopic knowledge and skill, were collected prior to training. Subjects were randomized to 5-week independent training sessions on a high-fidelity virtual reality arthroscopic simulator or on a bench-top arthroscopic setup, or to an untrained control group. Post-training, subjects were asked to perform a diagnostic arthroscopy on both simulators and in a simulated intraoperative environment on a cadaveric knee. A more difficult surprise task was also incorporated to evaluate skill transfer. Subjects were evaluated using the Global Rating Scale (GRS), the 14-point arthroscopic checklist, and a timer to determine procedural efficiency (time per task). Secondary outcomes focused on objective measures of virtual reality simulator motion analysis.

RESULTS

Trainees on both simulators demonstrated a significant improvement (p < 0.05) in arthroscopic skills compared with baseline scores and untrained controls, both in and ex vivo. The virtual reality simulation group consistently outperformed the bench-top model group in the diagnostic arthroscopy crossover tests and in the simulated cadaveric setup. Furthermore, the virtual reality group demonstrated superior skill transfer in the surprise skill transfer task.

CONCLUSIONS

Both high-fidelity and low-fidelity simulation trainings were effective in arthroscopic skill acquisition. High-fidelity virtual reality simulation was superior to bench-top simulation in the acquisition of arthroscopic skills, both in the laboratory and in vivo. Further clinical investigation is needed to interpret the importance of these results.

Computer-Simulated Arthroscopic Knee Surgery: Effects of Distraction on Resident Performance

Orthopedic surgeons cite "full focus" and "distraction control" as important factors for achieving excellent outcomes. Surgical simulation is a safe and cost-effective way for residents to practice surgical skills, and it is a suitable tool to study the effects of distraction on resident surgical performance. This study investigated the effects of distraction on arthroscopic knee simulator performance among residents at various levels of experience. The authors hypothesized that environmental distractions would negatively affect performance. Twenty-five orthopedic surgery residents performed a diagnostic knee arthroscopy computer simulation according to a checklist of structures to identify and tasks to complete. Participants were evaluated on arthroscopy time, number of chondral injuries, instances of looking down at their hands, and completion of checklist items. Residents repeated this task at least 2 weeks later while simultaneously answering distracting questions. During distracted simulation, the residents had significantly fewer completed checklist items (P<.02) compared with the initial simulation. Senior residents completed the initial simulation in less time (P<.001), with fewer chondral injuries (P<.005) and fewer instances of looking down at their hands (P<.012), compared with junior residents. Senior residents also completed 97% of the diagnostic checklist, whereas junior residents completed 89% (P<.019). During distracted simulation, senior residents continued to complete tasks more quickly (P<.006) and with fewer instances of looking down at their hands (P<.042). Residents at all levels appear to be susceptible to the detrimental effects of distraction when performing arthroscopic simulation. Addressing even straightforward questions intraoperatively may affect surgeon performance.

Arthroscopic skills assessment and use of box model for training in arthroscopic surgery using Sawbones - "FAST" workstation

PURPOSE

Arthroscopic skills training outside the operative room may decrease risks and errors by trainee surgeons. There is a need of simple objective method for evaluating proficiency and skill of arthroscopy trainees using simple bench model of arthroscopic simulator. The aim of this study is to correlate motor task performance to level of prior arthroscopic experience and establish benchmarks for training modules.

METHODS

Twenty orthopaedic surgeons performed a set of tasks to assess a) arthroscopic triangulation, b) navigation, c) object handling and d) meniscus trimming using SAWBONES "FAST" arthroscopy skills workstation. Time to completion and the errors were computed. The subjects were divided into four levels; "Novice", "Beginner", "Intermediate" and "Advanced" based on previous arthroscopy experience, for analyses of performance.

RESULTS

The task performance under transparent dome was not related to experience of the surgeon unlike opaque dome, highlighting the importance of hand-eye co-ordination required in arthroscopy. Median time to completion for each task improved as the level of experience increased and this was found to be statistically significant (p < .05) e.g. time for maze navigation (Novice - 166 s, Beginner - 135.5 s, Intermediate - 100 s, Advance - 97.5 s) and the similar results for all tasks. Majority (>85%) of subjects across all the levels reported improvement in performance with sequential tasks.

CONCLUSION

Use of the arthroscope requires visuo-spatial coordination which is a skill that develops with practice. This simple box model can reliably differentiate the arthroscopic skills based on experience and can be used to monitor progression of skills of trainees in institutions.

Global Rating Scales and Motion Analysis Are Valid Proficiency Metrics in Virtual and Benchtop Knee Arthroscopy Simulators

BACKGROUND

Work-hour restrictions and fatigue management strategies in surgical training programs continue to evolve in an effort to improve the learning environment and promote safer patient care. In response, training programs must reevaluate how various teaching modalities such as simulation can augment the development of surgical competence in trainees. For surgical simulators to be most useful, it is important to determine whether surgical proficiency can be reliably differentiated using them. To our knowledge, performance on both virtual and benchtop arthroscopy simulators has not been concurrently assessed in the same subjects.

QUESTIONS/PURPOSES

(1) Do global rating scales and procedure time differentiate arthroscopic expertise in virtual and benchtop knee models? (2) Can commercially available built-in motion analysis metrics differentiate arthroscopic expertise? (3) How well are performance measures on virtual and benchtop simulators correlated? (4) Are these metrics sensitive enough to differentiate by year of training?

METHODS

A cross-sectional study of 19 subjects (four medical students, 12 residents, and three staff) were recruited and divided into 11 novice arthroscopists (student to Postgraduate Year [PGY] 3) and eight proficient arthroscopists (PGY 4 to staff) who completed a diagnostic arthroscopy and loose-body retrieval in both virtual and benchtop knee models. Global rating scales (GRS), procedure times, and motion analysis metrics were used to evaluate performance.

RESULTS

The proficient group scored higher on virtual (14 ± 6 [95% confidence interval {CI}, 10-18] versus 36 ± 5 [95% CI, 32-40], p < 0.001) and benchtop (16 ± 8 [95% CI, 11-21] versus 36 ± 5 [95% CI, 31-40], p < 0.001) GRS scales. The proficient subjects completed nearly all tasks faster than novice subjects, including the virtual scope (579 ±169 [95% CI, 466-692] versus 358 ± 178 [95% CI, 210-507] seconds, p = 0.02) and benchtop knee scope + probe (480 ± 160 [95% CI, 373-588] versus 277 ± 64 [95% CI, 224-330] seconds, p = 0.002). The built-in motion analysis metrics also distinguished novices from proficient arthroscopists using the self-generated virtual loose body retrieval task scores (4 ± 1 [95% CI, 3-5] versus 6 ± 1 [95% CI, 5-7], p = 0.001). GRS scores between virtual and benchtop models were very strongly correlated (ρ = 0.93, p < 0.001). There was strong correlation between year of training and virtual GRS (ρ = 0.8, p < 0.001) and benchtop GRS (ρ = 0.87, p < 0.001) scores.

CONCLUSIONS

To our knowledge, this is the first study to evaluate performance on both virtual and benchtop knee simulators. We have shown that subjective GRS scores and objective motion analysis metrics and procedure time are valid measures to distinguish arthroscopic skill on both virtual and benchtop modalities. Performance on both modalities is well correlated. We believe that training on artificial models allows acquisition of skills in a safe environment. Future work should compare different modalities in the efficiency of skill acquisition, retention, and transferability to the operating room.

Simulation of Anterior Cruciate Ligament Reconstruction in a Dry Model

BACKGROUND

As the demand increases for demonstration of competence in surgical skill, the need for validated assessment tools also increases.

HYPOTHESIS/PURPOSE

The purpose of this study was to validate a dry knee model for the assessment of performance of anterior cruciate ligament reconstruction (ACLR). The hypothesis was that the combination of a checklist and a previously validated global rating scale would be a valid and reliable means of assessing ACLR when performed by residents in a dry model.

STUDY DESIGN

Controlled laboratory study.

METHODS

All residents, sports medicine staff, and fellows were invited to perform a hamstring ACLR using anteromedial drilling and Endobutton fixation on a dry model of an anterior cruciate ligament. Previous exposure to knee arthroscopy and ACLR was recorded. A detailed surgical manuscript and technique video were sent to all participants before the study. Residents were evaluated by staff surgeons with task-specific checklists created by use of a modified Delphi procedure and the Arthroscopic Surgical Skill Evaluation Tool (ASSET). Each procedure (hand movements and arthroscopic video) was recorded and scored by a fellow blinded to the year of training of each participant.

RESULTS

A total of 29 residents, 5 fellows, and 6 staff surgeons (40 participants total) performed an ACLR on the dry model. The internal reliability (Cronbach alpha) of the test when using the total ASSET score was very high (>0.9). One-way analysis of variance for the total ASSET score and the total checklist score demonstrated a difference between participants based on year of training (P < .001). Post hoc analysis demonstrated a significant difference in global ratings and checklist scores between junior residents (postgraduate year [PGY] 1-3) and senior residents (PGY 4 and 5) and senior residents and fellows (P < .05). A significant difference was seen between fellows and staff on the global rating (P < .05) but not on the checklist scores (P > .05). A good correlation was seen between the total ASSET score and prior exposure to knee arthroscopy (0.73) and ACLR (0.65). The interrater reliability (intraclass correlation coefficient) between the examiner ratings and the blinded assessor ratings for the total ASSET score was very high (>0.8).

CONCLUSION

The results of this study provide evidence that the performance of an ACLR in a dry model is a reliable method of assessing a resident's knowledge of the steps and instrumentation required, and the method shows evidence of validity.

CLINICAL RELEVANCE

These models may be used to ensure a minimal level of competence before residents and fellows perform ACLR in the operating room.

Assessing Diagnostic Arthroscopy Performance in the Operating Room Using the Arthroscopic Surgery Skill Evaluation Tool (ASSET)

PURPOSE

To determine the validity and reliability of using the Arthroscopic Surgery Skill Evaluation Tool (ASSET) to assess arthroscopic skill in the operating room.

METHODS

Eight orthopaedic residents, 1 sports medicine fellow, and 3 sports medicine faculty members recorded the diagnostic portion of 3 shoulder and 3 knee procedures in the operating room. Two blinded raters used the ASSET to assess each recorded procedure video. Criterion for a passing score on a procedure was attaining a 3 or greater in all 8 domains assessed.

RESULTS

In total 70 videos (36 shoulder, 34 knee) were evaluated by each rater. The attending/fellow group was assigned significantly higher mean ASSET scores compared with resident groups for both procedures (P = .01). The attending/fellow group also had the highest passing percentage (95.8%). Raters were in agreement for total ASSET scores assigned to both diagnostic arthroscopy of the shoulder (intraclass correlation coefficient [ICC] = 0.84) and knee (ICC = 0.81). Agreement on individual ASSET domains was moderate (ICC = 0.61 to 0.80) for all domains except safety and difficulty of procedure. Raters concurred on the pass-fail evaluation in 62 of 70 (88.6%) of procedures evaluated.

CONCLUSIONS

Using the ASSET to assess surgical skills in the operating room is feasible, reliable, and valid.

CLINICAL RELEVANCE

When combined with previously published results using the ASSET, the ASSET may provide a validated and reliable method for evaluating arthroscopic surgical skills in the surgical simulation lab and operating room.

The Shoulder Objective Practical Assessment Tool: Evaluation of a New Tool Assessing Residents Learning in Diagnostic Shoulder Arthroscopy

PURPOSE

To design and validate an objective practical assessment tool for diagnostic shoulder arthroscopy that would provide residents with a method to evaluate their progression in this field of surgery and to identify specific learning needs.

METHODS

We designed and evaluated the shoulder Objective Practical Assessment Tool (OPAT). The shoulder OPAT was designed by us, and scoring domains were created using a Delphi process. The shoulder OPAT was trialed by members of the British Elbow & Shoulder Society Education Committee for internal consistency and ease of use before being offered to other trainers and residents. Inter-rater reliability and intrarater reliability were calculated. One hundred forty orthopaedic residents, of varying seniority, within 5 training regions in the United Kingdom, were questioned regarding the tool. A pilot study of 6 residents was undertaken.

RESULTS

Internal consistency was 0.77 (standardized Cronbach α). Inter-rater reliability was 0.60, and intrarater reliability was 0.82. The Spearman correlation coefficient (r) between the global summary score for the shoulder OPAT and the current assessment tool used in postgraduate training for orthopaedic residents undertaking diagnostic shoulder arthroscopy equaled 0.74. Of the residents, 82% agreed or strongly agreed when asked if the shoulder OPAT would be a useful tool in monitoring progression and 72% agreed or strongly agreed with the introduction of the shoulder OPAT within the orthopaedic domain.

CONCLUSIONS

This study shows that the shoulder OPAT fulfills several aspects of reliability and validity when tested. Despite the inter-rater reliability being 0.60, we believe that the shoulder OPAT has the potential to play a role alongside the current assessment tool in the training of orthopaedic residents.

CLINICAL RELEVANCE

The shoulder OPAT can be used to assess residents during shoulder arthroscopy and has the potential for use in medical education, as well as arthroscopic skills training in the operating theater.

Orthopaedic Residents Improve Confidence and Knot-Tying Speed With a Skills Course

PURPOSE

To determine the effect of a knot-tying module, within an arthroscopic training course, on resident speed, resident confidence, and biomechanical quality of arthroscopically tied knots.

METHODS

Sixty-four participants (8 postgraduate year [PGY]-3 and 8 PGY-4 orthopaedic residents annually for 4 years) were enrolled in a 5-day training course, which included a daily knot-tying module. Self-assessed confidence was obtained by pre-course (day 1) and post-course (day 5) questionnaire. Each participant tied 5 sequential knots using an arthroscopic knot-tying station. Time per knot was recorded in seconds. Knots were later preloaded, cycled, and tested for peak load to failure and displacement change. Mean peak load to failure, displacement change, speed, and confidence were compared before and after training.

RESULTS

The mean time to complete 5 knots was significantly faster after training (12.8 minutes before the course [day 1] v 9.39 minutes after the course [day 5]) (P < .0001). Confidence improved from pre-course (mean, 3.3) to post-course (mean, 7.8) questionnaires (P < .0001). No statistically significant difference was found between peak force for pre-course (mean, 136 N) and post-course (mean, 138 N) knots (P = .076). No statistically significant difference was detected in mean displacement change (mean, 3.51 mm before the course v 3.57 mm after the course) (P = .61). Comparison of PGY-3 and PGY-4 residents was significant only for a higher pre-course confidence in PGY-4 residents (P = .02).

CONCLUSIONS

Participation in an arthroscopic knot-tying module improves resident speed and confidence in tying arthroscopic knots. Our data did not show a significant change in peak load to failure or loop security with training. These findings suggest that participation in a knot-tying module improves efficiency regarding arthroscopic knot tying by residents.

CLINICAL RELEVANCE

Residents who practice arthroscopic knot tying 5 days per year as part of an arthroscopic training course may be more efficient in the operating room.

The knee arthroscopy learning curve: quantitative assessment of surgical skills

PURPOSE

To assess orthopaedic trainees performing diagnostic knee arthroscopies and evaluate procedural competence using a novel statistical method, the Cumulative Summation Test for Learning Curve (LC-CUSUM).

METHODS

Twenty orthopaedic trainees in postgraduate year (PGY) 1 through 5 performed diagnostic knee arthroscopy and were evaluated intraoperatively with a validated 10-point knee task-specific checklist (TSCL) and 50-point global rating scale (GRS). A score of 40 points or greater (of 50 points) for the GRS and 8 points or greater (of 10 points) for the knee TSCL was considered to indicate a successful procedure. For the LC-CUSUM analysis, adequate performance was defined as a 10% failure rate, inadequate performance was defined as a 30% failure rate, and an acceptable deviation from adequate performance was defined as 10%. A limit h equal to 1.6 was selected to give a true-discovery rate of 90% and a false-discovery rate of 10% over 50 procedures.

RESULTS

A total of 340 consecutive procedures were performed by 20 trainees during the 12-month study period. The cumulative number of arthroscopic procedures performed by trainees before study start increased with increasing PGY. The median number of arthroscopic procedures performed per trainee was 16.5 (interquartile range, 14 to 21.75). Competency in knee arthroscopy for the TSCL was achieved by 8 trainees (40%), after a median of 16 procedures (interquartile range, 13 to 20), and for the GRS by only a single trainee (5%), after 14 procedures. Threshold-adjusted curves stratified by PGY level enabled multiple trainees to achieve competency for both the TSCL and GRS.

CONCLUSIONS

The LC-CUSUM can be successfully applied to knee arthroscopy to provide an individualized assessment of performance and quantitatively demonstrate competency for basic arthroscopic tasks.

CLINICAL RELEVANCE

The LC-CUSUM is an effective method to evaluate procedure competence in arthroscopic training and can provide objective feedback and benchmarks in the learning phase.

Validating a global rating scale to monitor individual resident learning curves during arthroscopic knee meniscal repair

PURPOSE

To determine whether a global rating scale (GRS) with construct validity can also be used to assess the learning curve of individual orthopaedic trainees during simulated arthroscopic knee meniscal repair.

METHODS

An established arthroscopic GRS was used to evaluate the technical skill of 19 orthopaedic residents performing a standardized arthroscopic meniscal repair in a bioskills laboratory. The residents had diagnostic knee arthroscopy experience but no experience with arthroscopic meniscal repair. Residents were videotaped performing an arthroscopic meniscal repair on 12 separate occasions. Their performance was assessed by use of the GRS and motion analysis objectively measuring the time taken to complete tasks, path length of the subject's hands, and number of hand movements. One author assessed all 228 videos, whereas 2 other authors rated 34 randomly selected videos, testing the interobserver reliability of the GRS. The validity of the GRS was tested against the motion analysis.

RESULTS

Objective assessment with motion analysis defined the surgeon's learning curve, showing significant improvement by each subject over 12 episodes (P < .0001). The GRS also showed a similar learning curve with significant improvements in performance (P < .0001). The median GRS score improved from 15 of 34 (interquartile range, 14 to 17) at baseline to 22 of 34 (interquartile range, 19 to 23) in the final period. There was a moderate correlation (P < .0001, Spearman test) between the GRS and motion analysis parameters (r = -0.58 for time, r = -0.58 for path length, and r = -0.51 for hand movements). The inter-rater reliability among 3 trained assessors using the GRS was excellent (Cronbach α = 0.88).

CONCLUSIONS

When compared with motion analysis, an established arthroscopic GRS, with construct validity, also offers a moderately feasible method to monitor the learning curve of individual residents during simulated knee meniscal repair.

CLINICAL RELEVANCE

An arthroscopic GRS can be used for monitoring skill improvement during knee meniscal repair and has the potential for use as a training and assessment tool in the real operating room.

Arthroscopic proficiency: methods in evaluating competency

BACKGROUND

The current paradigm of arthroscopic training lacks objective evaluation of technical ability and its adequacy is concerning given the accelerating complexity of the field. To combat insufficiencies, emphasis is shifting towards skill acquisition outside the operating room and sophisticated assessment tools. We reviewed (1) the validity of cadaver and surgical simulation in arthroscopic training, (2) the role of psychomotor analysis and arthroscopic technical ability, (3) what validated assessment tools are available to evaluate technical competency, and (4) the quantification of arthroscopic proficiency.

METHODS

The Medline and Embase databases were searched for published articles in the English literature pertaining to arthroscopic competence, arthroscopic assessment and evaluation and objective measures of arthroscopic technical skill. Abstracts were independently evaluated and exclusion criteria included articles outside the scope of knee and shoulder arthroscopy as well as original articles about specific therapies, outcomes and diagnoses leaving 52 articles cited in this review.

RESULTS

Simulated arthroscopic environments exhibit high levels of internal validity and consistency for simple arthroscopic tasks, however the ability to transfer complex skills to the operating room has not yet been established. Instrument and force trajectory data can discriminate between technical ability for basic arthroscopic parameters and may serve as useful adjuncts to more comprehensive techniques. There is a need for arthroscopic assessment tools for standardized evaluation and objective feedback of technical skills, yet few comprehensive instruments exist, especially for the shoulder. Opinion on the required arthroscopic experience to obtain proficiency remains guarded and few governing bodies specify absolute quantities.

CONCLUSIONS

Further validation is required to demonstrate the transfer of complex arthroscopic skills from simulated environments to the operating room and provide objective parameters to base evaluation. There is a deficiency of validated assessment tools for technical competencies and little consensus of what constitutes a sufficient case volume within the arthroscopy community.