HipSim - hip fracture surgery simulation utilizing the Learning Curve-Cumulative Summation test (LC-CUSUM)

Setting proficiency standards for simulation-based mastery learning of short antegrade femoral nail osteosynthesis: a multicenter study

BACKGROUND AND PURPOSE

Orthopedic trainees frequently perform short antegrade femoral nail osteosynthesis of trochanteric fractures, but virtual reality simulation-based training (SBT) with haptic feedback has been unavailable. We explored a novel simulator, with the aim of gathering validity evidence for an embedded test and setting a credible pass/fail standard allowing trainees to practice to proficiency.

PATIENTS AND METHODS

The research, conducted from May to September 2020 across 3 Danish simulation centers, utilized the Swemac TraumaVision simulator for short antegrade femoral nail osteosynthesis. The validation process adhered to Messick's framework, covering all 5 sources of validity evidence. Participants included novice groups, categorized by training to plateau (n = 14) or to mastery (n = 10), and experts (n = 9), focusing on their performance metrics and training duration.

RESULTS

The novices in the plateau group and experts had hands-on training for 77 (95% confidence interval [CI] 59-95) and 52 (CI 36-69) minutes while the plateau test score, defined as the average of the last 4 scores, was 75% (CI 65-86) and 96% (CI 94-98) respectively. The pass/fail standard was established at the average expert plateau test score of 96%. All novices in the mastery group could meet this standard and interestingly without increased hands-on training time (65 [CI 46-84] minutes).

CONCLUSION

Our study provides supporting validity evidence from all sources of Messick's framework for a simulation-based test in short antegrade nail osteosynthesis of intertrochanteric hip fracture and establishes a defensible pass/fail standard for mastery learning of SBT. Novices who practiced using mastery learning were able to reach the pre-defined pass/fail standard and outperformed novices without a set goal for external motivation.

Fluoroscopic image-based behavior analysis can objectively explain subjective expert assessment of wire navigation skill

Psychomotor skill and decision-making efficiency in surgical wire navigation can be objectively evaluated by analysis of intraoperative fluoroscopic image sequences. Prior work suggests that such image-based behavior analysis of operating room (OR) performance can predict performer experience level (R2  = 0.62) and agree with expert opinion (the current standard) on the quality of a final implant construct (R2  = 0.59). However, it is unclear how objective image-based evaluation compares with expert assessments for entire technical OR performances. This study examines the relationships between three key variables: (1) objective image-based criteria, (2) expert opinions, and (3) performing surgeon experience level. A paired-comparison survey of seven experts, based upon eight OR fluoroscopic wire navigation image sequences, shows that the experts' preferences are best explained by objective metrics that reflect psychomotor and decision-making behaviors which are counter-productive to successful implant placement, like image count (R2  = 0.83) and behavior tally (R2  = 0.74). One such behavior, adjustments away from goal, uniquely correlated well with all three key variables: a fluoroscopic image-based analysis composite score (R2  = 0.40), expert consensus (R2  = 0.76), and performer experience (R2  = 0.41). These results confirm that experts view less efficient technical behavior as indicative of lesser technical proficiency. While expert assessments of technical skill were reliable and consistent, neither individual nor consensus expert opinion appears to correlate with performer experience (R2  = 0.11).

Poor usability of computer-assisted navigation for hip fracture surgery

INTRODUCTION

The STRYKER ADAPT computer-assisted navigation system provides intraoperative feedback to the surgeon regarding implant placement of the Gamma3 nail. The usability of the ADAPT system has not been evaluated. The aim of the study was to investigate the perceived usability of the ADAPT system.

MATERIALS AND METHODS

This was a descriptive study with prospectively collected data. ADAPT was introduced at Aarhus University Hospital in February 2021. Prior to introduction, surgeons at the department attended a general introduction to the system. ADAPT was introduced to the surgical nurses and was on display at the surgical ward at more than one occasion, where personal introduction to the system was possible. After introduction, it was mandatory to use ADAPT when using the Gamma3 nail to treat intertrochanteric femur fractures. After each procedure, primary and an eventual supervisor answered a questionnaire, which encompassed the System Usability Scale (SUS) questionnaire. The SUS is a ten-item questionnaire regarding the perceived usability of a system. SUS scores were translated to adjectives, describing user experience on a 7-point adjective scale (worst imaginable, awful, poor, ok, good, excellent, best imaginable). User acceptability, defined as "not acceptable", "marginal" or "acceptable", was also used to interpret the SUS scores.

RESULTS

ADAPT was used in 50 procedures by 29 different surgeons, with varying skill-level. Median SUS-score after first-time use of ADAPT for all 29 surgeons was 43 (range: 5-60), which translated to "poor" or "not acceptable". For surgeons who performed ≥ 3 ADAPT-assisted procedures, there were no statistically significant difference in their first to latest SUS-score (median difference: 4.3, p = 0.5). In free text comments ADAPT was positively described as helpful in placement of K-wire and providing educational opportunities for inexperienced surgeons and negatively as inconsistent, slow, time consuming, and causing excessive fluoroscopy.

CONCLUSIONS

Usability and acceptability of ADAPT was rated as "poor" or "not acceptable" by the majority of operating surgeons. ADAPT has not been used at our institution based on these findings. The System Usability Scale may be used in further research exploring usability and acceptability of novel computer-assisted navigation systems for orthopaedic surgery.

Use of Virtual Reality in the Education of Orthopaedic Procedures: A Randomised Control Study in Early Validation of a Novel Virtual Reality Simulator

Background Virtual reality (VR) simulation is a potential solution to the barriers surgical trainees are facing. There needs to be validation for its implementation within current training. We aimed to compare VR simulation to traditional methods in acquiring surgical skills for a TFN-ADVANCED™ Proximal Femoral Nailing System (TFNA; DePuy Synthes, Auckland, New Zealand) femoral nailing system. Methods Thirty-one surgical trainees were randomised to two groups: traditional-training group (control group) and a VR-training group (intervention group) for insertion of a short cephalomedullary TFNA nail. Both groups then inserted the same TFNA system into saw-bone femurs. Surveys evaluated validity of the relevant activities, perception of simulation, confidence, stress and anxiety. The primary outcomes were tip-apex distance (TAD) and user anxiety/confidence levels. Secondary outcomes included number of screw- and nail-guidewire insertion attempts, the time taken to complete and user validity of the VR system. Results There was no statistical difference in TAD between the intervention and control groups (9mm vs 15mm, p=0.0734). The only TAD at risk of cut-out was in the control group (25mm). There was no statistical difference in time taken (2547.5ss vs 2395ss, p=0.668), nail guide-wire attempts (two for both groups, p=0.355) and screw guide-wire attempts (one for both groups, p=0.702). The control group versus intervention had higher anxiety levels (50% vs 33%) and had lower confidence (61% vs 84%). Interpretation There was no objective difference in performance on a saw-bone model between groups. However, this VR simulator resulted in more confidence and lower anxiety levels whilst performing a simulated TFNA. Whilst further studies with larger sample sizes and exploration of transfer validity to the operating theatre are required, this study does indicate potential benefits of VR within surgical training.

A Dedicated Simulator Training Curriculum Improves Resident Performance in Surgical Management of Pediatric Supracondylar Humerus Fractures

Background

The primary goal of including simulation in residency training is to improve technical skills while working outside of the operating room. Such simulation-related skill improvements have seldom been measured in the operating room. This is largely because uncontrolled variables, such as injury severity, patient comorbidity, and anatomical variation, can bias evaluation of an operating surgeon's skill. In this study, performance during the wire navigation phase of pediatric supracondylar humerus fracture fixation was quantitatively compared between 2 groups of orthopaedic residents: a standard training group consisting of residents who participated in a single simulator session of wire navigation training and an expanded training group consisting of residents who participated in a dedicated multifaceted wire navigation simulation training curriculum.

Methods

To evaluate performance in the operating room, the full sequence of fluoroscopic images collected during wire navigation was quantitatively analyzed. Objective performance metrics included number of fluoroscopic images acquired, duration from placement of the first wire to that of the final wire, and wire spread at the level of the fracture. These metrics were measured from 97 pediatric supracondylar humerus fracture pinning surgeries performed by 28 different orthopaedic residents.

Results

No differences were observed between the groups for wire spread in the final fluoroscopic images (t(94) = 0.75, p = 0.45), an important clinical objective of the surgery. Residents who received the expanded simulator training used significantly fewer fluoroscopic images (mean of 46 vs. 61 images, t(85) = 2.25, p < 0.03) and required less time from first to final wire placement (mean of 11.2 vs. 14.9 minutes, t(83) = 2.53, p = 0.013) than the standard training group. A post hoc review of Accreditation Council for Graduate Medical Education case logs for 24 cases from the standard training group and for 21 cases from the expanded training group indicated that, at the time of surgeries, residents who received expanded training had completed fewer comparable cases than residents in the standard training group (mean of 13 vs. 21, t(42) = 2.40 p = 0.02). Further regression analysis indicated that the expanded simulator training produced an effect comparable with that associated with completing 10.5 similar surgical case experiences.

Conclusions

This study demonstrates that training on a wire navigation simulator can lead to improved performance in the operating room on a critical skill for all orthopaedic residents. By taking fewer images and less time while maintaining sufficient pin spread, simulator-trained residents were objectively measured to have improved performance in comparison with residents who had not participated in the pediatric elbow simulator curriculum.

Clinical Relevance

As programs aim to provide safe and effective training for critical orthopaedic skills such as pinning a pediatric elbow, this study demonstrates a simulator curriculum that has demonstrated the transfer of skill from a learning environment to the operating room.

Current status of virtual reality simulation education for orthopedic residents: the need for a change in focus

Introduction

Advances in technology are changing surgical education. Simulation provides an important adjunct to operative experience. This pedagogy has arguably become more important in light of the COVID-19 pandemic, with resultant reduction in operative exposure for trainees. Virtual reality (VR) simulators may provide significant contribution to experiential learning; however, much of the investigative focus to date has, correctly, been on establishing validity evidence for these constructs. The aim of this work was to perform a scoping review to assess the current status of VR simulation education to determine curricular development efforts for orthopedic residents.

Methods

With a trained medical librarian, searches of PubMed, EMBASE, and Web of Science were conducted for all articles in the last 10 years (September 2011-September 2021). Controlled vocabulary Medical Subject Headings (MeSH) terms and natural language developed with subject matter experts describing virtual reality or VR simulation and orthopedic training were used. Two trained reviewers evaluated all abstracts for inclusion. Exclusion criteria were all articles that did not assess VR simulation education involving orthopedic residents. Data were extracted from the included full-text articles including: study design, type of participants, type of VR simulation, simulated orthopedic skill, type of educational event, learner assessment including Kirkpatrick's level, assessment of quality using the Medical Education Research Study Quality Instrument (MERSQI), and level of effectiveness (LoE).

Results

Initial search identified 1,394 articles, of which 61 were included in the final qualitative synthesis. The majority (54%) were published in 2019- 2021, 49% in Europe. The commonest VR simulator was ArthroS (23%) and the commonest simulated skill was knee arthroscopy (33%). The majority of studies (70%) focused on simulator validation. Twenty-three studies described an educational module or curriculum, and of the 21 (34%) educational modules, 43% were one-off events. Most modules (18/21, 86%) assessed learners at Kirkpatrick level 2. With regard to methodological quality, 44% of studies had MERSQI 11.5-15 and 89% of studies had LoE of 2. Two studies had LoE of 3.

Conclusion

Current literature pertaining to VR training for orthopedic residents is focused on establishing validity and rarely forms part of a curriculum. Where the focus is education, the majority are discrete educational modules and do not teach a comprehensive amalgam of orthopedic skills. This suggests focus is needed to embed VR simulation training within formal curricula efforts guided by the work of Kern, and assess the efficacy of these against patient outcomes.

Learning Curve of USgHIFU Ablation for Uterine Fibroids: A Multi-Center Prospective Study

OBJECTIVES

To verify the stability of high-intensity focused ultrasound (HIFU) technology and the feasibility of training programs with learning curve cumulative summation (LC-CUSUM).

METHODS

A total of 12 physicians and 720 cases were equally assigned to the learning group and the control group, with 6 physicians and 360 cases per group. The learning group was treated by physicians without HIFU experience and the control group was treated by experienced physicians. Nonperfused volume (NPV) ratio was assessed by contrast-enhanced magnetic resonance imaging. Technical failure was defined as NPV ratio of uterine fibroids <70% and/or major complication, while <80% was set as a stricter standard of training qualification. LC-CUSUM was used to analyze the learning curve.

RESULTS

Physicians with or without HIFU experience in both groups achieved matchable NPV ratios, where a NPV ratio of 92.52% (16.06) was achieved by experienced physicians and 93.82% (16.95) by inexperienced physicians. No major complication was observed. The results of LC-CUSUM analysis showed that, with the standards of the NPV ratio of 70% or 80%, the learning group mastered the technique on the 11th case and the 16th case, respectively, while the control group was stable.

CONCLUSIONS

HIFU technology stayed stable in operation, with good safety and sound effectiveness, and was easy to learn. NPV ratio of 70% was considered as an appropriate indicator of training qualification. HIFU has remarkable prospects in achieving a NPV ratio of ≥80% without safety being compromised.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

A review of participant recruitment transparency for sound validation of hip surgery simulators: a novel umbrella approach

Malposition of implants is associated with complications, higher wear and increased revision rates in total hip replacement (THR) along with surgeon inexperience. Training THR residents to reach expert proficiency is affected by the high cost and resource limitations of traditional training techniques. Research in extended reality (XR) technologies can overcome such barriers. These offer a platform for learning, objective skill-monitoring and, potentially, for automated certification. Prior to their incorporation into curricula however, thorough validation must be undertaken. As validity is heavily dependent on the participants recruited, there is a need to review, scrutinise and define recruitment criteria in the absence of pre-defined standards, for sound simulator validation. A systematic review on PubMed and IEEE databases was conducted. Training simulator validation research in fracture, arthroscopy and arthroplasty relating to the hip was included. 46 validation studies were reviewed. It was observed that there was no uniformity in reporting or recruitment criteria, rendering cross-comparison challenging. This work developed Umbrella categories to help prioritise recruitment, and has formulated a detailed template of fields and guidelines for reporting criteria so that, in future, research may come to a consensus as to recruitment criteria for a hip "expert" or "novice".

Reducing orthopaedic theatre exposure during the COVID-19 lockdown: does a shift towards virtual reality-based training offer a solution?

PerspectiveOrthopaedic training in the United Kingdom has changed little from the Halstedian apprenticeship model of graduated responsibility, with the mantra "see one, do one, teach one". Whilst still relevant in surgical teaching, the current and ongoing disruption to surgical training secondary to the coronavirus disease 2019 (COVID-19) outbreak highlights the need for alternative methods of experiential surgical learning, which allow for the development of the knowledge, skills, and attitudes of orthopaedic surgeons, to be sought.