Proficiency Levels and Validity Evidence for Scoring Metrics for a Virtual Reality and Inanimate Robotic Surgery Simulation Curriculum

OBJECTIVE

Our institution recently implemented a virtual reality (VR) skills curriculum for general surgery residents using the SimNow simulator. Based on a content alignment study, we revised the curriculum to include only 20 of 33 VR tasks and we added 3 previously validated inanimate tasks. The purpose of this study was to establish expert-derived proficiency levels for all tasks and to evaluate the validity of the scoring for the VR tasks.

DESIGN

Two expert robotic surgeons performed 5 repetitions of each VR and inanimate task. The trimmed mean (lowest scoring attempt and outliers [>2 standard deviations] were eliminated) was defined as the expert level for each task. For the VR tasks, expert levels were compared to resident performance to evaluate validity.

SETTING

This study was conducted at the University of Texas Southwestern Medical Center (Dallas, TX), a tertiary care academic teaching hospital.

PARTICIPANTS

Two expert robotic surgeons participated in this study. The data from 42 residents (PGY2-4) who completed the original curriculum was used to represent novice performance.

RESULTS

Comparison of expert levels and resident performance was statistically significant for 15 VR tasks (supporting validity) and approached significance (p = 0.06, 0.09) for 2 VR tasks; expert levels were designated as proficiency levels for these 17 tasks. Group comparisons were clearly not significant (p = 0.2-0.8) for 3 VR tasks; 2 of these 3 tasks were retained as introductory exercises (with 3 repetitions required) and 1 was excluded. For the 3 inanimate tasks, expert levels minus 2 standard deviations were designated as proficiency levels.

CONCLUSIONS

This analysis generated validity evidence for 15 VR tasks and established expert-derived proficiency levels for 17 VR tasks and 3 inanimate tasks. Our proposed curriculum now consists of 19 VR and 3 inanimate tasks using the selected proficiency levels. We anticipate that this design will maximize curriculum efficiency and effectiveness.

Baseline performance in a robotic virtual reality platform predicts rate of skill acquisition in a proficiency-based curriculum: a cohort study of surgical trainees

BACKGROUND

Residency programs must prepare to train the next generation of surgeons on the robotic platform. The purpose of this study was to determine if baseline skills of residents on a virtual reality (VR) robotic simulator before intern year predicted future performance in a proficiency-based curriculum.

METHODS

Across two academic years, 21 general surgery PGY-1s underwent the robotic surgery boot camp at the University of Texas Southwestern. During boot camp, subjects completed five previously validated VR tasks, and their performance metrics (score, time, and economy of motion [EOM]) were extracted retrospectively from their Intuitive learning accounts. The same metrics were assessed during their residency until they reached previously validated proficiency benchmarks. Outcomes were defined as the score at proficiency, attempts to reach proficiency, and time to proficiency. Spearman's rho and Mann-Whitney U tests were used; median (IQR) was reported. Significance level was set at p < 0.05.

RESULTS

Twenty-one residents completed at least three out of the five boot camp tasks and achieved proficiency in the former during residency. The median average score at boot camp was 12.3 (IQR: 5.14-18.5). The median average EOM at boot camp was 599.58 cm (IQR: 529.64-676.60). The average score at boot camp significantly correlated with lower time to achieve proficiency (p < 0.05). EOM at boot camp showed a significant correlation with attempts to proficiency and time to proficiency (p < 0.01). Residents with an average baseline EOM below the median showed a significant difference in attempts to proficiency (p < 0.05) and time to proficiency (p < 0.05) compared to those with EOMs above or equal to the median.

CONCLUSION

Residents with an innate ability to perform tasks with better EOM may acquire robotic surgery skills faster. Future investigators could explore how these innate differences impact performance throughout residency.

Home practice for robotic surgery: a randomized controlled trial of a low-cost simulation model

Pre-operative simulated practice allows trainees to learn robotic surgery outside the operating room without risking patient safety. While simulation practice has shown efficacy, simulators are expensive and frequently inaccessible. Cruff (J Surg Educ 78(2): 379-381, 2021) described a low-cost simulation model to learn hand movements for robotic surgery. Our study evaluates whether practice with low-cost home simulation models can improve trainee performance on robotic surgery simulators. Home simulation kits were adapted from those described by Cruff (J Surg Educ 78(2): 379-381, 2021). Hand controllers were modified to mimic the master tool manipulators (MTMs) on the da Vinci Skills Simulator (dVSS). Medical students completed two da Vinci exercises: Sea Spikes 1 (SS1) and Big Dipper Needle Driving (BDND). They were subsequently assigned to either receive a home simulation kit or not. Students returned two weeks later and repeated SS1 and BDND. Overall score, economy of motion, time to completion, and penalty subtotal were collected, and analyses of covariance were performed. Semi-structured interviews assessed student perceptions of the robotic simulation experience. Thirty-three medical students entered the study. Twenty-nine completed both sessions. The difference in score improvement between the experimental and control groups was not significant. In interviews, students provided suggestions to increase fidelity and usefulness of low-cost robotic home simulation. Low-cost home simulation models did not improve student performance on dVSS after two weeks of at-home practice. Interview data highlighted areas to focus future simulation efforts. Ongoing work is necessary to develop low-cost solutions to facilitate practice for robotic surgery and foster more inclusive and accessible surgical education.

Developing a Robotic Surgery Curriculum: Selection of Virtual Reality Drills for Content Alignment

INTRODUCTION

Despite the importance of simulation-based training for robotic surgery, there is no consensus about its training curricula. Recently, a virtual reality (VR) platform (SimNow, Intuitive, Inc) was introduced with 33 VR drills but without evidence of their validity. As part of our creating a new robotic VR curriculum, we assessed the drills' validity through content mapping and the alignment between learning goals and drill content.

METHODS

Three robotically trained surgeons content-mapped all 33 drills for how well the drills incorporated 15 surgery skills and also rated the drills' difficulty, usefulness, relevance, and uniqueness. Drills were added to the new curriculum based on consensus about ratings and historic learner data. The drills were grouped according to similar skill sets and arranged in order of complexity.

RESULTS

The 33 drills were judged to have 12/15 surgery skills as primary goals and 13/15 as secondary goals. Twenty of the 33 drills were selected for inclusion in the new curriculum; these had 11/15 skills as primary goals and 11/15 as secondary goals. However, skills regarding energy sources, atraumatic handling, blunt dissection, fine dissection, and running suturing were poorly represented in the drills. Three previously validated inanimate drills were added to the curriculum to address lacking skill domains.

CONCLUSIONS

We identified 20 of the 33 SimNow drills as a foundation for a robotic surgery curriculum based on content-oriented evidence. We added 3 other drills to address identified gaps in drill content.

Feasibility, effectiveness and transferability of a novel mastery-based virtual reality robotic training platform for general surgery residents

Background

The annual number of robotic surgical procedures is on the rise. Robotic surgery requires unique skills compared to other surgical approaches. Simulation allows basic robot skill acquisition and enhances patient safety. The purpose of this study was to evaluate the feasibility, effectiveness, and transferability of a mastery-based curriculum using a new virtual reality (VR) robotic simulator for surgery resident training.

Methods

Nineteen PGY2s and 22 PGY4s were enrolled. Residents completed a pretest and posttest consisting of five VR and three previously validated inanimate tasks. Training included practicing 33 VR tasks until a total score ≥ 90% (“mastery”) was achieved using automated metrics (time, economy of motion). Inanimate performance was evaluated by two trained, blinded raters using video review metrics (time, errors, and modified OSATS). Outcomes were defined as: curriculum feasibility (completion rate, training time, repetitions), training effectiveness (pre/post training skill improvement), and skill transferability (skill transfer to validated inanimate drills). Wilcoxon signed-rank and Mann–Whitney U tests were used; median (IQR) reported.

Results

Thirty-four of 41 residents (83%) achieved mastery on all 33 VR tasks; median training time was 7 h (IQR: 5′26″–8′52″). Pretest vs. post-test performance improved (all p < 0.001) according to all VR and Inanimate metrics for both PGY2 and PGY4 residents. Significant pretest performance differences were observed between PGY2 and PGY4 residents for VR but not inanimate tasks; no PGY2 vs. PGY4 posttest performance differences were observed for both VR and inanimate tasks.

Conclusion

This mastery-based VR curriculum was associated with a high completion rate and excellent feasibility. Significant performance improvements were noted for both the VR and inanimate tasks, supporting training effectiveness and skill transferability. Additional studies examining validity evidence may help further refine this curriculum.