Establishing objective benchmarks in robotic virtual reality simulation at the level of a competent surgeon using the RobotiX Mentor simulator

Evaluation of different approaches to define expert benchmark scores for new robotic training simulators based on the Medtronic HUGO™ RAS surgical robot experience

New robot-assisted surgery platforms being developed will be required to have proficiency-based simulation training available. Scoring methodologies and performance feedback for trainees are currently not consistent across all robotic simulator platforms. Also, there are virtually no prior publications on how VR simulation passing benchmarks have been established. This paper compares methods evaluated to determine the proficiency-based scoring thresholds (a.k.a. benchmarks) for the new Medtronic Hugo™ RAS robotic simulator. Nine experienced robotic surgeons from multiple disciplines performed the 49 skills exercises 5 times each. The data were analyzed in 3 different ways: (1) include all data collected, (2) exclude first sessions, (3) exclude outliers. Eliminating the first session discounts becoming familiar with the exercise. Discounting outliers allows removal of potentially erroneous data that may be due to technical issues, unexpected distractions, etc. Outliers were identified using a common statistical technique involving the interquartile range of the data. Using each method above, mean and standard deviations were calculated, and the benchmark was set at a value of 1 standard deviation above the mean. In comparison to including all the data, when outliers are excluded, fewer data points are removed than just excluding first sessions, and the metric benchmarks are made more difficult by an average of 11%. When first sessions are excluded, the metric benchmarks are made easier by an average of about 2%. In comparison with benchmarks calculated using all data points, excluding outliers resulted in the biggest change making the benchmarks more challenging. We determined that this method provided the best representation of the data. These benchmarks should be validated with future clinical training studies.

Transferability of the robot assisted and laparoscopic suturing learning curves

Robot assisted surgery (RAS) is increasingly used, and besides conventional minimally invasive surgery (cMIS) surgeons are challenged to learn an increased array of skills. This study aimed to assess the influence of both learning curves on each other. A prospective randomized crossover study was performed. Participants without cMIS or RAS experience (Groups 1 and 2), and cMIS experienced, (Group 3) were recruited. Three suturing tasks (intracorporal suturing, tilted plane and anastomosis needle transfer) were performed on the EoSim cMIS simulator or RobotiX RAS simulator up to twenty repetitions. Subsequently, Groups 1 and 2 performed the tasks on the other modality. Outcomes were simulator parameters, validated composite and pass/fail scores. In total forty-three participants were recruited. Overall RAS suturing was better in Group 1 (cMIS followed by RAS tasks) and 3 (RAS tasks) versus Group 2 (RAS followed by cMIS tasks) for time (163 s and 157 s versus 193 s p = 0.004, p = 0.001) and composite scores (92/100 and 91/100 versus 89/100 p = 0.008, p = 0.020). The cMIS suturing was better for Group 2 versus 1 (time 287 s versus 349 s p = 0.005, composite score 96/100 versus 94/100 p = 0.002). Significant differences from the RAS suturing pass/fail were reached earlier by Group 3, followed by Groups 1 and 2 (repetition six, nine and twelve). In cMIS suturing Group 2 reached significant differences from the pass/fail earlier than Group 1 (repetition four versus six). Transferability of skills was shown for cMIS and RAS, indicating that suturing experience on cMIS or RAS is beneficial in learning either approach.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

A systematic review of robotic surgery curricula using a contemporary educational framework

BACKGROUND

There has been a rising trend in robotic surgery. Thus, there is demand for a robotic surgery curriculum (RSC) for training surgical trainees and practicing surgeons. There are limited data available about current curricular designs and the extent to which they have incorporated educational frameworks. Our aim was to study the existing robotic surgery curricula using Kern's 6-step approach in curriculum development.

METHODS

A systematic review was conducted using PubMed, PubMed Central, Cochrane, Embase, and Scopus (we searched studies from 2001 to 2021). PRISMA Guidelines was used to guide the search. Curriculum designed for general surgery and its subspecialties were included. Urology and gynecology were excluded. The articles were reviewed by five reviewers.

RESULTS

Our review yielded 71 articles, including 39 curricula at 9 different settings. Using Kern's framework, we demonstrated that the majority of robotic surgery curricula contained all the elements of Kern's curricular design. However, there were significant deficiencies in important aspects of these curricula i.e., implementation, the quality of assessment tools for measurement of performance and evaluation of the educational value of these interventions. Most institutions used commercial virtual reality simulators (VRS) as the main component of their RSC and 23% of curricula only used VRS.

CONCLUSIONS

Although majority of these studies contained all the elements of Kern's framework, there are critical deficiencies in the components of existing curricula. Future curricula should be designed using established educational frameworks to improve the quality of robotic surgery training.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Assessment of validity evidence for the RobotiX robot assisted surgery simulator on advanced suturing tasks

Background

Robot assisted surgery has expanded considerably in the past years. Compared to conventional open or laparoscopic surgery, virtual reality (VR) training is an essential component in learning robot assisted surgery. However, for tasks to be implemented in a curriculum, the levels of validity should be studied for proficiency-based training. Therefore, this study was aimed to assess the validity evidence of advanced suturing tasks on a robot assisted VR simulator.

Method

Participants were voluntary recruited and divided in the robotic experienced, laparoscopic experienced or novice group, based on self-reported surgical experience. Subsequently, a questionnaire on a five-point Likert scale was completed to assess the content validity. Three component tasks of complex suturing were performed on the RobotiX simulator (Task1: tilted plane needle transfer, Task: 2 intracorporal suturing, Task 3: anastomosis needle transfer). Accordingly, the outcome of the parameters was used to assess construct validity between robotic experienced and novice participants. Composite scores (0–100) were calculated from the construct parameters and corresponding pass/fail scores with false positive (FP) and false negative (FN) percentages.

Results

Fifteen robotic experienced, 26 laparoscopic experienced and 29 novices were recruited. Overall content validity outcomes were scored positively on the realism (mean 3.7), didactic value (mean 4.0) and usability (mean 4.2). Robotic experienced participants significantly outperformed novices and laparoscopic experienced participants on multiple parameters on all three tasks of complex suturing. Parameters showing construct validity mainly consisted of movement parameters, needle precision and task completion time. Calculated composite pass/fail scores between robotic experienced and novice participants resulted for Task 1 in 73/100 (FP 21%, FN 5%), Task 2 in 85/100 (FP 28%, FN 4%) and Task 3 in 64/100 (FP 49%, FN 22%).

Conclusion

This study assessed the validity evidence on multiple levels of the three studied tasks. The participants score the RobotiX good on the content validity level. The composite pass/fail scores of Tasks 1 and 2 allow for proficiency-based training and could be implemented in a robot assisted surgery training curriculum.

Training novice robot surgeons: Proctoring provides same results as simulator-generated guidance

To understand the influence of proctored guidance versus simulator generated guidance (SGG) on the acquisition dexterity skills in novice surgeons learning RAS (robot assisted surgery). Prospective non-blinded 3-arm randomised controlled trial (RTC). Exclusion criteria: previous experience in RAS or robotic surgery simulation. The participants were assigned to three different intervention groups and received a different form of guidance: (1) proctored guidance, (2) simulator generated guidance, (3) no guidance, during training on virtual reality (VR) simulator. All participants were asked to complete multiple questionnaires. The training was the same in all groups with the exception of the intervention part. Catharina Hospital Eindhoven, The Netherlands. A total of 70 Dutch medical students, PhD-students, and surgical residents were included in the study. The participants were randomly assigned to one of the three groups. Overall, all the participants showed a significant improvement in their dexterity skills after the training. There was no significant difference in the improvement of surgical skills between the three different intervention groups. The proctored guidance group reported a higher participant satisfaction compared to the simulator-generated guidance group, which could indicate a higher motivation to continue the training. This study showed that novice surgeons. Significantly increase their dexterity skills in RAS after a short time of practicing on simulator. The lack of difference in results between the intervention groups could indicate there is a limited impact of "human proctoring" on dexterity skills during surgical simulation training. Since there is no difference between the intervention groups the exposure alone of novice surgeons to the robotic surgery simulator could possibly be sufficient to achieve a significant improvement of dexterity skills during the initial steps of RAS learning.

ACOG Simulation Working Group: A Needs Assessment of Simulation Training in OB/GYN Residencies and Recommendations for Future Research

OBJECTIVE

To evaluate current availability and needs of simulation training among obstetrics/gynecology (OB/GYN) residency programs.

DESIGN

Cross-sectional survey.

SETTING

Accreditation Council for Graduate Medical Education accredited OB/GYN residency programs in the United States.

PARTICIPANTS

Residency program directors, gynecology simulation faculty, obstetrics simulation faculty, and fourth-year residents.

RESULTS

Of 673 invited participants, 251 (37.3%) completed the survey. Among the survey responses, OB procedures were more broadly represented compared to the GYN procedures for simulation teaching: 8 (50%) of 16 OB procedures versus 4 (18.2%) of 22 GYN procedures had simulation teaching. Among the simulated procedures, a majority of residents and faculty reported that simulation teaching was available for operative vaginal delivery, postpartum hemorrhage, shoulder dystocia, perineal laceration repair, conventional laparoscopic procedures, and robotic surgery. There were significant differences between residents and faculty perceptions regarding the availability and needs of simulated procedures with a minority of residents having knowledge of Council on Resident Education in Obstetrics and Gynecology (47.2%) and American College of Obstetrics and Gynecology (27.8%) simulation tools compared to the majority of faculty (84.7% and 72.1%, respectively). More than 80% of trainees and faculty reported they felt the average graduating resident could perform vaginal, laparoscopic, and abdominal hysterectomies independently.

CONCLUSIONS

Simulation is now widely available for both gynecologic and obstetric procedures, but there remains tremendous heterogeneity between programs and the perceptions of residents, program directors, and faculty. The variations in simulation training and readiness for performing different procedures following residency support the need for objective, validated assessments of actual performance to better guide resident learning and faculty teaching efforts.

Training benchmarks based on validated composite scores for the RobotiX robot-assisted surgery simulator on basic tasks

The RobotiX robot-assisted virtual reality simulator aims to aid in the training of novice surgeons outside of the operating room. This study aimed to determine the validity evidence on multiple levels of the RobotiX simulator for basic skills. Participants were divided in either the novice, laparoscopic or robotic experienced group based on their minimally invasive surgical experience. Two basic tasks were performed: wristed manipulation (Task 1) and vessel energy dissection (Task 2). The performance scores and a questionnaire regarding the realism, didactic value, and usability were gathered (content). Composite scores (0–100), pass/fail values, and alternative benchmark scores were calculated. Twenty-seven novices, 21 laparoscopic, and 13 robotic experienced participants were recruited. Content validity evidence was scored positively overall. Statistically significant differences between novices and robotic experienced participants (construct) was found for movements left (Task 1 p = 0.009), movements right (Task 1 p = 0.009, Task 2 p = 0.021), path length left (Task 1 p = 0.020), and time (Task 1 p = 0.040, Task 2 p < 0.001). Composite scores were statistically significantly different between robotic experienced and novice participants for Task 1 (85.5 versus 77.1, p = 0.044) and Task 2 (80.6 versus 64.9, p = 0.001). The pass/fail score with false-positive/false-negative percentage resulted in a value of 75/100, 46/9.1% (Task 1) and 71/100, 39/7.0% (Task 2). Calculated benchmark scores resulted in a minority of novices passing multiple parameters. Validity evidence on multiple levels was assessed for two basic robot-assisted surgical simulation tasks. The calculated benchmark scores can be used for future surgical simulation training.

Concurrent, face, content, and construct validity of the RobotiX Mentor simulator for robotic basic skills

OBJECTIVES

To assess several criteria, such as concurrent, face, content, and construct validity of the RobotiX Mentor (RXM) simulator for basic robotic skills and to compare virtual and actual dry lab dome.

METHODS

A prospective study was conducted from December 2017 to May 2018 using RXM and a da Vinci Si robot. 37 subjects, divided into three groups according to their initial surgical training (expert, intermediate, and novice), were evaluated in terms of six representative exercises of basic robotic specific skills as recommended by the fundamentals of robotic surgery.

RESULTS

There was a correlation between the automatic data from the RXM and the subjective evaluation with the robot. The face and content validity, which were evaluated by the experts, were generally considered high (71.5% and 62.5%, respectively). Three levels (analysis of variance [ANOVA]; P = .01) and two levels (P = .001) of experience were clearly identified by the simulator.

CONCLUSION

Our study proves the concurrent validity and confirms the face, content, and construct validity of the RXM.

Educational note: teaching and training in robotic surgery. An opinion of the Minimally Invasive and Robotic Surgery Committee of the Brazilian College of Surgeons

ABSTRACT With the expansion of robotic surgical procedures, the acquisition of specific knowledge and skills for surgeons to reach proficiency seems essential before performing surgical procedures on humans. In this sense, the authors present a proposal to establish a certification based on objective and validated criteria for carrying out robotic procedures. A study was carried out by the Committee on Minimally Invasive and Robotic Surgery of the Brazilian College of Surgeons based on a reviewing strategy of the scientific literature. The study serves as a reference for the creation of a standard for the qualification and certification in robotic surgery according to a statement of the Brazilian Medical Association (AMB) announced on December 17, 2019. The standard proposes a minimum curriculum, integrating training and performance evaluation. The initial (pre-clinical) stage aims at knowledge and adaptation to a specific robotic platform and the development of psychomotor skills based on surgical simulation. Afterwards, the surgeon must accompany in person at least five surgeries in the specialty, participate as a bedside assistant in at least 10 cases and perform 10 surgeries under the supervision of a preceptor surgeon. The surgeon who completes all the steps will be considered qualified in robotic surgery in his specialty. The final certification must be issued by the specialty societies affiliated to AMB. The authors conclude that the creation of a norm for habilitation in robotic surgery should encourage Brazilian hospitals to apply objective qualification criteria for this type of procedure to qualify assistance.

Robot assisted versus laparoscopic suturing learning curve in a simulated setting

Background

Compared to conventional laparoscopy, robot assisted surgery is expected to have most potential in difficult areas and demanding technical skills like minimally invasive suturing. This study was performed to identify the differences in the learning curves of laparoscopic versus robot assisted suturing.

Method

Novice participants performed three suturing tasks on the EoSim laparoscopic augmented reality simulator or the RobotiX robot assisted virtual reality simulator. Each participant performed an intracorporeal suturing task, a tilted plane needle transfer task and an anastomosis needle transfer task. To complete the learning curve, all tasks were repeated up to twenty repetitions or until a time plateau was reached. Clinically relevant and comparable parameters regarding time, movements and safety were recorded. Intracorporeal suturing time and cumulative sum analysis was used to compare the learning curves and phases.

Results

Seventeen participants completed the learning curve laparoscopically and 30 robot assisted. Median first knot suturing time was 611 s (s) for laparoscopic versus 251 s for robot assisted (p < 0.001), and this was 324 s versus 165 (sixth knot, p < 0.001) and 257 s and 149 s (eleventh knot, p < 0.001) respectively on base of the found learning phases. The percentage of ‘adequate surgical knots’ was higher in the laparoscopic than in the robot assisted group. First knot: 71% versus 60%, sixth knot: 100% versus 83%, and eleventh knot: 100% versus 73%. When assessing the ‘instrument out of view’ parameter, the robot assisted group scored a median of 0% after repetition four. In the laparoscopic group, the instrument out of view increased from 3.1 to 3.9% (left) and from 3.0 to 4.1% (right) between the first and eleventh knot (p > 0.05).

Conclusion

The learning curve of minimally invasive suturing shows a shorter task time curve using robotic assistance compared to the laparoscopic curve. However, laparoscopic outcomes show good end results with rapid outcome improvement.

Electronic supplementary material

The online version of this article (10.1007/s00464-019-07263-2) contains supplementary material, which is available to authorized users.

A prospective study of the effect of video games on robotic surgery skills using the high-fidelity virtual reality RobotiX simulator

BACKGROUND

Robot-assisted surgery is a growing field. Prior video game experience might give advantage to novice robotic surgeons.

AIM

Assessing if prior video gaming experience gives advantage in performing high-fidelity virtual reality (VR)-simulated robotic surgery.

METHODS

In this observational study, 30 medical students and 2 interns (17 females; 15 males) with median age 25 years (range, 24-26 years) were recruited and subsequently divided into groups according to prior gaming experience; gamers (≥6 video game hours/week) vs nongamers (<6 video game hours/week). Participants performed VR-simulated urethrovesical anastomosis on RobotiX Mentor, which measured performance parameters. Participants answered a questionnaire for demographics and gaming experience. Groups were compared using Mann-Whitney U and multiple regression.

RESULTS

Gamers significantly outperformed nongamers in 3 of 24 performance metrics (p<0.05), and there was a trend toward better results for 7 of the 21 remaining metrics. Males outperformed females in 5 of 24 metrics (p<0.05) but were overrepresented among gamers.

CONCLUSION

Prior video game experience >6 hrs/week might give advantage in simulated robotic surgery. We recommend future studies testing this hypothesis to develop simulator programs for certification of robotic surgeons.