Robotic surgery training: construct validity of Global Evaluative Assessment of Robotic Skills (GEARS)

Classification of subtask types and skill levels in robot-assisted surgery using EEG, eye-tracking, and machine learning

BACKGROUND

Objective and standardized evaluation of surgical skills in robot-assisted surgery (RAS) holds critical importance for both surgical education and patient safety. This study introduces machine learning (ML) techniques using features derived from electroencephalogram (EEG) and eye-tracking data to identify surgical subtasks and classify skill levels.

METHOD

The efficacy of this approach was assessed using a comprehensive dataset encompassing nine distinct classes, each representing a unique combination of three surgical subtasks executed by surgeons while performing operations on pigs. Four ML models, logistic regression, random forest, gradient boosting, and extreme gradient boosting (XGB) were used for multi-class classification. To develop the models, 20% of data samples were randomly allocated to a test set, with the remaining 80% used for training and validation. Hyperparameters were optimized through grid search, using fivefold stratified cross-validation repeated five times. Model reliability was ensured by performing train-test split over 30 iterations, with average measurements reported.

RESULTS

The findings revealed that the proposed approach outperformed existing methods for classifying RAS subtasks and skills; the XGB and random forest models yielded high accuracy rates (88.49% and 88.56%, respectively) that were not significantly different (two-sample t-test; P-value = 0.9).

CONCLUSION

These results underscore the potential of ML models to augment the objectivity and precision of RAS subtask and skill evaluation. Future research should consider exploring ways to optimize these models, particularly focusing on the classes identified as challenging in this study. Ultimately, this study marks a significant step towards a more refined, objective, and standardized approach to RAS training and competency assessment.

Robotic colorectal surgery training: Portsmouth perspective

This study aims to discuss the principles and pillars of robotic colorectal surgery training and share the training pathway at Portsmouth Hospitals University NHS Trust. A narrative review is presented to discuss all the relevant and critical steps in robotic surgical training. Robotic training requires a stepwise approach, including theoretical knowledge, case observation, simulation, dry lab, wet lab, tutored programs, proctoring (in person or telementoring), procedure-specific training, and follow-up. Portsmouth Colorectal has an established robotic training model with a safe stepwise approach that has been demonstrated through perioperative and oncological results. Robotic surgery training should enable a trainee to use the robotic platform safely and effectively, minimize errors, and enhance performance with improved outcomes. Portsmouth Colorectal has provided such a stepwise training program since 2015 and continues to promote and augment safe robotic training in its field. Safe and efficient training programs are essential to upholding the optimal standard of care.

Use of Simulation For Training Advanced Colorectal Procedures

OBJECTIVE

Simulation training for minimally invasive colorectal procedures is in developing stages. This study aims to assess the impact of simulation on procedural knowledge and simulated performance in laparoscopic low anterior resection (LLAR) and robotic right colectomy (RRC).

DESIGN

LLAR and RRC simulation procedures were designed using human cadaveric models. Resident case experience and simulation selfassessments scores for operative ability and knowledge were collected before and after the simulation. Colorectal faculty assessed resident simulation performance using validated assessment scales (OSATS-GRS, GEARS). Paired t-tests, unpaired t-tests, Pearson's correlation, and descriptive statistics were applied in analyses.

SETTING

Barnes-Jewish Hospital/Washington University School of Medicine in St. Louis, Missouri.

PARTICIPANTS

Senior general surgery residents at large academic surgery program.

RESULTS

Fifteen PGY4/PGY5 general surgery residents participated in each simulation. Mean LLAR knowledge score increased overall from 10.0 ±  2.0 to 11.5  ±  1.6 of 15 points (p = 0.0018); when stratified, this increase remained significant for the PGY4 cohort only. Mean confidence in ability to complete LLAR increased overall from 2.0 ±  0.8 to 2.8  ± 0.9 on a 5-point rating scale (p = 0.0013); when stratified, this increase remained significant for the PGY4 cohort only. Mean total OSATS GRS score was 28  ±  6.3 of 35 and had strong positive correlation with previous laparoscopic colorectal experience (r = 0.64, p = 0.0092). Mean RRC knowledge score increased from 9.4 ±  2.2 to 11.1 ±  1.5 of 15 points (p = 0.0030); when stratified, this increase again remained significant for the PGY4 cohort only. Mean confidence in ability to complete RRC increased from 1.9 ±  0.9 to 3.2  ±  1.1 (p = 0.0002) and was significant for both cohorts.

CONCLUSIONS

Surgical trainees require opportunities to practice advanced minimally invasive colorectal procedures. Our simulation approach promotes increased procedural knowledge and resident confidence and offers a safe complement to live operative experience for trainee development. In the future, simulations will target trainees on the earlier part of the learning curve and be paired with live operative assessments to characterize longitudinal skill progression.

IDEAL-D Phase 0 Evaluation of the Avatera System in Robot-Assisted Prostate, Bladder and Renal Surgery

Purpose: To evaluate the utilization of novel Avatera system in urological operations according to the IDEAL-D framework recommendations for high-risk invasive surgical devices. Materials and Methods: Three surgeons attempted to perform 23 upper and lower urinary tract operations on human cadavers and in live porcine models using the Avatera system. Total operative time and the duration of the substeps were evaluated. Surgical performance was assessed with the Global Evaluative Assessment of Robotic Skills (GEARS) score. Suturing was rated using the technical checklist for the assessment of suturing in robotic surgery. Attending surgeons rated their satisfaction with the Avatera system on a scale of 1-5. Results and Limitation: Seventeen out of 18 operations performed on cadavers were completed, while one pyeloplasty was discontinued. All five operations performed in porcine models were completed. Although 1 pig was euthanized on the fifth postoperative day, its symptoms were unrelated to surgery. Mean GEARS and Suturing scores in the upper urinary tract were 29 ± 0.7 and 29.5 ± 0.95, respectively, and in the lower urinary 28.5 ± 1.2 and 29.5 ± 0.5, respectively. Surgeons' satisfaction was high or very high for all procedures. Conclusions: The Avatera system was associated with good surgical performance and high surgeons' satisfaction rates. All urological procedures performed were shown to be feasible, with comparable risks to other robot-assisted surgery systems.

"Doing well": Intraoperative entrustable professional activity assessments provided limited technical feedback

Background

Entrustable Professional Activities (EPAs) allow for the assessment of specific, observable, essential tasks in medical education. Since being developed in non-surgical fields, EPA assessments have been implemented in surgery to explore intraoperative entrustment. However, assessment burden is a significant problem for faculty, and it is unknown whether EPA assessments enable formative technical feedback. EPAs' formative utility could inform how surgical programs facilitate technical feedback for trainees. We aimed to assess the extent to which narrative comments provided through the Fellowship Council (FC) EPA assessments contained technical feedback.

Methods

The FC previously collected EPA assessments for subspecialty surgical fellows from September 2020 to October 2022. Two raters reviewed assessments' narrative comments for inclusion of each skill area that makes up part of the Objective Structured Assessment of Technical Skills (OSATS). A third rater reconciled discrepant ratings.

Results

During the study period, there were 3302 completed EPA assessments, including 1191 fellow self-assessments, 1124 faculty assessments, and 987 assessments without an identified assessor role. We found that assessments' narrative comments related to a median of two of the seven OSATS areas (IQR:1-2). There were no comments relevant to any of the seven OSATS areas in 16.0 % of all assessments.

Conclusions

In this review of narrative comments for EPA assessments from the FC, we found that limited technical feedback of the kind included in the OSATS was provided in many assessments. These results suggest benefit to adjusting the EPA form, enhancing faculty development, or continuing additional types of targeted technical assessment intraoperatively.

Key message

This analysis of narrative comments from fellowship EPA assessments showed that many assessments included limited technical feedback. To allow for continued technical feedback for fellows, these results highlight the need for further refinements of the EPA assessment form, additional faculty development, or ongoing use of other types of technical assessment.

AI-powered real-time annotations during urologic surgery: The future of training and quality metrics

INTRODUCTION AND OBJECTIVE

Real-time artificial intelligence (AI) annotation of the surgical field has the potential to automatically extract information from surgical videos, helping to create a robust surgical atlas. This content can be used for surgical education and qualitative initiatives. We demonstrate the first use of AI in urologic robotic surgery to capture live surgical video and annotate key surgical steps and safety milestones in real-time.

SUMMARY BACKGROUND DATA

While AI models possess the capability to generate automated annotations based on a collection of video images, the real-time implementation of such technology in urological robotic surgery to aid surgeon and training staff it is still pending to be studied.

METHODS

We conducted an educational symposium, which broadcasted 2 live procedures, a robotic-assisted radical prostatectomy (RARP) and a robotic-assisted partial nephrectomy (RAPN). A surgical AI platform system (Theator, Palo Alto, CA) generated real-time annotations and identified operative safety milestones. This was achieved through trained algorithms, conventional video recognition, and novel Video Transfer Network technology which captures clips in full context, enabling automatic recognition and surgical mapping in real-time.

RESULTS

Real-time AI annotations for procedure #1, RARP, are found in Table 1. The safety milestone annotations included the apical safety maneuver and deliberate views of structures such as the external iliac vessels and the obturator nerve. Real-time AI annotations for procedure #2, RAPN, are found in Table 1. Safety milestones included deliberate views of structures such as the gonadal vessels and the ureter. AI annotated surgical events included intraoperative ultrasound, temporary clip application and removal, hemostatic powder application, and notable hemorrhage.

CONCLUSIONS

For the first time, surgical intelligence successfully showcased real-time AI annotations of 2 separate urologic robotic procedures during a live telecast. These annotations may provide the technological framework for send automatic notifications to clinical or operational stakeholders. This technology is a first step in real-time intraoperative decision support, leveraging big data to improve the quality of surgical care, potentially improve surgical outcomes, and support training and education.

Standardization and Implementation of a Surgical Coaching Model for Pediatric Urology

To bridge gaps in proficiency and encourage life-long learning following training, coaching models have been utilized in multiple surgical fields; however, not within pediatric urology. In this review of our methodology, we describe the development of a coaching model at a single institution. In our initial experience, the perceived most beneficial aspect of the program was the goal setting process with logistics around debriefs being the most challenging. With our proposed coaching study, we aim to develop a model based upon prior coaching frameworks,1,2 that is feasible and universally adaptable to allow for further advancement of surgical coaching, particularly within the field of pediatric urology.

Use of targeted educational resources to improve robotic bariatric surgery training

BACKGROUND

Evidence for how to best train surgical residents for robotic bariatric procedures is lacking. We developed targeted educational resources to promote progression on the robotic bariatric learning curve. This study aimed to characterize the effect of resources on resident participation in robotic bariatric procedures.

METHODS

Performance metrics from the da Vinci Surgical System were retrospectively reviewed for sleeve gastrectomy (SG) and Roux-en-Y gastric bypass (RYGB) cases involving general surgery trainees with a single robotic bariatric surgeon. Pictorial case guides and narrated operative videos were developed for these procedures and disseminated to trainees. Percent active control time (%ACT)-amount of trainee console time spent in active instrument manipulations over total active time from both consoles-was the primary outcome measure following dissemination. One-way ANOVA, Student's t-tests, and Pearson correlations were applied.

RESULTS

From September 2020 to July 2021, 50 cases (54% SG, 46% RYGB) involving 14 unique trainees (PGY1-PGY5) were included. From November 2021 to May 2022 following dissemination, 29 cases (34% SG, 66% RYGB) involving 8 unique trainees were included. Mean %ACT significantly increased across most trainee groups following resource distribution: 21% versus 38% for PGY3s (p = 0.087), 32% versus 45% for PGY4s (p = 0.0009), and 38% versus 57% for PGY5s (p = 0.0015) and remained significant when stratified by case type. Progressive trainee %ACT was not associated with total active time for SG cases before or after intervention (pre r = - 0.0019, p = 0.9; post r = - 0.039, p = 0.9). It was moderately positively associated with total active time for RYGB cases before dissemination (r = 0.46, p = 0.027) but lost this association following intervention (r = 0.16, p = 0.5).

CONCLUSION

Use of targeted educational resources promoted increases in trainee participation in robotic bariatric procedures with more time spent actively operating at the console. As educators continue to develop robotic training curricula, efforts should include high-quality resource development for other sub-specialty procedures. Future work will examine the impact of increased trainee participation on clinical and patient outcomes.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills

BACKGROUND

Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment.

METHODS

Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed.

RESULTS

OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001).

CONCLUSION

OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.

Does Surgeon Experience Correlate with Crowd-Sourced Skill Assessment in Robotic Bariatric Surgery?

BACKGROUND

The Global Evaluative Assessment of Robotic Skills (GEARS) rubric provides a measure of skill in robotic surgery. We hypothesize surgery performed by more experienced operators will be associated with higher GEARS scores.

METHOD

Patients undergoing sleeve gastrectomy from 2016 to 2020 were analyzed. Three groups were defined by time in practice: less than 5, between 5 and 15, and more than 15 years. Continuous variables were compared with ANOVA and multivariable regression was performed.

RESULTS

Fourteen operators performing 154 cases were included. More experienced surgeons had higher GEARS scores and shorter operative times. On multivariable regression, operative time (P = 0.027), efficiency (P = .022), depth perception (P = 0.033), and bimanual dexterity (P = 0.047) were associated with experience.

CONCLUSIONS

In our video-based assessment (VBA) model, operative time and several GEARS subcomponent scores were associated with surgical experience. Further studies should determine the association between these metrics and surgical outcomes.

Surgical skill level classification model development using EEG and eye-gaze data and machine learning algorithms

The aim of this study was to develop machine learning classification models using electroencephalogram (EEG) and eye-gaze features to predict the level of surgical expertise in robot-assisted surgery (RAS). EEG and eye-gaze data were recorded from 11 participants who performed cystectomy, hysterectomy, and nephrectomy using the da Vinci robot. Skill level was evaluated by an expert RAS surgeon using the modified Global Evaluative Assessment of Robotic Skills (GEARS) tool, and data from three subtasks were extracted to classify skill levels using three classification models-multinomial logistic regression (MLR), random forest (RF), and gradient boosting (GB). The GB algorithm was used with a combination of EEG and eye-gaze data to classify skill levels, and differences between the models were tested using two-sample t tests. The GB model using EEG features showed the best performance for blunt dissection (83% accuracy), retraction (85% accuracy), and burn dissection (81% accuracy). The combination of EEG and eye-gaze features using the GB algorithm improved the accuracy of skill level classification to 88% for blunt dissection, 93% for retraction, and 86% for burn dissection. The implementation of objective skill classification models in clinical settings may enhance the RAS surgical training process by providing objective feedback about performance to surgeons and their teachers.

Active control time: an objective performance metric for trainee participation in robotic surgery

Trainee participation and progression in robotic general surgery remain poorly defined. Computer-assisted technology offers the potential to provide and track objective performance metrics. In this study, we aimed to validate the use of a novel metric-active control time (ACT)-for assessing trainee participation in robotic-assisted cases. Performance data from da Vinci Surgical Systems was retrospectively analyzed for all robotic cases involving trainees with a single minimally invasive surgeon over 10 months. The primary outcome metric was percent ACT-the amount of trainee console time spent in active system manipulations over total active time from both consoles. Kruskal-Wallis and Mann-Whitney U statistical tests were applied in analyses. A total of 123 robotic cases with 18 general surgery residents and 1 fellow were included. Of these, 56 were categorized as complex. Median %ACT was statistically different between trainee levels for all case types taken in aggregate (PGY1s 3.0% [IQR 2-14%], PGY3s 32% [IQR 27-66%], PGY4s 42% [IQR 26-52%], PGY5s 50% [IQR 28-70%], and fellow 61% [IQR 41-85%], p =  < 0.0001). When stratified by complexity, median %ACT was higher in standard versus complex cases for PGY5 (60% vs. 36%, p = 0.0002) and fellow groups (74% vs. 47%, p = 0.0045). In this study, we demonstrated an increase in %ACT with trainee level and with standard versus complex robotic cases. These findings are consistent with hypotheses, providing validity evidence for ACT as an objective measurement of trainee participation in robotic-assisted cases. Future studies will aim to define task-specific ACT to guide further robotic training and performance assessments.

Discrimination, Reliability, Sensitivity, and Specificity of Robotic Surgical Proficiency Assessment With Global Evaluative Assessment of Robotic Skills and Binary Scoring Metrics: Results From a Randomized Controlled Trial

Objective

To compare binary metrics and Global Evaluative Assessment of Robotic Skills (GEARS) evaluations of training outcome assessments for reliability, sensitivity, and specificity.

Background

GEARS-Likert-scale skills assessment are a widely accepted tool for robotic surgical training outcome evaluations. Proficiency-based progression (PBP) training is another methodology but uses binary performance metrics for evaluations.

Methods

In a prospective, randomized, and blinded study, we compared conventional with PBP training for a robotic suturing, knot-tying anastomosis task. Thirty-six surgical residents from 16 Belgium residency programs were randomized. In the skills laboratory, the PBP group trained until they demonstrated a quantitatively defined proficiency benchmark. The conventional group were yoked to the same training time but without the proficiency requirement. The final trial was video recorded and assessed with binary metrics and GEARS by robotic surgeons blinded to individual, group, and residency program. Sensitivity and specificity of the two assessment methods were evaluated with area under the curve (AUC) and receiver operating characteristics (ROC) curves.

Results

The PBP group made 42% fewer objectively assessed performance errors than the conventional group (P < 0.001) and scored 15% better on the GEARS assessment (P = 0.033). The mean interrater reliability for binary metrics and GEARS was 0.87 and 0.38, respectively. Binary total error metrics AUC was 97% and for GEARS 85%. With a sensitivity threshold of 0.8, false positives rates were 3% and 25% for, respectively, the binary and GEARS assessments.

Conclusions

Binary metrics for scoring a robotic VUA task demonstrated better psychometric properties than the GEARS assessment.

What credentials are required for robotic-assisted surgery in reconstructive and functional urology?

Introduction

The increasing popularity of robotic assisted surgery (RAS) as it is implemented in to sub specialities poses many challenges to ensuring standards in quality and safety. The area of Reconstructive and Functional Urology (RFU) has a wide range and largely complex heterogeneous procedures. In recent years RFU has started to incorporate RAS as the primary method to undertake these procedures due to improved vision, dexterity, and access to deep cavities. To ensure patient safety majority of institutions maintain minimal requirements to operate using RAS however across specialities and institutions these greatly vary.

Methods

A narrative review of all the relevant papers known to the author was conducted.

Results

Specific challenges facing RFU is the inability to rely on case numbers as a surrogate means to measure competency as well the ongoing consideration of how to differentiate between surgeons with robotic training and those with the clinical experience specific to RFU.

Conclusion

This review explores current models of training and credentialling and assess how it can be adapted to suggest a standardised guideline for RFU to ensure the highest standards of patient care.

Robotic Plastic Surgery Education: Developing a Robotic Surgery Training Program Specific to Plastic Surgery Trainees

Over the past two decades, the surgical community has increasingly embraced robotic-assisted surgery (RAS) due to its potential to enhance accuracy and decrease surgical morbidity. Plastic surgery as a field has been historically slow to incorporate RAS, with lack of adequate training posing as one of the most commonly cited barriers. To date, robot technology has been utilized for various reconstructive procedures including flap elevation and inset, pedicle dissection, and microvascular anastomosis. As RAS continues to integrate within plastic surgery procedures, the need for a structured RAS curriculum designed for plastic surgery trainees is rising. This article delineates the essential components of a plastic surgery-specific RAS curriculum and outlines current training models and assessment tools utilized across surgical subspecialties to date.

Developing surgical skill level classification model using visual metrics and a gradient boosting algorithm

Objective

Assessment of surgical skills is crucial for improving training standards and ensuring the quality of primary care. This study aimed to develop a gradient boosting classification model (GBM) to classify surgical expertise into inexperienced, competent, and experienced levels in robot-assisted surgery (RAS) using visual metrics.

Methods

Eye gaze data were recorded from 11 participants performing four subtasks; blunt dissection, retraction, cold dissection, and hot dissection using live pigs and the da Vinci robot. Eye gaze data were used to extract the visual metrics. One expert RAS surgeon evaluated each participant's performance and expertise level using the modified Global Evaluative Assessment of Robotic Skills (GEARS) assessment tool. The extracted visual metrics were used to classify surgical skill levels and to evaluate individual GEARS metrics. Analysis of Variance (ANOVA) was used to test the differences for each feature across skill levels.

Results

Classification accuracies for blunt dissection, retraction, cold dissection, and burn dissection were 95%, 96%, 96%, and 96%, respectively. The time to complete only the retraction was significantly different among the 3 skill levels (p-value = 0.04). Performance was significantly different for 3 categories of surgical skill level for all subtasks (p-values<0.01). The extracted visual metrics were strongly associated with GEARS metrics (R2>0.7 for GEARS metrics evaluation models).

Conclusions

Machine learning (ML) algorithms trained by visual metrics of RAS surgeons can classify surgical skill levels and evaluate GEARS measures. The time to complete a surgical subtask may not be considered a stand-alone factor for skill level assessment.

Crossover-effects in technical skills between laparoscopy and robot-assisted surgery

INTRODUCTION

Robot-assisted surgery is often performed by experienced laparoscopic surgeons. However, this technique requires a different set of technical skills and surgeons are expected to alternate between these approaches. The aim of this study is to investigate the crossover effects when switching between laparoscopic and robot-assisted surgery.

METHODS

An international multicentre crossover study was conducted. Trainees with distinctly different levels of experience were divided into three groups (novice, intermediate, expert). Each trainee performed six trials of a standardized suturing task using a laparoscopic box trainer and six trials using the da Vinci surgical robot. Both systems were equipped with the ForceSense system, measuring five force-based parameters for objective assessment of tissue handling skills. Statistical comparison was done between the sixth and seventh trial to identify transition effects. Unexpected changes in parameter outcomes after the seventh trial were further investigated.

RESULTS

A total of 720 trials, performed by 60 participants, were analysed. The expert group increased their tissue handling forces with 46% (maximum impulse 11.5 N/s to 16.8 N/s, p = 0.05), when switching from robot-assisted surgery to laparoscopy. When switching from laparoscopy to robot-assisted surgery, intermediates and experts significantly decreased in motion efficiency (time (sec), resp. 68 vs. 100, p = 0.05, and 44 vs. 84, p = 0.05). Further investigation between the seventh and ninth trial showed that the intermediate group increased their force exertion with 78% (5.1 N vs. 9.1 N, p = 0.04), when switching to robot-assisted surgery.

CONCLUSION

The crossover effects in technical skills between laparoscopic and robot-assisted surgery are highly depended on the prior experience with laparoscopic surgery. Where experts can alternate between approaches without impairment of technical skills, novices and intermediates should be aware of decay in efficiency of movement and tissue handling skills that could impact patient safety. Therefore, additional simulation training is advised to prevent from undesired events.

Do Skills Naturally Transfer Between Multiport and Single-Port Robotic Platforms? A Comparative Study in a Simulated Environment

Introduction and Objective: With introduction of the da Vinci single-port (SP) system, we evaluated which multiport (MP) robotic skills are naturally transferable to the SP platform. Methods: Three groups of urologists: Group 1 (5 inexperienced in MP and SP), Group 2 (5 experienced in MP without SP experience), and Group 3 (2 experienced in both MP and SP) were recruited to complete a validated urethrovesical anastomosis simulation using MP followed by SP robots. Performance was graded using both GEARS and RACE scales. Subjective cognitive load measurements (Surg-TLX and difficulty ratings [/20] of instrument collisions camera and EndoWrist movement) were collected. Results: GEARS and RACE scores for Groups 1 and 3 were maintained on switching from MP to SP (Group 3 scored significantly higher on both systems). Surg-TLX and difficulty scores were also maintained for both groups on switching from MP and SP except for a significant increase in SP camera movement (+7.2, p = 0.03) in Group 1 compared to Group 3 that maintained low scores on both. Group 2 demonstrated significant lower GEARS (-2.9, p = 0.047) and RACE (-5.1, p = 0.011) scores on SP vs MP. On subanalysis, GEARS subscores for force sensitivity and robotic control (-0.7, p = 0.04; -0.9, p = 0.02) and RACE subscores for needle entry, needle driving, and tissue approximation (-0.9, p = 0.01; -1.0, p = 0.02; -1.0, p < 0.01) significantly decreased. GEARS (depth perception, bimanual dexterity, and efficiency) and RACE subscores (needle positioning and suture placement) were maintained. All participants scored significantly lower in knot tying on the SP robot (-1.0, p = 0.03; -1.2, p = 0.02, respectively). Group 2 reported higher Surg-TLX (+13 pts, p = 0.015) and difficulty ratings on SP vs MP (+11.8, p < 0.01; +13.6, p < 0.01; +14 pts, p < 0.01). Conclusions: The partial skill transference across robots raises the question regarding SP-specific training for urologists proficient in MP. Novices maintained difficulty scores and cognitive load across platforms, suggesting that concurrent SP and MP training may be preferred.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Using AI and computer vision to analyze technical proficiency in robotic surgery

BACKGROUND

Intraoperative skills assessment is time-consuming and subjective; an efficient and objective computer vision-based approach for feedback is desired. In this work, we aim to design and validate an interpretable automated method to evaluate technical proficiency using colorectal robotic surgery videos with artificial intelligence.

METHODS

92 curated clips of peritoneal closure were characterized by both board-certified surgeons and a computer vision AI algorithm to compare the measures of surgical skill. For human ratings, six surgeons graded clips according to the GEARS assessment tool; for AI assessment, deep learning computer vision algorithms for surgical tool detection and tracking were developed and implemented.

RESULTS

For the GEARS category of efficiency, we observe a positive correlation between human expert ratings of technical efficiency and AI-determined total tool movement (r = - 0.72). Additionally, we show that more proficient surgeons perform closure with significantly less tool movement compared to less proficient surgeons (p < 0.001). For the GEARS category of bimanual dexterity, a positive correlation between expert ratings of bimanual dexterity and the AI model's calculated measure of bimanual movement based on simultaneous tool movement (r = 0.48) was also observed. On average, we also find that higher skill clips have significantly more simultaneous movement in both hands compared to lower skill clips (p < 0.001).

CONCLUSIONS

In this study, measurements of technical proficiency extracted from AI algorithms are shown to correlate with those given by expert surgeons. Although we target measurements of efficiency and bimanual dexterity, this work suggests that artificial intelligence through computer vision holds promise for efficiently standardizing grading of surgical technique, which may help in surgical skills training.

Parallel, component training in robotic total mesorectal excision

There has been widespread adoption of robotic total mesorectal excision (TME) for rectal cancer in recent years. There is now increasing interest in training robotic novice surgeons in robotic TME surgery using the principles of component-based learning. The aims of our study were to assess the feasibility of delivering a structured, parallel, component-based, training curriculum to surgical trainees and fellows. A prospective pilot study was undertaken between January 2021 and May 2021. A dedicated robotic training pathway was designed with two trainees trained in parallel per each robotic case based on prior experience, training grade and skill set. Component parts of each operation were allocated by the robotic trainer prior to the start of each case. Robotic proficiency was assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and the EARCS Global Assessment Score (GAS). Three trainees participated in this pilot study, performing a combined number of 52 TME resections. Key components of all 52 TME operations were performed by the trainees. GEARS scores improved throughout the study, with a mean overall baseline score of 17.3 (95% CI 15.1-1.4) compared to an overall final assessment mean score of 23.8 (95% CI 21.6-25.9), p = 0.003. The GAS component improved incrementally for all trainees at each candidate assessment (p < 0.001). Employing a parallel, component-based approach to training in robotic TME surgery is safe and feasible and can be used to train multiple trainees of differing grades simultaneously, whilst maintaining high-quality clinical outcomes.

Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA)

OBJECTIVE

As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence.

DESIGN

This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses.

SETTING

This study was conducted at an academic institution in a proctored exam setting.

PARTICIPANTS

Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment.

RESULTS

Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56).

CONCLUSIONS

Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

A resident robotic curriculum utilizing self-selection and a web-based feedback tool

To describe an obstetrics and gynecology residency robotic curriculum, facilitated by a web-based feedback and case-tracking tool, allowing for self-selection into advanced training. Phase I (Basic) was required for all residents and included online training modules, online assessment, and robotic bedside assistant dry lab. Phase II (Advanced) was elective console training. Before live surgery, 10 simulation drills completed to proficiency were required. A web-based tool was used for surgical feedback and case-tracking. Online assessments, drill reports, objective GEARS assessments, subjective feedback, and case-logs were reviewed (7/2018-6/2019). A satisfaction survey was reviewed. Twenty four residents completed Phase I training and 10 completed Phase II. To reach simulation proficiency, residents spent a median of 4.1 h performing required simulation drills (median of 10 (3, 26) attempts per drill) before live surgery. 128 post-surgical feedback entries were completed after performance as bedside assistant (75%, n = 96) and console surgeon (5.5%, n = 7). The most common procedure was hysterectomy 111/193 (58%). Resident console surgeons performed portions of 32 cases with a mean console time of 34.6 ± 19.5 min. Mean GEARS score 20.6 ± 3.7 (n = 28). Mean non-technical feedback results: communication (4.2 ± 0.8, n = 61), workload management (3.9 ± 0.9, n = 54), team skills (4.3 ± 0.8, n = 60). Residents completing > 50% of case assessed as "apprentice" 38.5% or "competent" 23% (n = 13). After curriculum change, 100% of surveyed attendings considered residents prepared for live surgical training, vs 17% (n = 6) prior to curriculum change [survey response rate 27/44 (61%)]. Attendings and residents were satisfied with curriculum; 95% and recommended continued use 90% (n = 19).This two-phase robotic curriculum allows residents to self-select into advanced training, alleviating many challenges of graduated robotic training.

Analysis of executional and procedural errors in dry‐lab robotic surgery experiments

Background

Analysing kinematic and video data can help identify potentially erroneous motions that lead to sub‐optimal surgeon performance and safety‐critical events in robot‐assisted surgery.

Methods

We develop a rubric for identifying task and gesture‐specific executional and procedural errors and evaluate dry‐lab demonstrations of suturing and needle passing tasks from the JIGSAWS dataset. We characterise erroneous parts of demonstrations by labelling video data, and use distribution similarity analysis and trajectory averaging on kinematic data to identify parameters that distinguish erroneous gestures.

Results

Executional error frequency varies by task and gesture, and correlates with skill level. Some predominant error modes in each gesture are distinguishable by analysing error‐specific kinematic parameters. Procedural errors could lead to lower performance scores and increased demonstration times but also depend on surgical style.

Conclusions

This study provides insights into context‐dependent errors that can be used to design automated error detection mechanisms and improve training and skill assessment.

Surgical Science - Simbionix Robotic Hysterectomy Simulator: Validating a New Tool

STUDY OBJECTIVE

To gather validity evidence for and determine acceptability of Surgical Science - Simbionix Hysterectomy Modules for the DaVinci Xi console simulation system and evaluate performance benchmarks between novice and experienced or expert surgeons.

DESIGN

Prospective education study (Messick validity framework) SETTING: Multi-center, academic medical institutions PARTICIPANTS: Residents, fellows, and faculty in Obstetrics and Gynecology were invited to participate at 3 institutions. Participants were categorized by experience level: less than 10 hysterectomies (novice), 10 to 50 hysterectomies (experienced), and greater than 50 hysterectomies (expert). A total of 10 novice, 10 experienced, and 14 expert surgeons were included.

INTERVENTIONS

Participants completed 4 simulator modules (ureter identification, bladder flap development, colpotomy, complete hysterectomy) and a qualitative survey. Simulator recordings were reviewed in duplicate by educators in minimally invasive gynecologic surgery using the Modified Global Evaluative Assessment of Robotic Skills (GEARS) rating scale.

MEASUREMENTS AND MAIN RESULTS

Most participants felt the simulator realistically simulated robotic hysterectomy (64.7%) and that feedback provided by the simulator was as or more helpful than feedback from previous simulators (88.2%) but less helpful than feedback provided in the OR (73.5%). Participants felt this simulator would be helpful for teaching junior residents. Simulator-generated metrics correlated with GEARS performance for bladder flap and ureter identification modules in multiple domains including total movements and total time for completion. GEARS performance for the bladder flap module correlated with experience level (novice vs experienced/expert) in domains of interest and total score but did not consistently correlate for the other procedural modules. Performance benchmarks were evaluated for the bladder flap module for each GEARS domain and total score.

CONCLUSION

The modules were well received by participants of all experience levels. Individual simulation modules appear to better discriminate between novice and experienced/expert users than overall simulator performance. Based on these data and participant feedback, use of individual modules in early residency education may be helpful for providing feedback and may ultimately serve as one component of determining readiness to perform robotic hysterectomy.

Assessment of the training program for Versius, a new innovative robotic system for use in minimal access surgery

Objectives

The Versius surgical system has been developed for use in robot-assisted minimal access surgery (MAS). This study aimed to evaluate the effectiveness of the Versius training program.

Design

A 3.5-day program following 10 hours of online didactic training. Participants were assessed during the technical training using the Global Evaluative Assessment of Robotic Skills (GEARS).

Setting

Dry box exercises were conducted in classrooms, and wet lab sessions simulated an operating room environment using cadaveric specimens.

Participants

Seventeen surgical teams participated; surgeons represented general, colorectal, obstetrics/gynecology, and urology specialties. All surgeons had previous laparoscopic MAS experience, while experience with robotics varied.

Main outcomes measures

Participants were scored on a five-point Likert Scale for each of six validated GEARS domains (depth perception, bimanual dexterity, efficiency, force sensitivity, autonomy, and robotic control). Additional metrics used to chart surgeon performance included: combined instrument path length; combined instrument angular path; and time taken to complete each task.

Results

Participants demonstrated an overall improvement in performance during the study, with a mean GEARS Score of 21.0 (SD: 1.9) in Assessment 1 increasing to 23.4 (SD: 2.9) in Validation. Greatest improvements were observed in the depth perception and robotic control domains. Greatest differences were observed when stratifying by robotic experience; those with extensive experience consistently scored higher than those with some or no experience.

Conclusions

The Versius training program is effective; participants were able to successfully operate the system by program completion, and more surgeons achieved intermediate-level and expert-level GEARS scores in Validation compared with Assessment 1.

OUP accepted manuscript

Background

This study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure.

Method

A prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated.

Results

Twenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 – Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019–0.008; GEARS scores, P = 0.029–0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15–41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58–64 per cent).

Conclusions

PBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.

Critical view of the myopectineal orifice: a scoring system to objectively evaluate transabdominal preperitoneal inguinal hernia repair

BACKGROUND

More than 20 million inguinal hernia repair (IHR) procedures are performed worldwide every year. The critical view of the myopectineal orifice (CV of the MPO) is a stepwise guide to the achievement and standardization of minimally invasive IHR (MI IHR). We propose a scoring system as an objective method for fulfillment of the CV of the MPO.

METHODS

The scoring system was employed for evaluation of the transabdominal preperitoneal (TAPP) technique in 15 video-recorded procedures. Two variants of the score were used: the simple CV of the MPO score (s-CVMPO score) and the extended CV of the MPO score (e-CVMPO score). The inter-rater agreement and internal consistency for both scores and the correlation between the two scores were assessed.

RESULTS

Inter-rater agreement with respect to satisfactory/unsatisfactory achievement of the CV of the MPO was high for both the s-CVMPO and e-CVMPO scores (κ = 1, p < 0.001). The Finn coefficient for inter-rater agreement was 0.97 for the s-CVMPO score and 0.99 for the e-CVMPO score (p < 0.001 for both). Both the s-CVMPO and e-CVMPO scores showed internal consistency with Cronbach's α of 0.89 and 0.87, respectively. The correlation coefficient between the two scores for the average score of each procedure was ρ = 0.96 (p < 0.001).

CONCLUSION

The CVMPO score is a reliable tool for expert evaluation of TAPP repair. Implementing the CVMPO score facilitates objective assessment of the safety and quality of the procedure.

Teaching and Training Surgeons in Robotic Colorectal Surgery

Robotic surgery is becoming more popular among practicing physicians as a new modality with improved visualization and mobility (1-2). As patients also desire minimally invasive procedures with quicker recoveries, there is a desire for new surgical residents and fellows to pursue robotic techniques in training (3-4). To develop a new colorectal robotics training program, an institution needs a well-formulated plan for the trainees and mentors with realistic expectations. The development of a robotics training program has potential obstacles, including increased initial cost, longer operative times, and overcoming learning curves. We have devised a four-phase training protocol for residents in colorectal surgical fellowship. Each of these phases attempts to create a curricular framework that outlines logical progression and sets expectations for trainees, Program Directors, and residency faculty. Phase zero begins prior to fellowship and is preparatory. Phase one focuses on an introduction to robotics with learning bedside console troubleshooting and simulation exercises. Phase Two prioritizes operative experience and safety while completing steps independently in a progressive fashion. Phase Three polishes the resident prior to graduation for future practice. We recommend frequent evaluation and open-mindedness while establishing a focused robotics program. The end goal is to graduate fellows with an equivalency certificate who can continue to practice colorectal robotic surgery.

Robotic lobectomy: how to teach thoracic residents

Robotic thoracic surgery emerged at the beginning of the 21st century and keep presenting the continuous development of its robotic systems, tools, and associated techniques. Strong clinical results including safety and oncological outcomes have fostered the dissemination of the robotic platform all over the world. However, there are still some safety concerns, especially regarding more elaborated procedures as lung resections, during the learning curve. In consequence, training programs for surgeons and surgery residents have been proposed to put into operation a strong and complete curriculum for robotic surgery and increase safety during the learning process. Also, the implementation of the training program makes the process complete and efficient. Lung lobectomies are complex procedures especially because of pulmonary arteries and pulmonary veins dissection, which demands quite accurate skills. Consequently, it is believed that specific training of thoracic surgery residents in robotic lobectomy is capital. The ideal curriculum must include technical content and broad psychomotor training using virtual reality models and also physical and animal models. Valid evaluation methods can be used from the first skill training to daily clinical practice. At the beginning as a console surgeon, the resident must initiate gradually with small procedures and progress to more complex surgeries before performing the whole lobectomy.

Introduction of robotic surgery does not negatively affect cardiothoracic surgery resident experience

The objective of this study was to evaluate the educational impact following the implementation of a robotic thoracic surgery program on cardiothoracic (CT) surgery trainees. We hypothesized that the introduction of a robotic thoracic surgery program would adversely affect the CT surgery resident experience, decreasing operative involvement and subsequent competency of surgical procedures. CT surgery residents and thoracic surgery attendings from a single academic institution were administered a recurring, electronic survey from September 2019 to September 2020 following each robotic thoracic surgery case. Surveys evaluated resident involvement and operative performance. This study was exempt from review by our Institutional Review Board. Attendings and residents completed surveys for 86 and 75 cases, respectively. Residents performed > 50% of the operation independently at the surgeon console in 66.2 and 73.3% of cases according to attending and resident responses, respectively. The proportion of trainees able to perform > 75% of the operation increased with each increasing year in training (p = 0.002). Based on the Global Evaluative Assessment of Robotic Skills grading tool, third-year residents averaged higher scores compared to first-year residents (22.9 versus 17.4 out of 30 possible points, p < 0.001), indicating that more extensive prior operative experience could shorten the learning curve of robotic thoracic surgery. CT surgery residents remain actively involved in an operative role during the establishment of a robotic thoracic surgery program. The transition to a robotic thoracic surgery platform appears feasible in a large academic setting without jeopardizing the educational experience of resident trainees.

Hanging up the surgical cap: Assessing the competence of aging surgeons

BACKGROUND

As the average age of surgeons continues to rise, determining when a surgeon should retire is an important public safety concern.

AIM

To investigate strategies used to determine competency in the industrial workplace that could be transferrable in the assessment of aging surgeons and to identify existing competency assessments of practicing surgeons.

METHODS

We searched websites describing non-medical professions within the United States where cognitive and physical competency are necessary for public safety. The mandatory age and certification process, including cognitive and physical requirements, were reported for each profession. Methods for determining surgical competency currently in use, and those existing in the literature, were also identified.

RESULTS

Four non-medical professions requiring mental and physical aptitude that involve public safety and have mandatory testing and/or retirement were identified: Airline pilots, air traffic controllers, firefighters, and United States State Judges. Nine late career practitioner policies designed to evaluate the ageing physician, including surgeons, were described. Six of these policies included subjective performance testing, 4 using peer assessment and 2 using dexterity testing. Six objective testing methods for evaluation of surgeon technical skill were identified in the literature. All were validated for surgical trainees. Only Objective Structured Assessment of Technical Skills (OSATS) was capable of distinguishing between surgeons of different skill level and showing a relationship between skill level and post-operative outcomes.

CONCLUSION

A surgeon should not be forced to hang up his/her surgical cap at a predetermined age, but should be able to practice for as long as his/her surgical skills are objectively maintained at the appropriate level of competency. The strategy of using skill-based simulations in evaluating non-medical professionals can be similarly used as part of the assessment of the ageing surgeons' surgical competency, showing who may require remediation or retirement.

Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery

BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome's point of view is clearly indicated by the available surgical error statistics.

Learning curve of robot-assisted transabdominal preperitoneal (rTAPP) inguinal hernia repair: a cumulative sum (CUSUM) analysis

BACKGROUND

Robot-assisted transabdominal preperitoneal inguinal hernia repair (rTAPP-IHR) is a safe and feasible approach for hernias of varying etiology. We aim to present a single surgeon's learning curve (LC) of this technique based on operative times, while accounting for bilaterality and complexity.

METHODS

This is a retrospective cohort analysis of patients who underwent rTAPP-IHR over a period of 5 years. Patients who underwent primary, recurrent, and complex (previous posterior repair, previous prostatectomy, scrotal, incarcerated) repairs were included. Cumulative and risk-adjusted cumulative sum analyses (CUSUM and RA-CUSUM) were used to depict the evolution of skin-to-skin times and complications/surgical site events (SSEs) with time, respectively.

RESULTS

A total of 371 patients were included in the study. Mean skin-to-skin times were stratified according to four subgroups: unilateral non-complex (46.8 min), unilateral complex (63.2 min), bilateral non-complex (70.9 min), and bilateral complex (102 min). A CUSUM-LC was then plotted using each procedures difference in operative time from its subgroup mean. The peak of the plot occurred at case number 138, which was used as a transition between 'early' and 'late' phases. The average operative time for the late phase was 15.9 min shorter than the early phase (p < 0.001). The RA-CUSUM, plotted using the weight of case complexity and unilateral/bilateral status, also showed decreasing SSE rates after the completion of 138 cases (early phase: 8.8% vs. late phase: 2.2%, p = 0.008). Overall complication rates did not differ significantly between the two phases.

CONCLUSIONS

Our study shows that regardless of bilateral or complex status, rTAPP operative times and SSE rates gradually decreased after completing 138 procedures. Previous laparoscopic experience, robotic team efficiency, and surgical knowledge are important considerations for a surgeon's LC.

Ex Vivo Porcine Model for Robot-Assisted Partial Nephrectomy Simulation at a High-Volume Tertiary Center: Resident Perception and Validation Assessment Using the Global Evaluative Assessment of Robotic Skills Tool

Introduction: With increased demands on surgeon productivity and outcomes, residency robotics training increasingly relies on simulations. The objective of this study is to assess the validity and effectiveness of an ex vivo porcine training model as a useful tool to improve surgical skill and confidence with robot-assisted partial nephrectomy (RAPN) among urology residents. Methods: A 2.5 cm circular area of ex vivo porcine kidneys was marked as the area of the tumor. Tumor excision and renorrhaphy was performed by trainees using a da Vinci Si robot. All residents ranging from postgraduate year (PGY) 2 to 5 participated in four training sessions during the 2017 to 2018 academic year. Each session was videorecorded and scored using the global evaluative assessment of robotic skills (GEARS) by faculty members. Results: Twelve residents completed the program. Initial mean GEARS score was 16.7 and improved by +1.4 with each subsequent session (p = 0.008). Initial mean excision, renorrhaphy, and total times were 8.2, 13.9, and 22.1 minutes, which improved by 1.6, 2.0, and 3.6 minutes, respectively (all p < 0.001). Residents' confidence at performing RAPN and robotic surgery increased after completing the courses (p = 0.012 and p < 0.001, respectively). Overall, residents rated that this program has greatly contributed to their skill (4/5) and confidence (4.1/5) in robotic surgery. Conclusions: An ex vivo porcine simulation model for RAPN and robotic surgery provides measurable improvement in GEARS score and reduction in procedural time, although significant differences for all PGY levels need to be confirmed with larger study participation. Adoption of this simulation in a urology residency curriculum may improve residents' skill and confidence in robotic surgery.

[New techniques and training methods for robot-assisted surgery and cost-benefit analysis of Ivor Lewis esophagectomy]

INTRODUCTION

Robotic surgery was introduced into general surgery more than 20 years ago. Shortly afterwards, Horgan performed the first robotic-assisted esophagectomy in 2003 in Chicago. The aim of this manuscript is to elucidate new developments and training methods in robotic surgery with a cost-benefit analysis for robotic-assisted Ivor Lewis esophagectomy.

METHODS

Systematic literature search regarding new technology and training methods for robotic surgery and cost analysis of intraoperative materials for hybrid and robotic-assisted Ivor Lewis esophagectomy.

RESULTS

Robotic-assisted esophageal surgery is complex and involves an extensive learning curve, which can be shortened with modern teaching methods. New robotic systems aim at the use of image-guided surgery and artificial intelligence. Robotic-assisted surgery of esophageal cancer is significantly more expensive that surgery without this technology.

CONCLUSION

Oncological short-term and long-term benefits need to be further evaluated to support the higher cost of robotic esophageal cancer surgery.

Applying the Delphi process for development of a hepatopancreaticobiliary robotic surgery training curriculum

BACKGROUND

Robotic hepatopancreaticobiliary (HPB) procedures are performed worldwide and establishing processes for safe adoption of this technology is essential for patient benefit. We report results of the Delphi process to define and optimize robotic training procedures for HPB surgeons.

METHODS

In 2019, a robotic HPB surgery panel with an interest in surgical training from the Americas and Europe was created and met. An e-consensus-finding exercise using the Delphi process was applied and consensus was defined as 80% agreement on each question. Iterations of anonymous voting continued over three rounds.

RESULTS

Members agreed on several points: there was need for a standardized robotic training curriculum for HPB surgery that considers experience of surgeons and based on a robotic hepatectomy includes a common approach for "basic robotic skills" training (e-learning module, including hardware description, patient selection, port placement, docking, troubleshooting, fundamentals of robotic surgery, team training and efficiency, and emergencies) and an "advanced technical skills curriculum" (e-learning, including patient selection information, cognitive skills, and recommended operative equipment lists). A modular approach to index procedures should be used with video demonstrations, port placement for index procedure, troubleshooting, and emergency scenario management information. Inexperienced surgeons should undergo training in basic robotic skills and console proficiency, transitioning to full procedure training of e-learning (video demonstration, simulation training, case observation, and final evaluation). Experienced surgeons should undergo basic training when using a new system (e-learning, dry lab, and operating room (OR) team training, virtual reality modules, and wet lab; case observations were unnecessary for basic training) and should complete the advanced index procedural robotic curriculum with assessment by wet lab, case observation, and OR team training.

CONCLUSIONS

Optimization and standardization of training and education of HPB surgeons in robotic procedures was agreed upon. Results are being incorporated into future curriculum for education in robotic surgery.

Correlation between operative time and crowd-sourced skills assessment for robotic bariatric surgery

INTRODUCTION

Operative time has been traditionally used as a proxy for surgical skill and is commonly utilized to measure the learning curve, assuming that faster operations indicate a more skilled surgeon. The Global Evaluative Assessment of Robotic Skills (GEARS) rubric is a validated Likert scale for evaluating technical skill. We hypothesize that operative time will not correlate with the GEARS score.

METHODS

Patients undergoing elective robotic sleeve gastrectomy at a single bariatric center of excellence hospital from January 2019 to March 2020 were captured in a prospectively maintained database. For step-specific scoring, videos were broken down into three steps: ligation of short gastric vessels, gastric transection, and oversewing the staple line. Overall and step-specific GEARS scores were assigned by crowd-sourced evaluators. Correlation between operative time and GEARS score was assessed with linear regression and calculation of the R² statistic.

RESULTS

Sixty-eight patients were included in the study, with a mean operative time of 112 ± 27.4 min. The mean GEARS score was 20.1 ± 0.81. Mean scores for the GEARS subcomponents were: bimanual dexterity 4.06 ± 0.17; depth perception 3.96 ± 0.24; efficiency 3.82 ± 0.19; force sensitivity 4.06 ± 0.20; robotic control 4.16 ± 0.21. Operative time and overall score showed no correlation (R² = 0.0146, p = 0.326). Step-specific times and scores showed weak correlation for gastric transection (R² = 0.0737, p = 0.028) and no correlation for ligation of short gastric vessels (R² = 0.0262, p = 0.209) or oversewing the staple line (R² = 0.0142, p = 0.344).

CONCLUSIONS

Operative time and crowd-sourced GEARS score were not correlated. Operative time and GEARS scores measure different performance characteristics, and future studies should consider using both a validated skills assessment tool and operative time for a more complete evaluation of skill.

Transoral Robotic Surgical Proficiency Via Real-Time Tactile Collision Awareness System

OBJECTIVES

In 2009, the Food and Drug Administration approved the use of the surgical robotic system for removal of benign and malignant conditions of the upper aerodigestive tract. This novel application of robotic-assisted surgery, termed transoral robotic surgery (TORS), places robotic instruments and camera system through the mouth to reach recessed areas of the pharynx and larynx. Over the successive decade, there was a rapid adoption of TORS with a surgical growth rate that continues to increase. Despite the rapid clinical acceptance, the field of TORS has not yet seen substantive changes or advances in the technical shortcomings, the lack of which has restricted objective TORS-specific surgical skills assessment as well as subsequent skills improvement efforts. One of the primary technical challenges of TORS is operating in a confined space, where the robotic system is maneuvered within the restrictive boundaries of the mouth and throat. Due to these confined boundaries of the pharynx, instruments can frequently collide with anatomic structures such as teeth and bone, producing anatomic collisions. Therefore, we hypothesized that anatomic collisions negatively impact TORS surgical performance. Secondarily, we hypothesized that avoidance of unwanted anatomic collisions could improve TORS surgical proficiency.

METHODS

Design and fidelity testing for a custom TORS training platform with an integrated anatomic collision-sensing system providing real-time tactile feedback is described. Following successful platform assembly and testing, validation study using the platform was carried through prospective surgical training with trial randomization. Twenty otolaryngology-head and neck surgery residents, each trainee performing three discrete mock surgical trials (n = 60), performed the initial system validation. Ten of the 20 residents were randomized to perform the surgical trials utilizing the real-time feedback system. The remaining 10 residents were randomized to perform the surgical trials without the feedback system, although the system still could record collision data. Surgical proficiency was measured by Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and tumor resection scores (categorical scale ranging 0-3, describing the adequacy of resection).

RESULTS

Major anatomic collisions (greater than 5N of force) negatively affected GEARS robotic skills. A mixed model analysis demonstrated that for every additional occurrence of a major collision, GEARS robotic skills assessment score would decrease by 0.29 points (P = .04). Real-time collision awareness created significantly fewer major (> 5 N) anatomic collisions with the tactile feedback system active (n = 30, mean collisions = 2.9 ± 4.2) as compared with trials without tactile feedback (n = 30, mean collisions = 12.53 ± 23.23) (P < .001). The second assessment measure of time to completion was unaffected by the presence of collisions or by the use of tactile feedback system. The third proficiency assessment was measured with tumor resection grading. Tumor resection scores was significantly (P = .02) improved with collision awareness system activated than trials without collision awareness.

CONCLUSION

In order to test our primary hypothesis, a novel TORS training platform was successfully developed that provides collision force measurements including frequency, severity, and duration of anatomic collisions. Additionally, the platform was modulated to provide real-time tactile feedback of the occurrence of out-of-field collisions. Utilizing this custom platform, our hypothesis that anatomic collisions during TORS diminishes surgical performance was supported. Additionally, our secondary hypothesis that subsequent reduction of anatomic collisions improves TORS proficiency was supported by the surgical trial. Dedicated investigation to characterize the effect size and clinical impact is required in order to translate this finding into training curriculums and into clinical utilization.

LEVEL OF EVIDENCE

II (Randomized trial) Laryngoscope, 130:S1-S17, 2020.

Development of a Transoral Robotic Surgery Training Platform

Transoral robotic surgery (TORS) presents unique challenges due to difficulty manipulating surgical instruments within the tight confines of the oral cavity. Collisions between the end effectors and anatomical structures can be visualized through the endoscope; however, instrument shaft collisions are outside of the field-of-view. Acquiring the requisite skill set to minimize these collisions is challenging due to the lack of an appropriate training platform. In this paper, we present a TORS training platform with an integrated collision sensing system and real-time haptic feedback. Preliminary testing involved the recruitment of 10 Otolaryngology residents assigned to `feedback' (N=5) and `no feedback' (N=5) groups. Each trainee performed three mock surgical procedures involving the resection of a tumor from the base of the tongue. Superior surgical performance was observed in the feedback group suggesting that haptic feedback will enhance the acquisition of surgical skills.

Incorporation and validation of clinically relevant performance metrics of simulation (CRPMS) into a novel full-immersion simulation platform for nerve-sparing robot-assisted radical prostatectomy (NS-RARP) utilizing three-dimensional printing and hydrogel casting technology

OBJECTIVES

To incorporate and validate clinically relevant performance metrics of simulation (CRPMS) into a hydrogel model for nerve-sparing robot-assisted radical prostatectomy (NS-RARP).

MATERIALS AND METHODS

Anatomically accurate models of the human pelvis, bladder, prostate, urethra, neurovascular bundle (NVB) and relevant adjacent structures were created from patient MRI by injecting polyvinyl alcohol (PVA) hydrogels into three-dimensionally printed injection molds. The following steps of NS-RARP were simulated: bladder neck dissection; seminal vesicle mobilization; NVB dissection; and urethrovesical anastomosis (UVA). Five experts (caseload >500) and nine novices (caseload <50) completed the simulation. Force applied to the NVB during the dissection was quantified by a novel tension wire sensor system fabricated into the NVB. Post-simulation margin status (assessed by induction of chemiluminescent reaction with fluorescent dye mixed into the prostate PVA) and UVA weathertightness (via a standard 180-mL leak test) were also assessed. Objective scoring, using Global Evaluative Assessment of Robotic Skills (GEARS) and Robotic Anastomosis Competency Evaluation (RACE), was performed by two blinded surgeons. GEARS scores were correlated with forces applied to the NVB, and RACE scores were correlated with UVA leak rates.

RESULTS

The expert group achieved faster task-specific times for nerve-sparing (P = 0.007) and superior surgical margin results (P = 0.011). Nerve forces applied were significantly lower for the expert group with regard to maximum force (P = 0.011), average force (P = 0.011), peak frequency (P = 0.027) and total energy (P = 0.003). Higher force sensitivity (subcategory of GEARS score) and total GEARS score correlated with lower nerve forces (total energy in Joules) applied to NVB during the simulation with a correlation coefficient (r value) of -0.66 (P = 0.019) and -0.87 (P = 0.000), respectively. Both total and force sensitivity GEARS scores were significantly higher in the expert group compared to the novice group (P = 0.003). UVA leak rate highly correlated with total RACE score r value = -0.86 (P = 0.000). Mean RACE scores were also significantly different between novices and experts (P = 0.003).

CONCLUSION

We present a realistic, feedback-driven, full-immersion simulation platform for the development and evaluation of surgical skills pertinent to NS-RARP. The correlation of validated objective metrics (GEARS and RACE) with our CRPMS suggests their application as a novel method for real-time assessment and feedback during robotic surgery training. Further work is required to assess the ability to predict live surgical outcomes.

Outcomes from the Delphi process of the Thoracic Robotic Curriculum Development Committee

OBJECTIVES

As the adoption of robotic procedures becomes more widespread, additional risk related to the learning curve can be expected. This article reports the results of a Delphi process to define procedures to optimize robotic training of thoracic surgeons and to promote safe performance of established robotic interventions as, for example, lung cancer and thymoma surgery.

METHODS

In June 2016, a working panel was spontaneously created by members of the European Society of Thoracic Surgeons (ESTS) and European Association for Cardio-Thoracic Surgery (EACTS) with a specialist interest in robotic thoracic surgery and/or surgical training. An e-consensus-finding exercise using the Delphi methodology was applied requiring 80% agreement to reach consensus on each question. Repeated iterations of anonymous voting continued over 3 rounds.

RESULTS

Agreement was reached on many points: a standardized robotic training curriculum for robotic thoracic surgery should be divided into clearly defined sections as a staged learning pathway; the basic robotic curriculum should include a baseline evaluation, an e-learning module, a simulation-based training (including virtual reality simulation, Dry lab and Wet lab) and a robotic theatre (bedside) observation. Advanced robotic training should include e-learning on index procedures (right upper lobe) with video demonstration, access to video library of robotic procedures, simulation training, modular console training to index procedure, transition to full-procedure training with a proctor and final evaluation of the submitted video to certified independent examiners.

CONCLUSIONS

Agreement was reached on a large number of questions to optimize and standardize training and education of thoracic surgeons in robotic activity. The production of the content of the learning material is ongoing.

Development of a technical checklist for the assessment of suturing in robotic surgery

BACKGROUND

With the increased use of simulation for surgical training, there is a need for objective forms of assessment to evaluate trainees. The Global Evaluative Assessment of Robotic Skills (GEARS) is widely used for assessing skills in robotic surgery, but there are no recognised checklist scoring systems. This study aimed to develop a checklist for suturing in robotic surgery.

METHODS

A suturing checklist for needle driving and knot tying was constructed following evaluation of participants performing urethrovesical anastomoses. Key procedural steps were identified from expert videos, while assessing novice videos allowed identification of common technical errors. 22 novice and 13 expert videos were marked on needle driving, while 18 novices and 10 experts were assessed on knot tying. Validation of the finalised checklist was performed with the assessment of 39 separate novices by an expert surgeon and compared to GEARS scoring.

RESULTS

The internal consistency of the preliminary checklist was high (Cronbach's alpha = 0.870 for needle driving items; 0.736 for knot tying items), and after removal of poorly correlating items, the final checklist contained 23 steps. Both the needle driving and knot tying categories discriminated between novices and experts, p < 0.005. While the GEARS score demonstrated construct validity for needle driving, it could not significantly differentiate between novices and experts for knot tying, p = 0.286. The needle driving category significantly correlated with the corresponding GEARS scores (r_s = 0.613, p < 0.005), but the correlation for knot tying was insignificant (r_s = 0.296, p = 0.127). The pilot data indicates the checklist significantly correlated with the GEARS score (p < 0.005).

CONCLUSION

This study reports the development of a valid assessment tool for suturing in robotic surgery. Given that checklists are simple to use, there is significant scope for this checklist to be used in surgical training.

The Society of European Robotic Gynaecological Surgery (SERGS) Pilot Curriculum for robot assisted gynecological surgery

Purpose

To set forth experiences in the context of the SERGS Pilot Curriculum—the first standardized educational program for robotic use in gynecological surgery—in terms of feasibility, effectiveness and potential for certification.

Methods

The Society of European Robotic Gynecological Surgery (SERGS) outlined a Pilot Curriculum for standardized education in robot-assisted laparoscopic gynecological surgery. Its feasibility and acceptance were checked in the form of a fellowship pilot program conducted at four European Centers of Excellence for robot-assisted surgery. Results and conclusions derived from this pilot program are presented.

Results

The SERGS Pilot Curriculum defines criteria for a standardized training and assessment of performance, boosts the learning curve of the candidate and increases contentment at work. Regarding face validity, it proves valuable as finally all candidates could perform the outlined procedure safely and efficiently without supervision.

Conclusion

Due to the immense increase of robotic procedures in gynecology standardized training curricula are indispensable. This seems highly necessary to ensure patients’ safety and surgical outcome. The SERGS Pilot Curriculum sets standards for a stepwise theoretical and practical training in gynecological robotic procedures. It seems feasible as instrument for accreditation as gynecologic robotic surgeon. Though as a general applicable guideline for systematic training in robot-assisted surgery, a definite curriculum should have a more definite timeline and implementation of a structured assessment of performance.

Electronic supplementary material

The online version of this article (10.1007/s00404-017-4612-5) contains supplementary material, which is available to authorized users.