Development and validation of the objective assessment of robotic suturing and knot tying skills for chicken anastomotic model

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review

BACKGROUND AND OBJECTIVE

Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways.

METHODS

We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification.

KEY FINDINGS AND LIMITATIONS

A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study.

CONCLUSIONS AND CLINICAL IMPLICATIONS

Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact.

PATIENT SUMMARY

Simulation-based training and programs in which progression is based on proficiency represent the new standard of quality for achieving surgical proficiency in urology. Studies have demonstrated the educational impact of these approaches. However, there are still no standardized training pathways for several urology procedures.

ESRU-ESU-YAU_UROTECH Survey on Urology Residents Surgical Training: Are We Ready for Simulation and a Standardized Program?

Background

Currently, the landscape of surgical training is undergoing rapid evolution, marked by the initial implementation of standardized surgical training programs, which are further facilitated by the emergence of new technologies. However, this proliferation is uneven across various countries and hospitals.

Objective

To offer a comprehensive overview of the existing surgical training programs throughout Europe, with a specific focus on the accessibility of simulation resources and standardized surgical programs.

Design setting and participants

A dedicated survey was designed and spread in May 2022 via the European Association of Urology (EAU) mail list, to Young Urologist Office (YUO), Junior membership, European Urology Residents Education Program participants between 2014 and 2022, and other urologists under 40 yr, and via the EAU Newsletter.

Intervention

A 64-item, online-based survey in accordance with the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) using the platform of Survey Monkey (Portland, OR, USA) was realized.

Outcome measurements and statistical analysis

The study involved an assessment of the demographic characteristics. Additionally, it explored the type of center, availability of various surgical approaches, presence of training infrastructure, participation in courses, organization of training, and participants' satisfaction with the training program. The level of satisfaction was evaluated using a Likert-5 scale. The subsequent sections delved into surgical training within the realms of open, laparoscopic, robotic, and endoscopic surgery, each explored separately. Finally, the investigation encompassed the presence of a structured training course and the availability of a duly validated final evaluation process.

Results and limitations

There were 375 responders with a completion rate of 82%. Among them, 75% were identified as male, 50.6% were young urologists, 31.7% were senior residents, and 17.6% were junior residents. A significant majority of participants (69.6%) were affiliated with academic centers. Regarding the presence of dry lab training facilities, only 50.3% of respondents indicated its availability. Among these centers, 46.7% were primarily focused on laparoscopy training. The availability of virtual and wet lab training centers was even more limited, with rates of 31.5% and 16.2%, respectively. Direct patient involvement was reported in 80.5% of cases for open surgery, 58.8% for laparoscopy, 25.0% for robotics, and 78.6% for endourology. It is worth noting that in <25% of instances, training followed a well-defined standardized program comprising both preclinical and clinical modular phases. Finally, the analysis of participant feedback showed that 49.7% of respondents expressed a satisfaction rating of either 4 or 5 points with respect to the training program. The limitations of our study include the low response rate, predominance of participants from academic centers, and absence of responses from individuals not affiliated with the EAU network.

Conclusions

The current distribution of surgical training centers falls short of ensuring widespread access to standardized training programs. Although dry lab facilities are relatively well spread, the availability of wet lab resources remains restricted. Additionally, it appears that many trainees' initial exposure to surgery occurs directly with patients. There is a pressing need for continued endeavors to establish uniform training routes and assessment techniques across various surgical methodologies.

Patient summary

Nowadays, the surgical training landscape is heterogeneous across different countries. The implementation of a standardized training methodology to enhance the overall quality of surgical training and thereby improving patient outcomes is needed.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

A cost-effective model for training in Robot-Assisted Sacrocolpopexy

BACKGROUND

The number of robotically assisted sacrocolpopexy procedures are increasing; therefore, experienced clinicians are needed. Simulation-based cadaver models are challenging in aspects of cost and availability. Therefore, we need to look at alternative and more cost-effective models.

OBJECTIVE

The objective of this video was to design a new surgical model for the training of robotic-assisted sacrocolpopexy, which is affordable and accessible.

METHODS

We used a whole chicken model to simulate the female pelvic floor. We used Medtronic's Hugo™ RAS system as the robotic console in that procedure. A vaginal cuff was prepared from the proventriculus (stomach), and a Y shaped mesh was secured to the ischium to simulate the sacrocolpopexy procedure.

CONCLUSION

This model is easily constructed and in our view is cost-effective. We have demonstrated a new valuable education tool that can serve as a practical simulation model to teach the sacrocolpopexy procedure and to improve trainees' skills. A larger cohort study size is essential to demonstrate the learning curve among young trainees using this simulation model.

A Multi-institution Study on the Association of Virtual Reality Skills with Continence Recovery after Robot-assisted Radical Prostatectomy

BACKGROUND

Virtual reality (VR) simulators are increasingly being used for surgical skills training. It is unclear what skills are best improved via VR, translate to live surgical skills, and influence patient outcomes.

OBJECTIVE

To assess surgeons in VR and live surgery using a suturing assessment tool and evaluate the association between technical skills and a clinical outcome.

DESIGN, SETTING, AND PARTICIPANTS

This prospective five-center study enrolled participants who completed VR suturing exercises and provided live surgical video. Graders provided skill assessments using the validated End-To-End Assessment of Suturing Expertise (EASE) suturing evaluation tool.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

A hierarchical Poisson model was used to compare skill scores among cohorts and evaluate the association of scores with clinical outcomes. Spearman's method was used to assess correlation between VR and live skills.

RESULTS AND LIMITATIONS

Ten novices, ten surgeons with intermediate expertise (median 64 cases, interquartile range [IQR] 6-80), and 26 expert surgeons (median 850 cases, IQR 375-3000) participated in this study. Intermediate and expert surgeons were significantly more likely to have ideal scores in comparison to novices for the subskills needle hold angle, wrist rotation, and wrist rotation needle withdrawal (p < 0.01). For both intermediate and expert surgeons, there was positive correlation between VR and live skills for needle hold angle (p < 0.05). For expert surgeons, there was a positive association between ideal scores for VR needle hold angle and driving smoothness subskills and 3-mo continence recovery (p < 0.05). Limitations include the size of the intermediate surgeon sample and clinical data limited to expert surgeons.

CONCLUSIONS

EASE can be used in VR to identify skills to improve for trainee surgeons. Technical skills that influence postoperative outcomes may be assessable in VR.

PATIENT SUMMARY

This study provides insights into surgical skills that translate from virtual simulation to live surgery and that have an impact on urinary continence after robot-assisted removal of the prostate. We also highlight the usefulness of virtual reality in surgical education.

What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature

A crucial element of any surgical training program is the ability to provide procedure-specific, objective, and reliable measures of performance. During robotic surgery, objective clinically relevant performance metrics (CRPMs) can provide tailored contextual feedback and correlate with clinical outcomes. This review aims to define CRPMs, assess their validity in robotic surgical training and compare CRPMs to existing measures of robotic performance. A systematic search of Medline and Embase databases was conducted in May 2022 following the PRISMA guidelines. The search terms included Clinically Relevant Performance Metrics (CRPMs) OR Clinically Relevant Outcome Measures (CROMs) AND robotic surgery. The study settings, speciality, operative context, study design, metric details, and validation status were extracted and analysed. The initial search yielded 116 citations, of which 6 were included. Citation searching identified 3 additional studies, resulting in 9 studies included in this review. Metrics were defined as CRPMs, CROMs, proficiency-based performance metrics and reference-procedure metrics which were developed using a modified Delphi methodology. All metrics underwent both contents and construct validation. Two studies found a strong correlation with GEARS but none correlated their metrics with patient outcome data. CRPMs are a validated and objective approach for assessing trainee proficiency. Evaluating CRPMs with other robotic-assessment tools will facilitate a multimodal metric evaluation approach to robotic surgery training. Further studies should assess the correlation with clinical outcomes. This review highlights there is significant scope for the development and validation of CRPMs to establish proficiency-based progression curricula that can be translated from a simulation setting into clinical practice.

Proficiency-based progression training for robotic surgery skills training: a randomized clinical trial

OBJECTIVE

To determine whether proficiency-based progression (PBP) training leads to better robotic surgical performance compared to traditional training (TT), given that the value of PBP training for learning robotic surgical skills is unclear.

MATERIALS AND METHODS

The PROVESA trial is a multicentric, prospective, randomized and blinded clinical study comparing PBP training with TT for robotic suturing and knot-tying anastomosis skills. A total of 36 robotic surgery-naïve junior residents were recruited from 16 training sites and 12 residency training programmes. Participants were randomly allocated to metric-based PBP training or the current standard of care TT, and compared at the end of training. The primary outcome was percentage of participants reaching the predefined proficiency benchmark. Secondary outcomes were the numbers of procedure steps and errors made.

RESULTS

Of the group that received TT, 3/18 reached the proficiency benchmark versus 12/18 of the PBP group (i.e. the PBP group were ~10 times as likely to demonstrate proficiency [P = 0.006]). The PBP group demonstrated a 51% reduction in number of performance errors from baseline to the final assessment (18.3 vs 8.9). The TT group demonstrated a marginal improvement (15.94 vs 15.44) in errors made.

CONCLUSIONS

The PROVESA trial is the first prospective randomized controlled trial on basic skills training in robotic surgery. Implementation of a PBP training methodology resulted in superior surgical performance for robotic suturing and knot-tying anastomosis performance. Compared to TT, better surgical quality could be obtained by implementing PBP training for basic skills in robotic surgery.

Does quality assured eLearning provide adequate preparation for robotic surgical skills; a prospective, randomized and multi-center study

Purpose

In particular after the onset of the COVID-19 pandemic, there was a precipitous rush to implement virtual and online learning strategies in surgery and medicine. It is essential to understand whether this approach is sufficient and adequate to allow the development of robotic basic surgical skills. The main aim of the authors was to verify if the quality assured eLearning is sufficient to prepare individuals to perform a basic surgical robotic task.

Methods

A prospective, randomized and multi-center study was conducted in September 2020 in the ORSI Academy, International surgical robotic training center. Forty-seven participants, with no experience but a special interest in robotic surgery, were matched and randomized into four groups who underwent a didactic preparation with different formats before carrying out a robotic suturing and anastomosis task. Didactic preparation methods ranged from a complete eLearning path to peer-reviewed published manuscripts describing the suturing, knot tying and task assessment metrics.

Results

The primary outcome was the percentage of trainees who demonstrated the quantitatively defined proficiency benchmark after learning to complete an assisted but unaided robotic vesico-urethral anastomosis task. The quantitatively defined benchmark was based on the objectively assessed performance (i.e., procedure steps completed, errors and critical errors) of experienced robotic surgeons for a proficiency-based progression (PBP) training course. None of the trainees in this study demonstrated the proficiency benchmarks in completing the robotic surgery task.

Conclusions

PBP-based e-learning methodology is an effective training method avoiding critical errors in the suturing and knotting task. Quality assured online learning is insufficient preparation for robotic suturing and knot tying anastomosis skills.

Trial registration

ClinicalTrials.gov Identifier: NCT04541615.

Potential Contenders for the Leadership in Robotic Surgery

Purpose: To summarize the scientific published literature on new robotic surgical platforms with potential use in the urological field, reviewing their evolution from presentation until the present day. Our goal is to describe the current characteristics and possible prospects for these platforms. Materials and Methods: A nonsystematic search of the PubMed, Cochrane library's Central, EMBASE, MEDLINE, and Scopus databases was conducted to identify scientific literature about new robotic platforms other than the Da Vinci^® system, reviewing their evolution from inception until December 2020. Only English language publications were included. The following keywords were used: "new robotic platforms," "Revo-I robot," "Versius robot," and "Senhance robot." All relevant English-language original studies were analyzed by one author (R.F.) and summarized after discussion with an independent third party (E.M., S.Y., S.P., and M.A.). Results: Since 1995, Intuitive Surgical, Inc., with the Da Vinci surgical system, is the leading company in the robotic surgical market. However, Revo-I^®, Versius^®, and Senhance^® are the other three platforms that recently appeared on the market with available articles published in peer-reviewed journals. Among these three new surgical systems, the Senhance robot has the most substantial scientific proof of its capacity to perform minimally invasive urological surgery and as such, it might become a contender of the Da Vinci robot. Conclusions: The Da Vinci surgical platform has allowed the diffusion of robotic surgery worldwide and showed the different advantages of this type of technique. However, its use has some drawbacks, especially its price. New robotic platforms characterized by unique features are under development. Of note, they might be less expensive compared with the Da Vinci robotic system. We found that these new platforms are still at the beginning of their technical and scientific validation. However, the Senhance robot is in a more advanced stage, with clinical studies supporting its full implementation.

Development and validation of the metric-based assessment of a robotic vessel dissection, vessel loop positioning, clip applying and bipolar coagulation task on an avian model

The evolution of robotic technology and its diffusion does not seem to have been adequately accompanied by the development and implementation of surgeon training programs that ensure skilled and safe device use at the start of the learning curve. The objective of the study is to develop and validate performance metrics for vessel dissection, vessel loop positioning, clip applying and bipolar coagulation using an avian model. Three robotic surgeons and a behavioral scientist characterized the performance metrics of the task according to the proficiency-based progression methodology. Fourteen experienced robotic surgeons from different European countries participated in a modified online Delphi consensus. Eight experienced surgeons and eight novices performed the robotic task twice. In the Delphi meeting, 100% consensus was reached on the performance metrics. Novice surgeons took 26 min to complete the entire task on trial 1 and 20 min on trial 2. Experts took 10.1 min and 9.5 min. On average the Expert Group completed the task 137% faster than the Novice Group. The amount of time to reach the vessel part of the task was also calculated. Novice surgeons took 26 min on trial 1 and 20 min on trial 2. Experts took 5.5 min and 4.8 min. On average the experts reached the vessel 200% faster than the novices. The Expert Group made 155% fewer performance errors than the Novice Group. The mean IRR of video-recorded performance assessments for all metrics was 0.96 (95% confidence intervals (CI) lower = 0.94-upper = 0.98). We report the development and validation for a standard and replicable basic robotic vessel dissection, vessel loop positioning, clip applying and bipolar coagulation task on an avian model. The development of objective performance metrics, based on a transparent and fair methodology (i.e., PBP), is the first fundamental step toward quality assured training. This task developed on the avian model proved to have good results in the validation study.

Objective assessment of intraoperative skills for robot-assisted radical prostatectomy (RARP): results from the ERUS Scientific and Educational Working Groups Metrics Initiative

OBJECTIVE

To develop and seek consensus from procedure experts on the metrics that best characterise a reference robot-assisted radical prostatectomy (RARP) and determine if the metrics distinguished between the objectively assessed RARP performance of experienced and novice urologists, as identifying objective performance metrics for surgical training in robotic surgery is imperative for patient safety.

MATERIALS AND METHODS

In Study 1, the metrics, i.e. 12 phases of the procedure, 81 steps, 245 errors and 110 critical errors for a reference RARP were developed and then presented to an international Delphi panel of 19 experienced urologists. In Study 2, 12 very experienced surgeons (VES) who had performed >500 RARPs and 12 novice urology surgeons performed a RARP, which was video recorded and assessed by two experienced urologists blinded as to subject and group. Percentage agreement between experienced urologists for the Delphi meeting and Mann-Whitney U- and Kruskal-Wallis tests were used for construct validation of the newly identified RARP metrics.

RESULTS

At the Delphi panel, consensus was reached on the appropriateness of the metrics for a reference RARP. In Study 2, the results showed that the VES performed ~4% more procedure steps and made 72% fewer procedure errors than the novices (P = 0.027). Phases VIIa and VIIb (i.e. neurovascular bundle dissection) best discriminated between the VES and novices.

LIMITATIONS

VES whose performance was in the bottom half of their group demonstrated considerable error variability and made five-times as many errors as the other half of the group (P = 0.006).

CONCLUSIONS

The international Delphi panel reached high-level consensus on the RARP metrics that reliably distinguished between the objectively scored procedure performance of VES and novices. Reliable and valid performance metrics of RARP are imperative for effective and quality assured surgical training.