Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

Evaluation status of current and emerging minimally invasive robotic surgical platforms

BACKGROUND

The rapid adoption of robotics within minimally invasive surgical specialties has also seen an explosion of new technology including multi- and single port, natural orifice transluminal endoscopic surgery (NOTES), endoluminal and "on-demand" platforms. This review aims to evaluate the validation status of current and emerging MIS robotic platforms, using the IDEAL Framework.

METHODS

A scoping review exploring robotic minimally invasive surgical devices, technology and systems in use or being developed was performed, including general surgery, gynaecology, urology and cardiothoracics. Systems operating purely outside the abdomen or thorax and endoluminal or natural orifice platforms were excluded. PubMed, Google Scholar, journal reports and information from the public domain were collected. Each company was approached via email for a virtual interview to discover more about the systems and to quality check data. The IDEAL Framework is an internationally accepted tool to evaluate novel surgical technology, consisting of four stages: idea, development/exploration, assessment, and surveillance. An IDEAL stage, synonymous with validation status in this review, was assigned by reviewing the published literature.

RESULTS

21 companies with 23 different robotic platforms were identified for data collection, 13 with national and/or international regulatory approval. Of the 17 multiport systems, 1 is fully evaluated at stage 4, 2 are stage 3, 6 stage 2b, 2 at stage 2a, 2 stage 1, and 4 at the pre-IDEAL stage 0. Of the 6 single-port systems none have been fully evaluated with 1 at stage 3, 3 at stage 1 and 2 at stage 0.

CONCLUSIONS

The majority of existing robotic platforms are currently at the preclinical to developmental and exploratory stage of evaluation. Using the IDEAL framework will ensure that emerging robotic platforms are fully evaluated with long-term data, to inform the surgical workforce and ensure patient safety.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills

BACKGROUND

Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment.

METHODS

Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed.

RESULTS

OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001).

CONCLUSION

OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.

Global assessment of surgical skills (GASS): validation of a new instrument to measure global technical safety in surgical procedures

BACKGROUND

Broad implementation of the American Board of Surgery's entrustable professional activities initiative will require assessment instruments that are reliable and easy to use. Existing assessment instruments of general laparoscopic surgical skills have limited reliability, efficiency, and validity across the spectrum of formative (low-stakes) and summative (high-stakes) assessments. A novel six-item global assessment of surgical skills (GASS) instrument was developed and evaluated with a focus upon safe versus unsafe surgical practice scoring rubric.

METHODS

The GASS was developed by iterative engagement with expert laparoscopic surgeons and includes six items (economy of motion, tissue handling, appreciating operative anatomy, bimanual dexterity, achievement of hemostasis, overall performance) with a uniform three-point scoring rubric ("poor-unsafe", "adequate-safe", "good-safe"). To test inter-rater reliability, a cross-sectional study of four bariatric surgeons with experience ranging from 4 to 28 years applied the GASS and the global operative assessment of laparoscopic skills (GOALS) to 30 consecutive Roux-en-Y gastric bypass procedure operative videos. Inter-rater reliability was assessed for a simplified dichotomous "safe" versus "unsafe" scoring rubric using Gwet's AC2.

RESULTS

The GASS inter-rater reliability was very high across all six domains (0.88-1.00). The GASS performed comparably to the GOALS inter-rater reliability scores (0.96-1.00). The economy of motion and bimanual dexterity items had the highest percentage of unsafe ratings (9.2% and 5.8%, respectively).

CONCLUSION

The GASS, a novel six-item instrument of general laparoscopic surgical skills, was designed with a simple scoring rubric (poor-safe, adequate-safe, good-safe) to minimize rater burden and focus feedback to trainees and promotion evaluations on safe surgical performance. Initial evaluation of the GASS is promising, demonstrating high inter-rater reliability. Future research will seek to assess the GASS against a broader spectrum of laparoscopic procedures.

Kinematic data profile and clinical outcomes in robotic inguinal hernia repairs: a pilot study

BACKGROUND

Surgical training requires clinical knowledge and technical skills to operate safely and optimize clinical outcomes. Technical skills are hard to measure. The Intuitive Data Recorder (IDR), (Sunnyvale, CA) allows for the measurement of technical skills using objective performance indicators (OPIs) from kinematic event data. Our goal was to determine whether OPIs improve with surgeon experience and whether they are correlated with clinical outcomes for robotic inguinal hernia repair (RIHR).

METHODS

The IDR was used to record RIHRs from six surgeons. Data were obtained from 98 inguinal hernia repairs from February 2022 to February 2023. Patients were called on postoperative days 5-10 and asked to take the Carolina Comfort Scale (CCS) survey to evaluate acute clinical outcomes. A Pearson test was run to determine correlations between OPIs from the IDR with a surgeon's yearly RIHR experience and with CCS scores. Linear regression was then run for correlated OPIs.

RESULTS

Multiple OPIs were correlated with surgeon experience. Specifically, for the task of peritoneal flap exploration, we found that 23 OPIs were significantly correlated with surgeons' 1-year RIHR case number. Total angular motion distance of the left arm instrument had a correlation of - 0.238 (95% CI - 0.417, - 0.042) for RIHR yearly case number. Total angular motion distance of right arm instrument was also negatively correlated with RIHR in 1 year with a correlation of - 0.242 (95% CI - 0.420, - 0.046). For clinical outcomes, wrist articulation of the surgeon's console positively correlated with acute sensation scores from the CCS with a correlation of 0.453 (95% CI 0.013, 0.746).

CONCLUSIONS

This study defines multiple OPIs that correlate with surgeon experience and with outcomes. Using this knowledge, surgical simulation platforms can be designed to teach patterns to surgical trainees that are associated with increased surgical experience and with improved postoperative outcomes.

Crossover-effects in technical skills between laparoscopy and robot-assisted surgery

INTRODUCTION

Robot-assisted surgery is often performed by experienced laparoscopic surgeons. However, this technique requires a different set of technical skills and surgeons are expected to alternate between these approaches. The aim of this study is to investigate the crossover effects when switching between laparoscopic and robot-assisted surgery.

METHODS

An international multicentre crossover study was conducted. Trainees with distinctly different levels of experience were divided into three groups (novice, intermediate, expert). Each trainee performed six trials of a standardized suturing task using a laparoscopic box trainer and six trials using the da Vinci surgical robot. Both systems were equipped with the ForceSense system, measuring five force-based parameters for objective assessment of tissue handling skills. Statistical comparison was done between the sixth and seventh trial to identify transition effects. Unexpected changes in parameter outcomes after the seventh trial were further investigated.

RESULTS

A total of 720 trials, performed by 60 participants, were analysed. The expert group increased their tissue handling forces with 46% (maximum impulse 11.5 N/s to 16.8 N/s, p = 0.05), when switching from robot-assisted surgery to laparoscopy. When switching from laparoscopy to robot-assisted surgery, intermediates and experts significantly decreased in motion efficiency (time (sec), resp. 68 vs. 100, p = 0.05, and 44 vs. 84, p = 0.05). Further investigation between the seventh and ninth trial showed that the intermediate group increased their force exertion with 78% (5.1 N vs. 9.1 N, p = 0.04), when switching to robot-assisted surgery.

CONCLUSION

The crossover effects in technical skills between laparoscopic and robot-assisted surgery are highly depended on the prior experience with laparoscopic surgery. Where experts can alternate between approaches without impairment of technical skills, novices and intermediates should be aware of decay in efficiency of movement and tissue handling skills that could impact patient safety. Therefore, additional simulation training is advised to prevent from undesired events.

Transferability of robotic console skills by early robotic surgeons: a multi-platform crossover trial of simulation training

Robotic surgical training is undergoing a period of transition now that new robotic operating platforms are entering clinical practice. As this occurs, training will need to be adapted to include strategies to train across various consoles. These new consoles differ in multiple ways, with some new vendors using flat screen open source 3D enhanced vision with glasses and differences in design will require surgeons to learn new skills. This process has parallels with aviation credentialling across different aircraft described as type rating. This study was designed to test the hypothesis that technical robotic console operating skills are transferrable across different robotic operating platforms. Ten participants sequentially completed four Mimic®(Surgical Science) simulation exercises on two different robotic operating platforms (DaVinci®, Intuitive Surgical and HUGO™ RAS, Medtronic). Ethical approval and informed consent were obtained for this study. Groups were balanced for key demographics including previous robotic simulator experience. Data for simulation metrics and time to proficiency were collected for each attempt at the simulated exercise and analysed. Qualitative feedback on multi-platform learning was sought via unstructured interviews and a questionnaire. Participants were divided into two groups of 5. Group 1 completed the simulation exercises on console A first then repeated these exercises on console B. Group 2 completed the simulated exercises on console B first then repeated these exercises on console A. Group 1 candidates adapted quicker to the second console and Group 2 candidates reached proficiency faster on the first console. Participants were slower on the second attempt of the final exercise regardless of their allocated group. Quality and efficiency metrics and risk and safety metrics were equivalent across consoles. The data from this investigation suggests that console operating skills are transferrable across different platforms. Overall risk and safety metrics are within acceptable limits regardless of the order of progression of console indicating that training can safely occur across multiple consoles contemporaneously. This data has implications for the design of training and certification as new platforms progress to market and supports a proficiency-based approach.

A resident robotic curriculum utilizing self-selection and a web-based feedback tool

To describe an obstetrics and gynecology residency robotic curriculum, facilitated by a web-based feedback and case-tracking tool, allowing for self-selection into advanced training. Phase I (Basic) was required for all residents and included online training modules, online assessment, and robotic bedside assistant dry lab. Phase II (Advanced) was elective console training. Before live surgery, 10 simulation drills completed to proficiency were required. A web-based tool was used for surgical feedback and case-tracking. Online assessments, drill reports, objective GEARS assessments, subjective feedback, and case-logs were reviewed (7/2018-6/2019). A satisfaction survey was reviewed. Twenty four residents completed Phase I training and 10 completed Phase II. To reach simulation proficiency, residents spent a median of 4.1 h performing required simulation drills (median of 10 (3, 26) attempts per drill) before live surgery. 128 post-surgical feedback entries were completed after performance as bedside assistant (75%, n = 96) and console surgeon (5.5%, n = 7). The most common procedure was hysterectomy 111/193 (58%). Resident console surgeons performed portions of 32 cases with a mean console time of 34.6 ± 19.5 min. Mean GEARS score 20.6 ± 3.7 (n = 28). Mean non-technical feedback results: communication (4.2 ± 0.8, n = 61), workload management (3.9 ± 0.9, n = 54), team skills (4.3 ± 0.8, n = 60). Residents completing > 50% of case assessed as "apprentice" 38.5% or "competent" 23% (n = 13). After curriculum change, 100% of surveyed attendings considered residents prepared for live surgical training, vs 17% (n = 6) prior to curriculum change [survey response rate 27/44 (61%)]. Attendings and residents were satisfied with curriculum; 95% and recommended continued use 90% (n = 19).This two-phase robotic curriculum allows residents to self-select into advanced training, alleviating many challenges of graduated robotic training.

Virtual reality simulation does not correlate with overall trainee robot-assisted laparoscopic hysterectomy performance

STUDY OBJECTIVE

While both simulator practice and intraoperative performance serve to inform surgical trainee training, the skill transfer from simulation to the intraoperative setting remains unclear. This study evaluates the correlation between trainee performance on virtual reality (VR) simulation and (1) overall intraoperative performance during robotic-assisted laparoscopic hysterectomy (RALH) procedures, and (2) suturing performance during vaginal cuff closure portion of the case.

DESIGN

Retrospective subgroup analysis of randomized controlled trial SETTING: Academic hospital PATIENTS: Patients with RALH (n=29) INTERVENTIONS: Gynecological trainees (n=21) performed simulation tasks using the da Vinci Skills Simulator on the day of surgery, prior to performing RALH. Attending surgeons assessed participants' intraoperative performance using Global Evaluative Assessment of Robotic Skills (GEARS). Performance of the vaginal cuff closure step was subsequently assessed using GEARS scoring of anonymized videos. Spearman's correlation was used to quantify the relationship between simulation and intraoperative performances.

MEASUREMENTS AND MAIN RESULTS

Trainees achieved a median intraoperative GEARS score of 18.5/30 (IQR:17-22) and a median total simulator score of 84.4/100 (IQR:78.1-87.5). More advanced residents exhibited worse overall simulator performance (median score 86.6/100 compared to 78.8/100, p=0.03) and similar intraoperative GEARS scores during overall RALH and vaginal cuff closure compared to less experienced trainees. Total simulation performance score was negatively correlated with GEARS Bimanual Dexterity (rho=-0.46, p=0.02) and Force Sensitivity subscores (rho=-0.39, p=0.05). There was no correlation between total GEARS intraoperative vaginal cuff closure scores and overall simulation performances; however, total Tubes simulation score was correlated with higher GEARS Force Sensitivity subscore (rho=0.73, p=0.048).

CONCLUSIONS

In this study, there was limited correlation between simulation score metrics and trainees' overall intraoperative performance. Furthermore, we identified that GEARS scores could not distinguish between similar trainee skill levels. These findings underscore the need to develop intraoperative assessment tools that can better discriminate different but similar skill levels.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Validation of a Novel Simulation-Based Test in Robot-Assisted Radical Prostatectomy

Purpose: To investigate validity evidence for a simulator-based test in robot-assisted radical prostatectomy (RARP). Materials and Methods: The test consisted of three modules on the RobotiX Mentor VR-simulator: Bladder Neck Dissection, Neurovascular Bundle Dissection, and Ureterovesical Anastomosis. Validity evidence was investigated by using Messick's framework by including doctors with different RARP experience: novices (who had assisted for RARP), intermediates (robotic surgeons, but not RARP surgeons), or experienced (RARP surgeons). The simulator metrics were analyzed, and Cronbach's alpha and generalizability theory were used to explore reliability. Intergroup comparisons were done with mixed-model, repeated measurement analysis of variance and the correlation between the number of robotic procedures and the mean test score were examined. A pass/fail score was established by using the contrasting groups' method. Results: Ten novices, 11 intermediates, and 6 experienced RARP surgeons were included. Six metrics could discriminate between groups and showed acceptable internal consistency reliability, Cronbach's alpha = 0.49, p < 0.001. Test-retest reliability was 0.75, 0.85, and 0.90 for one, two, and three repetitions of tests, respectively. Six metrics were combined into a simulator score that could discriminate between all three groups, p = 0.002, p < 0.001, and p = 0.029 for novices vs intermediates, novices vs experienced, and intermediates vs experienced, respectively. Total number of robotic operations and the mean score of the three repetitions were significantly correlated, Pearson's r = 0.74, p < 0.001. Conclusion: This study provides validity evidence for a simulator-based test in RARP. We determined a pass/fail level that can be used to ensure competency before proceeding to supervised clinical training.

Evaluating Non-operative Robotic Skills in Colorectal Surgical Training

BACKGROUND

Robotic-assisted surgery has become a common platform for performing colorectal procedures. Educators must determine how best to teach and train residents to use the technology safely. There is a paucity of literature on how non-operative skills are being taught and integrated into colorectal training. Herein we describe the implementation and assessment of a pilot simulation-based Robotic Colorectal Surgery Non-Technical Skills curriculum.

MATERIALS AND METHODS

Since 2017 six colon and rectal surgery residents participated in two scenarios: pelvic bleeding and CO₂ embolism. The scenarios were administered in a simulated operating room twice during the academic year (fall and spring), and audio-video recorded. In addition to self-assessment, videos were evaluated by faculty utilizing the validated Interpersonal and Cognitive Assessment for Robotic Surgery system. To understand the role of scenario difficulty with respect to perceived cognitive workload and performance residents completed a NASA-Task Load Index assessment form.

RESULTS

Between the fall and spring sessions residents significantly improved in intraoperative leadership skills for both the CO₂ embolism and bleeding scenarios, and decision-making and situational awareness for the embolism case. Assessment between resident (self) and expert (faculty) did not correlate (P < 0.05) for either scenario during the fall session. A correlation for both scenarios was appreciated following the spring session revealing resident non-technical skills improved over time. Other than for physical demand, NASA-Task Load Index scores were similar for both scenarios.

CONCLUSIONS

We were able to successfully develop and implement a pilot Robotic Colorectal Surgery Non-Technical Skills curriculum in a risk-free simulated environment. Non-technical skill curriculums should be considered for both training and assessment in robotic surgery.

How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

Introduction

A surgeon's technical skills are an important factor in delivering optimal patient care. Most existing methods to estimate technical skills remain subjective and resource intensive. Robotic-assisted surgery (RAS) provides a unique opportunity to develop objective metrics using key elements of intraoperative surgeon behavior which can be captured unobtrusively, such as instrument positions and button presses. Recent studies have shown that objective metrics based on these data (referred to as objective performance indicators [OPIs]) correlate to select clinical outcomes during robotic-assisted radical prostatectomy. However, the current OPIs remain difficult to interpret directly and, therefore, to use within structured feedback to improve surgical efficiencies.

Methods

We analyzed kinematic and event data from da Vinci surgical systems (Intuitive Surgical, Inc., Sunnyvale, CA, USA) to calculate values that can summarize the use of robotic instruments, referred to as OPIs. These indicators were mapped to broader technical skill categories of established training protocols. A data-driven approach was then applied to further sub-select OPIs that distinguish skill for each technical skill category within each training task. This subset of OPIs was used to build a set of logistic regression classifiers that predict the probability of expertise in that skill to identify targeted improvement and practice. The final, proposed feedback using OPIs was based on the coefficients of the logistic regression model to highlight specific actions that can be taken to improve.

Results

We determine that for the majority of skills, only a small subset of OPIs (2-10) are required to achieve the highest model accuracies (80-95%) for estimating technical skills within clinical-like tasks on a porcine model. The majority of the skill models have similar accuracy as models predicting overall expertise for a task (80-98%). Skill models can divide a prediction into interpretable categories for simpler, targeted feedback.

Conclusion

We define and validate a methodology to create interpretable metrics for key technical skills during clinical-like tasks when performing RAS. Using this framework for evaluating technical skills, we believe that surgical trainees can better understand both what can be improved and how to improve.

Novel virtual reality based training system for fine motor skills: Towards developing a robotic surgery training system

BACKGROUND

Training surgeons to use surgical robots are becoming part of surgical training curricula. We propose a novel method of training fine-motor skills such as Microscopic Selection Task (MST) for robot-assisted surgery using virtual reality (VR) with objective quantification of performance. We also introduce vibrotactile feedback (VTFB) to study its impact on training performance.

METHODS

We use a VR-based environment to perform MST with varying degrees of difficulties. Using a well-known human-computer interaction paradigm and incorporating VTFB, we quantify the performance: speed, precision and accuracy.

RESULTS

MST with VTFB showed statistically significant improvement in performance metrics leading to faster completion of MST with higher precision and accuracy compared to that without VTFB.

DISCUSSION

The addition of VTFB to VR-based training for robot-assisted surgeries may improve performance outcomes in real robotic surgery. VTFB, along with proposed performance metrics, can be used in training curricula for robot-assisted surgeries.

Training novice robot surgeons: Proctoring provides same results as simulator-generated guidance

To understand the influence of proctored guidance versus simulator generated guidance (SGG) on the acquisition dexterity skills in novice surgeons learning RAS (robot assisted surgery). Prospective non-blinded 3-arm randomised controlled trial (RTC). Exclusion criteria: previous experience in RAS or robotic surgery simulation. The participants were assigned to three different intervention groups and received a different form of guidance: (1) proctored guidance, (2) simulator generated guidance, (3) no guidance, during training on virtual reality (VR) simulator. All participants were asked to complete multiple questionnaires. The training was the same in all groups with the exception of the intervention part. Catharina Hospital Eindhoven, The Netherlands. A total of 70 Dutch medical students, PhD-students, and surgical residents were included in the study. The participants were randomly assigned to one of the three groups. Overall, all the participants showed a significant improvement in their dexterity skills after the training. There was no significant difference in the improvement of surgical skills between the three different intervention groups. The proctored guidance group reported a higher participant satisfaction compared to the simulator-generated guidance group, which could indicate a higher motivation to continue the training. This study showed that novice surgeons. Significantly increase their dexterity skills in RAS after a short time of practicing on simulator. The lack of difference in results between the intervention groups could indicate there is a limited impact of "human proctoring" on dexterity skills during surgical simulation training. Since there is no difference between the intervention groups the exposure alone of novice surgeons to the robotic surgery simulator could possibly be sufficient to achieve a significant improvement of dexterity skills during the initial steps of RAS learning.

A Prospective, Observational, Multicentre Study Concerning Nontechnical Skills in Robot-assisted Radical Cystectomy Versus Open Radical Cystectomy

Introduction and hypotheses

valuation of surgical skills, both technical and nontechnical, is possible through observations and video analysis. Besides technical failures, adverse outcomes in surgery can also be related to hampered communication, moderate teamwork, lack of leadership, and loss of situational awareness. Even though some surgeons are convinced about nontechnical skills being an important part of their professionalisation, there is paucity of data about a possible relationship between nontechnical skills and surgical outcome. In robot-assisted surgery, the surgeon sits behind the console and is at a remote position from the surgical field and team, making communication more important than in open surgery and conventional laparoscopy. A lack of structured research makes it difficult to assess the value of the different analysis methods for nontechnical skills, particularly in robot-assisted surgery. Our hypothesis includes the following: (1) introduction of robot-assisted surgery leads to an initial decay in nontechnical skills behaviour during the learning curve of the team, (2) nontechnical skills behaviour is more explicitly expressed in experienced robot-assisted surgery teams than in experienced open surgery teams, and (3) introduction of robot-assisted surgery leads to the development of different forms of nontechnical skills behaviour compared with open surgery.

Design

This study is a prospective, observational, multicentre, nonrandomised, case-control study including bladder cancer patients undergoing either an open radical cystectomy or a robot-assisted radical cystectomy at the Catharina Hospital Eindhoven, the Netherlands, or at the Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital Amsterdam. All patients are eligible for inclusion; there are no exclusion criteria. The Catharina Hospital Eindhoven, the Netherlands, performs on average 35 radical cystectomies a year. The Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital Amsterdam, performs on average 100 radical cystectomies a year.

Protocol overview

The choice of treatment is at the discretion of the patient and the surgeon. Patient results will be obtained prospectively. Pathology results as well as complications occurring within 90 d following surgery will be registered. Surgical complications will be registered according to the Clavien-Dindo system.

Measurements

Nontechnical skills will be observed using five different methods: (1) NOTSS: Nontechnical Skills for Surgeons; (2) Oxford NOTECHS II: a modified theatre team nontechnical skills scoring system; (3) OTAS: Observational Teamwork Assessment for Surgery; (4) Interpersonal and Cognitive Assessment for Robotic Surgery (ICARS): evaluation of nontechnical skills in robotic surgery; and (5) analysis of human factors. Technical skills in robot-assisted radical cystectomy will be analysed using two different methods: (1) GEARS: Global Evaluative Assessment of Robotic Skill and (2) GERT: Generic Error Rating Tool.

Safety criteria and reporting

Formal ethical approval has been provided by Medical research Ethics Committees United (MEC-U), The Netherlands (reference number W19.048). We hope to present the results of this study to the scientific community at conferences and in peer-reviewed journals.

Statistical analysis

Frequency statistics will be calculated for patient demographical data, and a Shapiro-Wilk test with p > 0.05 will be used to define normal distribution. Univariate analysis will be conducted to test for statistically significant differences in observation scores between open radical cystectomy and robot-assisted radical cystectomy cohorts across all variables, using independent sample t tests and Mann-Whitney U testing, as appropriate. A variable-selection strategy will be used to create multivariate models. Binary logistic regression will be conducted to calculate odds ratios and 95% confidence intervals for significant predictors on univariate analysis and clinically relevant covariates. Statistical significance is set at p < 0.05 based on a two-tailed comparison.

Summary

This study uses a structured approach to the analysis of nontechnical skills using extracorporeal videos of both open radical cystectomy and robot-assisted radical cystectomy surgeries, in order to obtain detailed data on nontechnical skills during open and minimally invasive surgeries. The results of this study could possibly be used to develop team-training programmes, specifically for the introduction of the surgical robot in relation to changes in nontechnical skills. Additional analysis of technical skills using the intracorporeal footage of the surgical robot will be used to elucidate the role of surgical skills and surgical events in nontechnical skills.

Artificial intelligence and robotics: a combination that is changing the operating room

PURPOSE

The aim of the current narrative review was to summarize the available evidence in the literature on artificial intelligence (AI) methods that have been applied during robotic surgery.

METHODS

A narrative review of the literature was performed on MEDLINE/Pubmed and Scopus database on the topics of artificial intelligence, autonomous surgery, machine learning, robotic surgery, and surgical navigation, focusing on articles published between January 2015 and June 2019. All available evidences were analyzed and summarized herein after an interactive peer-review process of the panel.

LITERATURE REVIEW

The preliminary results of the implementation of AI in clinical setting are encouraging. By providing a readout of the full telemetry and a sophisticated viewing console, robot-assisted surgery can be used to study and refine the application of AI in surgical practice. Machine learning approaches strengthen the feedback regarding surgical skills acquisition, efficiency of the surgical process, surgical guidance and prediction of postoperative outcomes. Tension-sensors on the robotic arms and the integration of augmented reality methods can help enhance the surgical experience and monitor organ movements.

CONCLUSIONS

The use of AI in robotic surgery is expected to have a significant impact on future surgical training as well as enhance the surgical experience during a procedure. Both aim to realize precision surgery and thus to increase the quality of the surgical care. Implementation of AI in master-slave robotic surgery may allow for the careful, step-by-step consideration of autonomous robotic surgery.

Linking surgical skills to postoperative outcomes: a Delphi study on the robot-assisted radical prostatectomy

OBJECTIVE

To develop an assessment instrument for the evaluation of surgical videos to elucidate the association between surgical skills and postoperative outcomes after a robot-assisted radical prostatectomy (RARP).

DESIGN

A Delphi study consisting of two consecutive online surveys and a consensus group meeting.

SETTING

Urology departments of general, teaching and university hospitals in the Netherlands.

PARTICIPANTS

All Dutch urologists with a specialization in RARP.

RESULTS

Of 18 invited experts, 12 (67%) participated in the first online survey. In the second round, 9 of the 18 invited experts participated (50%). The Delphi meeting was attended by 5 of the 18 (27%) invited experts. The panel identified seven surgical steps with a possible association to postoperative outcomes. The experts also expected an association between adverse postoperative outcomes and the frequency of camera removals, the number of stitches placed, the amount of bleeding, and the extent of coagulation. These factors were incorporated into an assessment instrument.

CONCLUSIONS

Experts in the field of RARP achieved consensus on 7 surgical steps and 4 aspects of the RARP procedure that may be related to adverse postoperative outcomes. The resulting assessment instrument will be tested in future research to determine its validity.

Understanding and Assessing Nontechnical Skills in Robotic Urological Surgery: A Systematic Review and Synthesis of the Validity Evidence

OBJECTIVE

Robotic urological surgery (RUS) has seen widespread adoption across institutions in the last decade. To match this rapid growth, it is imperative to develop a structured RUS curriculum that addresses both technical and nontechnical competencies. Emerging evidence has shown that nontechnical skills form a critical component of RUS training. The purpose of this review is to examine the validity evidence of available nontechnical skills assessment tools in RUS.

METHODS

A literature search of MEDLINE, EMBASE, and PsycINFO was conducted to identify primary articles using nontechnical skills assessment tools in RUS. Messick's validity framework and the Medical Education Research Study Quality Instrument were utilized to evaluate the quality of the validity evidence of the abstracted articles.

RESULTS

Of the 566 articles identified, 12 used nontechnical skills assessment tools in RUS. The metrics used ranged from self-assessment using global rating scales, to objective measures such as electroencephalography. The setting of these evaluations ranged from immersive and virtual reality-based simulators to live surgery.

CONCLUSIONS

Limited effort has been made to develop nontechnical skills assessment tools in RUS. Recently, there has been a shift from subjective to objective measures of nontechnical performance, as well as the development of assessments specific to RUS. However, the validity evidence supporting these nontechnical assessments is limited at this time, including their relationship to technical skills, and their impact on surgical outcomes.

Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy

OBJECTIVES

To evaluate automated performance metrics (APMs) and clinical data of experts and super-experts for four cardinal steps of robot-assisted radical prostatectomy (RARP): bladder neck dissection; pedicle dissection; prostate apex dissection; and vesico-urethral anastomosis.

SUBJECTS AND METHODS

We captured APMs (motion tracking and system events data) and synchronized surgical video during RARP. APMs were compared between two experience levels: experts (100-750 cases) and super-experts (2100-3500 cases). Clinical outcomes (peri-operative, oncological and functional) were then compared between the two groups. APMs and outcomes were analysed for 125 RARPs using multi-level mixed-effect modelling.

RESULTS

For the four cardinal steps selected, super-experts showed differences in select APMs compared with experts (P < 0.05). Despite similar PSA and Gleason scores, super-experts outperformed experts clinically with regard to peri-operative outcomes, with a greater lymph node yield of 22.6 vs 14.9 nodes, respectively (P < 0.01), less blood loss (125 vs 130 mL, respectively; P < 0.01), and fewer readmissions at 30 days (1% vs 13%, respectively; P = 0.02). A similar but nonsignificant trend was seen for oncological and functional outcomes, with super-experts having a lower rate of biochemical recurrence compared with experts (5% vs 15%, respectively; P = 0.13) and a higher continence rate at 3 months (36% vs 18%, respectively; P = 0.14).

CONCLUSION

We found that experts and super-experts differed significantly in select APMs for the four cardinal steps of RARP, indicating that surgeons do continue to improve in performance even after achieving expertise. We hope ultimately to identify associations between APMs and clinical outcomes to tailor interventions to surgeons and optimize patient outcomes.