Validation of a Novel Virtual Reality Robotic Simulator

Validity of task-specific metrics for assessment in perineal proctectomy

BACKGROUND

Perineal proctectomy is a complex procedure that requires advanced skills. Currently, there are no simulators for training in this procedure. As part of our objective of developing a virtual reality simulator, our goal was to develop and validate task-specific metrics for the assessment of performance for this procedure. We conducted a three-phase study to establish task-specific metrics, obtain expert consensus on the appropriateness of the developed metrics, and establish the discriminant validity of the developed metrics.

METHODS

In phase I, we utilized hierarchical task analysis to formulate the metrics. In phase II, a survey involving expert colorectal surgeons determined the significance of the developed metrics. Phase III was aimed at establishing the discriminant validity for novices (PGY1-3) and experts (PGY4-5 and faculty). They performed a perineal proctectomy on a rectal prolapse model. Video recordings were independently assessed by two raters using global ratings and task-specific metrics for the procedure. Total scores for both metrics were computed and analyzed using the Kruskal-Wallis test. A Mann-Whitney U test with Benjamini-Hochberg correction was used to evaluate between-group differences. Spearman's rank correlation coefficient was computed to assess the correlation between global and task-specific scores.

RESULTS

In phase II, a total of 23 colorectal surgeons were recruited and consensus was obtained on all the task-specific metrics. In phase III, participants (n = 22) included novices (n = 15) and experts (n = 7). There was a strong positive correlation between the global and task-specific scores (rs = 0.86; P < 0.001). Significant between-group differences were detected for both global (χ2 = 15.38; P < 0.001; df = 2) and task-specific (χ2 = 11.38; P = 0.003; df = 2) scores.

CONCLUSIONS

Using a biotissue rectal prolapse model, this study documented high IRR and significant discriminant validity evidence in support of video-based assessment using task-specific metrics.

Comparison of Attempts Needed for Veterinary Students to Reach Proficiency in a Basic and Advanced Robotic Simulator Task

Surgical training of students is one of the most difficult aspects of veterinary medical education. Competing interests of house officers, owners' wishes, and ethical concerns have led to increasing use of inanimate models for basic surgical skills training. Robotic simulators have benefits for psychomotor training but have not been previously investigated in veterinary medicine. Understanding the necessary practice time for new training devices is important when determining their potential value. The aim of this study was to compare the number of attempts needed for veterinary students to reach proficiency in both a basic and advanced robotic simulator task, and to assess the predictive nature of performance variables. Each student performed a basic and advanced tasks on the Mimic dV-Trainer™ until proficiency was reached. Students required a median of eight attempts (95% CI = 7-8, range: 6-11) to reach proficiency for the basic task versus 22 attempts (95% CI = 20-26, range: 11-62) for the advanced task. The median time required to complete training for the basic and advanced task was 13.5 minutes (range: 8-24 minutes) and 26.5 minutes (range: 11-82 minutes) respectively. The difference in task attempts supports the training protocol and confirms proficiency can be attained in a short period of time. The number of attempts to reach proficiency correlated with specific performance variables that can be used by educators to aid in training students on a robotic simulator. Continued investigations on robotic simulators should be performed to investigate their use in improving psychomotor skills in veterinary students.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Recent Advances in Surgical Simulation For Resident Education

PURPOSE OF REVIEW

Surgical simulation has become a cornerstone for the training of surgical residents, especially for urology residents. Urology as a specialty bolsters a diverse range of procedures requiring a variety of technical skills ranging from open and robotic surgery to endoscopic procedures. While hands-on supervised training on patients still remains the foundation of residency training and education, it may not be sufficient to achieve proficiency for graduation even if case minimums are achieved. It has been well-established that simulation-based education (SBE) can supplement residency training and achieve the required proficiency benchmarks.

RECENT FINDINGS

Low-fidelity modules, such as benchtop suture kits or laparoscopic boxes, can establish a strong basic skills foundation. Eventually, residents progress to high-fidelity models to refine application of technical skills and improve operative performance. Human cadavers and animal models remain the gold standard for procedural SBE. Recently, given the well-recognized financial and ethical costs associated with cadaveric and animal models, residency programs have shifted their investments toward virtual and more immersive simulations. Urology as a field has pushed the boundaries of SBE and has reached a level where unexplored modalities, e.g., 3D printing, augmented reality, and polymer casting, are widely utilized for surgical training as well as preparation for challenging cases at both the residents, attending and team training level.

Augmented and virtual reality in spine surgery

Augmented Reality (AR) and Virtual Reality (VR) have developed unprecedentedly in recent years, providing interesting opportunities for medical applications. Their integration into clinical assessment, surgical workflow, and training has shown tremendous potential to improve daily life activity in spine surgery. The paper explores the utilization of VR and AR in spine surgery, with their applications, benefits, challenges, and forthcoming prospects.

Best practice considerations on the assessment of robotic assisted surgical systems: results from an international consensus expert panel

BACKGROUND

Health technology assessments (HTAs) of robotic assisted surgery (RAS) face several challenges in assessing the value of robotic surgical platforms. As a result of using different assessment methods, previous HTAs have reached different conclusions when evaluating RAS. While the number of available systems and surgical procedures is rapidly growing, existing frameworks for assessing MedTech provide a starting point, but specific considerations are needed for HTAs of RAS to ensure consistent results. This work aimed to discuss different approaches and produce guidance on evaluating RAS.

METHODS

A consensus conference research methodology was adopted. A panel of 14 experts was assembled with international experience and representing relevant stakeholders: clinicians, health economists, HTA practitioners, policy makers, and industry. A review of previous HTAs was performed and seven key themes were extracted from the literature for consideration. Over five meetings, the panel discussed the key themes and formulated consensus statements.

RESULTS

A total of ninety-eight previous HTAs were identified from twenty-five total countries. The seven key themes were evidence inclusion and exclusion, patient- and clinician-reported outcomes, the learning curve, allocation of costs, appropriate time horizons, economic analysis methods, and robotic ecosystem/wider benefits.

CONCLUSIONS

Robotic surgical platforms are tools, not therapies. Their value varies according to context and should be considered across therapeutic areas and stakeholders. The principles set out in this paper should help HTA bodies at all levels to evaluate RAS. This work may serve as a case study for rapidly developing areas in MedTech that require particular consideration for HTAs.

Physiological Metrics of Surgical Difficulty and Multi-Task Requirement during Robotic Surgery Skills

Previous studies in robotic-assisted surgery (RAS) have studied cognitive workload by modulating surgical task difficulty, and many of these studies have relied on self-reported workload measurements. However, contributors to and their effects on cognitive workload are complex and may not be sufficiently summarized by changes in task difficulty alone. This study aims to understand how multi-task requirement contributes to the prediction of cognitive load in RAS under different task difficulties. Multimodal physiological signals (EEG, eye-tracking, HRV) were collected as university students performed simulated RAS tasks consisting of two types of surgical task difficulty under three different multi-task requirement levels. EEG spectral analysis was sensitive enough to distinguish the degree of cognitive workload under both surgical conditions (surgical task difficulty/multi-task requirement). In addition, eye-tracking measurements showed differences under both conditions, but significant differences of HRV were observed in only multi-task requirement conditions. Multimodal-based neural network models have achieved up to 79% accuracy for both surgical conditions.

Objective metrics for hand-sewn bowel anastomoses can differentiate novice from expert surgeons

BACKGROUND

Assessing performance automatically in a virtual reality trainer or from recorded videos is advantageous but needs validated objective metrics. The purpose of this study is to obtain expert consensus and validate task-specific metrics developed for assessing performance in double-layered end-to-end anastomosis.

MATERIALS AND METHODS

Subjects were recruited into expert (PGY 4-5, colorectal surgery residents, and attendings) and novice (PGY 1-3) groups. Weighted average scores of experts for each metric item, completion time, and the total scores computed using global and task-specific metrics were computed for assessment.

RESULTS

A total of 43 expert surgeons rated our task-specific metric items with weighted averages ranging from 3.33 to 4.5 on a 5-point Likert scale. A total of 20 subjects (10 novices and 10 experts) participated in validation study. The novice group completed the task significantly more slowly than the experienced group (37.67 ± 7.09 vs 25.47 ± 7.82 min, p = 0.001). In addition, both the global rating scale (23.47 ± 4.28 vs 28.3 ± 3.85, p = 0.016) and the task-specific metrics showed a significant difference in performance between the two groups (38.77 ± 2.83 vs 42.58 ± 4.56 p = 0.027) following partial least-squares (PLS) regression. Furthermore, PLS regression showed that only two metric items (Stay suture tension and Tool handling) could reliably differentiate the performance between the groups (20.41 ± 2.42 vs 24.28 ± 4.09 vs, p = 0.037).

CONCLUSIONS

Our study shows that our task-specific metrics have significant discriminant validity and can be used to evaluate the technical skills for this procedure.

dV-Trainer vs. da Vinci Simulator: Comparison of Virtual Reality Platforms for Robotic Surgery

BACKGROUND

A virtual reality (VR) curriculum performed on the da Vinci Simulation System (DVSS) was previously shown to be effective in training fellows. The dV-Trainer is a separate platform with similar features to the da Vinci console, but its efficacy and utility versus the DVSS simulator are not well known.

MATERIALS AND METHODS

A mastery-based VR curriculum was completed by surgical fellows on the DVSS (2014-2016) and on the dV-Trainer (2016-2018) at a large academic center. Pre-test/post-test scores were used to evaluate performance between the two groups. Data was collected prospectively.

RESULTS

Forty-six fellows enrolled in the curriculum: surgical oncology (n=31), hepatobiliary (n=5), head/neck (n=4), endocrine (n=2), cardiothoracic (n=2), gynecology (n=1) and transplant surgery (n=1). Twenty-four used the DVSS and twenty-two used the dV-Trainer. Compared to the DVSS, the dV-Trainer was associated with lower scores on 2 of 3 VR modules in the pre-test (P=0.027, P<0.001, respectively) and post-test (P=0.021, P<0.001, respectively). Fellows in the dV-Trainer era scored lower on inanimate drills as well. Average VR curriculum score was lower on the dV-Trainer (71.3% vs 83.34%, P<0.001). dV-Trainer users spent more time completing the pre-test and post-test; however, overall simulator time to complete the curriculum was not significantly different (297 vs 231 minutes, P=0.142). Both groups showed improvement in scores after completion of the VR curriculum.

CONCLUSIONS

The dV-Trainer simulator allows for more usability outside the operating room to complete VR modules; however, the DVSS simulator group outperformed the dV-Trainer group on the post-test.

Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery

BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome's point of view is clearly indicated by the available surgical error statistics.

Teaching and learning robotic surgery at the dual console: a video-based qualitative analysis

Robotic-assisted surgery (RAS) involves training processes and challenges that differ from open or laparoscopic surgery, particularly regarding the possibilities of observation and embodied guidance. The video recording and the dual-console system creates a potential opportunity for participation. Our research, conducted within the department of visceral surgery of a big Swiss, public, academic hospital, uses a methodology based on the co-analysis of video recordings with surgeons in self-confrontation interviews, to investigate the teaching activity of the lead surgeon supervising a surgeon in training at the dual console. Three short sequences have been selected for the paper. Our analysis highlights the skills-in-construction of the surgeon in training regarding communication with the operating team, fluency of working with three hands, and awareness of the whole operating site. It also shows the divergent necessities of enabling verbalization for professional training, while ensuring a quiet and efficient environment for medical performance. To balance these requirements, we argue that dedicated briefing and debriefing sessions may be particularly effective; we also suggest that the self-confrontation video technique may be valuable to support the verbalization on both the mentor’s and the trainee’s side during such debriefing, and to enhance the mentor’s reflexivity regarding didactic choices.

Validation of a Novel Simulation-Based Test in Robot-Assisted Radical Prostatectomy

Purpose: To investigate validity evidence for a simulator-based test in robot-assisted radical prostatectomy (RARP). Materials and Methods: The test consisted of three modules on the RobotiX Mentor VR-simulator: Bladder Neck Dissection, Neurovascular Bundle Dissection, and Ureterovesical Anastomosis. Validity evidence was investigated by using Messick's framework by including doctors with different RARP experience: novices (who had assisted for RARP), intermediates (robotic surgeons, but not RARP surgeons), or experienced (RARP surgeons). The simulator metrics were analyzed, and Cronbach's alpha and generalizability theory were used to explore reliability. Intergroup comparisons were done with mixed-model, repeated measurement analysis of variance and the correlation between the number of robotic procedures and the mean test score were examined. A pass/fail score was established by using the contrasting groups' method. Results: Ten novices, 11 intermediates, and 6 experienced RARP surgeons were included. Six metrics could discriminate between groups and showed acceptable internal consistency reliability, Cronbach's alpha = 0.49, p < 0.001. Test-retest reliability was 0.75, 0.85, and 0.90 for one, two, and three repetitions of tests, respectively. Six metrics were combined into a simulator score that could discriminate between all three groups, p = 0.002, p < 0.001, and p = 0.029 for novices vs intermediates, novices vs experienced, and intermediates vs experienced, respectively. Total number of robotic operations and the mean score of the three repetitions were significantly correlated, Pearson's r = 0.74, p < 0.001. Conclusion: This study provides validity evidence for a simulator-based test in RARP. We determined a pass/fail level that can be used to ensure competency before proceeding to supervised clinical training.

Robotic Surgery Training: Current Trends and Future Directions

The use of robotic surgery in urology has grown exponentially in the past 2 decades, but robotic surgery training has lagged behind. Most graduating residents report a lack of comfort independently performing common robotic urologic surgeries, despite an abundance of available resources. There is a general consensus on the key components of a comprehensive robotics curriculum, and well-validated tools have been developed to assess trainee competency. However, no single curriculum has emerged as the gold standard on which individual programs can build their own robotics curricula.

Simulation Training in Urology: State of the Art and Future Directions

There has been a major shift from the old paradigm of 'see one, do one, teach one' in medical training due in large part to resident work-hour restrictions and required oversight in the operating room. In response to this, advancements in technology have allowed for the introduction of more objective measures to assess the skill competency and proficiency of surgical trainees. Patient safety and trainee well-being are important drivers for this new model, and so surgical training programs are adopting simulation into their curriculum. Urology is uniquely positioned at the forefront of new emerging technologies in surgery, because of the field's commitment to safe and efficient minimally invasive surgery and endourological procedures. Due to these technically challenging procedures, urological training must incorporate these educational technologies to allow for objective skills assessment, skills transfer, and ultimately providing optimal patient care with the production of proficient and competent urological trainees.

Cost, training and simulation models for robotic-assisted surgery in pediatric urology

INTRODUCTION

Laparoscopic procedures in pediatric urology have been shown to be safe and effective over the last number of years. Coupled with this is the technological trend to provide minimally invasive options for even the most complex pediatric patients. Whilst robotic platforms continue to try to demonstrate superior patient outcomes in adults with mixed results, the utilization of robotic platforms for pediatric urology is increasing.

METHODS

A review of the current literature was undertaken to assess the evidence for training models and cost-effectiveness of robotic-assisted pediatric urology.

CONCLUSIONS

A growing body of evidence in this field has demonstrated that robotic platforms are safe and effective in children and can provide additional reconstructive benefits due to motion scaling, magnification, stereoscopic views, instrument dexterity and tremor reduction. The main drawbacks remain the financial implications associated with this platform through purchase, maintenance, and disposable costs. This review addresses some of the addresses issues pertaining to cost, training and simulation for robotic-assisted surgery in pediatric urology.

Comparison of effective teaching methods to achieve skill acquisition using a robotic virtual reality simulator: Expert proctoring versus an educational video versus independent training

BACKGROUND

To compare the superiority of teaching methods for acquiring a proficient level of surgical skill in a robotic surgery-naïve individual using a robotic virtual reality simulator.

METHODS

This study employed a prospective, randomized study design to assess student's learning curve. We divided 45 subjects into 3 groups: those with expert proctoring (group I), those who watched only an educational video (group II), and those with independent training (group III; n = 15 per group). The task used in this study was the Tube 2 and it imitates a vesicourethral anastomosis in robotic prostatectomy. The effects were analyzed by the time to end the task after overcoming the learning curve which is determined by several performance parameters.

RESULTS

The number of task repetitions required to reach the learning curve plateau was 45, 42, and 37 repetitions in groups I, II, and III, which means that there was continuous improvement in performing the task after 40 repetitions only in groups I and II. The mean time for completing the task during the stabilization period was significantly different between group I and group III and group II and group III, which means that the independent training method was inferior to the other methods (group I vs. group II vs. group III: 187.38 vs. 187.07 vs. 253.47 seconds, P < .001).

CONCLUSIONS

This study's findings showed that an educational video can be as beneficial as expert proctoring, which implies that the development of a standardized educational video would be worthwhile.

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

BACKGROUND

The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.

METHODS

Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module 'Guided Vaginal Cuff Closure' six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach's alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups' method.

RESULTS

The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach's alpha). The experienced surgeons' mean composite score for all six repetitions were significantly better than the novice surgeons' (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).

CONCLUSION

Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

Predictive Validation of a Robotic Virtual Reality Simulator: The Tube 3 module for Practicing Vesicourethral Anastomosis in Robot-Assisted Radical Prostatectomy

OBJECTIVE

To predict actual performance in real surgery when vesicourethral anastomosis (VUA) is performed in patients after Tube 3 module training of robot-naive surgeons.

METHODS

Forty-five patients were enrolled and divided into 3 groups according to chronological trends (each containing 15 patients). Three robot-naive surgeons in a single center completed VUA in robot-assisted radical prostatectomy (RARP) following robotic virtual reality simulator (RVRS) training. The practicing tool used in robotic virtual reality simulator was Tube 3, which was invented for the dV-Trainer that imitates a VUA in RARP. The effects of performance were investigated by analyzing the number of repetitions and the time required to complete the task until achieving the predetermined proficiency level.

RESULTS

The targeted time (predetermined proficiency level) for completing tasks of Tube 3 and the number of required task repetitions to achieve the proficiency level were 283.1 s and 36 times, respectively, whereas in actual VUA procedures, the number of required attempts was 24, with an average time of 14.9 minutes. The mean time for completing VUA in real surgery significantly decreased with serial cases among all surgeons (1-15 vs 16-30 vs 31-45 cases, P <.001), as well as comparisons between groups (P <.001).

CONCLUSION

The Tube 3 module can represent a valuable educational tool for procedure-specific robotic training by bridging the gap between safe acquisition of surgical skills and effective performance during actual VUA in RARP.

Head-to-Head Comparison of Three Virtual-Reality Robotic Surgery Simulators

Background and Objectives:

There are several different commercially available virtual-reality robotic simulators, but very little comparative data. We compared the face and content validity of 3 robotic surgery simulators and their pricing and availability.

Methods:

Fifteen participants completed one task on each of the following: dV-Trainer (dVT; Mimic Technologies, Inc., Seattle, Washington, USA), da Vinci Skills Simulator (dVSS; Intuitive Surgical Inc., Sunnyvale, California, USA), and RobotiX Mentor (RM; 3D Systems, Rock Hill, South Carolina, USA). Participants completed previously validated face and content validity questionnaires and a demographics questionnaire. Statistical analysis was then performed on the scores.

Results:

Participants had a mean age of 29.6 (range, 25–41) years. Most were surgical trainees, having performed a mean of 8.6 robotic primary surgeries. For face validity, ANOVA showed a significant difference favoring the dVSS over the dVT (P = .001), and no significant difference between the RM, dVSS, and dVT. Content validity revealed similar results, with a significant difference between the dVSS and dVT (P = .021), a trend toward a difference between the RM and dVT (P = .092), and no difference between the dVSS and RM (P = .99).

Conclusion:

All simulators demonstrated evidence of face and content validity, with significantly higher scores for the dVSS; it is also the least costly ($80,000 for the simulator), although it is frequently unavailable because of intra-operative use. The dVT and RM have similar face and content validity, are slightly more expensive, and are readily available.

Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

OBJECTIVES

To systematically review and synthesise the validity evidence supporting intraoperative and simulation-based assessments of technical skill in urological robot-assisted surgery (RAS), and make evidence-based recommendations for the implementation of these assessments in urological training.

MATERIALS AND METHODS

A literature search of the Medline, PsycINFO and Embase databases was performed. Articles using technical skill and simulation-based assessments in RAS were abstracted. Only studies involving urology trainees or faculty were included in the final analysis.

RESULTS

Multiple tools for the assessment of technical robotic skill have been published, with mixed sources of validity evidence to support their use. These evaluations have been used in both the ex vivo and in vivo settings. Performance evaluations range from global rating scales to psychometrics, and assessments are carried out through automation, expert analysts, and crowdsourcing.

CONCLUSION

There have been rapid expansions in approaches to RAS technical skills assessment, both in simulated and clinical settings. Alternative approaches to assessment in RAS, such as crowdsourcing and psychometrics, remain under investigation. Evidence to support the use of these metrics in high-stakes decisions is likely insufficient at present.

Systematic review of measurement tools to assess surgeons' intraoperative cognitive workload

BACKGROUND

Surgeons in the operating theatre deal constantly with high-demand tasks that require simultaneous processing of a large amount of information. In certain situations, high cognitive load occurs, which may impact negatively on a surgeon's performance. This systematic review aims to provide a comprehensive understanding of the different methods used to assess surgeons' cognitive load, and a critique of the reliability and validity of current assessment metrics.

METHODS

A search strategy encompassing MEDLINE, Embase, Web of Science, PsycINFO, ACM Digital Library, IEEE Xplore, PROSPERO and the Cochrane database was developed to identify peer-reviewed articles published from inception to November 2016. Quality was assessed by using the Medical Education Research Study Quality Instrument (MERSQI). A summary table was created to describe study design, setting, specialty, participants, cognitive load measures and MERSQI score.

RESULTS

Of 391 articles retrieved, 84 met the inclusion criteria, totalling 2053 unique participants. Most studies were carried out in a simulated setting (59 studies, 70 per cent). Sixty studies (71 per cent) used self-reporting methods, of which the NASA Task Load Index (NASA-TLX) was the most commonly applied tool (44 studies, 52 per cent). Heart rate variability analysis was the most used real-time method (11 studies, 13 per cent).

CONCLUSION

Self-report instruments are valuable when the aim is to assess the overall cognitive load in different surgical procedures and assess learning curves within competence-based surgical education. When the aim is to assess cognitive load related to specific operative stages, real-time tools should be used, as they allow capture of cognitive load fluctuation. A combination of both subjective and objective methods might provide optimal measurement of surgeons' cognition.

Training in Robotic Surgery-an Overview

PURPOSE OF REVIEW

There has been a rapid and widespread adoption of the robotic surgical system with a lag in the development of a comprehensive training and credentialing framework. A literature search on robotic surgical training techniques and benchmarks was conducted to provide an evidence-based road map for the development of a robotic surgical skills for the novice robotic surgeon.

RECENT FINDINGS

A structured training curriculum is suggested incorporating evidence-based training techniques and benchmarks for progress. This usually involves sequential progression from observation, case assisting, acquisition of basic robotic skills in the dry and wet lab setting along with achievement of individual and team-based non-technical skills, modular console training under supervision, and finally independent practice. Robotic surgical training must be based on demonstration of proficiency and safety in executing basic robotic skills and procedural tasks prior to independent practice.

Milestone assessment of minimally invasive surgery in Pediatric Urology fellowship programs

INTRODUCTION

Minimally invasive surgery has become an important aspect of Pediatric Urology fellowship training. In 2014, the Accreditation Council for Graduate Medical Education published the Pediatric Urology Milestone Project as a metric of fellow proficiency in multiple facets of training, including laparoscopic/robotic procedures.

OBJECTIVE

The present study assessed trends in minimally invasive surgery training and utilization of the Milestones among recent Pediatric Urology fellows.

STUDY DESIGN

Using an electronic survey instrument, Pediatric Urology fellowship program directors and fellows who completed their clinical year in 2015 were surveyed. Participants were queried regarding familiarity with the Milestone Project, utilization of the Milestones, robotic/laparoscopic case volume and training experience, and perceived competency with robotic/laparoscopic surgery at the start and end of the fellowship clinical year according to Milestone criteria. Responses were accepted between August and November 2015.

RESULTS

Surveys were distributed via e-mail to 35 fellows and 30 program directors. Sixteen fellows (46%) and 14 (47%) program directors responded. All fellows reported some robotic experience prior to fellowship, and 69% performed >50 robotic/laparoscopic surgeries during residency. Fellow robotic/laparoscopic case volume varied: three had 1-10 cases (19%), four had 11-20 cases (25%), and nine had >20 cases (56%). Supplementary or robotic training modalities included simulation (9), animal models (6), surgical videos (7), and courses (2). Comparison of beginning and end of fellowship robotic/laparoscopic Milestone assessment (Summary Fig.) revealed scores of <3 in (10) 62% of fellow self-assessments and 10 (75%) of program director assessments. End of training Milestone scores >4 were seen in 12 (75%) of fellow self-assessment and eight (57%) of program director assessments.

DISCUSSION

An improvement in robotic/laparoscopic Milestone scores by both fellow self-assessment and program director assessment was observed during the course of training; however, 43% of program directors rated their fellow below the graduation target of a Milestone score of 4.

CONCLUSION

The best ways to teach minimally invasive surgery in fellowship training must be critically considered.

Diffusion of Robotic Technology Into Urologic Practice has Led to Improved Resident Physician Robotic Skills

OBJECTIVE

To investigate whether propagation of robotic technology into urologic practice and training programs has improved baseline urology resident trainee robotic skills.

DESIGN

Questionnaires were completed by each urology resident trainee participating in a training course and asked about access to robotic simulation, robot experience, and console time. Baseline resident trainee scores on the Mimic Robotic Simulator (Mimic Technologies, Inc., Seattle, WA) from 27 participants of 2012 course were compared with the 2015 scores of 34 trainees on 4 standard Mimic exercises using Wilcoxon rank-sum tests. p = 0.05 or less were considered statistically significant.

PARTICIPANTS AND SETTING

Totally, 34 resident trainees from 17 programs in the Southeast Section of the American Urological Association participated in an annual 2-day robotic training course.

RESULTS

Overall score, economy of motion score, and time to complete exercise were all significantly better in the 2015 trainee group compared with the 2012 trainee group (p < 0.001) for the Peg Board 1, Camera Targeting 2, and Energy Dissection exercises. Overall scores for needle targeting improved between 2012 and 2015 (p = 0.04). Trainee access to a simulator was not associated with overall score on any of the 4 exercises in the 2015 group. In the 2015 group, actual robotic console time was associated with better overall scores in Camera Targeting 2 (p = 0.02) and Peg Board 1 (p = 0.04).

CONCLUSIONS

Baseline resident trainee performance on basic robotic simulator exercises has improved over the past 3 years irrespective of robotic simulator access or console time.

Effect of Spatial Cognitive Ability on Gain in Robot-Assisted Surgical Skills of Urological Surgeons

BACKGROUND

Although previous studies have demonstrated the needs for a spatial cognitive ability that can give an accurate understanding of the position, orientation, and size and form of the objects in endoscopic surgery, there has been no study on the relationship between the skills of robot-assisted surgery and spatial cognitive ability.

OBJECTIVE

To assess the effect of spatial cognitive ability on gain in robot-assisted surgical skills of urological surgeons.

MATERIALS AND METHODS

The robot-assisted surgery skills of 24 urological surgeons who had no previous experience with the Mimic dV-Trainer (MdVT) and had not been the main surgeon in robot-assisted surgery and 20 volunteer medical students who had no previous experience of the MdVT were assessed by using a program consisting of 4 kinds of tasks. Their performances were recorded using a built-in scoring algorithm. Their spatial cognitive abilities were also assessed using a mental rotation test.

RESULTS

Although there was a significant correlation between the spatial cognitive ability and a score of 2 for the more difficult tasks for student groups using the MdVT, there was no significant correlation between them for all tasks for groups of urological surgeons.

CONCLUSION

The results of the present study indicate that differences in spatial cognitive ability in urological surgeons have no effect on the gain in fundamental robot-assisted surgery skills whereas there was a significant correlation between the spatial cognitive ability and fundamental robot-assisted surgical skills in the volunteers.

The Surgical Skill of a Novice Trainee Manifests in Time-Consuming Exercises of a Virtual Simulator Rather Than a Quick-Finishing Counterpart: A Concurrent Validity Study Using an Urethrovesical Anastomosis Model

OBJECTIVE

The purpose of this study is to determine an optimal training curriculum using a robotic virtual simulator (RVS) that enables unexperienced trainees to perform a complex task in a hands-on setting.

PATIENTS AND METHODS

This study was conducted in 2 phases. In the RVS phase, 43 participants sequentially completed 12 exercises consistent with all primary exercises in the EndoWrist manipulation and advanced needle-driving category, until the overall score reached more than 80% by repeated practice. In the hands-on phase using a robotic surgical system, 10 randomly selected trainees performed 8 sutures once, simulating urethrovesical anastomosis, and the console time was recorded.

RESULTS

The median total time and total attempts for the RVS phase was 195.2 minutes and 54 times, respectively. The trainees were divided by median total time, and times to accomplish each RVS exercise were then compared between the early- and the late-completion groups; among 12 exercises trained, 6 exercises (prolonged course) requiring significantly more time in the late-completion group were identified. The prolonged course occupied 88.18% of the total time and 77.61% of the total attempts. For participants the in hands-on phase, a multiple linear regression model showed that the time to accomplish the prolonged course was a single independent predictor of the console time (R(2) = 0.524, B = 0.05; p = 0.018).

CONCLUSION

After establishment of a high standard cutoff score, the time spent for the prolonged course showed a significant association with console time in hands-on training simulating urethrovesical anastomosis, implying educational efficacy of training involving time-consuming exercise in performance of a complex task.

Learning Experiences in Robotic-Assisted Laparoscopic Surgery

With the use and adoption of computer-assisted laparoscopic technology gaining more prominence, important issues pertaining to the learning process are raised. Several modalities can be incorporated into a training program for robotic surgical development. The role and utility of various methods, including didactic instruction, virtual reality simulators, dry and wet laboratories, bedside assistance, mentoring, as well as proctorship, are still in the process of being assessed and validated. Integration of robotic training in residency and fellowship programs as well as the formation of a structured didactic robotic curriculum continues to be a challenge. Finally, methods to assess competency of training and the process for credentialing robotic surgeons still require further structuring and codification.

Concurrent and predictive validation of robotic simulator Tube 3 module

Purpose

We previously described a new procedure specific module (Tube 3) to allow the practice of vesicourethral anastomosis after robot-assisted radical prostatectomy. Herein, we report a predetermined proficiency level of Tube 3 and preliminary validation to explore whether this new module can lead to performance improvement in the da Vinci system.

Materials and Methods

Eight urology residents and three urology fellows performed the Tube 3 module 1 hour daily for 7 days. The learning curve was depicted through a scatterplot and the stable point was identified through the cumulative sum chart. Concurrent and predictive validations were performed with the da Vinci system. The mean time to complete the task and end product rating score between Tube 3 training group and no Tube 3 training group were compared.

Results

Concerning the learning curve, about 41 repetitions comprising about 5 hours were needed to achieve this stable point when the mean time to complete Tube of 384 seconds was set as a target. With regarding to the concurrent and predictive validation, there significant differences were evident in the mean time to complete 16 needle passages and the vesicourethral anastomosis and the end product rating score.

Conclusions

The virtual reality (VR) simulator can yield sufficient improvement in technical performance in Tube 3 within 5 hours. The acquired proficiency can be transferable to the vesicourethral anastomosis using the da Vinci system.

A Systematic Review of Virtual Reality Simulators for Robot-assisted Surgery

CONTEXT

No single large published randomized controlled trial (RCT) has confirmed the efficacy of virtual simulators in the acquisition of skills to the standard required for safe clinical robotic surgery. This remains the main obstacle for the adoption of these virtual simulators in surgical residency curricula.

OBJECTIVE

To evaluate the level of evidence in published studies on the efficacy of training on virtual simulators for robotic surgery.

EVIDENCE ACQUISITION

In April 2015 a literature search was conducted on PubMed, Web of Science, Scopus, Cochrane Library, the Clinical Trials Database (US) and the Meta Register of Controlled Trials. All publications were scrutinized for relevance to the review and for assessment of the levels of evidence provided using the classification developed by the Oxford Centre for Evidence-Based Medicine.

EVIDENCE SYNTHESIS

The publications included in the review consisted of one RCT and 28 cohort studies on validity, and seven RCTs and two cohort studies on skills transfer from virtual simulators to robot-assisted surgery. Simulators were rated good for realism (face validity) and for usefulness as a training tool (content validity). However, the studies included used various simulation training methodologies, limiting the assessment of construct validity. The review confirms the absence of any consensus on which tasks and metrics are the most effective for the da Vinci Skills Simulator and dV-Trainer, the most widely investigated systems. Although there is consensus for the RoSS simulator, this is based on only two studies on construct validity involving four exercises. One study on initial evaluation of an augmented reality module for partial nephrectomy using the dV-Trainer reported high correlation (r=0.8) between in vivo porcine nephrectomy and a virtual renorrhaphy task according to the overall Global Evaluation Assessment of Robotic Surgery (GEARS) score. In one RCT on skills transfer, the experimental group outperformed the control group, with a significant difference in overall GEARS score (p=0.012) during performance of urethrovesical anastomosis on an inanimate model. Only one study included assessment of a surgical procedure on real patients: subjects trained on a virtual simulator outperformed the control group following traditional training. However, besides the small numbers, this study was not randomized.

CONCLUSIONS

There is an urgent need for a large, well-designed, preferably multicenter RCT to study the efficacy of virtual simulation for acquisition competence in and safe execution of clinical robotic-assisted surgery.

PATIENT SUMMARY

We reviewed the literature on virtual simulators for robot-assisted surgery. Validity studies used various simulation training methodologies. It is not clear which exercises and metrics are the most effective in distinguishing different levels of experience on the da Vinci robot. There is no reported evidence of skills transfer from simulation to clinical surgery on real patients.

Current state of virtual reality simulation in robotic surgery training: a review

BACKGROUND

Worldwide, the annual number of robotic surgical procedures continues to increase. Robotic surgical skills are unique from those used in either open or laparoscopic surgery. The acquisition of a basic robotic surgical skill set may be best accomplished in the simulation laboratory. We sought to review the current literature pertaining to the use of virtual reality (VR) simulation in the acquisition of robotic surgical skills on the da Vinci Surgical System.

MATERIALS AND METHODS

A PubMed search was conducted between December 2014 and January 2015 utilizing the following keywords: virtual reality, robotic surgery, da Vinci, da Vinci skills simulator, SimSurgery Educational Platform, Mimic dV-Trainer, and Robotic Surgery Simulator. Articles were included if they were published between 2007 and 2015, utilized VR simulation for the da Vinci Surgical System, and utilized a commercially available VR platform.

RESULTS

The initial search criteria returned 227 published articles. After all inclusion and exclusion criteria were applied, a total of 47 peer-reviewed manuscripts were included in the final review.

CONCLUSIONS

There are many benefits to utilizing VR simulation for robotic skills acquisition. Four commercially available simulators have been demonstrated to be capable of assessing robotic skill. Three of the four simulators demonstrate the ability of a VR training curriculum to improve basic robotic skills, with proficiency-based training being the most effective training style. The skills obtained on a VR training curriculum are comparable with those obtained on dry laboratory simulation. The future of VR simulation includes utilization in assessment for re-credentialing purposes, advanced procedural-based training, and as a warm-up tool prior to surgery.

An effective repetitive training schedule to achieve skill proficiency using a novel robotic virtual reality simulator

PURPOSE

A robotic virtual reality simulator (Mimic dV-Trainer) can be a useful training method for the da Vinci surgical system. Herein, we investigate several repetitive training schedules and determine which is the most effective.

METHODS

A total of 30 medical students were enrolled and were divided into 3 groups according to the training schedule. Group 1 performed the task 1 hour daily for 4 consecutive days, group II performed the task on once per week for 1 hour for 4 consecutive weeks, and group III performed the task for 4 consecutive hours in 1 day. The effects of training were investigated by analyzing the number of repetitions and the time required to complete the "Tube 2" simulation task when the learning curve plateau was reached. The point at which participants reached a stable score was evaluated using the cumulative sum control graph.

RESULTS

The average time to complete the task at the learning curve plateau was 150.3 seconds in group I, 171.9 seconds in group II, and 188.5 seconds in group III. The number of task repetitions required to reach the learning curve plateau was 45 repetitions in group I, 36 repetitions in group II, and 39 repetitions in group III. Therefore, there was continuous improvement in the time required to perform the task after 40 repetitions in group I only. There was a significant correlation between improvement in each trial interval and attempt, and the correlation coefficient (0.924) in group I was higher than that in group II (0.899) and group III (0.838).

CONCLUSION

Daily 1-hour practice sessions performed for 4 consecutive days resulted in the best final score, continuous score improvement, and effective training while minimizing fatigue. This repetition schedule can be used for effectively training novices in future.

Learning tools and simulation in robotic surgery: state of the art

Robotic surgery has emerged as a new technology over the last decade and has brought with it new challenges, particularly in terms of teaching and training. To overcome these challenges, robotic courses, virtual simulation, and dual consoles have been successfully introduced. In fact, there are several simulators currently on the market that have proven to be a valid option for training, especially for the novice trainee. Robotic courses have also found success around the world, allowing participants to implement robotic programs at their institution, typically with the help of a proctor. More recently, the dual console has enabled two surgeons to be operating at the same time. Having one experienced surgeon and one trainee each at his or her own console has made it an obvious choice for training. Although these methods have been successfully introduced, the data remain relatively scarce concerning their role in training. The aim of this article was to review the various methods and tools involved in the training of surgeons in robotic surgery.

Development of a standardised training curriculum for robotic surgery: a consensus statement from an international multidisciplinary group of experts

OBJECTIVES

To explore the views of experts about the development and validation of a robotic surgery training curriculum, and how this should be implemented.

MATERIALS AND METHODS

An international expert panel was invited to a structured session for discussion. The study was of a mixed design, including qualitative and quantitative components based on focus group interviews during the European Association of Urology (EAU) Robotic Urology Section (ERUS) (2012), EAU (2013) and ERUS (2013) meetings. After introduction to the aims, principles and current status of the curriculum development, group responses were elicited. After content analysis of recorded interviews generated themes were discussed at the second meeting, where consensus was achieved on each theme. This discussion also underwent content analysis, and was used to draft a curriculum proposal. At the third meeting, a quantitative questionnaire about this curriculum was disseminated to attendees to assess the level of agreement with the key points.

RESULTS

In all, 150 min (19 pages) of the focus group discussion was transcribed (21 316 words). Themes were agreed by two raters (median agreement κ 0.89) and they included: need for a training curriculum (inter-rater agreement κ 0.85); identification of learning needs (κ 0.83); development of the curriculum contents (κ 0.81); an overview of available curricula (κ 0.79); settings for robotic surgery training ((κ 0.89); assessment and training of trainers (κ 0.92); requirements for certification and patient safety (κ 0.83); and need for a universally standardised curriculum (κ 0.78). A training curriculum was proposed based on the above discussions.

CONCLUSION

This group proposes a multi-step curriculum for robotic training. Studies are in process to validate the effectiveness of the curriculum and to assess transfer of skills to the operating room.

[Objective surgery -- advanced robotic devices and simulators used for surgical skill assessment]

Robotic assistance became a leading trend in minimally invasive surgery, which is based on the global success of laparoscopic surgery. Manual laparoscopy requires advanced skills and capabilities, which is acquired through tedious learning procedure, while da Vinci type surgical systems offer intuitive control and advanced ergonomics. Nevertheless, in either case, the key issue is to be able to assess objectively the surgeons' skills and capabilities. Robotic devices offer radically new way to collect data during surgical procedures, opening the space for new ways of skill parameterization. This may be revolutionary in MIS training, given the new and objective surgical curriculum and examination methods. The article reviews currently developed skill assessment techniques for robotic surgery and simulators, thoroughly inspecting their validation procedure and utility. In the coming years, these methods will become the mainstream of Western surgical education.

Impact of laparoscopic experience on virtual robotic simulator dexterity

BACKGROUND:

Different skills are required for robotic surgery and laparoscopic surgery. We hypothesized that the laparoscopic experience would not affect the performance with the da Vinci^® system. A virtual robotic simulator was used to estimate the operator's robotic dexterity.

MATERIALS AND METHODS:

The performance of 11 surgical fellows with laparoscopic experience and 14 medical students were compared using the dV-trainer^®. Each subject completed three virtual endo-wrist modules (“Pick and Place,” “Peg Board,“ and “Match Board”). Performance was recorded using a built-in scoring algorithm.

RESULTS:

In the Peg Board module, the performance of surgical fellows was better in terms of the number of instrument collisions and number of drops (P < 0.05). However, no significant differences were found in the percentage scores of the three endo-wrist modules between the groups.

CONCLUSION:

Robotic dexterity was not significantly affected by laparoscopic experience in this study. Laparoscopic experience is not an important factor for learning robotic skills.

A review of training research and virtual reality simulators for the da Vinci surgical system

PHENOMENON: Virtual reality simulators are the subject of several recent studies of skills training for robot-assisted surgery. Yet no consensus exists regarding what a core skill set comprises or how to measure skill performance. Defining a core skill set and relevant metrics would help surgical educators evaluate different simulators.

APPROACH

This review draws from published research to propose a core technical skill set for using the da Vinci surgeon console. Publications on three commercial simulators were used to evaluate the simulators' content addressing these skills and associated metrics.

FINDINGS

An analysis of published research suggests that a core technical skill set for operating the surgeon console includes bimanual wristed manipulation, camera control, master clutching to manage hand position, use of third instrument arm, activating energy sources, appropriate depth perception, and awareness of forces applied by instruments. Validity studies of three commercial virtual reality simulators for robot-assisted surgery suggest that all three have comparable content and metrics. However, none have comprehensive content and metrics for all core skills. INSIGHTS: Virtual reality simulation remains a promising tool to support skill training for robot-assisted surgery, yet existing commercial simulator content is inadequate for performing and assessing a comprehensive basic skill set. The results of this evaluation help identify opportunities and challenges that exist for future developments in virtual reality simulation for robot-assisted surgery. Specifically, the inclusion of educational experts in the development cycle alongside clinical and technological experts is recommended.