Development and Validation of an Objective Scoring Tool for Robot-Assisted Radical Prostatectomy: Prostatectomy Assessment and Competency Evaluation

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review

BACKGROUND AND OBJECTIVE

Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways.

METHODS

We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification.

KEY FINDINGS AND LIMITATIONS

A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study.

CONCLUSIONS AND CLINICAL IMPLICATIONS

Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact.

PATIENT SUMMARY

Simulation-based training and programs in which progression is based on proficiency represent the new standard of quality for achieving surgical proficiency in urology. Studies have demonstrated the educational impact of these approaches. However, there are still no standardized training pathways for several urology procedures.

Implementation and Validation of an Automated, Longitudinal Robotic Surgical Evaluation and Feedback Program at a High-volume Center and Impact on Training

Background

Surgical education lacks a standardized, proficiency-based approach to evaluation and feedback.

Objective

To assess the implementation and reception (ie, feasibility) of an automated, standardized, longitudinal surgical skill assessment and feedback system, and identify baseline trainee (resident and fellow) characteristics associated with achieving proficiency in robotic surgery while learning robotic-assisted laparoscopic prostatectomy.

Design setting and participants

A quality improvement study assessing a pilot of a surgical experience tracking program was conducted over 1 yr. Participants were six fellows, eight residents, and nine attending surgeons at a tertiary cancer center.

Intervention

Trainees underwent baseline self-assessment. After each surgery, an evaluation was completed independently by the trainee and attending surgeons. Performance was rated on a five-point anchored Likert scale (trainees were considered "proficient" when attending surgeons' rating was ≥4). Technical skills were assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and Prostatectomy Assessment and Competency Evaluation (PACE).

Outcome measurements and statistical analysis

Program success and utility were assessed by evaluating completion rates, evaluation completion times, and concordance rates between attending and trainee surgeons, and exit surveys. Baseline characteristics were assessed to determine associations with achieving proficiency.

Results and limitations

Completion rates for trainees and attending surgeons were 72% and 77%, respectively. Fellows performed more steps/cases than residents (median [interquartile range]: 5 [3-7] and 3 [2-4], respectively; p < 0.01). Prior completion of robotics or laparoscopic skill courses and surgical experience measures were associated with achieving proficiency in multiple surgical steps and GEARS domains. Interclass correlation coefficients on individual components were 0.27-0.47 on GEARS domains.

Conclusions

An automated surgical experience tracker with structured, longitudinal evaluation and feedback can be implemented with good participation and minimal participant time commitment, and can guide curricular development in a proficiency-based education program by identifying modifiable factors associated with proficiency, individualizing education, and identifying improvement areas within the education program.

Patient summary

An automated, standardized, longitudinal surgical skill assessment and feedback system can be implemented successfully in surgical education settings and used to inform education plans and predict trainee proficiency.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Navigating Now and Next: Recent Advances and Future Horizons in Robotic Radical Prostatectomy

Robotic-assisted radical prostatectomy (RARP) has become the leading approach for radical prostatectomy driven by innovations aimed at improving functional and oncological outcomes. The initial advancement in this field was transperitoneal multiport robotics, which has since undergone numerous technical modifications. These enhancements include the development of extraperitoneal, transperineal, and transvesical approaches to radical prostatectomy, greatly facilitated by the advent of the Single Port (SP) robot. This review offers a comprehensive analysis of these evolving techniques and their impact on RARP. Additionally, we explore the transformative role of artificial intelligence (AI) in digitizing robotic prostatectomy. AI advancements, particularly in automated surgical video analysis using computer vision technology, are unprecedented in their scope. These developments hold the potential to revolutionize surgeon feedback and assessment and transform surgical documentation, and they could lay the groundwork for real-time AI decision support during surgical procedures in the future. Furthermore, we discuss future robotic platforms and their potential to further enhance the field of RARP. Overall, the field of minimally invasive radical prostatectomy for prostate cancer has been an incubator of innovation over the last two decades. This review focuses on some recent developments in robotic prostatectomy, provides an overview of the next frontier in AI innovation during prostate cancer surgery, and highlights novel robotic platforms that may play an increasing role in prostate cancer surgery in the future.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Structured Resident Training in Robotic Surgery: Recommendations of the Robotic Surgery Education Working Group

OBJECTIVE

A universal resident robotic surgery training pathway that maximizes proficiency and safety has not been defined by a consensus of surgical educators or by surgical societies. The objective of the Robotic Surgery Education Working Group was to develop a universal curriculum pathway and leverage digital tools to support resident education.

DESIGN

The two lead authors (JP and YN) contacted potential members of the Working Group. Members were selected based on their authorship of peer-review publications, their experience as minimally invasive and robotic surgeons, their reputations, and their ability to commit the time involved to work collaboratively and efficiently to reach consensus regarding best practices in robotic surgery education. The Group's approach was to reach 100% consensus to provide a transferable curriculum that could be applied to the vast majority of resident programs.

SETTING

Virtual and in-person meetings in the United States.

PARTICIPANTS

Eight surgeons (2 females and 6 males) from five academic medical institutions (700-1541 beds) and three community teaching hospitals (231-607 beds) in geographically diverse locations comprised the Working Group. They represented highly specialized general surgeons and educators in their mid-to-late careers. All members were experienced minimally invasive surgeons and had national reputations as robotic surgery educators.

RESULTS

The surgeons initially developed and agreed upon questions for each member to consider and respond to individually via email. Responses were collated and consolidated to present on an anonymized basis to the Group during an in-person day-long meeting. The surgeons self-facilitated and honed the agreed upon responses of the Group into a 5-level Robotic Surgery Curriculum Pathway, which each member agreed was relevant and expressed their convictions and experience.

CONCLUSIONS

The current needs for a universal robotic surgery training curriculum are validated objective and subjective measures of proficiency, access to simulation, and a digital platform that follows a resident from their first day of residency through training and their entire career. Refinement of current digital solutions and continued innovation guided by surgical educators is essential to build and maintain a scalable, multi-institutional supported curriculum.

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

BACKGROUND

Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery.

METHODS

A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB.

CONCLUSION

AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

3D printed patient-specific prostate cancer models to guide nerve-sparing robot-assisted radical prostatectomy: a systematic review

Precise knowledge of each patient's index cancer and surrounding anatomy is required for nerve-sparing robot-assisted radical prostatectomy (NS-RARP). Complementary to this, 3D printing has proven its utility in improving the visualisation of complex anatomy. This is the first systematic review to critically assess the potential of 3D printed patient-specific prostate cancer models in improving visualisation and the practice of NS-RARP. A literature search of PubMed and OVID Medline databases was performed using the terms "3D Printing", "Robot Assisted Radical Prostatectomy" and related index terms as per the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Eight articles were included; six were identified via database searches, to which a further two articles were located via a snowballing approach. Eight papers were identified for review. There were five prospective single centre studies, one case series, one technical report and one letter to the editor. Of these articles, five publications (62.5%) reported on the utility of 3D printed models for NS-RARP planning. Two publications (25%) utilised 3D printed prostate models for simulation and training, and two publications (25%) used the models for patient engagement. Despite the nascency of the field, 3D printed models are emerging in the uro-oncological literature as a useful tool in visualising complex anatomy. This has proven useful in NS-RARP for preoperative planning, simulation and patient engagement. However, best practice guidelines, the future regulatory landscape, and health economic considerations need to be addressed before this synergy of new technologies is ready for the mainstream.

What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature

A crucial element of any surgical training program is the ability to provide procedure-specific, objective, and reliable measures of performance. During robotic surgery, objective clinically relevant performance metrics (CRPMs) can provide tailored contextual feedback and correlate with clinical outcomes. This review aims to define CRPMs, assess their validity in robotic surgical training and compare CRPMs to existing measures of robotic performance. A systematic search of Medline and Embase databases was conducted in May 2022 following the PRISMA guidelines. The search terms included Clinically Relevant Performance Metrics (CRPMs) OR Clinically Relevant Outcome Measures (CROMs) AND robotic surgery. The study settings, speciality, operative context, study design, metric details, and validation status were extracted and analysed. The initial search yielded 116 citations, of which 6 were included. Citation searching identified 3 additional studies, resulting in 9 studies included in this review. Metrics were defined as CRPMs, CROMs, proficiency-based performance metrics and reference-procedure metrics which were developed using a modified Delphi methodology. All metrics underwent both contents and construct validation. Two studies found a strong correlation with GEARS but none correlated their metrics with patient outcome data. CRPMs are a validated and objective approach for assessing trainee proficiency. Evaluating CRPMs with other robotic-assessment tools will facilitate a multimodal metric evaluation approach to robotic surgery training. Further studies should assess the correlation with clinical outcomes. This review highlights there is significant scope for the development and validation of CRPMs to establish proficiency-based progression curricula that can be translated from a simulation setting into clinical practice.

Assessing the efficacy of dissection gestures in robotic surgery

Our group previously defined a dissection gesture classification system that deconstructs robotic tissue dissection into its most elemental yet meaningful movements. The purpose of this study was to expand upon this framework by adding an assessment of gesture efficacy (ineffective, effective, or erroneous) and analyze dissection patterns between groups of surgeons of varying experience. We defined three possible gesture efficacies as ineffective (no meaningful effect on the tissue), effective (intended effect on the tissue), and erroneous (unintended disruption of the tissue). Novices (0 prior robotic cases), intermediates (1-99 cases), and experts (≥ 100 cases) completed a robotic dissection task in a dry-lab training environment. Video recordings were reviewed to classify each gesture and determine its efficacy, then dissection patterns between groups were analyzed. 23 participants completed the task, with 9 novices, 8 intermediates with median caseload 60 (IQR 41-80), and 6 experts with median caseload 525 (IQR 413-900). For gesture selection, we found increasing experience associated with increasing proportion of overall dissection gestures (p = 0.009) and decreasing proportion of retraction gestures (p = 0.009). For gesture efficacy, novices performed the greatest proportion of ineffective gestures (9.8%, p < 0.001), intermediates commit the greatest proportion of erroneous gestures (26.8%, p < 0.001), and the three groups performed similar proportions of overall effective gestures, though experts performed the greatest proportion of effective retraction gestures (85.6%, p < 0.001). Between groups of experience, we found significant differences in gesture selection and gesture efficacy. These relationships may provide insight into further improving surgical training.

Development and Validation of an Objective Scoring Tool for Robot-Assisted Partial Nephrectomy: Scoring for Partial Nephrectomy

Objective: To develop a structured and objective scoring tool for assessment of robot-assisted partial nephrectomy (RAPN): Scoring for Partial Nephrectomy (SPaN). Materials and Methods: Content development: RAPN was deconstructed into 6 domains by a multi-institutional panel of 10 expert robotic surgeons. Performance on each domain was represented on a Likert scale of 1 to 5, with specific descriptions of anchors 1, 3, and 5. Content validation: The Delphi methodology was utilized to achieve consensus about the description of each anchor for each domain in terms of appropriateness of the skill assessed, objectiveness, clarity, and unambiguous wording. The content validity index (CVI) of ≥0.75 was set as cutoff for consensus. Reliability: 15 de-identified videos of RAPN were utilized to determine the inter-rater reliability using linearly weighted percent agreement, and Construct validation of SPaN was described in terms of median scores and odds ratios. Results: The expert panel reached consensus (CVI ≥0.75) after 2 rounds. Consensus was achieved for 36 (67%) statements in the first round and 18 (33%) after the second round. The final six-domain SPaN included Exposure of the kidney; Identification and dissection of the ureter and gonadal vessels; Dissection of the hilum; Tumor localization and exposure; Clamping and tumor resection; and Renorrhaphy. The linearly weighted percent agreement was >0.75 for all domains. There was no difference between median scores for any domain between attendings and trainees. Conclusion: Despite the lack of significant construct validity, SPaN is a structured, reliable, and procedure-specific tool that can objectively assesses technical proficiency for RAPN.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Training in robot-assisted surgery

PURPOSE OF REVIEW

The aim of this narrative review is to identify the currently available training programs in the field of robotic-assisted surgery.

RECENT FINDINGS

The lack of well trained robotic surgeons represents the main obstacle facing robotic surgery. This calls for development of structured robotic training programs in order to improve patient safety and outcomes. Structured curriculum should consist of theoretical training (e-learning, case observation), preclinical simulation-based training (virtual reality simulation, dry and wet lab), clinical modular training, and final evaluation. Tele-mentoring might reveal mandatory in future curricula development. To date, several training programs are currently available for robotic surgery. However, the majority lack of clinical modular training that consists of progressive, proficiency-based training through surgical steps with increasing levels of complexity. Moreover, many of these curricula lack validation.

SUMMARY

This recent literature analysis suggests that there is an urgent need to develop and validate new structured training curricula for robotic surgery. This allows to improve the skills of the surgeons and of their team and to prevent patients from being used as a training module optimizing their safety.

A Prospective, Observational, Multicentre Study Concerning Nontechnical Skills in Robot-assisted Radical Cystectomy Versus Open Radical Cystectomy

Introduction and hypotheses

valuation of surgical skills, both technical and nontechnical, is possible through observations and video analysis. Besides technical failures, adverse outcomes in surgery can also be related to hampered communication, moderate teamwork, lack of leadership, and loss of situational awareness. Even though some surgeons are convinced about nontechnical skills being an important part of their professionalisation, there is paucity of data about a possible relationship between nontechnical skills and surgical outcome. In robot-assisted surgery, the surgeon sits behind the console and is at a remote position from the surgical field and team, making communication more important than in open surgery and conventional laparoscopy. A lack of structured research makes it difficult to assess the value of the different analysis methods for nontechnical skills, particularly in robot-assisted surgery. Our hypothesis includes the following: (1) introduction of robot-assisted surgery leads to an initial decay in nontechnical skills behaviour during the learning curve of the team, (2) nontechnical skills behaviour is more explicitly expressed in experienced robot-assisted surgery teams than in experienced open surgery teams, and (3) introduction of robot-assisted surgery leads to the development of different forms of nontechnical skills behaviour compared with open surgery.

Design

This study is a prospective, observational, multicentre, nonrandomised, case-control study including bladder cancer patients undergoing either an open radical cystectomy or a robot-assisted radical cystectomy at the Catharina Hospital Eindhoven, the Netherlands, or at the Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital Amsterdam. All patients are eligible for inclusion; there are no exclusion criteria. The Catharina Hospital Eindhoven, the Netherlands, performs on average 35 radical cystectomies a year. The Netherlands Cancer Institute, Antoni van Leeuwenhoek Hospital Amsterdam, performs on average 100 radical cystectomies a year.

Protocol overview

The choice of treatment is at the discretion of the patient and the surgeon. Patient results will be obtained prospectively. Pathology results as well as complications occurring within 90 d following surgery will be registered. Surgical complications will be registered according to the Clavien-Dindo system.

Measurements

Nontechnical skills will be observed using five different methods: (1) NOTSS: Nontechnical Skills for Surgeons; (2) Oxford NOTECHS II: a modified theatre team nontechnical skills scoring system; (3) OTAS: Observational Teamwork Assessment for Surgery; (4) Interpersonal and Cognitive Assessment for Robotic Surgery (ICARS): evaluation of nontechnical skills in robotic surgery; and (5) analysis of human factors. Technical skills in robot-assisted radical cystectomy will be analysed using two different methods: (1) GEARS: Global Evaluative Assessment of Robotic Skill and (2) GERT: Generic Error Rating Tool.

Safety criteria and reporting

Formal ethical approval has been provided by Medical research Ethics Committees United (MEC-U), The Netherlands (reference number W19.048). We hope to present the results of this study to the scientific community at conferences and in peer-reviewed journals.

Statistical analysis

Frequency statistics will be calculated for patient demographical data, and a Shapiro-Wilk test with p > 0.05 will be used to define normal distribution. Univariate analysis will be conducted to test for statistically significant differences in observation scores between open radical cystectomy and robot-assisted radical cystectomy cohorts across all variables, using independent sample t tests and Mann-Whitney U testing, as appropriate. A variable-selection strategy will be used to create multivariate models. Binary logistic regression will be conducted to calculate odds ratios and 95% confidence intervals for significant predictors on univariate analysis and clinically relevant covariates. Statistical significance is set at p < 0.05 based on a two-tailed comparison.

Summary

This study uses a structured approach to the analysis of nontechnical skills using extracorporeal videos of both open radical cystectomy and robot-assisted radical cystectomy surgeries, in order to obtain detailed data on nontechnical skills during open and minimally invasive surgeries. The results of this study could possibly be used to develop team-training programmes, specifically for the introduction of the surgical robot in relation to changes in nontechnical skills. Additional analysis of technical skills using the intracorporeal footage of the surgical robot will be used to elucidate the role of surgical skills and surgical events in nontechnical skills.

An optimal prognostic model based on gene expression for clear cell renal cell carcinoma

Clear cell renal cell carcinoma (ccRCC) is the most prevalent type of RCC; however, prognostic prediction tools for ccRCC are scant. Developing mRNA or long non-coding RNA (lncRNA)-based risk assessment tools may improve the prognosis in patients with ccRCC. RNA-sequencing and prognostic data from patients with ccRCC were downloaded from The Cancer Genome Atlas and the European Bioinformatics Institute Array database at the National Center for Biotechnology Information. Differentially expressed (DE) RNAs (DERs) and prognostic DERs were screened between less favorable and favorable prognoses using the limma package in R 3.4.1, and analyzed using univariate and multivariate Cox regression analyses, respectively. Risk score models were constructed using optimal combinations of DEmRNAs and DElncRNAs identified using the Least Absolute Shrinkage And Selection Operator Cox regression model of the penalized package. Associations between risk score models and overall survival time were evaluated. Independent prognostic clinical factors were screened using univariate and multivariate Cox regression analyses, and nomogram models were constructed. Gene Ontology biological processes and Kyoto Encyclopedia of Genes and Genomes pathway enrichment analyses were conducted using the clusterProfiler package in R3.4.1. A total of 451 DERs were identified, including 404 mRNAs and 47 lncRNAs, between less favorable and favorable prognoses, and 269 DERs, including 233 mRNAs and 36 lncRNAs, were identified as independent prognostic factors. Optimal combinations including 10 DEmRNAs or 10 DElncRNAs were screened using four risk score models based on the status or expression levels of the 10 DEmRNAs or 10 DElncRNAs. The model based on the expression levels of the 10 DEmRNAs had the highest prognostic power. These prognostic DEmRNAs may be involved in biological processes associated with the inflammatory response, complement and coagulation cascades and neuroactive ligand-receptor interaction pathways. The present validated risk assessment tool based on the expression levels of these 10 DEmRNAs may help to identify patients with ccRCC at a high risk of mortality. These 10 DEmRNAs in optimal combinations may serve as prognostic biomarkers and help to elucidate the pathogenesis of ccRCC.

Analysis of the video motion tracking system "Kinovea" to assess surgical movements during robot-assisted radical prostatectomy

BACKGROUNDS

Robot-assisted surgery facilitated the possibility to evaluate the surgeon's skills by recording and evaluating the robot surgical images. The aim of this study was to investigate the possibility of using a computer programme (Kinovea) for objective assessment of surgical movements in previously recorded in existing robot-assisted radical prostatectomy (RARP) videos.

METHODS

Twelve entire RARP procedures were analysed by a trained researcher using the computer programme "Kinovea" to perform semi-automated assessment of surgical movements.

RESULTS

Data analysis showed Kinovea was on average able to automatically assess only 22% of the total surgical duration per video of the robot-assisted surgery. On average, it lasted 4 hours of continued monitoring by the researcher to assess one RARP using Kinovea.

CONCLUSION

Although we proved it is technically possible to use the Kinovea system in retrospective analysis of surgical movement in robot-assisted surgery, the acquired data do not give a comprehensive enough analysis of the video to be used in skills assessment.

A novel risk score model for stomach adenocarcinoma based on the expression levels of 10 genes

Stomach adenocarcinoma (STAD) accounts for 95% of cases of malignant gastric cancer, which is the third leading cause of cancer-associated mortality worldwide. The pathogenesis and effective diagnosis of STAD have become popular topics for research in the previous decade. In the present study, high-throughput RNA sequencing expression profiles and clinical data from patients with STAD were obtained from The Cancer Genome Atlas database and were used as a training dataset to screen differentially expressed genes (DEGs). Prognostic DEGs were identified using univariate Cox regression analysis and were further screened by the least absolute shrinkage and selection operator regularization regression algorithm. The resulting genes were used to construct a risk score model, the validation and effectiveness evaluation of which were performed on an independent dataset downloaded from the Gene Expression Omnibus database. Stratified and functional pathway (gene set enrichment) analyses were performed on groups with different estimated prognosis. A total of 92 genes significantly associated with STAD prognosis were obtained by univariate Cox regression analysis, and 10 prognosis-associated DEGs; hemoglobin b, chromosome 4 open reading frame 48, Dickkopf WNT signaling pathway inhibitor 1, coagulation factor V, serpin family E member 1, transmembrane protein 200A, NADPH oxidase organizer 1, C-X-C motif chemokine ligand 3, mannosidase endo-α-like and tripartite motif-containing 31; were selected for the development of the risk score model. The reliability of this prognostic method was verified using a validation set, and the results of multivariate Cox analysis indicated that the risk score may serve as an independent prognostic factor. In functional DEG analysis, eight Kyoto Encyclopedia of Genes and Genomes pathways were identified to be significantly associated with STAD risk factors. Thus, the 10-gene risk score model established in the present study was regarded as credible. This risk assessment tool may help identify patients with a high risk of STAD, and the proposed prognostic mRNAs may be useful in elucidating STAD pathogenesis.

Video review for measuring and improving skill in urological surgery

Interest is growing within the urological surgery community for objective assessments of technical skill. Surgical video review relies on the use of objective assessment tools to evaluate both global and procedure-specific skill. These evaluations provide structured feedback to surgeons with the aim of improving technique, which has been associated with patient outcomes. Currently, skill assessments can be performed by using expert peer-review, crowdsourcing or computer-based methods. Given the relationship between skill and patient outcomes, surgeons might be required in the future to provide empirical evidence of their technical skill for certification, employment, credentialing and quality improvement. Interventions such as coaching and skills workshops incorporating video review might help surgeons improve their skill, with the ultimate goal of improving patient outcomes.

Does YouTube include high-quality resources for training on laparoscopic and robotic radical prostatectomy?

PURPOSE

Our aim was to assess the educational quality of the YouTube video content related to laparoscopic and robotic radical prostatectomy (RP).

METHODS

An objective scoring tool named as Prostatectomy Assessment and Competency Evaluation (PACE) score was used to measure and quantify seven critical steps in RP including bladder drop, preparation of the prostate, bladder neck dissection, posterior/seminal vesicle dissection, neurovascular bundle preservation, apical dissection, and urethro-vesical anastomosis. A five-point scale was used for grading the seven steps, where a score of 1 and 5 represented the lowest and ideal performance, respectively. Additionally, descriptive statistics including the upload time, video length, view count, number of comments, likes, and dislikes were all recorded.

RESULTS

Of the 1688 videos (551 from laparoscopic RP, 567 from robotic RP, and 570 from robot-assisted RP), 226 videos were analyzed after excluding duplicate and irrelevant videos. Robotic/robot-assisted RP videos were found to be statistically longer than laparoscopic RP videos (p = 0.016). The PACE score of urethro-vesical anastomosis step in robotic RP videos was statistically higher than laparoscopic RP videos (p = 0.021). A weak but significant positive correlation between the video length and total PACE score (rho: 0.51; p = 0.04 for laparoscopic RP and rho: 0.43; p = 0.03 for robotic/robot-assisted RP) was found. A weak but positive correlation was also determined between number of likes and total PACE score (rho: 0.39; p = 0.02) for robotic/robot-assisted RP videos.

CONCLUSIONS

Although YouTube website includes high-quality videos for both laparoscopic and robotic/robot-assisted RP, there is no objective parameter to predict the educational quality of the videos.

A novel risk score system for assessment of ovarian cancer based on co-expression network analysis and expression level of five lncRNAs

Background

Ovarian cancer (OC) is the most deadly gynaecological cancer, contributing significantly to female cancer-related deaths worldwide. Improving the outlook for OC patients depends on the identification of more reliable prognostic biomarkers for early diagnosis and survival prediction. The various roles of long non-coding RNAs (lncRNAs) in OC have attracted increasing attention. This study aimed to identify a lncRNA-based signature for survival prediction in OC patients.

Methods

RNA expression data and clinical information from a large number of OC patients were downloaded from a public database. These data were regarded as a training set to construct a weighed gene co-expression network analysis (WGCNA) network, mine stable modules, and screen differentially expressed lncRNAs. The prognostic lncRNAs were screened using univariate Cox regression analysis and the optimal prognosis lncRNA combination was screened using a Cox-PH model. The finalised lncRNA combination was used to construct the risk score system, which was validated and assessed for effectiveness using other independent datasets. Further functional pathway enrichment was performed using gene set enrichment analysis (GSEA).

Results

A co-expression network was constructed and four stable modules with OC-related biological functions were obtained. A total of 19 lncRNAs significantly related to prognosis of ovarian cancer were obtained using univariate Cox regression analysis, and the 5 prognostic signature lncRNAs GAS5, HCP5, PART1, SNHG11, and SNHG5 were used to establish a risk assessment system. The reliability of the prognostic scoring system was further confirmed using validation sets, which indicated that the risk assessment system could be used as an independent prognostic factor. Pathway enrichment analysis revealed that the network modules related to the above five prognostic genes were significantly associated with cell local adhesion, cancer signaling pathways, JAK-STAT signalling, and endogenous cell receptor interaction.

Conclusions

The risk score system established in this study could provide a novel reliable method to identify individuals at high risk of OC. In addition, the five prognostic lncRNAs identified here are promising potential prognostic biomarkers that could help to elucidate the pathogenesis of OC.

Design and Validation of a Low-Cost, High-Fidelity Model for Urethrovesical Anastomosis in Radical Prostatectomy

OBJECTIVE

We sought to develop and validate a low-cost, high-fidelity robotic surgical model for the urethrovesical anastomosis component of the robot-assisted laparoscopic radical prostatectomy.

MATERIALS AND METHODS

A novel simulation model was constructed using a 3D-printed model of the male bony pelvis from CT scan data and silicone molds to recreate the soft tissue aspects. Using a da Vinci Si surgical robot, urology faculty and trainees performed simulated urethrovesical anastomosis. Each participant was given 12 minutes to complete the simulation. A survey established face validity, content validity, and acceptability. Simulation runs were evaluated by three blinded reviewers. The anastomosis was graded by two reviewers for suture placement accuracy and anastomosis quality. These factors were compared with robotic experience to establish construct validity.

RESULTS

Twenty participants took part in the initial validation of this model. Groups were defined as experts (surgical faculty), intermediate (fellows and chief residents), and novices (junior residents). Likert scores (1-5 scale, top score 5) examining face validity, content validity, and acceptability were 3.49 ± 0.43, 4.15 ± 0.23, and 4.02 ± 0.19, respectively. Construct validity was excellent based on the model's ability to stratify groups. All evaluated metrics were statistically different between the three levels of training. Total material cost was $2.50 per model.

CONCLUSIONS

We developed a novel low-cost robotic simulation of the urethrovesical anastomosis for robot-assisted radical prostatectomy. The model discerns robotic skill level across all levels of training and was found favorable by participants showing excellent face, content, and construct validities.

Linking surgical skills to postoperative outcomes: a Delphi study on the robot-assisted radical prostatectomy

OBJECTIVE

To develop an assessment instrument for the evaluation of surgical videos to elucidate the association between surgical skills and postoperative outcomes after a robot-assisted radical prostatectomy (RARP).

DESIGN

A Delphi study consisting of two consecutive online surveys and a consensus group meeting.

SETTING

Urology departments of general, teaching and university hospitals in the Netherlands.

PARTICIPANTS

All Dutch urologists with a specialization in RARP.

RESULTS

Of 18 invited experts, 12 (67%) participated in the first online survey. In the second round, 9 of the 18 invited experts participated (50%). The Delphi meeting was attended by 5 of the 18 (27%) invited experts. The panel identified seven surgical steps with a possible association to postoperative outcomes. The experts also expected an association between adverse postoperative outcomes and the frequency of camera removals, the number of stitches placed, the amount of bleeding, and the extent of coagulation. These factors were incorporated into an assessment instrument.

CONCLUSIONS

Experts in the field of RARP achieved consensus on 7 surgical steps and 4 aspects of the RARP procedure that may be related to adverse postoperative outcomes. The resulting assessment instrument will be tested in future research to determine its validity.

Understanding and Assessing Nontechnical Skills in Robotic Urological Surgery: A Systematic Review and Synthesis of the Validity Evidence

OBJECTIVE

Robotic urological surgery (RUS) has seen widespread adoption across institutions in the last decade. To match this rapid growth, it is imperative to develop a structured RUS curriculum that addresses both technical and nontechnical competencies. Emerging evidence has shown that nontechnical skills form a critical component of RUS training. The purpose of this review is to examine the validity evidence of available nontechnical skills assessment tools in RUS.

METHODS

A literature search of MEDLINE, EMBASE, and PsycINFO was conducted to identify primary articles using nontechnical skills assessment tools in RUS. Messick's validity framework and the Medical Education Research Study Quality Instrument were utilized to evaluate the quality of the validity evidence of the abstracted articles.

RESULTS

Of the 566 articles identified, 12 used nontechnical skills assessment tools in RUS. The metrics used ranged from self-assessment using global rating scales, to objective measures such as electroencephalography. The setting of these evaluations ranged from immersive and virtual reality-based simulators to live surgery.

CONCLUSIONS

Limited effort has been made to develop nontechnical skills assessment tools in RUS. Recently, there has been a shift from subjective to objective measures of nontechnical performance, as well as the development of assessments specific to RUS. However, the validity evidence supporting these nontechnical assessments is limited at this time, including their relationship to technical skills, and their impact on surgical outcomes.

Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy

OBJECTIVES

To evaluate automated performance metrics (APMs) and clinical data of experts and super-experts for four cardinal steps of robot-assisted radical prostatectomy (RARP): bladder neck dissection; pedicle dissection; prostate apex dissection; and vesico-urethral anastomosis.

SUBJECTS AND METHODS

We captured APMs (motion tracking and system events data) and synchronized surgical video during RARP. APMs were compared between two experience levels: experts (100-750 cases) and super-experts (2100-3500 cases). Clinical outcomes (peri-operative, oncological and functional) were then compared between the two groups. APMs and outcomes were analysed for 125 RARPs using multi-level mixed-effect modelling.

RESULTS

For the four cardinal steps selected, super-experts showed differences in select APMs compared with experts (P < 0.05). Despite similar PSA and Gleason scores, super-experts outperformed experts clinically with regard to peri-operative outcomes, with a greater lymph node yield of 22.6 vs 14.9 nodes, respectively (P < 0.01), less blood loss (125 vs 130 mL, respectively; P < 0.01), and fewer readmissions at 30 days (1% vs 13%, respectively; P = 0.02). A similar but nonsignificant trend was seen for oncological and functional outcomes, with super-experts having a lower rate of biochemical recurrence compared with experts (5% vs 15%, respectively; P = 0.13) and a higher continence rate at 3 months (36% vs 18%, respectively; P = 0.14).

CONCLUSION

We found that experts and super-experts differed significantly in select APMs for the four cardinal steps of RARP, indicating that surgeons do continue to improve in performance even after achieving expertise. We hope ultimately to identify associations between APMs and clinical outcomes to tailor interventions to surgeons and optimize patient outcomes.

Dynamic changes of brain functional states during surgical skill acquisition

There is lack of a standardized measure of technical proficiency and skill acquisition for robot-assisted surgery (RAS). Learning surgical skills, in addition to the interaction with the machine and the new surgical environment adds to the complexity of the learning process. Moreover, evaluation of surgeon performance in operating room is required to optimize patient safety. In this study, we investigated the dynamic changes of RAS trainee’s brain functional states by practice. We also developed brain functional state measurements to find the relationship between RAS skill acquisition (especially human-machine interaction skills) and reconfiguration of brain functional states. This relationship may help in providing trainees with helpful, structured feedback regarding skills requiring improvement and will help in tailoring training activities.

The Importance of Technical and Non-technical Skills in Robotic Surgery Training

We discuss the differences in cognitive (thinking) and other non-technical skills (NTS) in robotic surgery training compared to other approaches to surgery. Recognition of the importance of NTS and cognitive training will aid the development of robotic surgery curricula.

Development of a technical checklist for the assessment of suturing in robotic surgery

BACKGROUND

With the increased use of simulation for surgical training, there is a need for objective forms of assessment to evaluate trainees. The Global Evaluative Assessment of Robotic Skills (GEARS) is widely used for assessing skills in robotic surgery, but there are no recognised checklist scoring systems. This study aimed to develop a checklist for suturing in robotic surgery.

METHODS

A suturing checklist for needle driving and knot tying was constructed following evaluation of participants performing urethrovesical anastomoses. Key procedural steps were identified from expert videos, while assessing novice videos allowed identification of common technical errors. 22 novice and 13 expert videos were marked on needle driving, while 18 novices and 10 experts were assessed on knot tying. Validation of the finalised checklist was performed with the assessment of 39 separate novices by an expert surgeon and compared to GEARS scoring.

RESULTS

The internal consistency of the preliminary checklist was high (Cronbach's alpha = 0.870 for needle driving items; 0.736 for knot tying items), and after removal of poorly correlating items, the final checklist contained 23 steps. Both the needle driving and knot tying categories discriminated between novices and experts, p < 0.005. While the GEARS score demonstrated construct validity for needle driving, it could not significantly differentiate between novices and experts for knot tying, p = 0.286. The needle driving category significantly correlated with the corresponding GEARS scores (r_s = 0.613, p < 0.005), but the correlation for knot tying was insignificant (r_s = 0.296, p = 0.127). The pilot data indicates the checklist significantly correlated with the GEARS score (p < 0.005).

CONCLUSION

This study reports the development of a valid assessment tool for suturing in robotic surgery. Given that checklists are simple to use, there is significant scope for this checklist to be used in surgical training.

Use of Automated Performance Metrics to Measure Surgeon Performance during Robotic Vesicourethral Anastomosis and Methodical Development of a Training Tutorial

PURPOSE

We sought to develop and validate automated performance metrics to measure surgeon performance of vesicourethral anastomosis during robotic assisted radical prostatectomy. Furthermore, we sought to methodically develop a standardized training tutorial for robotic vesicourethral anastomosis.

MATERIALS AND METHODS

We captured automated performance metrics for motion tracking and system events data, and synchronized surgical video during robotic assisted radical prostatectomy. Nonautomated performance metrics were manually annotated by video review. Automated and nonautomated performance metrics were compared between experts with 100 or more console cases and novices with fewer than 100 cases. Needle driving gestures were classified and compared. We then applied task deconstruction, cognitive task analysis and Delphi methodology to develop a standardized robotic vesicourethral anastomosis tutorial.

RESULTS

We analyzed 70 vesicourethral anastomoses with a total of 1,745 stitches. For automated performance metrics experts outperformed novices in completion time (p <0.01), EndoWrist® articulation (p <0.03), instrument movement efficiency (p <0.02) and camera manipulation (p <0.01). For nonautomated performance metrics experts had more optimal needle to needle driver positioning, fewer needle driving attempts, a more optimal needle entry angle and less tissue trauma (each p <0.01). We identified 14 common robotic needle driving gestures. Random gestures were associated with lower efficiency (p <0.01), more attempts (p <0.04) and more trauma (p <0.01). The finalized tutorial contained 66 statements and figures. Consensus among 8 expert surgeons was achieved after 2 rounds, including among 58 (88%) after round 1 and 8 (12%) after round 2.

CONCLUSIONS

Automated performance metrics can distinguish surgeon expertise during vesicourethral anastomosis. The expert vesicourethral anastomosis technique was associated with more efficient movement and less tissue trauma. Standardizing robotic vesicourethral anastomosis and using a methodically developed tutorial may help improve robotic surgical training.

Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

OBJECTIVES

To systematically review and synthesise the validity evidence supporting intraoperative and simulation-based assessments of technical skill in urological robot-assisted surgery (RAS), and make evidence-based recommendations for the implementation of these assessments in urological training.

MATERIALS AND METHODS

A literature search of the Medline, PsycINFO and Embase databases was performed. Articles using technical skill and simulation-based assessments in RAS were abstracted. Only studies involving urology trainees or faculty were included in the final analysis.

RESULTS

Multiple tools for the assessment of technical robotic skill have been published, with mixed sources of validity evidence to support their use. These evaluations have been used in both the ex vivo and in vivo settings. Performance evaluations range from global rating scales to psychometrics, and assessments are carried out through automation, expert analysts, and crowdsourcing.

CONCLUSION

There have been rapid expansions in approaches to RAS technical skills assessment, both in simulated and clinical settings. Alternative approaches to assessment in RAS, such as crowdsourcing and psychometrics, remain under investigation. Evidence to support the use of these metrics in high-stakes decisions is likely insufficient at present.

Development and validation of surgical training tool: cystectomy assessment and surgical evaluation (CASE) for robot-assisted radical cystectomy for men

BACKGROUND

We aimed to develop a structured scoring tool: cystectomy assessment and surgical evaluation (CASE) that objectively measures and quantifies performance during robot-assisted radical cystectomy (RARC) for men.

METHODS

A multinational 10-surgeon expert panel collaborated towards development and validation of CASE. The critical steps of RARC in men were deconstructed into nine key domains, each assessed by five anchors. Content validation was done utilizing the Delphi methodology. Each anchor was assessed in terms of context, score concordance, and clarity. The content validity index (CVI) was calculated for each aspect. A CVI ≥ 0.75 represented consensus, and this statement was removed from the next round. This process was repeated until consensus was achieved for all statements. CASE was used to assess de-identified videos of RARC to determine reliability and construct validity. Linearly weighted percent agreement was used to assess inter-rater reliability (IRR). A logit model for odds ratio (OR) was used to assess construct validation.

RESULTS

The expert panel reached consensus on CASE after four rounds. The final eight domains of the CASE included: pelvic lymph node dissection, development of the peri-ureteral space, lateral pelvic space, anterior rectal space, control of the vascular pedicle, anterior vesical space, control of the dorsal venous complex, and apical dissection. IRR > 0.6 was achieved for all eight domains. Experts outperformed trainees across all domains.

CONCLUSION

We developed and validated a reliable structured, procedure-specific tool for objective evaluation of surgical performance during RARC. CASE may help differentiate novice from expert performances.

Utilizing Machine Learning and Automated Performance Metrics to Evaluate Robot-Assisted Radical Prostatectomy Performance and Predict Outcomes

PURPOSE

Surgical performance is critical for clinical outcomes. We present a novel machine learning (ML) method of processing automated performance metrics (APMs) to evaluate surgical performance and predict clinical outcomes after robot-assisted radical prostatectomy (RARP).

MATERIALS AND METHODS

We trained three ML algorithms utilizing APMs directly from robot system data (training material) and hospital length of stay (LOS; training label) (≤2 days and >2 days) from 78 RARP cases, and selected the algorithm with the best performance. The selected algorithm categorized the cases as "Predicted as expected LOS (pExp-LOS)" and "Predicted as extended LOS (pExt-LOS)." We compared postoperative outcomes of the two groups (Kruskal-Wallis/Fisher's exact tests). The algorithm then predicted individual clinical outcomes, which we compared with actual outcomes (Spearman's correlation/Fisher's exact tests). Finally, we identified five most relevant APMs adopted by the algorithm during predicting.

RESULTS

The "Random Forest-50" (RF-50) algorithm had the best performance, reaching 87.2% accuracy in predicting LOS (73 cases as "pExp-LOS" and 5 cases as "pExt-LOS"). The "pExp-LOS" cases outperformed the "pExt-LOS" cases in surgery time (3.7 hours vs 4.6 hours, p = 0.007), LOS (2 days vs 4 days, p = 0.02), and Foley duration (9 days vs 14 days, p = 0.02). Patient outcomes predicted by the algorithm had significant association with the "ground truth" in surgery time (p < 0.001, r = 0.73), LOS (p = 0.05, r = 0.52), and Foley duration (p < 0.001, r = 0.45). The five most relevant APMs, adopted by the RF-50 algorithm in predicting, were largely related to camera manipulation.

CONCLUSION

To our knowledge, ours is the first study to show that APMs and ML algorithms may help assess surgical RARP performance and predict clinical outcomes. With further accrual of clinical data (oncologic and functional data), this process will become increasingly relevant and valuable in surgical assessment and training.

A decade of robot-assisted radical prostatectomy training: Time-based metrics and qualitative grading for fellows and residents

OBJECTIVES

As modern urology residency and fellowship training in robot-assisted surgery evolves toward standardized curricula (didactics, dry/wet-laboratory exercises, and surgical assistance), additional tools are needed to evaluate on-console performance. At the start of our robotics program in 2006, we set-up a time- and quality-based evaluation program and aim to consolidate this data into a simple set of metrics for self-evaluation.

MATERIALS AND METHODS

Using our index procedure of robot-assisted radical prostatectomy (RARP), we prospectively collected data on 2,215 cases over 10 years from 6 faculty surgeons and 94 trainees (43 urologic oncology fellows and 51 urology residents). The steps of the operation were divided into 11 consistent steps, and the metrics included time to completion and quality using a 6-level grading system. Time metrics were consolidated into quartiles for benchmarking.

RESULTS

The median times for trainees to complete each step were 15% to 120% higher than those of the staff (P<0.001). Each step can be presented with quartile-based time metrics by pooled trainee and staff results. Steps performed by trainees were carefully chosen for a high success rate, and on our Likert-like scale were graded 4 to 5 in more than 95% of cases. There were no grade 0 (very poor) cases, and grades 1 (multiple technical errors) and 2 (could not be completed but without safety issues) were rare (<1%).

CONCLUSIONS

RARP training can be evaluated with a time-based metric that allows a quartile-based comparison to a large experience of trainees and staff. As a trainee progress through a rotation, these benchmarks can assist in prioritizing the need for more attention to a basic step vs. progression to more advanced steps.

Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study

PURPOSE

We explore and validate objective surgeon performance metrics using a novel recorder ("dVLogger") to directly capture surgeon manipulations on the da Vinci® Surgical System. We present the initial construct and concurrent validation study of objective metrics during preselected steps of robot-assisted radical prostatectomy.

MATERIALS AND METHODS

Kinematic and events data were recorded for expert (100 or more cases) and novice (less than 100 cases) surgeons performing bladder mobilization, seminal vesicle dissection, anterior vesicourethral anastomosis and right pelvic lymphadenectomy. Expert/novice metrics were compared using mixed effect statistical modeling (construct validation). Expert reviewers blindly rated seminal vesicle dissection and anterior vesicourethral anastomosis using GEARS (Global Evaluative Assessment of Robotic Skills). Intraclass correlation measured inter-rater variability. Objective metrics were correlated to corresponding GEARS metrics using Spearman's test (concurrent validation).

RESULTS

The performance of 10 experts (mean 810 cases, range 100 to 2,000) and 10 novices (mean 35 cases, range 5 to 80) was evaluated in 100 robot-assisted radical prostatectomy cases. For construct validation the experts completed operative steps faster (p <0.001) with less instrument travel distance (p <0.01), less aggregate instrument idle time (p <0.001), shorter camera path length (p <0.001) and more frequent camera movements (p <0.03). Experts had a greater ratio of dominant-to-nondominant instrument path distance for all steps (p <0.04) except anterior vesicourethral anastomosis. For concurrent validation the median experience of 3 expert reviewers was 300 cases (range 200 to 500). Intraclass correlation among reviewers was 0.6-0.7. For anterior vesicourethral anastomosis and seminal vesicle dissection, kinematic metrics had low associations with GEARS metrics.

CONCLUSIONS

Objective metrics revealed experts to be more efficient and directed during preselected steps of robot-assisted radical prostatectomy. Objective metrics had limited associations to GEARS. These findings lay the foundation for developing standardized metrics for surgeon training and assessment.