Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy

Novel workflow analysis of robot-assisted hysterectomy through objective performance indicators: a pilot study

Introduction

The curriculum for a da Vinci surgeon in gynecology requires special training before a surgeon performs their first independent case, but standardized, objective assessments of a trainee's workflow or skills learned during clinical cases are lacking. This pilot study presents a methodology to evaluate intraoperative surgeon behavior in hysterectomy cases through standardized surgical step segmentation paired with objective performance indicators (OPIs) calculated directly from robotic data streams. This method can provide individual case analysis in a truly objective capacity.

Materials and methods

Surgical data from six robot-assisted total laparoscopic hysterectomies (rTLH) performed by two experienced surgeons was collected prospectively using an Intuitive Data Recorder. Each rTLH video was annotated and segmented into specific, functional surgical steps based on the recorded video. Once annotated, OPIs were compared through workflow analysis and across surgeons during two critical surgical steps: colpotomy and vaginal cuff closure.

Results

Through visualization of the individual steps over time, we observe workflow consistencies and variabilities across individual surgeons of a similar experience level at the same hospital, creating unique surgeon behavior signatures across each surgical case. OPI differences across surgeons were observed for both the colpotomy and vaginal cuff closure steps, specifically reflecting camera movement, energy usage and clutching behaviors. Comparing colpotomy and vaginal cuff closure time needed for the step and the events of energy use were significantly different (p < 0.001). For the comparison between the two surgeons only the event count for camera movement during colpotomy showed significant differences (p = 0.03).

Conclusion

This pilot study presents a novel methodology to analyze and compare individual rTLH procedures with truly objective measurements. Through collection of robotic data streams and standardized segmentation, OPI measurements for specific rTLH surgery steps can be reliably calculated and compared to those of other surgeons. This provides opportunity for critical standardization to the gynecology field, which can be integrated into individualized training plans in the future. However, more studies are needed to establish context surrounding these metrics in gynecology.

Objective performance indicators during specific steps of robotic right colectomy can differentiate surgeon expertise

BACKGROUND

Current surgical assessment tools are subjective and nonscalable. Objective performance indicators, calculated from robotic systems data, provide automated data regarding surgeon movements and robotic arm kinematics. We identified objective performance indicators that significantly differed among expert and trainee surgeons during specific steps of robotic right colectomy.

METHODS

Endoscopic videos were annotated to delineate surgical steps during robotic right colectomies. Objective performance indicators were compared during mesenteric dissection, ascending colon mobilization, hepatic flexure mobilization, and bowel preparation for transection.

RESULTS

Twenty-five robotic right colectomy procedures (461 total surgical steps) performed by 2 experts and 8 trainees were analyzed. Experts exhibited faster camera acceleration and jerk during all steps, as well as faster dominant and nondominant arm acceleration and dominant arm jerk during all steps except distal bowel preparation. During mesenteric dissection, experts used faster camera and dominant arm velocity. During medial-to-lateral ascending colon mobilization, experts used less-dominant wrist yaw and pitch, faster nondominant arm velocity, shorter dominant arm path length, and shorter moving times for camera, dominant arm, and nondominant arm. During lateral-to-medial ascending colon mobilization, experts had faster dominant and nondominant arm velocity and third-arm acceleration. During hepatic flexure mobilization, experts exhibited more camera movements, greater velocity for camera, dominant and nondominant arms, and faster third-arm acceleration. During distal bowel preparation, experts used greater dominant wrist articulation, faster camera velocity, and longer nondominant arm path length. During proximal bowel preparation, experts demonstrated faster nondominant arm velocity.

CONCLUSION

Objective performance indicators can differentiate experts from trainees during distinct steps of robotic right colectomy. These automated, objective and scalable metrics can provide personalized feedback for trainees.

Association of Crowd-Sourced Assessment of Technical Skills (CSATSTM) and Outcomes of Robotic Assisted Radical Prostatectomy

OBJECTIVE

To investigate if use of the Crowd-Sourced Assessment of Technical Skills (CSATSTM) platform and video peer review with constructive feedback is associated with improvement in technical skill and patient outcomes for robotic-assisted laparoscopic prostatectomy (RALP).

METHODS

Five fellowship-trained urologists voluntarily submitted RALP cases for CSATSTM Global Evaluative Assessment of Robotic Skills (GEARS) scoring and expert narrative review between April 15, 2022 - April 30, 2023. Surgeon-selected and randomly-selected cases were reviewed. Surgeons underwent local peer review of videos with constructive feedback. Change in GEARS scores and frequency of post-operative outcomes over the 12-month periods before and during the study were analyzed in logistic regression models. Bias was assessed with sensitivity analysis comparing surgeon-selected to randomly-selected cases.

RESULTS

GEARS scores for randomly-selected versus surgeon-selected cases did not differ significantly. Overall GEARS score correlated positively with annual surgical RALP volume (r=0.39, p=0.003) and negatively with years in practice (r=-0.34, p=0.01). After adjusting for confounders, there was no significant improvement in overall GEARS Score (0.01±0.02/month, p=0.48); but likelihood of sepsis (Odds Ratio 0.07, 95% CI 0.01-1.00, p=0.05) and pelvic fluid collection (Odds Ratio 0.09, 95% CI 0.01-0.99, p=0.049) were significantly decreased during the intervention period (n=165) compared to the prior 12 months (n=144). No outcome increased in likelihood (p>0.05).

CONCLUSIONS

Integration of CSATSTM and local video peer review is associated with significant improvement in patient outcomes after RALP, despite no significant change in surgeons' GEARS scores. This is the first study demonstrating improvement in patient RALP outcomes after implementation of such a paradigm in practicing surgeons.

Quantification of surgical workflow during robotic proctectomy

BACKGROUND

Surgical workflow assessments offer insight regarding procedure variability. We utilised an objective method to evaluate workflow during robotic proctectomy (RP).

METHODS

We annotated 31 RPs and used Spearman's correlation to measure the correlation of step time and step visit frequency with console time (CT) and total operative time (TOT).

RESULTS

Strong correlations were seen with CT and step times for inferior mesenteric vein dissection and ligation (ρ = 0.60, ρ = 0.60), lateral-to-medial splenic flexure mobilisation (SFM) (ρ = 0.63), left rectal dissection (ρ = 0.64) and mesorectal division (ρ = 0.71). CT correlated strongly with medial-to-lateral (ρ = 0.75) and supracolic SFM visit frequency (ρ = 0.65). TOT correlated strongly with initial exposure time (ρ = 0.60), and medial-to-lateral (ρ = 0.67) and supracolic SFM visit frequency (ρ = 0.65).

CONCLUSION

This study correlates surgical steps with CT and TOT through standardised annotation, providing an objective approach to quantify workflow.

Evaluation status of current and emerging minimally invasive robotic surgical platforms

BACKGROUND

The rapid adoption of robotics within minimally invasive surgical specialties has also seen an explosion of new technology including multi- and single port, natural orifice transluminal endoscopic surgery (NOTES), endoluminal and "on-demand" platforms. This review aims to evaluate the validation status of current and emerging MIS robotic platforms, using the IDEAL Framework.

METHODS

A scoping review exploring robotic minimally invasive surgical devices, technology and systems in use or being developed was performed, including general surgery, gynaecology, urology and cardiothoracics. Systems operating purely outside the abdomen or thorax and endoluminal or natural orifice platforms were excluded. PubMed, Google Scholar, journal reports and information from the public domain were collected. Each company was approached via email for a virtual interview to discover more about the systems and to quality check data. The IDEAL Framework is an internationally accepted tool to evaluate novel surgical technology, consisting of four stages: idea, development/exploration, assessment, and surveillance. An IDEAL stage, synonymous with validation status in this review, was assigned by reviewing the published literature.

RESULTS

21 companies with 23 different robotic platforms were identified for data collection, 13 with national and/or international regulatory approval. Of the 17 multiport systems, 1 is fully evaluated at stage 4, 2 are stage 3, 6 stage 2b, 2 at stage 2a, 2 stage 1, and 4 at the pre-IDEAL stage 0. Of the 6 single-port systems none have been fully evaluated with 1 at stage 3, 3 at stage 1 and 2 at stage 0.

CONCLUSIONS

The majority of existing robotic platforms are currently at the preclinical to developmental and exploratory stage of evaluation. Using the IDEAL framework will ensure that emerging robotic platforms are fully evaluated with long-term data, to inform the surgical workforce and ensure patient safety.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Structured Resident Training in Robotic Surgery: Recommendations of the Robotic Surgery Education Working Group

OBJECTIVE

A universal resident robotic surgery training pathway that maximizes proficiency and safety has not been defined by a consensus of surgical educators or by surgical societies. The objective of the Robotic Surgery Education Working Group was to develop a universal curriculum pathway and leverage digital tools to support resident education.

DESIGN

The two lead authors (JP and YN) contacted potential members of the Working Group. Members were selected based on their authorship of peer-review publications, their experience as minimally invasive and robotic surgeons, their reputations, and their ability to commit the time involved to work collaboratively and efficiently to reach consensus regarding best practices in robotic surgery education. The Group's approach was to reach 100% consensus to provide a transferable curriculum that could be applied to the vast majority of resident programs.

SETTING

Virtual and in-person meetings in the United States.

PARTICIPANTS

Eight surgeons (2 females and 6 males) from five academic medical institutions (700-1541 beds) and three community teaching hospitals (231-607 beds) in geographically diverse locations comprised the Working Group. They represented highly specialized general surgeons and educators in their mid-to-late careers. All members were experienced minimally invasive surgeons and had national reputations as robotic surgery educators.

RESULTS

The surgeons initially developed and agreed upon questions for each member to consider and respond to individually via email. Responses were collated and consolidated to present on an anonymized basis to the Group during an in-person day-long meeting. The surgeons self-facilitated and honed the agreed upon responses of the Group into a 5-level Robotic Surgery Curriculum Pathway, which each member agreed was relevant and expressed their convictions and experience.

CONCLUSIONS

The current needs for a universal robotic surgery training curriculum are validated objective and subjective measures of proficiency, access to simulation, and a digital platform that follows a resident from their first day of residency through training and their entire career. Refinement of current digital solutions and continued innovation guided by surgical educators is essential to build and maintain a scalable, multi-institutional supported curriculum.

A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy

BACKGROUND

The increased digitization in robotic surgical procedures today enables surgeons to quantify their movements through data captured directly from the robotic system. These calculations, called objective performance indicators (OPIs), offer unprecedented detail into surgical performance. In this study, we link case- and surgical step-specific OPIs to case complexity, surgical experience and console utilization, and post-operative clinical complications across 87 robotic cholecystectomy (RC) cases.

METHODS

Videos of RCs performed by a principal surgeon with and without fellows were segmented into eight surgical steps and linked to patients' clinical data. Data for OPI calculations were extracted from an Intuitive Data Recorder and the da Vinci ® robotic system. RC cases were each assigned a Nassar and Parkland Grading score and categorized as standard or complex. OPIs were compared across complexity groups, console attributions, and post-surgical complication severities to determine objective relationships across variables.

RESULTS

Across cases, differences in camera control and head positioning metrics of the principal surgeon were observed when comparing standard and complex cases. Further, OPI differences across the principal surgeon and the fellow(s) were observed in standard cases and include differences in arm swapping, camera control, and clutching behaviors. Monopolar coagulation energy usage differences were also observed. Select surgical step duration differences were observed across complexities and console attributions, and additional surgical task analyses determine the adhesion removal and liver bed hemostasis steps to be the most impactful steps for case complexity and post-surgical complications, respectively.

CONCLUSION

This is the first study to establish the association between OPIs, case complexities, and clinical complications in RC. We identified OPI differences in intra-operative behaviors and post-surgical complications dependent on surgeon expertise and case complexity, opening the door for more standardized assessments of teaching cases, surgical behaviors and case complexities.

Factors affecting upper limb ergonomics in robotic colorectal surgery

The aim of the study was to examine the factors which may influence suboptimal ergonomic surgeon hand positioning during robotic colorectal surgery (RCS). An observational study of 11 consecutive RCS cases from June 2022 to August 2022 was performed. Continuous video footage of RCS cases was analysed concurrently with video recordings of surgeon's hand positions at the console. The outcome studied was the frequency with which either hand remained in a suboptimal ergonomic position outside the predetermined double box outlines, as marked on the surgeon's video, for >1 min. Situations which resulted in poor upper limb ergonomics were dissection in the peripheral operating field location, left-hand use, use of the stapler, dissection of the main mesenteric blood vessels, and multi-quadrant surgery. Being aware of situations when suboptimal ergonomic positions occur can allow surgeons to consciously compensate by using the clutch or pausing to take a rest break.

What does this paper add to the literature?

The study is important because it is the first to look at factors which may influence poor upper limb ergonomics during non-simulated RCS. By recognizing these factors and compensating for them, it may improve surgeon ergonomics with resultant better performance.

A Multi-institution Study on the Association of Virtual Reality Skills with Continence Recovery after Robot-assisted Radical Prostatectomy

BACKGROUND

Virtual reality (VR) simulators are increasingly being used for surgical skills training. It is unclear what skills are best improved via VR, translate to live surgical skills, and influence patient outcomes.

OBJECTIVE

To assess surgeons in VR and live surgery using a suturing assessment tool and evaluate the association between technical skills and a clinical outcome.

DESIGN, SETTING, AND PARTICIPANTS

This prospective five-center study enrolled participants who completed VR suturing exercises and provided live surgical video. Graders provided skill assessments using the validated End-To-End Assessment of Suturing Expertise (EASE) suturing evaluation tool.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

A hierarchical Poisson model was used to compare skill scores among cohorts and evaluate the association of scores with clinical outcomes. Spearman's method was used to assess correlation between VR and live skills.

RESULTS AND LIMITATIONS

Ten novices, ten surgeons with intermediate expertise (median 64 cases, interquartile range [IQR] 6-80), and 26 expert surgeons (median 850 cases, IQR 375-3000) participated in this study. Intermediate and expert surgeons were significantly more likely to have ideal scores in comparison to novices for the subskills needle hold angle, wrist rotation, and wrist rotation needle withdrawal (p < 0.01). For both intermediate and expert surgeons, there was positive correlation between VR and live skills for needle hold angle (p < 0.05). For expert surgeons, there was a positive association between ideal scores for VR needle hold angle and driving smoothness subskills and 3-mo continence recovery (p < 0.05). Limitations include the size of the intermediate surgeon sample and clinical data limited to expert surgeons.

CONCLUSIONS

EASE can be used in VR to identify skills to improve for trainee surgeons. Technical skills that influence postoperative outcomes may be assessable in VR.

PATIENT SUMMARY

This study provides insights into surgical skills that translate from virtual simulation to live surgery and that have an impact on urinary continence after robot-assisted removal of the prostate. We also highlight the usefulness of virtual reality in surgical education.

Quantification of Surgical Workflow during Robotic Proctectomy

Aim

Assessments of surgical workflow offer insight regarding procedure variability, case complexity and surgeon proficiency. We utilize an objective method to evaluate step-by-step workflow and step transitions during robotic proctectomy (RP).

Methods

We annotated 31 RPs using a procedure-specific annotation card. Using Spearman's correlation, we measured strength of association of step time and step visit frequency with console time (CT) and total operative time (TOT).

Results

Across 31 RPs, a mean (± standard deviation) of 49.0 (± 20.3) steps occurred per procedure. Mean CT and TOT were 213 (± 90) and 283 (± 108) minutes. Posterior mesorectal dissection required most visits (8.7 ± 5.0), while anastomosis required most time (18.0 [± 8.5] minutes). Inferior mesenteric vein (IMV) ligation required least visits (1.0 ± 0.0) and lowest duration (0.9 [± 0.5] minutes). Strong correlations were seen with CT and step times for IMV dissection and ligation (ρ = 0.60 for both), lateral-to-medial splenic flexure mobilization (SFM) (ρ = 0.63), left rectal dissection (ρ = 0.64) and mesorectal division (ρ = 0.71). CT correlated strongly with medial-to-lateral and supracolic SFM visit frequency (ρ = 0.75 and ρ = 0.65). There were strong correlations with TOT and initial exposure time (ρ = 0.60), as well as visit frequency for medial-to-lateral (ρ = 0.67) and supracolic SFM (ρ = 0.65). Descending colon mobilization was nodal, rectal mobilization convergent and rectal transection divergent.

Conclusion

This study correlates individual surgical steps with CT and TOT through standardized annotation. It provides an objective approach to quantify workflow.

Automated performance metrics, learning curve and robotic colorectal surgery

BACKGROUND

The aim of this study was to evaluate the usefulness of Automated Performance Metrics (APMs) in assessing the learning curve.

METHODS

A retrospective review of 85 consecutive patients who underwent total robotic colorectal surgery at a single institution between August 2020 and October 2022 was performed. Patient demographics, operation type, and APMs were collected and analysed. Cumulative summation technique (CUSUM) was used to construct learning curves of surgeon console time (SCT), use of the fourth arm, clutch activation, instrument off screen (number and duration), and cut electrocautery activation.

RESULTS

Two phases with 50 and 35 cases were identified from the CUSUM graph for SCT. The SCT was significantly different between the two phases (176 and 251 min, p < 0.002). After adjustment for SCT, the APMs were not significantly different between the two phases.

CONCLUSIONS

Most APMs do not offer additional learning curve information when compared with SCT analysis alone.

Kinematic data profile and clinical outcomes in robotic inguinal hernia repairs: a pilot study

BACKGROUND

Surgical training requires clinical knowledge and technical skills to operate safely and optimize clinical outcomes. Technical skills are hard to measure. The Intuitive Data Recorder (IDR), (Sunnyvale, CA) allows for the measurement of technical skills using objective performance indicators (OPIs) from kinematic event data. Our goal was to determine whether OPIs improve with surgeon experience and whether they are correlated with clinical outcomes for robotic inguinal hernia repair (RIHR).

METHODS

The IDR was used to record RIHRs from six surgeons. Data were obtained from 98 inguinal hernia repairs from February 2022 to February 2023. Patients were called on postoperative days 5-10 and asked to take the Carolina Comfort Scale (CCS) survey to evaluate acute clinical outcomes. A Pearson test was run to determine correlations between OPIs from the IDR with a surgeon's yearly RIHR experience and with CCS scores. Linear regression was then run for correlated OPIs.

RESULTS

Multiple OPIs were correlated with surgeon experience. Specifically, for the task of peritoneal flap exploration, we found that 23 OPIs were significantly correlated with surgeons' 1-year RIHR case number. Total angular motion distance of the left arm instrument had a correlation of - 0.238 (95% CI - 0.417, - 0.042) for RIHR yearly case number. Total angular motion distance of right arm instrument was also negatively correlated with RIHR in 1 year with a correlation of - 0.242 (95% CI - 0.420, - 0.046). For clinical outcomes, wrist articulation of the surgeon's console positively correlated with acute sensation scores from the CCS with a correlation of 0.453 (95% CI 0.013, 0.746).

CONCLUSIONS

This study defines multiple OPIs that correlate with surgeon experience and with outcomes. Using this knowledge, surgical simulation platforms can be designed to teach patterns to surgical trainees that are associated with increased surgical experience and with improved postoperative outcomes.

Differences Between Attending and Trainee Surgeon Performance Using Objective Performance Indicators During Robot-Assisted Lobectomy

OBJECTIVE

Existing approaches for assessing surgical performance are subjective and prone to bias. In contrast, utilizing digital kinematic and system data from the surgical robot allows the calculation of objective performance indicators (OPIs) that may differentiate technical skill and competency. This study compared OPIs of trainees and attending surgeons to assess differences during robotic lobectomy (RL).

METHODS

There were 50 cardiothoracic surgery residents and 7 attending surgeons who performed RL on a left upper lobectomy of an ex vivo perfused model. A novel recorder simultaneously captured video and data from the system and instruments. The lobectomy was annotated into discrete tasks, and OPIs were analyzed for both hands during 6 tasks: exposure of the superior pulmonary vein, upper division of the pulmonary artery and bronchus, and the stapling of these structures.

RESULTS

There were significant differences between attendings and trainees in all tasks. Among 20 OPIs during exposure tasks, significant differences were observed for the left hand in 31 of 60 (52%) of OPIs and for the right hand in 42 of 60 (70%). During stapling tasks, significant differences were observed for the stapling hand in 28 of 60 (47%) of OPIs and for the nonstapling hand in 14 of 60 (25%).

CONCLUSIONS

Use of a novel data and video recorder to generate OPIs for both hands revealed significant differences in the operative gestures performed by trainees compared to attendings during RL. This method of assessing performance has potential for establishing objective competency benchmarks and use for tracking progress.

A novel approach to quantifying surgical workflow in robotic-assisted lobectomy

INTRODUCTION

Understanding surgical workflow is critical for optimizing efficiencies and outcomes; however, most research evaluating workflow is impacted by observer subjectivity, limiting its reproducibility, scalability, and actionability. To address this, we developed a novel approach to quantitatively describe workflow within robotic-assisted lobectomy (RL). We demonstrate the utility of this approach by analysing features of surgical workflow that correlate with procedure duration.

METHODS

RL was deconstructed into 12 tasks by expert thoracic surgeons. Task start and stop times were annotated across videos of 10 upper RLs (5 right and 5 left). Markov Networks were used to estimate both the likelihood of transitioning from one task to another and each task-transition entropy (i.e. complexity). Associations between the frequency with which each task was revisited intraoperatively and procedure duration were assessed using Pearson's correlation coefficient.

RESULTS

Entropy calculations identified fissure dissection and hilar node dissection as tasks with especially complex transitions, while mediastinal lymph node dissection and division of pulmonary veins were less complex. The number of transitions to three tasks significantly correlated with case duration (fissure dissection (R = 0.69, p = 0.01), dissect arteries (R = 0.59, p = 0.03), and divide arteries (R = 0.63, p = 0.03)).

CONCLUSION

This pilot demonstrates the feasibility of objectively quantifying workflow between RL tasks and introduces entropy as a new metric of task-transition complexity. These innovative measures of surgical workflow enable detailed characterization of a given surgery and might indicate behaviour that impacts case progression. We discuss how these measures can serve as a foundation and be combined with relevant clinical information to better understand factors influencing surgical inefficiency.

Relating process and outcome metrics for meaningful and interpretable cannulation skill assessment: A machine learning paradigm

BACKGROUND AND OBJECTIVES

The quality of healthcare delivery depends directly on the skills of clinicians. For patients on hemodialysis, medical errors or injuries caused during cannulation can lead to adverse outcomes, including potential death. To promote objective skill assessment and effective training, we present a machine learning approach, which utilizes a highly-sensorized cannulation simulator and a set of objective process and outcome metrics.

METHODS

In this study, 52 clinicians were recruited to perform a set of pre-defined cannulation tasks on the simulator. Based on data collected by sensors during their task performance, the feature space was then constructed based on force, motion, and infrared sensor data. Following this, three machine learning models- support vector machine (SVM), support vector regression (SVR), and elastic net (EN)- were constructed to relate the feature space to objective outcome metrics. Our models utilize classification based on the conventional skill classification labels as well as a new method that represents skill on a continuum.

RESULTS

With less than 5% of trials misplaced by two classes, the SVM model was effective in predicting skill based on the feature space. In addition, the SVR model effectively places both skill and outcome on a fine-grained continuum (versus discrete divisions) that is representative of reality. As importantly, the elastic net model enabled the identification of a set of process metrics that highly impact outcomes of the cannulation task, including smoothness of motion, needle angles, and pinch forces.

CONCLUSIONS

The proposed cannulation simulator, paired with machine learning assessment, demonstrates definite advantages over current cannulation training practices. The methods presented here can be adopted to drastically increase the effectiveness of skill assessment and training, thereby potentially improving clinical outcomes of hemodialysis treatment.

Ratio of Economy of Motion: A New Objective Performance Indicator to Assign Consoles During Dual-Console Robotic Proctectomy

BACKGROUND

Our group investigates objective performance indicators (OPIs) to analyze robotic colorectal surgery. Analyses of OPI data are difficult in dual-console procedures (DCPs) as there is currently no reliable, efficient, or scalable technique to assign console-specific OPIs during a DCP. We developed and validated a novel metric to assign tasks to appropriate surgeons during DCPs.

METHODS

A colorectal surgeon and fellow reviewed 21 unedited, dual-console proctectomy videos with no information to identify the operating surgeons. The reviewers watched a small number of random tasks and assigned "attending" or "trainee" to each task. Based on this sampling, the remainder of task assignments for each procedure was extrapolated. In parallel, we applied our newly developed OPI, ratio of economy of motion (rEOM), to assign consoles. Results from the 2 methods were compared.

RESULTS

A total of 1811 individual surgical tasks were recorded during 21 proctectomy videos. A median of 6.5 random tasks (137 total) were reviewed during each video, and the remainder of task assignments were extrapolated based on the 7.6% of tasks audited. The task assignment agreement was 91.2% for video review vs rEOM, with rEOM providing ground truth. It took 2.5 hours to manually review video and assign tasks. Ratio of economy of motion task assignment was immediately available based on OPI recordings and automated calculation.

DISCUSSION

We developed and validated rEOM as an accurate, efficient, and scalable OPI to assign individual surgical tasks to appropriate surgeons during DCPs. This new resource will be useful to everyone involved in OPI research across all surgical specialties.

Omitting routine cystography after RARP: Analysis of complications and readmission rates in suprapubic and transurethral drained patients

OBJECTIVES

Robot-assisted radical prostatectomy (RARP) has become the therapy of choice for local treatment of prostate cancer. Postoperatively, urologists perform cystography before removing urinary catheters due to concerns about the integrity of the vesicourethral anastomosis. This study aims to evaluate the safety of waiving cystography before early catheter removal after RARP.

METHODS

A total of 514 patients from two tertiary referral centers who underwent RARP were retrospectively included. Patients received postoperative urinary drainage by transurethral (TUC) or suprapubic catheter (SPC). During the first year, both centers performed routine cystography before removing TUC or SPC on postoperative day 5. In the following year, management changed and catheters were removed without cystography unless indicated by the surgeon. Demographic and perioperative data were analyzed. Postoperative complications and readmission rates were compared between standard cystography (StCG), no cystography (NCG), and selective cystography (SCG).

RESULTS

Groups were comparable regarding demographic and oncological parameters. Analysis showed no significant difference regarding major complications and readmission rates between standard and no cystography (p = 0.155 and 0.998 respectively). Omitting routine cystography did not lead to inferior postoperative courses regardless of both urinary drainage used and tumor stage. Subgroup analysis showed an increase of major complications in SCG patients when compared with NCG (p = 0.003) while readmissions remained comparable (p = 0.554).

CONCLUSION

Waiving routine cystography before early catheter removal after RARP appears to be safe and feasible regardless of urinary drainage. However, the selective cystogram at the surgeon's request still plays a role in monitoring patients with an elevated risk profile.

Surgical gestures as a method to quantify surgical performance and predict patient outcomes

How well a surgery is performed impacts a patient's outcomes; however, objective quantification of performance remains an unsolved challenge. Deconstructing a procedure into discrete instrument-tissue "gestures" is a emerging way to understand surgery. To establish this paradigm in a procedure where performance is the most important factor for patient outcomes, we identify 34,323 individual gestures performed in 80 nerve-sparing robot-assisted radical prostatectomies from two international medical centers. Gestures are classified into nine distinct dissection gestures (e.g., hot cut) and four supporting gestures (e.g., retraction). Our primary outcome is to identify factors impacting a patient's 1-year erectile function (EF) recovery after radical prostatectomy. We find that less use of hot cut and more use of peel/push are statistically associated with better chance of 1-year EF recovery. Our results also show interactions between surgeon experience and gesture types-similar gesture selection resulted in different EF recovery rates dependent on surgeon experience. To further validate this framework, two teams independently constructe distinct machine learning models using gesture sequences vs. traditional clinical features to predict 1-year EF. In both models, gesture sequences are able to better predict 1-year EF (Team 1: AUC 0.77, 95% CI 0.73-0.81; Team 2: AUC 0.68, 95% CI 0.66-0.70) than traditional clinical features (Team 1: AUC 0.69, 95% CI 0.65-0.73; Team 2: AUC 0.65, 95% CI 0.62-0.68). Our results suggest that gestures provide a granular method to objectively indicate surgical performance and outcomes. Application of this methodology to other surgeries may lead to discoveries on methods to improve surgery.

Objective performance indicators of cardiothoracic residents are associated with vascular injury during robotic-assisted lobectomy on porcine models

Surgical training relies on subjective feedback on resident technical performance by attending surgeons. A novel data recorder connected to a robotic-assisted surgical platform captures synchronized kinematic and video data during an operation to calculate quantitative, objective performance indicators (OPIs). The aim of this study was to determine if OPIs during initial task of a resident's robotic-assisted lobectomy (RL) correlated with bleeding during the procedure. Forty-six residents from the 2019 Thoracic Surgery Directors Association Resident Boot Camp completed RL on an ex vivo perfused porcine model while continuous video and kinematic data were recorded. For this pilot study, RL was segmented into 12 tasks and OPIs were calculated for the initial major task. Cases were reviewed for major bleeding events and OPIs of bleeding cases were compared to those who did not. Data from 42 residents were complete and included in the analysis. 10/42 residents (23.8%) encountered bleeding: 10/40 residents who started with superior pulmonary vein exposure and 0/2 residents who started with pulmonary artery exposure. Twenty OPIs for both hands were assessed during the initial task. Six OPIs related to instrument usage or smoothness of motion were significant for bleeding. Differences were statistically significant for both hands (p < 0.05). OPIs showing bimanual asymmetry indicated lower proficiency. This study demonstrates that kinematic and video analytics can establish a correlation between objective performance metrics and bleeding events in an ex vivo perfused lobectomy. Further study could assist in the development of focused exercises and simulation on objective domains to help improve overall performance and reducing complications during RL.

Laparoscopic and Robot-Assisted Hepatic Surgery: An Historical Review

Hepatic surgery is a rapidly expanding component of abdominal surgery and is performed for a wide range of indications. The introduction of laparoscopic cholecystectomy in 1987 was a major change in abdominal surgery. Laparoscopic surgery was widely and rapidly adopted throughout the world for cholecystectomy initially and then applied to a variety of other procedures. Laparoscopic surgery became regularly applied to hepatic surgery, including segmental and major resections as well as organ donation. Many operations progressed from open surgery to laparoscopy to robot-assisted surgery, including colon resection, pancreatectomy, splenectomy thyroidectomy, adrenalectomy, prostatectomy, gastrectomy, and others. It is difficult to prove a data-based benefit using robot-assisted surgery, although laparoscopic and robot-assisted surgery of the liver are not inferior regarding major outcomes. When laparoscopic surgery initially became popular, many had concerns about its use to treat malignancies. Robot-assisted surgery is being used to treat a variety of benign and malignant conditions, and studies have shown no deterioration in outcomes. Robot-assisted surgery for the treatment of malignancies has become accepted and is now being used at more centers. The outcomes after robot-assisted surgery depend on its use at specialized centers, the surgeon's personal experience backed up by extensive training and maintenance of international registries. Robot-assisted hepatic surgery has been shown to be associated with slightly less intraoperative blood loss and shorter hospital lengths of stay compared to open surgery. Oncologic outcomes have been maintained, and some studies show higher rates of R0 resections. Patients who need surgery for liver lesions should identify a surgeon they trust and should not be concerned with the specific operative approach used. The growth of robot-assisted surgery of the liver has occurred in a stepwise approach which is very different from the frenzy that was seen with the introduction of laparoscopic cholecystectomy. This approach allowed the identification of areas for improvement, many of which are at the nexus of engineering and medicine. Further improvements in robot-assisted surgery depend on the combined efforts of engineers and surgeons.

Survival Analysis Using Surgeon Skill Metrics and Patient Factors to Predict Urinary Continence Recovery After Robot-assisted Radical Prostatectomy

BACKGROUND

It has been shown that metrics recorded for instrument kinematics during robotic surgery can predict urinary continence outcomes.

OBJECTIVE

To evaluate the contributions of patient and treatment factors, surgeon efficiency metrics, and surgeon technical skill scores, especially for vesicourethral anastomosis (VUA), to models predicting urinary continence recovery following robot-assisted radical prostatectomy (RARP).

DESIGN, SETTING, AND PARTICIPANTS

Automated performance metrics (APMs; instrument kinematics and system events) and patient data were collected for RARPs performed from July 2016 to December 2017. Robotic Anastomosis Competency Evaluation (RACE) scores during VUA were manually evaluated. Training datasets included: (1) patient factors; (2) summarized APMs (reported over RARP steps); (3) detailed APMs (reported over suturing phases of VUA); and (4) technical skills (RACE). Feature selection was used to compress the dimensionality of the inputs.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

The study outcome was urinary continence recovery, defined as use of 0 or 1 safety pads per day. Two predictive models (Cox proportional hazards [CoxPH] and deep learning survival analysis [DeepSurv]) were used.

RESULTS AND LIMITATIONS

Of 115 patients undergoing RARP, 89 (77.4%) recovered their urinary continence and the median recovery time was 166 d (interquartile range [IQR] 82-337). VUAs were performed by 23 surgeons. The median RACE score was 28/30 (IQR 27-29). Among the individual datasets, technical skills (RACE) produced the best models (C index: CoxPH 0.695, DeepSurv: 0.708). Among summary APMs, posterior/anterior VUA yielded superior model performance over other RARP steps (C index 0.543-0.592). Among detailed APMs, metrics for needle driving yielded top-performing models (C index 0.614-0.655) over other suturing phases. DeepSurv models consistently outperformed CoxPH; both approaches performed best when provided with all the datasets. Limitations include feature selection, which may have excluded relevant information but prevented overfitting.

CONCLUSIONS

Technical skills and "needle driving" APMs during VUA were most contributory. The best-performing model used synergistic data from all datasets.

PATIENT SUMMARY

One of the steps in robot-assisted surgical removal of the prostate involves joining the bladder to the urethra. Detailed information on surgeon performance for this step improved the accuracy of predicting recovery of urinary continence among men undergoing this operation for prostate cancer.

A Probabilistic Approach To Surgical Tasks and Skill Metrics

Identifying and quantifying the activities that compose surgery is essential for effective interventions, computer-aided analyses and the advancement of surgical data science. For example, recent studies have shown that objective metrics (referred to as objective performance indicators, OPIs) computed during key surgical tasks correlate with surgeon skill and clinical outcomes. Unambiguous identification of these surgical tasks can be particularly challenging for both human annotators and algorithms. Each surgical procedure has multiple approaches, each surgeon has their own level of skill, and the initiation and termination of surgical tasks can be subject to interpretation. As such, human annotators and machine learning models face the same basic problem, accurately identifying the boundaries of surgical tasks despite variable and unstructured information. For use in surgeon feedback, OPIs should also be robust to the variability and diversity in this data. To mitigate this difficulty, we propose a probabilistic approach to surgical task identification and calculation of OPIs. Rather than relying on tasks that are identified by hard temporal boundaries, we demonstrate an approach that relies on distributions of start and stop times, for a probabilistic interpretation of when the task was performed. We first use hypothetical data to outline how this approach is superior to other conventional approaches. Then we present similar analyses on surgical data. We find that when surgical tasks are identified by their individual probabilities, the resulting OPIs are less sensitive to noise in the identification of the start and stop times. These results suggest that this probabilistic approach holds promise for the future of surgical data science.

Measuring Cannulation Skills for Hemodialysis: Objective Versus Subjective Assessment

Lack of cannulation skill during hemodialysis treatments results in poor clinical outcomes due to infiltration and other cannulation-related trauma. Unfortunately, training of patient care technicians and nurses, specifically on the "technical" aspects of cannulation, has traditionally not received much attention. Simulators have been successfully deployed in many medical specialties for assessment and training of clinical skills. However, simulators have not been as widely used in nursing, especially in the context of training clinical personnel in the dialysis unit. We designed a state-of-the-art simulator for quantifying skill for hemodialysis cannulation. In this study, 52 nurses and patient care technicians with varying levels of clinical experience performed 16 cannulations on the simulator with different fistula properties. We formulated a composite metric for objectively measuring overall success of cannulation and compared this metric with subjective assessment by experts. In addition, we examined if years of clinical experience correlated with objective and subjective scores for cannulation skill. Results indicated that, while subjective and objective metrics generally correlated with each other, the objective metric was more precise and better suited for quantifying cannulation skill. Further, the simulator-based objective metric provides several advantages over subjective ratings, including providing fine-grained assessment of skill, consistency in measurement unaffected by subjective biases, and basing assessment on a more complete evaluation of performance. Years of clinical experience, however, demonstrated little correlation with either method of skill assessment. The methods presented for cannulation skill assessment in this study, if widely applied, could result in improved cannulation skill among our PCTs and nurses, which could positively impact patient outcomes in a tangible way.

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy

INTRODUCTION

Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon's technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)-objective measures related to instrument movements-can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes.

METHODS

A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query "robot-assisted surgery OR robotic surgery AND performance metrics" between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed.

RESULTS

The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies.

CONCLUSIONS

APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required.

Road to automating robotic suturing skills assessment: Battling mislabeling of the ground truth

OBJECTIVE

To automate surgeon skills evaluation using robotic instrument kinematic data. Additionally, to implement an unsupervised mislabeling detection algorithm to identify potentially mislabeled samples that can be removed to improve model performance.

METHODS

Video recordings and instrument kinematic data were derived from suturing exercises completed on the Mimic FlexVR robotic simulator. A structured human consensus-building process was developed to determine Robotic Anastomosis Competency Evaluation technical scores across 3 human graders. A 2-layer long short-term memory-based classification model used instrument kinematic data to automate suturing skills assessment. An unsupervised label analyzer (NoiseRank) was used to identify potential mislabeling of skills data. Performance of the long short-term memory model's technical skill score prediction was measured by best area under the curve over the training runs. NoiseRank outputted a ranked list of rated skills assessments based on likelihood of mislabeling.

RESULTS

22 surgeons performed 226 suturing attempts, which were broken down into 1,404 individual skill assessment points. Automation of needle entry angle, needle driving, and needle withdrawal technical skill scores performed better (area under the curve 0.698-0.705) than needle positioning (0.532) at baseline using all available data. Potential mislabels were subsequently identified by NoiseRank and removed, improving model performance across all domains (area under the curve 0.551-0.766).

CONCLUSION

Using ground truth labels from human graders and robotic instrument kinematic data, machine learning models have automated assessment of detailed suturing technical skills with good performance. Further, an unsupervised mislabeling detection algorithm projected mislabeled data, allowing for their removal and subsequent improvement of model performance.

How the use of the artificial intelligence could improve surgical skills in urology: state of the art and future perspectives

PURPOSE OF REVIEW

As technology advances, surgical training has evolved in parallel over the previous decade. Training is commonly seen as a way to prepare surgeons for their day-to-day work; however, more importantly, it allows for certification of skills to ensure maximum patient safety. This article reviews advances in the use of machine learning and artificial intelligence for improvements of surgical skills in urology.

RECENT FINDINGS

Six studies have been published, which met the inclusion criteria. All articles assessed the application of artificial intelligence in improving surgical training. Different approaches were taken, such as using machine learning to identify and classify suturing gestures, creating automated objective evaluation reports, and determining surgical technical skill levels to predict clinical outcomes. The articles illustrated the continuously growing role of artificial intelligence to address the difficulties currently present in evaluating urological surgical skills.

SUMMARY

Artificial intelligence allows us to efficiently analyze the surmounting data related to surgical training and use it to come to conclusions that normally would require human intelligence. Although these metrics have been shown to predict surgeon expertise and surgical outcomes, evidence is still scarce regarding their ability to directly improve patient outcomes. Considering this, current active research is growing on the topic of deep learning-based computer vision to provide automated metrics needed for real-time surgeon feedback.

Motion Smoothness Metrics for Cannulation Skill Assessment: What Factors Matter?

Medical training simulators have the potential to provide remote and automated assessment of skill vital for medical training. Consequently, there is a need to develop "smart" training devices with robust metrics that can quantify clinical skills for effective training and self-assessment. Recently, metrics that quantify motion smoothness such as log dimensionless jerk (LDLJ) and spectral arc length (SPARC) are increasingly being applied in medical simulators. However, two key questions remain about the efficacy of such metrics: how do these metrics relate to clinical skill, and how to best compute these metrics from sensor data and relate them with similar metrics? This study addresses these questions in the context of hemodialysis cannulation by enrolling 52 clinicians who performed cannulation in a simulated arteriovenous (AV) fistula. For clinical skill, results demonstrate that the objective outcome metric flash ratio (FR), developed to measure the quality of task completion, outperformed traditional skill indicator metrics (years of experience and global rating sheet scores). For computing motion smoothness metrics for skill assessment, we observed that the lowest amount of smoothing could result in unreliable metrics. Furthermore, the relative efficacy of motion smoothness metrics when compared with other process metrics in correlating with skill was similar for FR, the most accurate measure of skill. These results provide guidance for the computation and use of motion-based metrics for clinical skill assessment, including utilizing objective outcome metrics as ideal measures for quantifying skill.

Motion analysis of the JHU-ISI Gesture and Skill Assessment Working Set II: learning curve analysis

PURPOSE

The Johns Hopkins-Intuitive Gesture and Skill Assessment Working Set (JIGSAWS) dataset is used to develop robotic surgery skill assessment tools, but there has been no detailed analysis of this dataset. The aim of this study is to perform a learning curve analysis of the existing JIGSAWS dataset.

METHODS

Five trials were performed in JIGSAWS by eight participants (four novices, two intermediates and two experts) for three exercises (suturing, knot-tying and needle passing). Global Rating Scores and time, path length and movements were analyzed quantitatively and qualitatively by graphical analysis.

RESULTS

There are no significant differences in Global Rating Scale scores over time. Time in the suturing exercise and path length in needle passing had significant differences. Other kinematic parameters were not significantly different. Qualitative analysis shows a learning curve only for suturing. Cumulative sum analysis suggests completion of the learning curve for suturing by trial 4.

CONCLUSIONS

The existing JIGSAWS dataset does not show a quantitative learning curve for Global Rating Scale scores, or most kinematic parameters which may be due in part to the limited size of the dataset. Qualitative analysis shows a learning curve for suturing. Cumulative sum analysis suggests completion of the suturing learning curve by trial 4. An expanded dataset is needed to facilitate subset analyses.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Is Experience in Hemodialysis Cannulation Related to Expertise? A Metrics-based Investigation for Skills Assessment

Cannulation is not only one of the most common medical procedures but also fraught with complications. The skill of the clinician performing cannulation directly impacts cannulation outcomes. However, current methods of teaching this skill are deficient, relying on subjective demonstrations and unrealistic manikins that have limited utility for skills training. Furthermore, of the factors that hinders effective continuing medical education is the assumption that clinical experience results in expertise. In this work, we examine if objective metrics acquired from a novel cannulation simulator are able to distinguish between experienced clinicians and established experts, enabling the measurement of true expertise. Twenty-two healthcare professionals, who practiced cannulation with varying experience, performed a simulated arteriovenous fistula cannulation task on the simulator. Four clinicians were peer-identified as experts while the others were designated to the experienced group. The simulator tracked the motion of the needle (via an electromagnetic sensor), rendered blood flashback function (via an infrared light sensor), and recorded pinch forces exerted on the needle (via force sensing elements). Metrics were computed based on motion, force, and other sensor data. Results indicated that, with near 80% of accuracy using both logistic regression and linear discriminant analysis, the objective metrics differentiated between experts and the experienced, including identifying needle motion and finger force as two prominent features that distinguished between the groups. Furthermore, results indicated that expertise was not correlated with years of experience, validating the central hypothesis of the study. These insights contribute to structured and standardized medical skills training by enabling a meaningful definition of expertise and could potentially lead to more effective skills training methods.

Expert Consensus Recommendations for Robotic Surgery Credentialing

OBJECTIVE

To define criteria for robotic credentialing using expert consensus.

BACKGROUND

A recent review of institutional robotic credentialing policies identified significant variability and determined current policies are largely inadequate to ensure surgeon proficiency and may threaten patient safety.

METHODS

28 national robotic surgery experts were invited to participate in a consensus conference. After review of available institutional policies and discussion, the group developed a 91 proposed criteria. Using a modified Delphi process the experts were asked to indicate their agreement with the proposed criteria in three electronic survey rounds after the conference. Criteria that achieved 80% or more in agreement (consensus) in all rounds were included in the final list.

RESULTS

All experts agreed that there is a need for standardized robotic surgery credentialing criteria across institutions that promote surgeon proficiency. 49 items reached consensus in the first round, 19 in the second, and 8 in the third for a total of 76 final items. Experts agreed that privileges should be granted based on video review of surgical performance and attainment of clearly defined objective proficiency benchmarks. Parameters for ongoing outcome monitoring were determined and recommendations for technical skills training, proctoring, and performance assessment were defined.

CONCLUSIONS

Using a systematic approach, detailed credentialing criteria for robotic surgery were defined. Implementation of these criteria uniformly across institutions will promote proficiency of robotic surgeons and has the potential to positively impact patient outcomes.

Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience

Automated performance metrics objectively measure surgeon performance during a robot-assisted radical prostatectomy. Machine learning has demonstrated that automated performance metrics, especially during the vesico-urethral anastomosis of the robot-assisted radical prostatectomy, are predictive of long-term outcomes such as continence recovery time. This study focuses on automated performance metrics during the vesico-urethral anastomosis, specifically on stitch versus sub-stitch levels, to distinguish surgeon experience. During the vesico-urethral anastomosis, automated performance metrics, recorded by a systems data recorder (Intuitive Surgical, Sunnyvale, CA, USA), were reported for each overall stitch (C^total) and its individual components: needle handling/targeting (C¹), needle driving (C²), and suture cinching (C³) (Fig 1, A). These metrics were organized into three datasets (GlobalSet [whole stitch], RowSet [independent sub-stitches], and ColumnSet [associated sub-stitches] (Fig 1, B) and applied to three machine learning models (AdaBoost, gradient boosting, and random forest) to solve two classifications tasks: experts (≥100 cases) versus novices (<100 cases) and ordinary experts (≥100 and <2,000 cases) versus super experts (≥2,000 cases). Classification accuracy was determined using analysis of variance. Input features were evaluated through a Jaccard index. From 68 vesico-urethral anastomoses, we analyzed 1,570 stitches broken down into 4,708 sub-stitches. For both classification tasks, ColumnSet best distinguished experts (n = 8) versus novices (n = 9) and ordinary experts (n = 5) versus super experts (n = 3) at an accuracy of 0.774 and 0.844, respectively. Feature ranking highlighted Endowrist articulation and needle handling/targeting as most important in classification. Surgeon performance measured by automated performance metrics on a granular sub-stitch level more accurately distinguishes expertise when compared with summary automated performance metrics over whole stitches.

Development and validation of non-guided bladder-neck and neurovascular-bundle dissection modules of the RobotiX-Mentor® full-procedure robotic-assisted radical prostatectomy virtual reality simulation

Background

Full‐procedure virtual reality (VR) simulator training in robotic‐assisted radical prostatectomy (RARP) is a new tool in surgical education.

Methods

Description of the development of a VR RARP simulation model, (RobotiX‐Mentor®) including non‐guided bladder neck (ngBND) and neurovascular bundle dissection (ngNVBD) modules, and assessment of face, content, and construct validation of the ngBND and ngNVBD modules by robotic surgeons with different experience levels.

Results

Simulator and ngBND/ngNVBD modules were rated highly by all surgeons for realism and usability as training tool. In the ngBND‐task construct, validation was not achieved in task‐specific performance metrics. In the ngNVBD, task‐specific performance of the expert/intermediately experienced surgeons was significantly better than that of novices.

Conclusions

We proved face and content validity of simulator and both modules, and construct validity for generic metrics of the ngBND module and for generic and task‐specific metrics of the ngNVBD module.

How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

Introduction

A surgeon's technical skills are an important factor in delivering optimal patient care. Most existing methods to estimate technical skills remain subjective and resource intensive. Robotic-assisted surgery (RAS) provides a unique opportunity to develop objective metrics using key elements of intraoperative surgeon behavior which can be captured unobtrusively, such as instrument positions and button presses. Recent studies have shown that objective metrics based on these data (referred to as objective performance indicators [OPIs]) correlate to select clinical outcomes during robotic-assisted radical prostatectomy. However, the current OPIs remain difficult to interpret directly and, therefore, to use within structured feedback to improve surgical efficiencies.

Methods

We analyzed kinematic and event data from da Vinci surgical systems (Intuitive Surgical, Inc., Sunnyvale, CA, USA) to calculate values that can summarize the use of robotic instruments, referred to as OPIs. These indicators were mapped to broader technical skill categories of established training protocols. A data-driven approach was then applied to further sub-select OPIs that distinguish skill for each technical skill category within each training task. This subset of OPIs was used to build a set of logistic regression classifiers that predict the probability of expertise in that skill to identify targeted improvement and practice. The final, proposed feedback using OPIs was based on the coefficients of the logistic regression model to highlight specific actions that can be taken to improve.

Results

We determine that for the majority of skills, only a small subset of OPIs (2-10) are required to achieve the highest model accuracies (80-95%) for estimating technical skills within clinical-like tasks on a porcine model. The majority of the skill models have similar accuracy as models predicting overall expertise for a task (80-98%). Skill models can divide a prediction into interpretable categories for simpler, targeted feedback.

Conclusion

We define and validate a methodology to create interpretable metrics for key technical skills during clinical-like tasks when performing RAS. Using this framework for evaluating technical skills, we believe that surgical trainees can better understand both what can be improved and how to improve.

Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons’ skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson’s correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.

The what? How? And Who? Of video based assessment

BACKGROUND

Currently, there is significant variability in the development, implementation and overarching goals of video review for assessment of surgical performance.

METHODS

This paper evaluates the current methods in which video review is used for evaluation of surgical performance and identifies which processes are critical for successful, widespread implementation of video-based assessment.

RESULTS

Despite the advances in video capture technology and growing interest in video-based assessment, there is a notable gap in the implementation and longitudinal use of formative and summative assessment using video.

CONCLUSION

Validity, scalability and discoverability are current but removable barriers to video-based assessment.

Analysis of the video motion tracking system "Kinovea" to assess surgical movements during robot-assisted radical prostatectomy

BACKGROUNDS

Robot-assisted surgery facilitated the possibility to evaluate the surgeon's skills by recording and evaluating the robot surgical images. The aim of this study was to investigate the possibility of using a computer programme (Kinovea) for objective assessment of surgical movements in previously recorded in existing robot-assisted radical prostatectomy (RARP) videos.

METHODS

Twelve entire RARP procedures were analysed by a trained researcher using the computer programme "Kinovea" to perform semi-automated assessment of surgical movements.

RESULTS

Data analysis showed Kinovea was on average able to automatically assess only 22% of the total surgical duration per video of the robot-assisted surgery. On average, it lasted 4 hours of continued monitoring by the researcher to assess one RARP using Kinovea.

CONCLUSION

Although we proved it is technically possible to use the Kinovea system in retrospective analysis of surgical movement in robot-assisted surgery, the acquired data do not give a comprehensive enough analysis of the video to be used in skills assessment.

Comparison of clinical outcomes and automated performance metrics in robot-assisted radical prostatectomy with and without trainee involvement

PURPOSE

In this study, we investigate the effect of trainee involvement on surgical performance, as measured by automated performance metrics (APMs), and outcomes after robot-assisted radical prostatectomy (RARP).

METHODS

We compared APMs (instrument tracking, EndoWrist® articulation, and system events data) and clinical outcomes for cases with varying resident involvement. Four of 12 standardized RARP steps were designated critical ("cardinal") steps. Comparison 1: cases where the attending surgeon performed all four cardinal steps (Group A) and cases where a trainee was involved in at least one cardinal step (Group B). Comparison 2, where Group A is split into Groups C and D: cases where attending performs the whole case (Group C) vs. cases where a trainee performed at least one non-cardinal step (Group D). Mann-Whitney U and Chi-squared tests were used for comparisons.

RESULTS

Comparison 1 showed significant differences in APM profiles including camera movement time, third instrument usage, dominant instrument moving time, velocity, articulation, as well as non-dominant instrument moving time and articulation (all favoring Group A p < 0.05). There was a significant difference in re-admission rates (10.9% in Group A vs 0% in Group B, p < 0.02), but not for post-operative outcomes. Comparison 2 demonstrated a significant difference in dominant instrument articulation (p < 0.05) but not in post-operative outcomes.

CONCLUSIONS

Trainee involvement in RARP is safe. The degree of trainee involvement does not significantly affect major clinical outcomes. APM profiles are less efficient when trainees perform at least one cardinal step but not during non-cardinal steps.

Using video review to understand the technical variation of robot-assisted radical prostatectomy in a statewide surgical collaborative

PURPOSE

Video assessment is an emerging tool for understanding surgical technique. Patient outcomes after robot-assisted radical prostatectomy (RARP) may be linked to technical aspects of the procedure. In an effort to refine surgical approaches and improve outcomes, we sought to understand technical variation for the key steps of RARP in a surgical collaborative.

METHODS

The Michigan Urological Surgery Improvement Collaborative (MUSIC) is a statewide quality improvement collaborative with the aim of improving prostate cancer care. MUSIC surgeons were invited to submit representative complete videos of nerve-sparing RARP for blinded analysis. We also analyzed peri-operative outcomes from these surgeons in the registry.

RESULTS

Surgical video data from 20 unique surgeons identified many variations in technique and time to complete different steps. Common to all surgeons was a transperitoneal approach and a running urethrovesical anastomosis. Prior to anastomosis, 25% surgeons undertook a posterior reconstruction and 30% employed urethral suspension. 65% surgeons approached the seminal vesicle anteriorly. For control of the dorsal vein complex, suture ligation was used in 60%, and vascular stapler was 15%. The majority (80%) of surgeons employed clips for managing pedicles. In examining patient outcomes for surgeons, peri-operative outcomes were not correlated with surgeon's operative time; however, surgeons with an EBL > 400 ml had significant difference among the five different techniques employed.

CONCLUSIONS

Despite the worldwide popularity of RARP, the operation is still far from standardized. Correlating variation in technique with clinical outcomes may help provide objective data to support best practices with the goal to improve patient outcomes.

Using objective robotic automated performance metrics and task-evoked pupillary response to distinguish surgeon expertise

PURPOSE

In this study, we investigate the ability of automated performance metrics (APMs) and task-evoked pupillary response (TEPR), as objective measures of surgeon performance, to distinguish varying levels of surgeon expertise during generic robotic surgical tasks. Additionally, we evaluate the association between APMs and TEPR.

METHODS

Participants completed ten tasks on a da Vinci Xi Surgical System (Intuitive Surgical, Inc.), each representing a surgical skill type: EndoWrist^® manipulation, needle targeting, suturing/knot tying, and excision/dissection. Automated performance metrics (instrument motion tracking, EndoWrist^® articulation, and system events data) and TEPR were recorded by a systems data recorder (Intuitive Surgical, Inc.) and Tobii Pro Glasses 2 (Tobii Technologies, Inc.), respectively. The Kruskal-Wallis test determined significant differences between groups of varying expertise. Spearman's rank correlation coefficient measured associations between APMs and TEPR.

RESULTS

Twenty-six participants were stratified by robotic surgical experience: novice (no prior experience; n = 9), intermediate (< 100 cases; n = 9), and experts (≥ 100 cases; n = 8). Several APMs differentiated surgeon experience including task duration (p < 0.01), time active of instruments (p < 0.03), linear velocity of instruments (p < 0.04), and angular velocity of dominant instrument (p < 0.04). Task-evoked pupillary response distinguished surgeon expertise for three out of four task types (p < 0.04). Correlation trends between APMs and TEPR revealed that expert surgeons move more slowly with high cognitive workload (ρ < - 0.60, p < 0.05), while novices move faster under the same cognitive experiences (ρ > 0.66, p < 0.05).

CONCLUSIONS

Automated performance metrics and TEPR can distinguish surgeon expertise levels during robotic surgical tasks. Furthermore, under high cognitive workload, there can be a divergence in robotic movement profiles between expertise levels.

Effect of surgeon experience and bony pelvic dimensions on surgical performance and patient outcomes in robot-assisted radical prostatectomy

OBJECTIVES

To evaluate the effects of surgeon experience, body habitus, and bony pelvic dimensions on surgeon performance and patient outcomes after robot-assisted radical prostatectomy (RARP).

PATIENTS, SUBJECTS AND METHODS

The pelvic dimensions of 78 RARP patients were measured on preoperative magnetic resonance imaging and computed tomography by three radiologists. Surgeon automated performance metrics (APMs [instrument motion tracking and system events data, i.e., camera movement, third-arm swap, energy use]) were obtained by a systems data recorder (Intuitive Surgical, Sunnyvale, CA, USA) during RARP. Two analyses were performed: Analysis 1, examined effects of patient characteristics, pelvic dimensions and prior surgeon RARP caseload on APMs using linear regression; Analysis 2, the effects of patient body habitus, bony pelvic measurement, and surgeon experience on short- and long-term outcomes were analysed by multivariable regression.

RESULTS

Analysis 1 showed that while surgeon experience affected the greatest number of APMs (P < 0.044), the patient's body mass index, bony pelvic dimensions, and prostate size also affected APMs during each surgical step (P < 0.043, P < 0.046, P < 0.034, respectively). Analysis 2 showed that RARP duration was significantly affected by pelvic depth (β = 13.7, P = 0.039) and prostate volume (β = 0.5, P = 0.024). A wider and shallower pelvis was less likely to result in a positive margin (odds ratio 0.25, 95% confidence interval [CI] 0.09-0.72). On multivariate analysis, urinary continence recovery was associated with surgeon's prior RARP experience (hazard ratio [HR] 2.38, 95% CI 1.18-4.81; P = 0.015), but not on pelvic dimensions (HR 1.44, 95% CI 0.95-2.17).

CONCLUSION

Limited surgical workspace, due to a narrower and deeper pelvis, does affect surgeon performance and patient outcomes, most notably in longer surgery time and an increased positive margin rate.

Novel evaluation of surgical activity recognition models using task-based efficiency metrics

PURPOSE

Surgical task-based metrics (rather than entire procedure metrics) can be used to improve surgeon training and, ultimately, patient care through focused training interventions. Machine learning models to automatically recognize individual tasks or activities are needed to overcome the otherwise manual effort of video review. Traditionally, these models have been evaluated using frame-level accuracy. Here, we propose evaluating surgical activity recognition models by their effect on task-based efficiency metrics. In this way, we can determine when models have achieved adequate performance for providing surgeon feedback via metrics from individual tasks.

METHODS

We propose a new CNN-LSTM model, RP-Net-V2, to recognize the 12 steps of robotic-assisted radical prostatectomies (RARP). We evaluated our model both in terms of conventional methods (e.g., Jaccard Index, task boundary accuracy) as well as novel ways, such as the accuracy of efficiency metrics computed from instrument movements and system events.

RESULTS

Our proposed model achieves a Jaccard Index of 0.85 thereby outperforming previous models on RARP. Additionally, we show that metrics computed from tasks automatically identified using RP-Net-V2 correlate well with metrics from tasks labeled by clinical experts.

CONCLUSION

We demonstrate that metrics-based evaluation of surgical activity recognition models is a viable approach to determine when models can be used to quantify surgical efficiencies. We believe this approach and our results illustrate the potential for fully automated, postoperative efficiency reports.

Review of strategies and factors to maximize cost-effectiveness of robotic hysterectomies and myomectomies in benign gynecological disease

Common benign gynecologic procedures include hysterectomies and myomectomies, with hysterectomy being the most common gynecologic procedure in the United States [1]. While historically performed via laparotomy, the field of gynecologic surgery was revolutionized with the advent of laparoscopic techniques, with the most recent advancement being the introduction of robotic-assisted surgery in 2005. Robotic surgery has all the benefits of laparoscopic surgery such as decreased blood loss, quicker return to activities, and shorter length of hospital stay. Additional robotic-specific advantages include but are not limited to improved ergonomics, 3D visualization, and intuitive surgical movements. Despite these advantages, one of the most commonly cited drawbacks of robotic surgery is the associated cost. While the initial cost to purchase the robotic console and its associated maintenance costs are relatively high, robotic surgery can be cost-effective when utilized correctly.This article reviews application strategies and factors that can offset traditional costs and maximize the benefits of robotic surgery.