Association of Surgical Skill Assessment With Clinical Outcomes in Cancer Surgery

Development and evaluation of a societal core robotic surgery accreditation curriculum for the UK

Standardised proficiency-based progression is the cornerstone of safe robotic skills acquisition, however, is currently lacking within surgical training curricula. Expert consensuses have defined a modular pathway to accredit surgeons. This study aimed to address the lack of a formal, pre-clinical core robotic skills, proficiency-based accreditation curriculum in the UK. Novice robotic participants underwent a four-day pre-clinical core robotic skills curriculum incorporating multimodal assessment. Modifiable-Global Evaluative Assessment of Robotic Skills (M-GEARS), VR-automated performance metrics (APMs) and Objective Clinical Human Reliability Analysis (OCHRA) error methodology assessed performance at the beginning and end of training. Messick's validity concept and a curriculum evaluation model were utilised. Feedback was collated. Proficiency-based progression, benchmarking, tool validity and reliability was assessed through comparative and correlational statistical methods. Forty-seven participants were recruited. Objective assessment of VR and dry models across M-GEARS, APMs and OCHRA demonstrated significant improvements in technical skill (p < 0.001). Concurrent validity between assessment tools demonstrated strong correlation in dry and VR tasks (r = 0.64-0.92, p < 0.001). OCHRA Inter-rater reliability was excellent (r = 0.93, p < 0.001 and 81% matched error events). A benchmark was established with M-GEARS and for the curriculum at 80%. Thirty (63.82%) participants passed. Feedback was 5/5 stars on average, with 100% recommendation. Curriculum evaluation fulfilled all five domains of Messick's validity. Core robotic surgical skills training can be objectively evaluated and benchmarked to provide accreditation in basic robotic skills. A strategy is necessary to enrol standardised curricula into national surgical training at an early stage to ensure patient safety.

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review

INTRODUCTION

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity.

MATERIAL AND METHODS

A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis.

RESULTS

A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes.

CONCLUSIONS

Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Validation of a novel, low-fidelity virtual reality simulator and an artificial intelligence assessment approach for peg transfer laparoscopic training

Simulators are widely used in medical education, but objective and automatic assessment is not feasible with low-fidelity simulators, which can be solved with artificial intelligence (AI) and virtual reality (VR) solutions. The effectiveness of a custom-made VR simulator and an AI-based evaluator of a laparoscopic peg transfer exercise was investigated. Sixty medical students were involved in a single-blinded randomised controlled study to compare the VR simulator with the traditional box trainer. A total of 240 peg transfer exercises from the Fundamentals of Laparoscopic Surgery programme were analysed. The experts and AI-based software used the same criteria for evaluation. The algorithm detected pitfalls and measured exercise duration. Skill improvement showed no significant difference between the VR and control groups. The AI-based evaluator exhibited 95% agreement with the manual assessment. The average difference between the exercise durations measured by the two evaluation methods was 2.61 s. The duration of the algorithmic assessment was 59.47 s faster than the manual assessment. The VR simulator was an effective alternative practice compared with the training box simulator. The AI-based evaluation produced similar results compared with the manual assessment, and it could significantly reduce the evaluation time. AI and VR could improve the effectiveness of basic laparoscopic training.

Surgical quality assessment of critical view of safety in 283 laparoscopic cholecystectomy videos by surgical residents and surgeons

INTRODUCTION

Surgical quality assessment has improved the efficacy and efficiency of surgical training and has the potential to optimize the surgical learning curve. In laparoscopic cholecystectomy (LC), the critical view of safety (CVS) can be assessed with a 6-point competency assessment tool (CAT), a task commonly performed by experienced surgeons. The aim of this study is to determine the capability of surgical residents to perform this assessment.

METHODS

Both surgeons and surgical residents assessed unedited LC videos using a 6-point CVS, a CAT, using an online video assessment platform. The CAT consists of the following three criteria: 1. clearance of hepatocystic triangle, 2. cystic plate, and 3. two structures connect to the gallbladder, with a maximum of 2 points available for each criterion. A higher score indicates superior surgical performance. The intraclass correlation coefficient (ICC) was employed to assess the inter-rater reliability between surgeons and surgical residents.

RESULTS

In total, 283 LC videos were assessed by 19 surgeons and 31 surgical residents. The overall ICC for all criteria was 0.628. Specifically, the ICC scores were 0.504 for criterion 1, 0.639 for criterion 2, and 0.719 for the criterion involving the two structures connected to the gallbladder. Consequently, only the criterion regarding clearance of the hepatocystic triangle exhibited fair agreement, whereas the other two criteria, as well as the overall scores, demonstrated good agreement. In 71% of cases, both surgeons and surgical residents scored a total score either ranging from 0 to 4 or from 5 to 6.

CONCLUSION

Compared to the gold standard, i.e., the surgeons' assessments, surgical residents are equally skilled at assessing critical view of safety (CVS) in laparoscopic cholecystectomy (LC) videos. By incorporating video-based assessments of surgical procedures into their training, residents could potentially enhance their learning pace, which may result in better clinical outcomes.

International consensus panel for transurethral resection of bladder tumours metrics: assessment of face and content validity

OBJECTIVE

To develop performance metrics that objectively define a reference approach to a transurethral resection of bladder tumours (TURBT) procedure, seek consensus on the performance metrics from a group of international experts.

METHODS

The characterisation of a reference approach to a TURBT procedure was performed by identifying phases and explicitly defined procedure events (i.e., steps, errors, and critical errors). An international panel of experienced urologists (i.e., Delphi panel) was then assembled to scrutinise the metrics using a modified Delphi process. Based on the panel's feedback, the proposed metrics could be edited, supplemented, or deleted. A voting process was conducted to establish the consensus level on the metrics. Consensus was defined as the panel majority (i.e., >80%) agreeing that the metric definitions were accurate and acceptable. The number of metric units before and after the Delphi meeting were presented.

RESULTS

A core metrics group (i.e., characterisation group) deconstructed the TURBT procedure. The reference case was identified as an elective TURBT on a male patient, diagnosed after full diagnostic evaluation with three or fewer bladder tumours of ≤3 cm. The characterisation group identified six procedure phases, 60 procedure steps, 43 errors, and 40 critical errors. The metrics were presented to the Delphi panel which included 15 experts from six countries. After the Delphi, six procedure phases, 63 procedure steps, 47 errors, and 41 critical errors were identified. The Delphi panel achieved a 100% consensus.

CONCLUSION

Performance metrics to characterise a reference approach to TURBT were developed and an international panel of experts reached 100% consensus on them. This consensus supports their face and content validity. The metrics can now be used for a proficiency-based progression training curriculum for TURBT.

Current trends and barriers to video management and analytics as a tool for surgeon skilling

BACKGROUND

The benefits of intraoperative recording are well published in the literature; however, few studies have identified current practices, barriers, and subsequent solutions. The objective of this study was to better understand surgeon's current practices and perceptions of video management and gather blinded feedback on a new surgical video recording product with the potential to address these barriers effectively.

METHODS

A structured questionnaire was used to survey 230 surgeons (general, gynecologic, and urologic) and hospital administrators across the US and Europe regarding their current video recording practices. The same questionnaire was used to evaluate a blinded concept describing a new intraoperative recording solution.

RESULTS

54% of respondents reported recording eligible cases, with the majority recording less than 35% of their total eligible caseload. Reasons for not recording included finding no value in recording simple procedures, forgetting to record, lack of access to equipment, legal concerns, labor intensity, and difficulty accessing videos. Among non-recording surgeons, 65% reported considering recording cases to assess surgical techniques, document practice, submit to conferences, share with colleagues, and aid in training. 35% of surgeons rejected recording due to medico-legal concerns, lack of perceived benefit, concerns about secure storage, and price. Regarding the concept of a recording solution, 74% of all respondents were very likely or quite likely to recommend the product for adoption at their facility. Appealing features to current recorders included the product's ease of use, use of AI to maintain patient and staff privacy, lack of manual downloads, availability of full-length procedural videos, and ease of access and storage. Non-recorders found the immediate access to videos and maintenance of patient/staff privacy appealing.

CONCLUSION

Tools that address barriers to recording, accessing, and managing surgical case videos are critical for improving surgical skills. Touch Surgery Enterprise is a valuable tool that can help overcome these barriers.

The utility of synoptic operation reports in colorectal surgery: a systematic review

PURPOSE

Accurate documentation is crucial in surgical patient care. Synoptic reports (SR) are structured checklist-based reports that offer a standardised alternative to traditional narrative reports (NR). This systematic review aims to assess the completeness of SR compared to NR in colorectal cancer (CRC) surgery. Secondary outcomes include the time to completion, surgeon satisfaction, educational value, research value, and barriers to implementation.

METHODS

Prospective or retrospective studies that assessed SR compared to NR in colorectal cancer surgery procedures were identified through a systematic search of Ovid MEDLINE, Embase (Ovid), CIHNAL Plus with Full Text (EBSCOhost), and Cochrane. One thousand two articles were screened, and eight studies met the inclusion criteria after full-text review of 17 papers.

RESULTS

Analysis included 1797 operative reports (NR, 729; SR, 1068). Across studies reporting this outcome, the completeness of documentation was significantly higher in SR (P < 0.001). Reporting of secondary outcomes was limited, with a predominant focus on research value. Several studies demonstrated significantly reduced data extraction times when utilising SR. Surgeon satisfaction with SR was high, and these reports were seen as valuable tools for research and education. Barriers to implementation included integrating SR into existing electronic medical records (EMR) and surgeon concerns regarding increased administrative burden.

CONCLUSIONS

SR offer advantages in completeness, data extraction, and communication compared to NR. Surgeons perceive them as beneficial for research, quality improvement, and teaching. This review supports the necessity for development of user-friendly SR that seamlessly integrate into pre-existing EMRs, optimising patient care and enhancing the quality of CRC surgical documentation.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

Adoption of routine surgical video recording: a nationwide freedom of information act request across England and Wales

Background

Surgical video contains data with significant potential to improve surgical outcome assessment, quality assurance, education, and research. Current utilisation of surgical video recording is unknown and related policies/governance structures are unclear.

Methods

A nationwide Freedom of Information (FOI) request concerning surgical video recording, technology, consent, access, and governance was sent to all acute National Health Service (NHS) trusts/boards in England/Wales between 20th February and 20th March 2023.

Findings

140/144 (97.2%) trusts/boards in England/Wales responded to the FOI request. Surgical procedures were routinely recorded in 22 trusts/boards. The median estimate of consultant surgeons routinely recording their procedures was 20%. Surgical video was stored on internal systems (n = 27), third-party products (n = 29), and both (n = 9). 32/140 (22.9%) trusts/boards ask for consent to record procedures as part of routine care. Consent for recording included non-clinical purposes in 55/140 (39.3%) trusts/boards. Policies for surgeon/patient access to surgical video were available in 48/140 (34.3%) and 32/140 (22.9%) trusts/boards, respectively. Surgical video was used for non-clinical purposes in 64/140 (45.7%) trusts/boards. Governance policies covering surgical video recording, use, and/or storage were available from 59/140 (42.1%) trusts/boards.

Interpretation

There is significant heterogeneity in surgical video recording practices in England and Wales. A minority of trusts/boards routinely record surgical procedures, with large variation in recording/storage practices indicating scope for NHS-wide coordination. Revision of surgical video consent, accessibility, and governance policies should be prioritised by trusts/boards to protect key stakeholders. Increased availability of surgical video is essential for patients and surgeons to maximally benefit from the ongoing digital transformation of surgery.

Funding

KL is supported by an NIHR Academic Clinical Fellowship and acknowledges infrastructure support for this research from the National Institute for Health Research (NIHR) Imperial Biomedical Research Centre (BRC).

Nationwide Association of Surgical Performance of Minimally Invasive Esophagectomy With Patient Outcomes

Importance

Suboptimal surgical performance is hypothesized to be associated with less favorable patient outcomes in minimally invasive esophagectomy (MIE). Establishing this association may lead to programs that promote better surgical performance of MIE and improve patient outcomes.

Objective

To investigate associations between surgical performance and postoperative outcomes after MIE.

Design, Setting, and Participants

In this nationwide cohort study of 15 Dutch hospitals that perform more than 20 MIEs per year, 7 masked expert MIE surgeons assessed surgical performance using videos and a previously developed and validated competency assessment tool (CAT). Each hospital submitted 2 representative videos of MIEs performed between November 4, 2021, and September 13, 2022. Patients registered in the Dutch Upper Gastrointestinal Cancer Audit between January 1, 2020, and December 31, 2021, were included to examine patient outcomes.

Exposure

Hospitals were divided into quartiles based on their MIE-CAT performance score. Outcomes were compared between highest (top 25%) and lowest (bottom 25%) performing quartiles. Transthoracic MIE with gastric tube reconstruction.

Main Outcome and Measure

The primary outcome was severe postoperative complications (Clavien-Dindo ≥3) within 30 days after surgery. Multilevel logistic regression, with clustering of patients within hospitals, was used to analyze associations between performance and outcomes.

Results

In total, 30 videos and 970 patients (mean [SD] age, 66.6 [9.1] years; 719 men [74.1%]) were included. The mean (SD) MIE-CAT score was 113.6 (5.5) in the highest performance quartile vs 94.1 (5.9) in the lowest. Severe postoperative complications occurred in 18.7% (41 of 219) of patients in the highest performance quartile vs 39.2% (40 of 102) in the lowest (risk ratio [RR], 0.50; 95% CI, 0.24-0.99). The highest vs the lowest performance quartile showed lower rates of conversions (1.8% vs 8.9%; RR, 0.21; 95% CI, 0.21-0.21), intraoperative complications (2.7% vs 7.8%; RR, 0.21; 95% CI, 0.04-0.94), and overall postoperative complications (46.1% vs 65.7%; RR, 0.54; 95% CI, 0.24-0.96). The R0 resection rate (96.8% vs 94.2%; RR, 1.03; 95% CI, 0.97-1.05) and lymph node yield (mean [SD], 38.9 [14.7] vs 26.2 [9.0]; RR, 3.20; 95% CI, 0.27-3.21) increased with oncologic-specific performance (eg, hiatus dissection, lymph node dissection). In addition, a high anastomotic phase score was associated with a lower anastomotic leakage rate (4.6% vs 17.7%; RR, 0.14; 95% CI, 0.06-0.31).

Conclusions and Relevance

These findings suggest that better surgical performance is associated with fewer perioperative complications for patients with esophageal cancer on a national level. If surgical performance of MIE can be improved with MIE-CAT implementation, substantially better patient outcomes may be achievable.

Evaluation of the prognostic performance of different cutoff values of lymph node ratio staging system for stage III colorectal cancer

This study attempted to compare the prognostic performance of lymph node ratio (LNR) staging system with different cutoff values relative to American Joint Committee on Cancer (AJCC) pN staging system in stage III colorectal cancer (CRC). Overall, 45,069 patients from the SEER dataset and 69 patients from the Second Affiliated Hospital of Nanjing Medical University (the External set) who underwent surgical resection of the primary tumor and were diagnosed with stage III CRC by postoperative pathology were included. Patients were divided into three subgroups based on the LNR cutoff used in previous studies, Kaplan-Meier curves were plotted, and log-rank test was used to compare the differences among groups in terms of cancer-specific survival (CSS). Cox regression model was applied for survival analysis. To evaluate the discriminatory power of different lymph node staging systems, Harrell's C statistic(C-index) and Akaike's Information Criterion (AIC) were applied. A set of optimal cutoff values (0.11; 0.36; 0.66) of LNR staging system with the most considerable discriminatory power to the prognosis in patients with stage III CRC (SEER set: C-index = 0.714; AIC = 58,942.46, External set: C-index = 0.809; AIC = 164.36) were obtained, and both were superior to the AJCC pN staging system (SEER set: C-index = 0.708; AIC = 59,071.20, External set: C-index = 0.788; AIC = 167.06). For evaluating the prognostic efficacy of patients with stage III colorectal cancer, the cutoff value (0.11; 0.36; 0.66) of LNR staging system had the best discrimination and prognostic ability, which was superior to LNR staging system under other cutoff values and AJCC pN staging system.

A Video-Based Procedure-Specific Competency Assessment Tool for Minimally Invasive Esophagectomy

Importance

Minimally invasive esophagectomy (MIE) is a complex procedure with substantial learning curves. In other complex minimally invasive procedures, suboptimal surgical performance has convincingly been associated with less favorable patient outcomes as assessed by peer review of the surgical procedure.

Objective

To develop and validate a procedure-specific competency assessment tool (CAT) for MIE.

Design, Setting, and Participants

In this international quality improvement study, a procedure-specific MIE-CAT was developed and validated. The MIE-CAT contains 8 procedural phases, and 4 quality components per phase are scored with a Likert scale ranging from 1 to 4. For evaluation of the MIE-CAT, intraoperative MIE videos performed by a single surgical team in the Esophageal Center East Netherlands were peer reviewed by 18 independent international MIE experts (with more than 120 MIEs performed). Each video was assessed by 2 or 3 blinded experts to evaluate feasibility, content validity, reliability, and construct validity. MIE-CAT version 2 was composed with refined content aimed at improving interrater reliability. A total of 32 full-length MIE videos from patients who underwent MIE between 2011 and 2020 were analyzed. Data were analyzed from January 2021 to January 2023.

Exposure

Performance assessment of transthoracic MIE with an intrathoracic anastomosis.

Main Outcomes and Measures

Feasibility, content validity, interrater and intrarater reliability, and construct validity, including correlations with both experience of the surgical team and clinical parameters, of the developed MIE-CAT.

Results

Experts found the MIE-CAT easy to understand and easy to use to grade surgical performance. The MIE-CAT demonstrated good intrarater reliability (range of intraclass correlation coefficients [ICCs], 0.807 [95% CI, 0.656 to 0.892] for quality component score to 0.898 [95% CI, 0.846 to 0.932] for phase score). Interrater reliability was moderate (range of ICCs, 0.536 [95% CI, -0.220 to 0.994] for total MIE-CAT score to 0.705 [95% CI, 0.473 to 0.846] for quality component score), and most discrepancies originated in the lymphadenectomy phases. Hypothesis testing for construct validity showed more than 75% of hypotheses correct: MIE-CAT performance scores correlated with experience of the surgical team (r = 0.288 to 0.622), blood loss (r = -0.034 to -0.545), operative time (r = -0.309 to -0.611), intraoperative complications (r = -0.052 to -0.319), and severe postoperative complications (r = -0.207 to -0.395). MIE-CAT version 2 increased usability. Interrater reliability improved but remained moderate (range of ICCs, 0.666 to 0.743), and most discrepancies between raters remained in the lymphadenectomy phases.

Conclusions and Relevance

The MIE-CAT was developed and its feasibility, content validity, reliability, and construct validity were demonstrated. By providing insight into surgical performance of MIE, the MIE-CAT might be used for clinical, training, and research purposes.

Free-Viewpoint Video in Open Surgery: Development of Surgical Arena 360

OBJECTIVE

We aimed to apply the free-viewpoint video technology developed and introduced mainly for sports spectators to an open surgical video recording system.

DESIGN

Prospective feasibility study.

SETTING

University of Tsukuba Hospital, Ibaraki, Japan.

PARTICIPANTS

Patients who underwent open pancreaticoduodenectomy for pancreatic cancer between December 2022 and March 2023 were included. The gastrojejunal anastomosis was the subject of the recording.

RESULTS

Four surgeries were recorded with Surgical Arena 360, which is the free-viewpoint video system that we developed. The feasibility of performing a series of surgical procedures without interrupting the surgeon's line of sight or manipulation was demonstrated in all cases.

CONCLUSIONS

Our study revealed that Surgical Arena 360, an open surgical video recording system developed by applying free-viewpoint video technology, can provide new insights into surgical support and clinical knowledge to surgeons by enabling secure capture of the open surgical field from multiple angles.

Kinematic metrics and surgeon experience in robotic cholecystectomies: a pilot study on breaking down technical performance

BACKGROUND

Recent studies have correlated surgical skill measured by video-based assessment with improved clinical outcomes. Certain automated measures of operative performance in robotic surgery can be gathered beyond video review called objective performance indicators (OPIs). We explore the relationship between OPIs, surgeon experience, and postoperative recovery, hypothesizing that more efficient dissection will be associated with experience.

METHODS

Fifty-six robotic cholecystectomies between February 2022 and March 2023 were recorded at a large tertiary referral center. Surgeon experience and clinical outcomes data from the EMR were obtained for all 56 cases with 10 completing the QOL survey. Two steps of robotic cholecystectomies were reviewed: dissection of Calot's triangle (DCT) and dissection of the gallbladder from the liver (DGL). Postoperative recovery was measured using the SF-36 well-being survey. Univariate analysis was conducted using Pearson's coefficient.

RESULTS

Increased operative experience was associated with more efficient camera and instrument movements. DCT had 7 and DGL had 31 of 41 OPIs that correlated with experience. With respect to DGL, more experienced surgeons had reduced step duration and instrument path length and increased camera and instrument speeds.

CONCLUSIONS

Several OPIs correlate with surgical experience and may form the basis of more instructive feedback for trainees and less experienced surgeons in improving intraoperative technique.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Nationwide standardization of minimally invasive right hemicolectomy for colon cancer and development and validation of a video-based competency assessment tool (the Right study)

BACKGROUND

Substantial variation exists when performing a minimally invasive right hemicolectomy (MIRH) due to disparities in training, expertise and differences in implementation of innovations. This study aimed to achieve national consensus on an optimal and standardized MIRH technique for colon cancer and to develop and validate a video-based competency assessment tool (CAT) for MIRH.

METHOD

Statements covering all elements of MIRH were formulated. Subsequently, the Delphi technique was used to reach consensus on a standardized MIRH among 76 colorectal surgeons from 43 different centres. A CAT was developed based on the Delphi results. Nine surgeons assessed the same 12 unedited full-length videos using the CAT, allowing evaluation of the intraclass correlation coefficient (ICC).

RESULTS

After three Delphi rounds, consensus (≥80% agreement) was achieved on 23 of the 24 statements. Consensus statements included the use of low intra-abdominal pressure, detailed anatomical outline how to perform complete mesocolic excision with central vascular ligation, the creation of an intracorporeal anastomosis, and specimen extraction through a Pfannenstiel incision using a wound protector. The CAT included seven consecutive steps to measure competency of the MIRH and showed high consistency among surgeons with an overall ICC of 0.923.

CONCLUSION

Nationwide consensus on a standardized and optimized technique of MIRH was reached. The CAT developed showed excellent interrater reliability. These achievements are crucial steps to an ongoing nationwide quality improvement project (the Right study).

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills

BACKGROUND

Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment.

METHODS

Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed.

RESULTS

OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001).

CONCLUSION

OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.

Laparoscopic but not open surgical skills can be transferred to robot-assisted surgery: A systematic review and meta-analysis

BACKGROUND

With an increase in robot-assisted surgery across all specialties, adequate training and credentialing strategies need to be identified to ensure patients safety. The meta-analysis assesses the transferability of technical surgical skills between laparoscopic surgery, open surgery, and robot-assisted surgery.

DESIGN

A systematic search was conducted in Medline, Cochrane Central Register of Controlled Trials, and Web of Science. Outcomes were categorized into time, process, product, and composite outcome measures and pooled separately using Hedges'g (standardized mean difference [SMD]). Subgroup analyses were performed to assess the effect of study design, virtual reality platforms and task difficulty.

RESULTS

Out of 14,120 screened studies, 30 were included in the qualitative synthesis and 26 in the quantitative synthesis. Technical surgical skill transfer was demonstrated from laparoscopic to robot-assisted surgery (composite: SMD 0.40, 95%-confidence interval [CI] [0.19; 0.62], time: SMD 0.62, CI [0.33; 0.91]) and vice versa (composite: SMD 0.66, CI [0.33; 0.99], time [basic skills]: SMD 0.36, CI [0.01; 0.72]). No skill transfer was seen from open to robot-assisted surgery with limited available data.

CONCLUSION

Technical surgical skills can be transferred from laparoscopic to robot-assisted surgery and vice versa. Robot-assisted and laparoscopic surgical skills training and credentialing should not be regarded separately, but a reasonable combination could shorten overall training times and increase efficiency. Previous experience in open surgery should not be considered as an imperative prerequisite for training in robot-assisted surgery. Recommendations for studies assessing skill transfer are proposed to increase comparability and significance of future studies.

PROSPERO REGISTRATION NUMBER

PROSPERO CRD42018104507.

Powerful medical education improves health care quality and return on investment

INTRODUCTION

Powerful medical education (PME) involves the use of new technologies informed by the science of expertise that are embedded in laboratories and organizations that value evidence-based education and support innovation. This contrasts with traditional medical education that relies on a dated apprenticeship model that yields uneven results. PME involves an amalgam of features, conditions and assumptions, and contextual variables that comprise an approach to developing clinical competence grounded in education impact metrics including efficiency and cost-effectiveness.

METHODS

This article is a narrative review based on SANRA criteria and informed by realist review principles. The review addresses the PME model with an emphasis on mastery learning and deliberate practice principles drawn from the new science of expertise. Pub Med, Scopus, and Web of Science search terms include medical education, the science of expertise, mastery learning, translational outcomes, cost effectiveness, and return on investment. Literature coverage is comprehensive with selective citations.

RESULTS

PME is described as an integrated set of twelve features embedded in a group of seven conditions and assumptions and four context variables. PME is illustrated via case examples that demonstrate improved ventilator patient management learning outcomes compared to traditional clinical education and mastery learning of breaking bad news communication skills. Evidence also shows that PME of physicians and other health care providers can have translational, downstream effects on patient care practices, patient outcomes, and return on investment. Several translational health care quality improvements that derive from PME include reduced infections; better communication among physicians, patients, and families; exceptional birth outcomes; more effective patient education; and return on investment.

CONCLUSIONS

The article concludes with challenges to hospitals, health systems, and medical education organizations that are responsible for producing physicians who are expected to deliver safe, effective, and cost-conscious health care.

Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos

BACKGROUND

Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model.

METHODS

Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score.

RESULTS

The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04).

CONCLUSION

The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.

Use of artificial intelligence in total mesorectal excision in rectal cancer surgery: State of the art and perspectives

BACKGROUND

Colorectal cancer is a major public health problem, with 1.9 million new cases and 953000 deaths worldwide in 2020. Total mesorectal excision (TME) is the standard of care for the treatment of rectal cancer and is crucial to prevent local recurrence, but it is a technically challenging surgery. The use of artificial intelligence (AI) could help improve the performance and safety of TME surgery.

AIM

To review the literature on the use of AI and machine learning in rectal surgery and potential future developments.

METHODS

Online scientific databases were searched for articles on the use of AI in rectal cancer surgery between 2020 and 2023.

RESULTS

The literature search yielded 876 results, and only 13 studies were selected for review. The use of AI in rectal cancer surgery and specifically in TME is a rapidly evolving field. There are a number of different AI algorithms that have been developed for use in TME, including algorithms for instrument detection, anatomical structure identification, and image-guided navigation systems.

CONCLUSION

AI has the potential to revolutionize TME surgery by providing real-time surgical guidance, preventing complications, and improving training. However, further research is needed to fully understand the benefits and risks of AI in TME surgery.

Global Cancer Surgery: pragmatic solutions to improve cancer surgery outcomes worldwide

The first Lancet Oncology Commission on Global Cancer Surgery was published in 2015 and serves as a landmark paper in the field of cancer surgery. The Commission highlighted the burden of cancer and the importance of cancer surgery, while documenting the many inadequacies in the ability to deliver safe, timely, and affordable cancer surgical care. This Commission builds on the first Commission by focusing on solutions and actions to improve access to cancer surgery globally, developed by drawing upon the expertise from cancer surgery leaders across the world. We present solution frameworks in nine domains that can improve access to cancer surgery. These nine domains were refined to identify solutions specific to the six WHO regions. On the basis of these solutions, we developed eight actions to propel essential improvements in the global capacity for cancer surgery. Our initiatives are broad in scope, pragmatic, affordable, and contextually applicable, and aimed at cancer surgeons as well as leaders, administrators, elected officials, and health policy advocates. We envision that the solutions and actions contained within the Commission will address inequities and promote safe, timely, and affordable cancer surgery for every patient, regardless of their socioeconomic status or geographic location.

Best Evidence for Each Surgical Step in Minimally Invasive Right Hemicolectomy: A Systematic Review

Objective

The aim of this study was to systematically review the literature for each surgical step of the minimally invasive right hemicolectomy (MIRH) for non-locally advanced colon cancer, to define the most optimal procedure with the highest level of evidence.

Background

High variability exists in the way MIRH is performed between surgeons and hospitals, which could affect patients' postoperative and oncological outcomes.

Methods

A systematic search using PubMed was performed to first identify systematic reviews and meta-analyses, and if there were none then landmark papers and consensus statements were systematically searched for each key step of MIRH. Systematic reviews were assessed using the AMSTAR-2 tool, and selection was based on highest quality followed by year of publication.

Results

Low (less than 12 mmHg) intra-abdominal pressure (IAP) gives higher mean quality of recovery compared to standard IAP. Complete mesocolic excision (CME) is associated with lowest recurrence and highest 5-year overall survival rates, without worsening short-term outcomes. Routine D3 versus D2 lymphadenectomy showed higher LN yield, but more vascular injuries, and no difference in overall and disease-free survival. Intracorporeal anastomosis is associated with better intra- and postoperative outcomes. The Pfannenstiel incision gives the lowest chance of incisional hernias compared to all other extraction sites.

Conclusion

According to the best available evidence, the most optimal MIRH for colon cancer without clinically involved D3 nodes entails at least low IAP, CME with D2 lymphadenectomy, an intracorporeal anastomosis and specimen extraction through a Pfannenstiel incision.

National implementation of an optimal standardised technique for right-sided colon cancer: protocol of an interventional sequential cohort study (Right study)

PURPOSE

Minimally invasive right hemicolectomy (MIRH) is the cornerstone of treatment for patients with right-sided colon cancer. This operation has evolved during recent decades, with many innovations and improvements but this has also resulted in high variability of uptake with subsequent substantial variableness. The aim of this ongoing study is to identify current surgical variations, determine the most optimal and standardised MIRH and nationally train and implement that technique to improve short-term clinical and long-term oncological outcomes.

METHODS

The Right study is a national multicentre prospective interventional sequential cohort study. Firstly, current local practice was evaluated. Subsequently, a standardised surgical technique for right-sided colon cancer was determined using the Delphi consensus method, and this procedure was trained during hands-on courses. The standardised MIRH will be implemented with proctoring (implementation cohort), after which the performance will be monitored (consolidation cohort). Patients who will receive a minimally invasive (extended) right hemicolectomy for cT1-3N0-2M0 colon cancer will be included. The primary outcome is patient safety reflected in the 90-day overall complication rate according to the Clavien-Dindo classification. Secondary outcomes will include intraoperative complications, 90-day mortality rate, number of resected tumour-positive lymph nodes, completeness of mesocolic excision, surgical quality score, locoregional and distant recurrence and 5-year overall survival. A total number of 1095 patients (365 per cohort) will be included.

DISCUSSION

The Right study is designed to safely implement the best surgical practice concerning patients with right-sided colon cancer aiming to standardise and improve the surgical quality of MIRH at a national level.

TRIAL REGISTRATION

ClinicalTrials.gov: NCT04889456, May 2021.

Crowd-sourced and expert video assessment in minimally invasive esophagectomy

BACKGROUND

Video-based assessment by experts may structurally measure surgical performance using procedure-specific competency assessment tools (CATs). A CAT for minimally invasive esophagectomy (MIE-CAT) was developed and validated previously. However, surgeon's time is scarce and video assessment is time-consuming and labor intensive. This study investigated non-procedure-specific assessment of MIE video clips by MIE experts and crowdsourcing, collective surgical performance evaluation by anonymous and untrained laypeople, to assist procedure-specific expert review.

METHODS

Two surgical performance scoring frameworks were used to assess eight MIE videos. First, global performance was assessed with the non-procedure-specific Global Operative Assessment of Laparoscopic Skills (GOALS) of 64 procedural phase-based video clips < 10 min. Each clip was assessed by two MIE experts and > 30 crowd workers. Second, the same experts assessed procedure-specific performance with the MIE-CAT of the corresponding full-length video. Reliability and convergent validity of GOALS for MIE were investigated using hypothesis testing with correlations (experience, blood loss, operative time, and MIE-CAT).

RESULTS

Less than 75% of hypothesized correlations between GOALS scores and experience of the surgical team (r < 0.3), blood loss (r = - 0.82 to 0.02), operative time (r = - 0.42 to 0.07), and the MIE-CAT scores (r = - 0.04 to 0.76) were met for both crowd workers and experts. Interestingly, experts' GOALS and MIE-CAT scores correlated strongly (r = 0.40 to 0.79), while crowd workers' GOALS and experts' MIE-CAT scores correlations were weak (r = - 0.04 to 0.49). Expert and crowd worker GOALS scores correlated poorly (ICC ≤ 0.42).

CONCLUSION

GOALS assessments by crowd workers lacked convergent validity and showed poor reliability. It is likely that MIE is technically too difficult to assess for laypeople. Convergent validity of GOALS assessments by experts could also not be established. GOALS might not be comprehensive enough to assess detailed MIE performance. However, expert's GOALS and MIE-CAT scores strongly correlated indicating video clip (instead of full-length video) assessments could be useful to shorten assessment time.

Technical skill assessment in minimally invasive surgery using artificial intelligence: a systematic review

BACKGROUND

Technical skill assessment in surgery relies on expert opinion. Therefore, it is time-consuming, costly, and often lacks objectivity. Analysis of intraoperative data by artificial intelligence (AI) has the potential for automated technical skill assessment. The aim of this systematic review was to analyze the performance, external validity, and generalizability of AI models for technical skill assessment in minimally invasive surgery.

METHODS

A systematic search of Medline, Embase, Web of Science, and IEEE Xplore was performed to identify original articles reporting the use of AI in the assessment of technical skill in minimally invasive surgery. Risk of bias (RoB) and quality of the included studies were analyzed according to Quality Assessment of Diagnostic Accuracy Studies criteria and the modified Joanna Briggs Institute checklists, respectively. Findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement.

RESULTS

In total, 1958 articles were identified, 50 articles met eligibility criteria and were analyzed. Motion data extracted from surgical videos (n = 25) or kinematic data from robotic systems or sensors (n = 22) were the most frequent input data for AI. Most studies used deep learning (n = 34) and predicted technical skills using an ordinal assessment scale (n = 36) with good accuracies in simulated settings. However, all proposed models were in development stage, only 4 studies were externally validated and 8 showed a low RoB.

CONCLUSION

AI showed good performance in technical skill assessment in minimally invasive surgery. However, models often lacked external validity and generalizability. Therefore, models should be benchmarked using predefined performance metrics and tested in clinical implementation studies.

Research progress of calcium carbonate nanomaterials in cancer therapy: challenge and opportunity

Cancer has keeping the main threat to the health of human being. Its overall survival rate has shown rare substantial progress in spite of the improving diagnostic and treatment techniques for cancer in recent years. Indeed, such classic strategies for malignant tumor as surgery, radiation and chemotherapy have been developed and bring more hope to the patients, but still been accompanied by certain limitations, which include the challenge of managing large wound sizes, systemic toxic side effects, and harmful to the healthy tissues caused by imprecise alignment with tumors in radiotherapy. Furthermore, immunotherapy exhibits a limited therapeutic effect in advanced tumors which is reported only up to 25%-30%. The combination of nanomaterials and cancer treatment offers new hope for cancer patients, demonstrating strong potential in the field of medical research. Among the extensively utilized nanomaterials, calcium carbonate nanomaterials (CCNM) exhibit a broad spectrum of biomedical applications due to their abundant availability, cost-effectiveness, and exceptional safety profile. CCNM have the potential to elevate intracellular Ca2+ levels in tumor cells, trigger the mitochondrial damage and ultimately lead to tumor cell death. Moreover, compared with other types of nanomaterials, CCNM exhibit remarkable advantages as delivery systems owing to their high loading capacity, biocompatibility and biodegradability. The purpose of this review is to provide an overview of CCNM synthesis, focusing on summarizing its diverse roles in cancer treatment and the benefits and challenges associated with CCNM in cancer therapy. Hoping to present the significance of CCNM as for the clinical application, and summarize information for the design of CCNM and other types of nanomaterials in the future.

Automatic Surgical Skill Assessment System Based on Concordance of Standardized Surgical Field Development Using Artificial Intelligence

Importance

Automatic surgical skill assessment with artificial intelligence (AI) is more objective than manual video review-based skill assessment and can reduce human burden. Standardization of surgical field development is an important aspect of this skill assessment.

Objective

To develop a deep learning model that can recognize the standardized surgical fields in laparoscopic sigmoid colon resection and to evaluate the feasibility of automatic surgical skill assessment based on the concordance of the standardized surgical field development using the proposed deep learning model.

Design, Setting, and Participants

This retrospective diagnostic study used intraoperative videos of laparoscopic colorectal surgery submitted to the Japan Society for Endoscopic Surgery between August 2016 and November 2017. Data were analyzed from April 2020 to September 2022.

Interventions

Videos of surgery performed by expert surgeons with Endoscopic Surgical Skill Qualification System (ESSQS) scores higher than 75 were used to construct a deep learning model able to recognize a standardized surgical field and output its similarity to standardized surgical field development as an AI confidence score (AICS). Other videos were extracted as the validation set.

Main Outcomes and Measures

Videos with scores less than or greater than 2 SDs from the mean were defined as the low- and high-score groups, respectively. The correlation between AICS and ESSQS score and the screening performance using AICS for low- and high-score groups were analyzed.

Results

The sample included 650 intraoperative videos, 60 of which were used for model construction and 60 for validation. The Spearman rank correlation coefficient between the AICS and ESSQS score was 0.81. The receiver operating characteristic (ROC) curves for the screening of the low- and high-score groups were plotted, and the areas under the ROC curve for the low- and high-score group screening were 0.93 and 0.94, respectively.

Conclusions and Relevance

The AICS from the developed model strongly correlated with the ESSQS score, demonstrating the model's feasibility for use as a method of automatic surgical skill assessment. The findings also suggest the feasibility of the proposed model for creating an automated screening system for surgical skills and its potential application to other types of endoscopic procedures.

Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial

Importance

In cardiac device implant training, there is no common system to objectively assess trainees' ability to perform tasks at predetermined performance levels before in vivo practice; therefore, patients are potentially exposed to risks related to operators' early learning curve.

Objective

To assess the effect on implant performance quality of novel metrics-based simulation training to proficiency (proficiency-based progression [PBP]) vs traditional simulation-based training (SBT).

Design, Setting, and Participants

In this prospective randomized trial, conducted between March 8, 2022 and November 24, 2022, novice implanters were randomized (blinded) 1:1 to participate in an SBT curriculum (procedural knowledge e-learning and in-person simulation training) at an international skills training center, with proficiency demonstration requirements at each training stage for advancing (PBP approach) or without the requirements. Ultimately, trainees performed a cardiac resynchronization therapy (CRT) implant using virtual reality simulation. The procedure was video-recorded and subsequently scored using previously validated metrics by 2 independent assessors blinded to group. Physicians who had already implanted more than 20 pacemakers or defibrillators and fewer than 200 CRT systems as the first operator were eligible. Thirty-two implanters from 10 countries voluntarily enrolled in the training program and were randomized; 30 (15 per group) started and completed training. Data analysis was performed from November 27 to December 22, 2022.

Intervention

Training with PBP vs SBT.

Main Outcome and Measures

The primary outcome comprised 4 objectively assessed performance metrics derived from the video-recordings: number of procedural steps completed, errors, critical errors, and all errors combined.

Results

Baseline experience of the 30 participants (19 [63%] male; mean [SD] number of years in implant practice, 2.0 [1.8]; median [IQR] number of implanted pacemakers or defibrillators, 47.5 [30.0-115.0]; median [IQR] number of implanted CRT systems, 3.0 [1.25-10.0]) was similar between study groups. Compared with the SBT group, the PBP group completed 27% more procedural steps (median [IQR], 31 [30-32] vs 24 [22-27]; P < .001) and made 73% fewer errors (median [IQR], 2 [1-3] vs 7 [5-8]; P < .001), 84% fewer critical errors (median [IQR], 1 [0-1] vs 3 [3-5]; P < .001), and 77% fewer all errors combined (errors plus critical errors) (median [IQR], 3 [1-3] vs 11 [8-12]; P < .001); 14 of the 15 PBP trainees (93%) demonstrated the predefined target performance level vs 0 of the 15 SBT trainees.

Conclusions and Relevance

In this randomized trial, the PBP approach to novice implanter training generated superior objectively assessed performance vs SBT. If implemented broadly and systematically, PBP training may ensure safe and effective performance standards before trainees proceed to (supervised) in vivo practice. Future studies are needed to verify implications on procedure-related patient complications.

Trial Registration

ClinicalTrials.gov Identifier: NCT05952908.

SurgSmart: an artificial intelligent system for quality control in laparoscopic cholecystectomy: an observational study

BACKGROUND

The rate of bile duct injury in laparoscopic cholecystectomy (LC) continues to be high due to low critical view of safety (CVS) achievement and the absence of an effective quality control system. The development of an intelligent system enables the automatic quality control of LC surgery and, eventually, the mitigation of bile duct injury. This study aims to develop an intelligent surgical quality control system for LC and using the system to evaluate LC videos and investigate factors associated with CVS achievement.

MATERIALS AND METHODS

SurgSmart, an intelligent system capable of recognizing surgical phases, disease severity, critical division action, and CVS automatically, was developed using training datasets. SurgSmart was also applied in another multicenter dataset to validate its application and investigate factors associated with CVS achievement.

RESULTS

SurgSmart performed well in all models, with the critical division action model achieving the highest overall accuracy (98.49%), followed by the disease severity model (95.45%) and surgical phases model (88.61%). CVSI, CVSII, and CVSIII had an accuracy of 80.64, 97.62, and 78.87%, respectively. CVS was achieved in 4.33% in the system application dataset. In addition, the analysis indicated that surgeons at a higher hospital level had a higher CVS achievement rate. However, there was still considerable variation in CVS achievement among surgeons in the same hospital.

CONCLUSIONS

SurgSmart, the surgical quality control system, performed admirably in our study. In addition, the system's initial application demonstrated its broad potential for use in surgical quality control.

Video-based tools for surgical quality assessment of technical skills in laparoscopic procedures: a systematic review

BACKGROUND

Quality of surgery has substantial impact on both short- and long-term clinical outcomes. This stresses the need for objective surgical quality assessment (SQA) for education, clinical practice and research purposes. The aim of this systematic review was to provide a comprehensive overview of all video-based objective SQA tools in laparoscopic procedures and their validity to objectively assess surgical performance.

METHODS

PubMed, Embase.com and Web of Science were systematically searched by two reviewers to identify all studies focusing on video-based SQA tools of technical skills in laparoscopic surgery performed in a clinical setting. Evidence on validity was evaluated using a modified validation scoring system.

RESULTS

Fifty-five studies with a total of 41 video-based SQA tools were identified. These tools were used in 9 different fields of laparoscopic surgery and were divided into 4 categories: the global assessment scale (GAS), the error-based assessment scale (EBAS), the procedure-specific assessment tool (PSAT) and artificial intelligence (AI). The number of studies focusing on these four categories were 21, 6, 31 and 3, respectively. Twelve studies validated the SQA tool with clinical outcomes. In 11 of those studies, a positive association between surgical quality and clinical outcomes was found.

CONCLUSION

This systematic review included a total of 41 unique video-based SQA tools to assess surgical technical skills in various domains of laparoscopic surgery. This study suggests that validated SQA tools enable objective assessment of surgical performance with relevance for clinical outcomes, which can be used for training, research and quality improvement programs.

Assessment methods in laparoscopic colorectal surgery: a systematic review of available instruments

BACKGROUND

Laparoscopic surgery has become the golden standard for many procedures, requiring new skills and training methods. The aim of this review is to appraise literature on assessment methods for laparoscopic colorectal procedures and quantify these methods for implementation in surgical training.

MATERIALS AND METHODS

PubMed, Embase and Cochrane Central Register of Controlled Trials databases were searched in October 2022 for studies reporting learning and assessment methods for laparoscopic colorectal surgery. Quality was scored using the Downs and Black checklist. Included articles were categorized in procedure-based assessment methods and non-procedure-based assessment methods. A second distinction was made between capability for formative and/or summative assessment.

RESULTS

In this systematic review, nineteen studies were included. These studies showed large heterogeneity despite categorization. Median quality score was 15 (range 0-26). Fourteen studies were categorized as procedure-based assessment methods (PBA), and five as non-procedure-based assessment methods. Three studies were applicable for summative assessment.

CONCLUSIONS

The results show a considerable diversity in assessment methods with varying quality and suitability. To prevent a sprawl of assessment methods, we argue for selection and development of available high-quality assessment methods. A procedure-based structure combined with an objective assessment scale and possibility for summative assessment should be cornerstones.

Automatic purse-string suture skill assessment in transanal total mesorectal excision using deep learning-based video analysis

BACKGROUND

Purse-string suture in transanal total mesorectal excision is a key procedural step. The aims of this study were to develop an automatic skill assessment system for purse-string suture in transanal total mesorectal excision using deep learning and to evaluate the reliability of the score output from the proposed system.

METHODS

Purse-string suturing extracted from consecutive transanal total mesorectal excision videos was manually scored using a performance rubric scale and computed into a deep learning model as training data. Deep learning-based image regression analysis was performed, and the purse-string suture skill scores predicted by the trained deep learning model (artificial intelligence score) were output as continuous variables. The outcomes of interest were the correlation, assessed using Spearman's rank correlation coefficient, between the artificial intelligence score and the manual score, purse-string suture time, and surgeon's experience.

RESULTS

Forty-five videos obtained from five surgeons were evaluated. The mean(s.d.) total manual score was 9.2(2.7) points, the mean(s.d.) total artificial intelligence score was 10.2(3.9) points, and the mean(s.d.) absolute error between the artificial intelligence and manual scores was 0.42(0.39). Further, the artificial intelligence score significantly correlated with the purse-string suture time (correlation coefficient = -0.728) and surgeon's experience (P< 0.001).

CONCLUSION

An automatic purse-string suture skill assessment system using deep learning-based video analysis was shown to be feasible, and the results indicated that the artificial intelligence score was reliable. This application could be expanded to other endoscopic surgeries and procedures.

The RECOURSE Study: Long-term Oncologic Outcomes Associated With Robotically Assisted Minimally Invasive Procedures for Endometrial, Cervical, Colorectal, Lung, or Prostate Cancer: A Systematic Review and Meta-analysis

OBJECTIVE

To assess long-term outcomes with robotic versus laparoscopic/thoracoscopic and open surgery for colorectal, urologic, endometrial, cervical, and thoracic cancers.

BACKGROUND

Minimally invasive surgery provides perioperative benefits and similar oncological outcomes compared with open surgery. Recent robotic surgery data have questioned long-term benefits.

METHODS

A systematic review and meta-analysis of cancer outcomes based on surgical approach was conducted based on Preferred Reporting Items for Systematic Reviews and Meta-analyses guidelines using Pubmed, Scopus, and Embase. Hazard ratios for recurrence, disease-free survival (DFS), and overall survival (OS) were extracted/estimated using a hierarchical decision tree and pooled in RevMan 5.4 using inverse-variance fixed-effect (heterogeneity nonsignificant) or random effect models.

RESULTS

Of 31,204 references, 199 were included (7 randomized, 23 database, 15 prospective, 154 retrospective studies)-157,876 robotic, 68,007 laparoscopic/thoracoscopic, and 234,649 open cases. Cervical cancer: OS and DFS were similar between robotic and laparoscopic [1.01 (0.56, 1.80), P =0.98] or open [1.18 (0.99, 1.41), P =0.06] surgery; 2 papers reported less recurrence with open surgery [2.30 (1.32, 4.01), P =0.003]. Endometrial cancer: the only significant result favored robotic over open surgery [OS; 0.77 (0.71, 0.83), P <0.001]. Lobectomy: DFS favored robotic over thoracoscopic surgery [0.74 (0.59, 0.93), P =0.009]; OS favored robotic over open surgery [0.93 (0.87, 1.00), P =0.04]. Prostatectomy: recurrence was less with robotic versus laparoscopic surgery [0.77 (0.68, 0.87), P <0.0001]; OS favored robotic over open surgery [0.78 (0.72, 0.85), P <0.0001]. Low-anterior resection: OS significantly favored robotic over laparoscopic [0.76 (0.63, 0.91), P =0.004] and open surgery [0.83 (0.74, 0.93), P =0.001].

CONCLUSIONS

Long-term outcomes were similar for robotic versus laparoscopic/thoracoscopic and open surgery, with no safety signal or indication requiring further research (PROSPERO Reg#CRD42021240519).

Development of biotissue training models for anastomotic suturing in pancreatic surgery

BACKGROUND

Anastomotic suturing is the Achilles heel of pancreatic surgery. Especially in laparoscopic and robotically assisted surgery, the pancreatic anastomosis should first be trained outside the operating room. Realistic training models are therefore needed.

METHODS

Models of the pancreas, small bowel, stomach, bile duct, and a realistic training torso were developed for training of anastomoses in pancreatic surgery. Pancreas models with soft and hard textures, small and large ducts were incrementally developed and evaluated. Experienced pancreatic surgeons (n = 44) evaluated haptic realism, rigidity, fragility of tissues, and realism of suturing and knot tying.

RESULTS

In the iterative development process the pancreas models showed high haptic realism and highest realism in suturing (4.6 ± 0.7 and 4.9 ± 0.5 on 1-5 Likert scale, soft pancreas). The small bowel model showed highest haptic realism (4.8 ± 0.4) and optimal wall thickness (0.1 ± 0.4 on -2 to +2 Likert scale) and suturing behavior (0.1 ± 0.4). The bile duct models showed optimal wall thickness (0.3 ± 0.8 and 0.4 ± 0.8 on -2 to +2 Likert scale) and optimal tissue fragility (0 ± 0.9 and 0.3 ± 0.7).

CONCLUSION

The biotissue training models showed high haptic realism and realistic suturing behavior. They are suitable for realistic training of anastomoses in pancreatic surgery which may improve patient outcomes.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN)

RAPN training usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee capacity to perform RAPN at predetermined performance levels prior to in-vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. The authors sought to examine the reliability, construct and discriminative validity of objective intraoperative performance metrics which best characterize the optimal and suboptimal performance of a reference approach for training novice RAPN surgeons. Seven Novice and 9 Experienced RAPN surgeons video recorded one or two independently performed RAPN procedures in the human. The videos were anonymized and two experienced urology surgeons were trained to reliably score RAPN performance, using previously developed metrics. The assessors were blinded to the performing surgeon, hospital and surgeon group. They independently scored surgeon RAPN performance. Novice and Experienced group performance scores were compared for procedure steps completed and errors made. Each group was divided at the median for Total Errors score, and subgroup scores (i.e., Novice HiErrs and LoErrs, Experienced HiErrs and LoErrs) were compared. The mean inter-rater reliability (IRR) for scoring was 0.95 (range 0.84-1). Compared with Novices, Experienced RAPN surgeons made 69% fewer procedural Total Errors. This difference was accentuated when the LoErr Expert RAPN surgeon's performance was compared with the HiErrs Novice RAPN surgeon's performance with an observed 170% fewer Total Errors. GEARS showed poor reliability (Mean IRR = 0.44; range 0.0-0.8), for scoring RAPN surgical performance. The RAPN procedure metrics reliably distinguish Novice and Experienced surgeon performances. They further differentiated performance levels within a group with similar experiences. Reliable and valid metrics will underpin quality-assured novice RAPN surgical training.

The Surgical Learning Curve: Does Robotic Technical Skill Explain Differences in Operative Performance?

Background: Prior studies on technical skills use small collections of videos for assessment. However, there is likely heterogeneity of performance among surgeons and likely improvement after training. If technical skill explains these differences, then it should vary among practicing surgeons and improve over time. Materials and Methods: Sleeve gastrectomy cases (n = 162) between July 2018 and January 2021 at one health system were included. Global evaluative assessment of robotic skills (GEARS) scores were assigned by crowdsourced evaluators. Videos were manually annotated. Analysis of variance was used to compare continuous variables between surgeons. Tamhane's post hoc test was used to define differences between surgeons with the eta-squared value for effect size. Linear regression was used for temporal changes. A P value <.05 was considered significant. Results: Variations in operative time discriminated between individuals (e.g., between 2 surgeons, means were 91 and 112 minutes, Tamhane's = 0.001). Overall, GEARS scores did not vary significantly (e.g., between those 2 surgeons, means were 20.32 and 20.6, Tamhane's = 0.151). Operative time and total GEARS score did not change over time (R² = 0.0001-0.096). Subcomponent scores showed idiosyncratic temporal changes, although force sensitivity increased among all (R² = 0.172-0.243). For a novice surgeon, phase-adjusted operative time (R² = 0.24), but not overall GEARS scores (R² = 0.04), improved over time. Conclusions: GEARS scores showed less variability and did not improve with time for a novice surgeon. Improved technical skill does not explain the learning curve of a novice surgeon or variation among surgeons. More work could define valid surrogate metrics for performance analysis.

Association between Surgical Technical Skills and Clinical Outcomes: A Systematic Literature Review and Meta-Analysis

Background

A systematic literature review and meta-analysis was conducted to assess the association between intraoperative surgical skill and clinical outcomes.

Methods

Peer-reviewed, original research articles published through August 31, 2021 were identified from PubMed and Embase. From the 1,513 potential articles, seven met eligibility requirements, reporting on 151 surgeons and 17,932 procedures. All included retrospective assessment of operative videos. Associations between surgical skill and outcomes were assessed by pooling odds ratios (OR) using random-effects models with the inverse variance method. Eligible studies included pancreaticoduodenectomy, gastric bypass, laparoscopic gastrectomy, prostatectomy, colorectal, and hemicolectomy procedures.

Results

Meta-analytic pooling identified significant associations between the highest vs. lowest quartile of surgical skill and reoperation (OR: 0.44; 95% confidence interval [CI]: 0.23, 0.83), hemorrhage (OR: 0.66; 95% CI, 0.65, 0.68), obstruction (OR: 0.33; 95% CI, 0.30, 0.35), and any medical complication (OR: 0.23, 95% CI, 0.19, 0.27). Nonsignificant inverse associations were noted between skill and readmission, emergency department visit, mortality, leak, infection, venous thromboembolism, and cardiac and pulmonary complications.

Conclusions

Overall, surgeon technical skill appears to predict clinical outcomes. However, there are surprisingly few articles that evaluate this association. The authors recommend a thoughtful approach for the development of a comprehensive surgical quality infrastructure that could significantly reduce the challenges identified by this study.

Development of a procedure-specific tool for skill assessment in left- and right-sided laparoscopic complete mesocolic excision

AIM

To (1) develop an assessment tool for laparoscopic complete mesocolic excision (LCME) and (2) report evidence of its content validity.

METHOD

Assessment statements were revealed through (1) semi-structured expert interviews and (2) consensus by the Delphi method, both involving an expert panel of five LCME surgeons. All experts were interviewed and then asked to rate LCME describing statements from 1 (strongly disagree) to 5 (strongly agree). Responses were returned anonymously to the panel until consensus was reached. Statements were directly included as content in the assessment tool if ≥60% of the experts responded "agree" or "strongly agree" (ratings 4 and 5), with the remaining responses being "neither agree nor disagree" (rating 3). Interclass correlation coefficient (ICC) was calculated for expert agreement evaluation. All included statements were subsequently reformulated as tool items and approved by the experts.

RESULTS

Four Delphi rounds were performed to reach consensus. Disagreement was reported for statements describing instrument handling around pancreas; visualisation of landmarks before inferior mesenteric artery ligation; lymphadenectomy around the inferior mesenteric artery, and division of the terminal ileum and transverse colon. ICC in the last Delphi-round was 0.84. The final tool content included 73 statements, converted to 48 right- and 40 left-sided items for LCME assessment.

CONCLUSION

A procedure-specific, video-based tool, named complete mesocolic excision competency assessment tool (CMECAT), has been developed for LCME skill assessment. In the future, we hope it can facilitate assessment of LCME surgeons, resulting in improved patient outcome after colon cancer surgery.

Validation study of a skill assessment tool for education and outcome prediction of laparoscopic distal gastrectomy

BACKGROUND

The Japanese operative-rating scale for laparoscopic distal gastrectomy (JORS-LDG) was developed through cognitive task analysis together with the Delphi method to measure intraoperative performance during laparoscopic distal gastrectomy. This study aimed to investigate the value of this rating scale as an educational tool and a surgical outcome predictor in laparoscopic distal gastrectomy.

METHODS

The surgical performance of laparoscopic distal gastrectomy was assessed by the first assistant, through self-evaluation in the operating room and by video raters blind to the case. We evaluated inter-rater reliability, internal consistency, and correlations between the JORS-LDG scores and the evaluation methods, patient characteristics, and surgical outcomes.

RESULTS

Fifty-four laparoscopic distal gastrectomy procedures performed by 40 surgeons at 16 institutions were evaluated in the operating room and with video recordings using the proposed rating scale. The video inter-rater reliability was > 0.8. Participating surgeons were divided into the low, intermediate, and high groups based on their total scores. The number of laparoscopic surgeries and laparoscopic gastrectomy procedures performed differed significantly among the groups according to laparoscopic distal gastrectomy skill levels. The low, intermediate, and high groups also differed in terms of median operating times (311, 266, and 229 min, respectively, P < 0.001), intraoperative complication rates (27.8, 11.8, and 0%, respectively, P = 0.01), and postoperative complication rates (22.2, 0, and 0%, respectively, P = 0.002).

CONCLUSIONS

The JORS-LDG is a reliable and valid measure for laparoscopic distal gastrectomy training and could be useful in predicting surgical outcomes.

Surgomics: personalized prediction of morbidity, mortality and long-term outcome in surgery using machine learning on multimodal data

BACKGROUND

Personalized medicine requires the integration and analysis of vast amounts of patient data to realize individualized care. With Surgomics, we aim to facilitate personalized therapy recommendations in surgery by integration of intraoperative surgical data and their analysis with machine learning methods to leverage the potential of this data in analogy to Radiomics and Genomics.

METHODS

We defined Surgomics as the entirety of surgomic features that are process characteristics of a surgical procedure automatically derived from multimodal intraoperative data to quantify processes in the operating room. In a multidisciplinary team we discussed potential data sources like endoscopic videos, vital sign monitoring, medical devices and instruments and respective surgomic features. Subsequently, an online questionnaire was sent to experts from surgery and (computer) science at multiple centers for rating the features' clinical relevance and technical feasibility.

RESULTS

In total, 52 surgomic features were identified and assigned to eight feature categories. Based on the expert survey (n = 66 participants) the feature category with the highest clinical relevance as rated by surgeons was "surgical skill and quality of performance" for morbidity and mortality (9.0 ± 1.3 on a numerical rating scale from 1 to 10) as well as for long-term (oncological) outcome (8.2 ± 1.8). The feature category with the highest feasibility to be automatically extracted as rated by (computer) scientists was "Instrument" (8.5 ± 1.7). Among the surgomic features ranked as most relevant in their respective category were "intraoperative adverse events", "action performed with instruments", "vital sign monitoring", and "difficulty of surgery".

CONCLUSION

Surgomics is a promising concept for the analysis of intraoperative data. Surgomics may be used together with preoperative features from clinical data and Radiomics to predict postoperative morbidity, mortality and long-term outcome, as well as to provide tailored feedback for surgeons.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review

BACKGROUND

Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes.

METHODS

Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis.

RESULTS

A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results.

CONCLUSION

Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy

BACKGROUND

A computer vision (CV) platform named EndoDigest was recently developed to facilitate the use of surgical videos. Specifically, EndoDigest automatically provides short video clips to effectively document the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). The aim of the present study is to validate EndoDigest on a multicentric dataset of LC videos.

METHODS

LC videos from 4 centers were manually annotated with the time of the cystic duct division and an assessment of CVS criteria. Incomplete recordings, bailout procedures and procedures with an intraoperative cholangiogram were excluded. EndoDigest leveraged predictions of deep learning models for workflow analysis in a rule-based inference system designed to estimate the time of the cystic duct division. Performance was assessed by computing the error in estimating the manually annotated time of the cystic duct division. To provide concise video documentation of CVS, EndoDigest extracted video clips showing the 2 min preceding and the 30 s following the predicted cystic duct division. The relevance of the documentation was evaluated by assessing CVS in automatically extracted 2.5-min-long video clips.

RESULTS

144 of the 174 LC videos from 4 centers were analyzed. EndoDigest located the time of the cystic duct division with a mean error of 124.0 ± 270.6 s despite the use of fluorescent cholangiography in 27 procedures and great variations in surgical workflows across centers. The surgical evaluation found that 108 (75.0%) of the automatically extracted short video clips documented CVS effectively.

CONCLUSIONS

EndoDigest was robust enough to reliably locate the time of the cystic duct division and efficiently video document CVS despite the highly variable workflows. Training specifically on data from each center could improve results; however, this multicentric validation shows the potential for clinical translation of this surgical data science tool to efficiently document surgical safety.

What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature

A crucial element of any surgical training program is the ability to provide procedure-specific, objective, and reliable measures of performance. During robotic surgery, objective clinically relevant performance metrics (CRPMs) can provide tailored contextual feedback and correlate with clinical outcomes. This review aims to define CRPMs, assess their validity in robotic surgical training and compare CRPMs to existing measures of robotic performance. A systematic search of Medline and Embase databases was conducted in May 2022 following the PRISMA guidelines. The search terms included Clinically Relevant Performance Metrics (CRPMs) OR Clinically Relevant Outcome Measures (CROMs) AND robotic surgery. The study settings, speciality, operative context, study design, metric details, and validation status were extracted and analysed. The initial search yielded 116 citations, of which 6 were included. Citation searching identified 3 additional studies, resulting in 9 studies included in this review. Metrics were defined as CRPMs, CROMs, proficiency-based performance metrics and reference-procedure metrics which were developed using a modified Delphi methodology. All metrics underwent both contents and construct validation. Two studies found a strong correlation with GEARS but none correlated their metrics with patient outcome data. CRPMs are a validated and objective approach for assessing trainee proficiency. Evaluating CRPMs with other robotic-assessment tools will facilitate a multimodal metric evaluation approach to robotic surgery training. Further studies should assess the correlation with clinical outcomes. This review highlights there is significant scope for the development and validation of CRPMs to establish proficiency-based progression curricula that can be translated from a simulation setting into clinical practice.