European expert consensus on a structured approach to training robotic-assisted low anterior resection using performance metrics

European Robotic Surgery Consensus (ERSC): Protocol for the development of a consensus in robotic training for gastrointestinal surgery trainees

BACKGROUND

The rapid adoption of robotic surgical systems across Europe has led to a critical gap in training and credentialing for gastrointestinal (GI) surgeons. Currently, there is no existing standardised curriculum to guide robotic training, assessment and certification for GI trainees. This manuscript describes the protocol to achieve a pan-European consensus on the essential components of a comprehensive training programme for GI robotic surgery through a five-stage process.

METHODS AND ANALYSIS

In Stage 1, a Steering Committee, consisting of international experts, trainees and educationalists, has been established to lead and coordinate the consensus development process. In Stage 2, a systematic review of existing multi-specialty robotic training curricula will be performed to inform the formulation of key position statements. In Stage 3, a comprehensive survey will be disseminated across Europe to capture the current state of robotic training and identify potential challenges and opportunities for improvement. In Stage 4, an international panel of GI surgeons, trainees, and robotic theatre staff will participate in a three-round Delphi process, seeking ≥ 70% agreement on crucial aspects of the training curriculum. Industry and patient representatives will be involved as external advisors throughout this process. In Stage 5, the robotic training curriculum for GI trainees will be finalised in a dedicated consensus meeting, culminating in the production of an Explanation and Elaboration (E&E) document.

REGISTRATION DETAILS

The study protocol has been registered on the Open Science Framework (https://osf.io/br87d/).

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

The deconstructed procedural description in robotic colorectal surgery

Increasing robotic surgical utilisation in colorectal surgery internationally has strengthened the need for standardised training. Deconstructed procedural descriptions identify components of an operation that can be integrated into proficiency-based progression training. This approach allows both access to skill level appropriate training opportunities and objective and comparable assessment. Robotic colorectal surgery has graded difficulty of operative procedures lending itself ideally to component training. Developing deconstructed procedural descriptions may assist in the structure and progression components in robotic colorectal surgical training. There is no currently published guide to procedural descriptions in robotic colorectal surgical or assessment of their training utility. This scoping review was conducted in June 2022 following the PRISMA-ScR guidelines to identify which robotic colorectal surgical procedures have available component-based procedural descriptions. Secondary aims were identifying the method of development of these descriptions and how they have been adapted in a training context. 20 published procedural descriptions were identified covering 8 robotic colorectal surgical procedures with anterior resection the most frequently described procedure. Five publications included descriptions of how the procedural description has been utilised for education and training. From these publications terminology relating to using deconstructed procedural descriptions in robotic colorectal surgical training is proposed. Development of deconstructed robotic colorectal procedural descriptions (DPDs) in an international context may assist in the development of a global curriculum of component operating competencies supported by objective metrics. This will allow for standardisation of robotic colorectal surgical training and supports a proficiency-based training approach.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Use of a High-Fidelity Training Simulator for Minimally Invasive Lumbar Decompression Increases Working Knowledge and Technical Skills Among Orthopedic and Neurosurgical Trainees

STUDY DESIGN

Prospective comparative study.

OBJECTIVE

To quantify the educational benefit to surgical trainees of using a high-fidelity simulator to perform minimally invasive (MIS) unilateral laminotomy for bilateral decompression (ULBD) for lumbar stenosis.

METHODS

Twelve orthopedic and neurologic surgery residents performed three MIS ULBD procedures over 2 weeks on a simulator guided by established AO Spine metrics. Video recording of each surgery was rated by three blinded, independent experts using a global rating scale. The learning curve was evaluated with attention to technical skills, skipped steps, occurrence of errors, and timing. A knowledge gap analysis evaluating participants' current vs desired ability was performed after each trial.

RESULTS

From trial 1 to 3, there was a decrease in average procedural time by 31.7 minutes. The cumulative number of skipped steps and surgical errors decreased from 25 to 6 and 24 to 6, respectively. Overall surgical proficiency improved as indicated by video rating of efficiency and smoothness of surgical maneuvers, most notably with knowledge and handling of instruments. The greatest changes were noted in junior rather than senior residents. Average knowledge gap analysis significantly decreased by 30% from the first to last trial (P = .001), signifying trainees performed closer to their desired technical goal.

CONCLUSION

Procedural metrics for minimally invasive ULBD in combination with a realistic surgical simulator can be used to improve the skills and confidence of trainees. Surgical simulation may offer an important educational complement to traditional methods of skill acquisition and should be explored further with other MIS techniques.

Integrating a tumour appropriate transanal or robotic assisted approach to total mesorectal excision in high-volume rectal cancer practice is safe and cost-effective

Total mesorectal excision (TME) is accepted as the gold standard for oncological resection in rectal cancer. The best approach to TME is debated and often surgeons will select a preferred approach. In this study, we aimed to describe how both robotic (R-TME) and transanal (TaTME) TME can be integrated into high-volume rectal cancer surgeon practice with a comparison of clinical and oncological outcomes and cost analysis. A prospective comparative cohort study was performed in a high-volume rectal cancer centre comparing the previous 50 R-TME and 50 TaTME performed by the same surgeon. A comparison of tumour characteristics was performed to highlight a specific role for each technique. Clinical outcomes (operative duration, length of stay (LOS) and perioperative morbidity), cancer quality indicators (resection margin and completeness of TME) and cost analysis were compared. Statistical analysis was performed using IBM SPSS, version 20. R-TME was preferred in mid-rectal cancer, compared to TaTME preferred in low rectal cancer (9 cm vs. 5 cm, p < 0.001). Operative duration was longer in R-TME compared to TaTME (265 vs. 179 min, p < 0.001). Major complications (CD III-IV complications) were experienced in 10% of R-TME and 14% of TaTME (p = 0.476). A 98% (n = 49) clear R0 resection margin was achieved with both R-TME and TaTME and mesorectum quality defined as 'complete' in 86% (n = 43) in R-TME and 82% (n = 41) in TaTME. Length of hospital stay was shorter in R-TME (5 vs. 7 days, p = 0.624). An overall difference of €131 was observed favouring TaTME. In high-volume rectal cancer surgery practice, both R-TME and TaTME can be practised and tailored according to patients and tumour characteristics, with comparable clinical and cancer outcomes and is cost-effective.

Developing procedural metrics for training in class II posterior composite resin restorations: Outcomes from an international Delphi panel

OBJECTIVE

The objective of this study was to develop and operationally define procedural performance metrics for the Class II posterior composite resin restoration procedure and to obtain face and content validity through a consensus meeting.

METHODS

A Metrics Group consisting of 4 experienced Consultants in Restorative Dentistry, an experienced member of staff from the Restorative Dentistry department in CUDSH and a senior behavioural scientist and education/training expert deconstructed the performance of the Class II posterior composite resin restoration and defined performance metrics. At a modified Delphi meeting, 20 experts in the field of Restorative Dentistry from 11 different Dental Institutions critiqued these metrics and their operational definitions before reaching consensus.

RESULTS

Initially performance metrics consisting of 15 Phases, 45 steps, 42 errors and 34 critical errors were identified that characterize the performance of the Class II posterior resin composite procedure. During the Delphi panel these were modified and consensus was reached on 15 Phases (with a change to the initial sequence) with 46 Steps (1 added, 13 modified), 37 Errors (2 added, 1 deleted, 6 reclassified as Critical Error), and 43 Critical Errors (9 added). Consensus on the resulting metrics was obtained and face and content validity verified.

CONCLUSIONS

It is possible to develop and objectively define performance metrics that comprehensively characterize a Class II posterior composite resin restoration. It is also possible to achieve consensus on the metrics from a Delphi panel of experts and to confirm the face and content validity of those procedure metrics.

A systematic review of robotic surgery curricula using a contemporary educational framework

BACKGROUND

There has been a rising trend in robotic surgery. Thus, there is demand for a robotic surgery curriculum (RSC) for training surgical trainees and practicing surgeons. There are limited data available about current curricular designs and the extent to which they have incorporated educational frameworks. Our aim was to study the existing robotic surgery curricula using Kern's 6-step approach in curriculum development.

METHODS

A systematic review was conducted using PubMed, PubMed Central, Cochrane, Embase, and Scopus (we searched studies from 2001 to 2021). PRISMA Guidelines was used to guide the search. Curriculum designed for general surgery and its subspecialties were included. Urology and gynecology were excluded. The articles were reviewed by five reviewers.

RESULTS

Our review yielded 71 articles, including 39 curricula at 9 different settings. Using Kern's framework, we demonstrated that the majority of robotic surgery curricula contained all the elements of Kern's curricular design. However, there were significant deficiencies in important aspects of these curricula i.e., implementation, the quality of assessment tools for measurement of performance and evaluation of the educational value of these interventions. Most institutions used commercial virtual reality simulators (VRS) as the main component of their RSC and 23% of curricula only used VRS.

CONCLUSIONS

Although majority of these studies contained all the elements of Kern's framework, there are critical deficiencies in the components of existing curricula. Future curricula should be designed using established educational frameworks to improve the quality of robotic surgery training.

What are clinically relevant performance metrics in robotic surgery? A systematic review of the literature

A crucial element of any surgical training program is the ability to provide procedure-specific, objective, and reliable measures of performance. During robotic surgery, objective clinically relevant performance metrics (CRPMs) can provide tailored contextual feedback and correlate with clinical outcomes. This review aims to define CRPMs, assess their validity in robotic surgical training and compare CRPMs to existing measures of robotic performance. A systematic search of Medline and Embase databases was conducted in May 2022 following the PRISMA guidelines. The search terms included Clinically Relevant Performance Metrics (CRPMs) OR Clinically Relevant Outcome Measures (CROMs) AND robotic surgery. The study settings, speciality, operative context, study design, metric details, and validation status were extracted and analysed. The initial search yielded 116 citations, of which 6 were included. Citation searching identified 3 additional studies, resulting in 9 studies included in this review. Metrics were defined as CRPMs, CROMs, proficiency-based performance metrics and reference-procedure metrics which were developed using a modified Delphi methodology. All metrics underwent both contents and construct validation. Two studies found a strong correlation with GEARS but none correlated their metrics with patient outcome data. CRPMs are a validated and objective approach for assessing trainee proficiency. Evaluating CRPMs with other robotic-assessment tools will facilitate a multimodal metric evaluation approach to robotic surgery training. Further studies should assess the correlation with clinical outcomes. This review highlights there is significant scope for the development and validation of CRPMs to establish proficiency-based progression curricula that can be translated from a simulation setting into clinical practice.

OUP accepted manuscript

Background

This study aimed to evaluate the use of binary metric-based (proficiency-based progression; PBP) performance assessments and global evaluative assessment of robotic skills (GEARS) of a robotic-assisted low anterior rectal resection (RA-LAR) procedure.

Method

A prospective study of video analysis of RA-LAR procedures was carried out using the PBP metrics with binary parameters previously developed, and GEARS. Recordings were collected from five novice surgeons (≤30 RA-LAR previously performed) and seven experienced surgeons (>30 RA-LAR previously performed). Two consultant colorectal surgeons were trained to be assessors in the use of PBP binary parameters to evaluate the procedure phases, surgical steps, errors, and critical errors in male and female patients and GEARS scores. Novice and experienced surgeons were categorized and assessed using PBP metrics and GEARS; mean scores obtained were compared for statistical purpose. Also, the inter-rater reliability (IRR) of these assessment tools was evaluated.

Results

Twenty unedited recordings of RA-LAR procedures were blindly assessed. Overall, using PBP metric-based assessment, a subgroup of experienced surgeons made more errors (20 versus 16, P = 0.158) and critical errors (9.2 versus 7.8, P = 0.417) than the novice group, although not significantly. However, during the critical phase of RA-LAR, experienced surgeons made significantly fewer errors than the novice group (95% CI of the difference, Lower = 0.104 – Upper = 5.155, df = 11.9, t = 2.23, p = 0.042), and a similar pattern was observed for critical errors. The PBP metric and GEARS assessment tools distinguished between the objectively assessed performance of experienced and novice colorectal surgeons performing RA-LAR (total error scores with PBP metrics, P = 0.019–0.008; GEARS scores, P = 0.029–0.025). GEARS demonstrated poor IRR (mean IRR 0.49) and weaker discrimination between groups (15–41 per cent difference). PBP binary metrics demonstrated good IRR (mean 0.94) and robust discrimination particularly for total error scores (58–64 per cent).

Conclusions

PBP binary metrics seem to be useful for metric-based training for surgeons learning RA-LAR procedures.