International expert consensus on a scientific approach to training novice cardiac resynchronization therapy implanters using performance quality metrics

Surgical skill simulation training to proficiency reduces procedural errors among novice cardiac device implanters: a randomized study

AIMS

In cardiac device implantation, having both surgical skills and ability to manipulate catheter/lead/wire is crucial. Few cardiologists, however, receive formal surgical training prior to implanting. Skills are mostly acquired directly on-the-job and surgical technique varies across institutions; suboptimal approaches may increase complications. We investigated how novel proficiency-based progression (PBP) simulation training impacts the surgical quality of implantations, compared to traditional simulation (SIM) training.

METHODS AND RESULTS

In this international prospective study, novice implanters were randomized (blinded) 1:1 to participate in a simulation-based procedure training curriculum, with proficiency demonstration requirements for advancing (PBP approach) or without (SIM). Ultimately, trainees performed the surgical tasks of an implant on a porcine tissue that was video-recorded and then scored by two independent assessors (blinded to group), using previously validated performance metrics. Primary outcomes were the number of procedural Steps Completed, Critical Errors, Errors (non-critical), and All Errors Combined. Thirty novice implanters from 10 countries participated. Baseline experiences were similar between groups. Compared to SIM-trained, the PBP-trained group completed on average 11% more procedural Steps (P < 0.001) and made 61.2% fewer Critical Errors (P < 0.001), 57.1% fewer Errors (P = 0.140), and 60.7% fewer All Errors Combined (P = 0.001); 11/15 (73%) PBP trainees demonstrated the predefined target performance level vs. 3/15 SIM trainees (20%) in the video-recorded performance.

CONCLUSION

Proficiency-based progression training produces superior objectively assessed novice operators' surgical performance in device implantation compared with traditional (simulation) training. Systematic PBP incorporation into formal academic surgical skills training is recommended before in vivo device practice. Future studies will quantify PBP training's effect on surgery-related device complications.

Development and validation of metrics for a new RAPN training model

Robot-assisted partial nephrectomy (RAPN) is a complex and index procedure that urologists need to learn how to perform safely. No validated performance metrics specifically developed for a RAPN training model (TM) exist. A Core Metrics Group specifically adapted human RAPN metrics to be used in a newly developed RAPN TM, explicitly defining phases, steps, errors, and critical errors. A modified Delphi meeting concurred on the face and content validation of the new metrics. One hundred percent consensus was achieved by the Delphi panel on 8 Phases, 32 Steps, 136 Errors and 64 Critical Errors. Two trained assessors evaluated recorded video performances of novice and expert RAPN surgeons executing an emulated RAPN in the newly developed TM. There were no differences in procedure Steps completed by the two groups. Experienced RAPN surgeons made 34% fewer Total Errors than the Novice group. Performance score for both groups was divided at the median score using Total Error scores, into HiError and LoError subgroups. The LowErrs Expert RAPN surgeons group made 118% fewer Total Errors than the Novice HiErrs group. Furthermore, the LowErrs Expert RAPN surgeons made 77% fewer Total Errors than the HiErrs Expert RAPN surgeons. These results established construct and discriminative validity of the metrics. The authors described a novel RAPN TM and its associated performance metrics with evidence supporting their face, content, construct, and discriminative validation. This report and evidence support the implementation of a simulation-based proficiency-based progression (PBP) training program for RAPN.

Effect of Metrics-Based Simulation Training to Proficiency on Procedure Quality and Errors Among Novice Cardiac Device Implanters: The IMPROF Randomized Trial

Importance

In cardiac device implant training, there is no common system to objectively assess trainees' ability to perform tasks at predetermined performance levels before in vivo practice; therefore, patients are potentially exposed to risks related to operators' early learning curve.

Objective

To assess the effect on implant performance quality of novel metrics-based simulation training to proficiency (proficiency-based progression [PBP]) vs traditional simulation-based training (SBT).

Design, Setting, and Participants

In this prospective randomized trial, conducted between March 8, 2022 and November 24, 2022, novice implanters were randomized (blinded) 1:1 to participate in an SBT curriculum (procedural knowledge e-learning and in-person simulation training) at an international skills training center, with proficiency demonstration requirements at each training stage for advancing (PBP approach) or without the requirements. Ultimately, trainees performed a cardiac resynchronization therapy (CRT) implant using virtual reality simulation. The procedure was video-recorded and subsequently scored using previously validated metrics by 2 independent assessors blinded to group. Physicians who had already implanted more than 20 pacemakers or defibrillators and fewer than 200 CRT systems as the first operator were eligible. Thirty-two implanters from 10 countries voluntarily enrolled in the training program and were randomized; 30 (15 per group) started and completed training. Data analysis was performed from November 27 to December 22, 2022.

Intervention

Training with PBP vs SBT.

Main Outcome and Measures

The primary outcome comprised 4 objectively assessed performance metrics derived from the video-recordings: number of procedural steps completed, errors, critical errors, and all errors combined.

Results

Baseline experience of the 30 participants (19 [63%] male; mean [SD] number of years in implant practice, 2.0 [1.8]; median [IQR] number of implanted pacemakers or defibrillators, 47.5 [30.0-115.0]; median [IQR] number of implanted CRT systems, 3.0 [1.25-10.0]) was similar between study groups. Compared with the SBT group, the PBP group completed 27% more procedural steps (median [IQR], 31 [30-32] vs 24 [22-27]; P < .001) and made 73% fewer errors (median [IQR], 2 [1-3] vs 7 [5-8]; P < .001), 84% fewer critical errors (median [IQR], 1 [0-1] vs 3 [3-5]; P < .001), and 77% fewer all errors combined (errors plus critical errors) (median [IQR], 3 [1-3] vs 11 [8-12]; P < .001); 14 of the 15 PBP trainees (93%) demonstrated the predefined target performance level vs 0 of the 15 SBT trainees.

Conclusions and Relevance

In this randomized trial, the PBP approach to novice implanter training generated superior objectively assessed performance vs SBT. If implemented broadly and systematically, PBP training may ensure safe and effective performance standards before trainees proceed to (supervised) in vivo practice. Future studies are needed to verify implications on procedure-related patient complications.

Trial Registration

ClinicalTrials.gov Identifier: NCT05952908.

Objective assessment of intraoperative skills for robot-assisted partial nephrectomy (RAPN)

RAPN training usually takes place in-vivo and methods vary across countries/institutions. No common system exists to objectively assess trainee capacity to perform RAPN at predetermined performance levels prior to in-vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes. The authors sought to examine the reliability, construct and discriminative validity of objective intraoperative performance metrics which best characterize the optimal and suboptimal performance of a reference approach for training novice RAPN surgeons. Seven Novice and 9 Experienced RAPN surgeons video recorded one or two independently performed RAPN procedures in the human. The videos were anonymized and two experienced urology surgeons were trained to reliably score RAPN performance, using previously developed metrics. The assessors were blinded to the performing surgeon, hospital and surgeon group. They independently scored surgeon RAPN performance. Novice and Experienced group performance scores were compared for procedure steps completed and errors made. Each group was divided at the median for Total Errors score, and subgroup scores (i.e., Novice HiErrs and LoErrs, Experienced HiErrs and LoErrs) were compared. The mean inter-rater reliability (IRR) for scoring was 0.95 (range 0.84-1). Compared with Novices, Experienced RAPN surgeons made 69% fewer procedural Total Errors. This difference was accentuated when the LoErr Expert RAPN surgeon's performance was compared with the HiErrs Novice RAPN surgeon's performance with an observed 170% fewer Total Errors. GEARS showed poor reliability (Mean IRR = 0.44; range 0.0-0.8), for scoring RAPN surgical performance. The RAPN procedure metrics reliably distinguish Novice and Experienced surgeon performances. They further differentiated performance levels within a group with similar experiences. Reliable and valid metrics will underpin quality-assured novice RAPN surgical training.

International Expert Consensus on Metric-based Characterization of Robot-assisted Partial Nephrectomy

BACKGROUND

Robot-assisted partial nephrectomy (RAPN) training usually takes place in vivo, and methods vary across countries/institutions. No common system exists to objectively assess trainee ability to perform RAPN at predetermined performance levels prior to in vivo practice. The identification of objective performance metrics for RAPN training is a crucial starting point to improve training and surgical outcomes.

OBJECTIVE

We sought to identify objective performance metrics that best characterize a reference approach to RAPN, and obtain face and content validity from procedure experts through a modified Delphi meeting.

DESIGN, SETTING, AND PARTICIPANTS

During a series of online meetings, a core metrics team of three RAPN experts and a senior behavioral scientist performed a detailed task deconstruction of a transperitoneal left-sided RAPN procedure.

OUTCOME MEASUREMENTS AND STATISTICAL ANALYSIS

Based on published guidelines, manufacturers' instructions, and unedited videos of RAPN, the team identified performance metrics that constitute an optimal approach for training purposes. The metrics were then subjected to an in-person modified international Delphi panel meeting with 19 expert surgeons.

RESULTS AND LIMITATIONS

Eleven procedure phases, with 64 procedure steps, 43 errors, and 39 critical errors, were identified. After the modified Delphi process, the international expert panel added 13 metrics (two steps), six were deleted, and three were modified; 100% panel consensus on the resulting metrics was obtained. Limitations are that the metrics are applicable only to left-sided RAPN cases and some might have been excluded.

CONCLUSIONS

Performance metrics that accurately characterize RAPN procedure were developed by a core group of experts. The metrics were then presented to and endorsed by an international panel of very experienced peers. Reliable and valid metrics underpin effective, quality-assured, structured surgical training for RAPN.

PATIENT SUMMARY

We organize a meeting among robot-assisted partial nephrectomy (RAPN) experts to identify and reach consensus on objective performance metrics for RAPN training. The metrics are a crucial starting point to improve and quality assure surgical training and patients' clinical outcomes.

Metric-Based Simulation Training to Proficiency for Endovascular Thrombectomy in Ischemic Stroke

Ischemic stroke is one of the leading causes of death and long-term disability in the West. Mechanical revascularization techniques are considered the standard of care for large vessel occlusive stroke. Traditional apprenticeship models involve doctors training their skills on patients. Simulation platforms have long been recognized as an alternative to this. There has however been very little robust assessment of the training outcomes achieved on some of these platforms. At best, these simulations increase understanding of the procedural process and may help improve some technical skills; at worst they may instill bad habits and poor technique. The prerequisite of any simulation process must be to teach what to do, with which devices, in the correct sequence as well as what not to do. It should provide valid metric-based feedback to the trainee that is objective, transparent, and fair for formative and summative performance feedback. It should deliver a training program that measures the performance progress of trainees against expert benchmarks—benchmarks that represent an evidence-based peer-reviewed standard. In this paper, we present a perspective for PBP training for thrombectomy based on our experience with the process of procedure characterization, metric validation, and early experience of using this approach for proficiency training. Patient outcomes are not only determined by optimal performance in the Angio Suite but also by an efficient patient procedure pathway. There will be value in utilizing the PBP training standard not only for the procedure itself but also for the constituent elements of the stroke pathway to further improve treatment outcomes for ischemic stroke patients.

EHRA expert consensus statement and practical guide on optimal implantation technique for conventional pacemakers and implantable cardioverter-defibrillators: endorsed by the Heart Rhythm Society (HRS), the Asia Pacific Heart Rhythm Society (APHRS), and the Latin-American Heart Rhythm Society (LAHRS)

With the global increase in device implantations, there is a growing need to train physicians to implant pacemakers and implantable cardioverter-defibrillators. Although there are international recommendations for device indications and programming, there is no consensus to date regarding implantation technique. This document is founded on a systematic literature search and review, and on consensus from an international task force. It aims to fill the gap by setting standards for device implantation.

A Systematic Review and Meta-analysis on the Impact of Proficiency-based Progression Simulation Training on Performance Outcomes

OBJECTIVE

To analyze all published prospective, randomized, and blinded clinical studies on the proficiency-based progression (PBP) training using objective performance metrics.

BACKGROUND

The benefit of PBP methodology to learning clinical skills in comparison to conventional training is not settled.

METHODS

Search of PubMed, Cochrane library's Central, EMBASE, MEDLINE, and Scopus databases, from inception to 1st March 2020. Two independent reviewers extracted the data. The Medical Education Research Study Quality Instrument (MERSQI) was used to assess the methodological quality of included studies. Results were pooled using biased corrected standardized mean difference and ratio-of-means. Summary effects were evaluated using a series of fixed and random effects models. The primary outcome was the number of procedural errors performed comparing PBP and non-PBP-based training pathways. Secondary outcomes were the number of procedural steps completed and the time to complete the task/procedure.

RESULTS

From the initial pool of 468 studies, 12 randomized clinical studies with a total of 239 participants were included in the analysis. In comparison to the non-PBP training, ratio-of-means results showed that PBP training reduced the number of performance errors by 60% (P < 0.001) and procedural time by 15% (P = 0.003) and increased the number of steps performed by 47% (P < 0.001).

CONCLUSIONS AND RELEVANCE

Our systematic review and meta-analysis confirms that PBP training in comparison to conventional or quality assured training improved trainees' performances, by decreasing procedural errors and procedural time, while increasing the number of correct steps taken when compared to standard simulation-based training.

Objective assessment of intraoperative skills for robot-assisted radical prostatectomy (RARP): results from the ERUS Scientific and Educational Working Groups Metrics Initiative

OBJECTIVE

To develop and seek consensus from procedure experts on the metrics that best characterise a reference robot-assisted radical prostatectomy (RARP) and determine if the metrics distinguished between the objectively assessed RARP performance of experienced and novice urologists, as identifying objective performance metrics for surgical training in robotic surgery is imperative for patient safety.

MATERIALS AND METHODS

In Study 1, the metrics, i.e. 12 phases of the procedure, 81 steps, 245 errors and 110 critical errors for a reference RARP were developed and then presented to an international Delphi panel of 19 experienced urologists. In Study 2, 12 very experienced surgeons (VES) who had performed >500 RARPs and 12 novice urology surgeons performed a RARP, which was video recorded and assessed by two experienced urologists blinded as to subject and group. Percentage agreement between experienced urologists for the Delphi meeting and Mann-Whitney U- and Kruskal-Wallis tests were used for construct validation of the newly identified RARP metrics.

RESULTS

At the Delphi panel, consensus was reached on the appropriateness of the metrics for a reference RARP. In Study 2, the results showed that the VES performed ~4% more procedure steps and made 72% fewer procedure errors than the novices (P = 0.027). Phases VIIa and VIIb (i.e. neurovascular bundle dissection) best discriminated between the VES and novices.

LIMITATIONS

VES whose performance was in the bottom half of their group demonstrated considerable error variability and made five-times as many errors as the other half of the group (P = 0.006).

CONCLUSIONS

The international Delphi panel reached high-level consensus on the RARP metrics that reliably distinguished between the objectively scored procedure performance of VES and novices. Reliable and valid performance metrics of RARP are imperative for effective and quality assured surgical training.

European expert consensus on a structured approach to training robotic-assisted low anterior resection using performance metrics

AIM

The aim was to develop and operationally define 'performance metrics' that characterize a reference approach to robotic-assisted low anterior resection (RA-LAR) and to obtain face and content validity through a consensus meeting.

METHOD

Three senior colorectal surgeons with robotic experience and a senior behavioural scientist formed the Metrics Group. We used published guidelines, training materials, manufacturers' instructions and unedited videos of RA-LAR to deconstruct the operation into defined, measurable components - performance metrics (i.e. procedure phases, steps, errors and critical errors). The performance metrics were then subjected to detailed critique by 18 expert colorectal surgeons in a modified Delphi process.

RESULTS

Performance metrics for RA-LAR had 15 procedure phases, 128 steps, 89 errors and 117 critical errors in women, 88 errors and 118 critical errors in men. After the modified Delphi process the final performance metrics consisted of 14 procedure phases, 129 steps, 88 errors and 115 critical errors in women, 87 errors and 116 critical errors in men. After discussion by the Delphi panel, all procedure phases received unanimous consensus apart from phase I (patient positioning and preparation, 83%) and phase IV (docking, 94%).

CONCLUSION

A robotic rectal operation can be broken down into procedure phases, steps, with errors and critical errors, known as performance metrics. The face and content of these metrics have been validated by a large group of expert robotic colorectal surgeons from Europe. We consider the metrics essential for the development of a structured training curriculum and standardized procedural assessment for RA-LAR.