Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

Intraoperative fluoroscopy skills in distal radius fracture surgery: valid and reliable assessment on a novel immersive virtual reality simulator

BACKGROUND AND PURPOSE

 Orthopedic trainees must be able to perform intraoperative fluoroscopy imaging to assess the surgical result after volar locking plate surgeries of distal radius fractures. Guided by Messick's contemporary validity framework, the aim of our study was to gather evidence of validity for a test of proficiency for intraoperative imaging of a distal radius fracture using a novel immersive virtual reality simulator.

METHODS

 11 novices and 9 experienced surgeons employed at orthopedic departments completed 2 individual simulator sessions. At each session the participants performed 3 repetitions of an intraoperative fluoroscopic control of a distal radius fracture, consisting of 5 different fluoroscopic views. Several performance metrics were automatically recorded by the simulator and compared between the 2 groups.

RESULTS

 Simulator metrics for 3 of the 5 fluoroscopic views could discriminate between novices and experienced surgeons. An estimated composite score based on these 3 views showed good test-retest reliability, ICC = 0.82 (confidence interval 0.65-0.92; P < 0.001). A discriminatory standard was set at a composite score of 6.15 points resulting in 1 false positive (i.e., novice scoring better than the standard), and 1 false negative (i.e., experienced surgeon scoring worse than the standard).

CONCLUSION

 This study provided validity evidence from all 5 sources of Messick's contemporary validity framework (content, response process, internal structure, relationship with other variables, and consequences) for a simulation-based test of proficiency in intraoperative fluoroscopic control of a distal radius fracture fixated by a volar locking plate.

Medical Students Cannot Assess Robotic Surgeons Performing Radical Prostatectomy

INTRODUCTION

Medical students have previously been shown to be just as effective for video rating as experts. We want to compare medical students to experienced surgeons as video assessors of simulated robot-assisted radical prostatectomy (RARP) performance.

MATERIALS AND METHODS

Video recordings of three RARP modules on the RobotiX (formerly Simbionix) simulator from a previous study were used. Five novice surgeons, five experienced robotic surgeons, and five experienced robotic surgeons in RARP performed a total of 45 video-recorded procedures. The videos were assessed with the modified Global Evaluative Assessment of Robotic Skills tool as both full-length and an edited edition that only included the first 5 minutes of the procedure.

RESULTS

Fifty medical students and two experienced RARP surgeons (ES) performed a total of 680 video ratings of full-length videos and 5-minute videos (2-9 ratings per video). Medical students and ES showed poor agreement for both full-length videos and 5-minute videos (0.29 and -0.13, respectively). Medical students could not discriminate between the skill level of the surgeons in either full-length videos or 5-minute videos ( P = 0.053-0.36 and P = 0.21-0.82), whereas ES could discriminate between novice surgeons and experienced surgeons (full-length, P < 0.001, and 5 minutes, P = 0.007) and intermediate and experienced surgeons (full-length, P = 0.001, and 5 minutes, P = 0.01) in both full-length videos and 5-minute videos.

CONCLUSION

We found that medical students cannot be used to assess RARP because they showed poor agreement with the ES rating for both full-length videos and 5-minute videos. Medical students could not discriminate between surgical skill levels.

Validity of A Novel Simulator for Percutaneous Transforaminal Endoscopic Discectomy

BACKGROUND

Percutaneous transforaminal endoscopic discectomy (PTED) has steep learning curves and a high incidence of complications, but currently, efficient and economical training methods are lacking. This study aimed to validate a novel simulator for PTED.

METHODS

The simulated PTED included puncturing and establishing the working channel (PEWC) and endoscopic discectomy, with the PEWC being the tested module. Eleven experts and 21 novices were included and introduced to the simulator and tasks; all participants completed the PEWC. Outcomes included: total operation time, number of fluoroscopy for positioning the working sheath, number of spinal risk region invasion, Global Rating Scale (GRS) and a modified GRS, etc. The Mann-Whitney U test was used to compare 2 groups. Spearman's correlation coefficient analyzed continuous variables.

RESULTS

Experts outperformed novices in total operation time (P = 0.001), requiring fewer number of fluoroscopies for positioning the working sheath (P = 0.003). Additionally, experts had a lower number of spinal risk region invasions (P = 0.016) and higher scores on both the GRS (P < 0.001) and modified GRS (P < 0.001). PTED experience correlated with GRS scores (P = 0.001) and modified GRS (P < 0.001). The overall realism scored a median of 4 (3.75-5), and educational value had a median of 4 (range 3-5).

CONCLUSIONS

This study demonstrates the validity of the novel simulator, revealing significant associations between PTED experience and performance metrics in a simulated PEWC setting. Furthermore, the PEWC module also offers a good realistic design and high education value according to experts.

Role of simulation-based training and assessment to improve brachytherapy competency among radiation oncology residents

Simulation is a technique used in healthcare to replicate clinical scenarios and improve patient safety, efficacy, and efficiency. Simulation-based medical education facilitates training and assessment in healthcare without increasing risk to patients, supported by ample evidence from surgical/procedural specialties. Simulation in radiation oncology has been leveraged to an extent, with successful examples of both screen-based and hands-on simulators that have improved confidence and performance in trainees. In the current era, evidence substantiates a significant deficit in brachytherapy procedure education, with radiation oncology residents reporting low confidence in this procedural skill, largely attributable to insufficient caseloads at some centers. Simulation-based medical education can facilitate structured training and competency-based assessment in brachytherapy skills. This review discusses existing advances and future directions in brachytherapy simulation, using examples from simulation in surgical specialties.

Generating a Prioritized List of Operative Procedures for Simulation-based Assessment of General Surgery Trainees Through Consensus

OBJECTIVE

To develop appropriate content for high-stakes simulation-based assessments of operative competence in general surgery training through consensus.

BACKGROUND

Valid methods of summative operative competence assessment are required by competency-based training programs in surgery.

METHOD

An online Delphi consensus study was conducted. Procedures were derived from the competency expectations outlined by the Joint Committee on Surgical Training Curriculum 2021, and subsequent brainstorming. Procedures were rated according to their perceived importance, perceived procedural risk, how frequently they are performed, and simualtion feasibility by a purposive sample of 30 surgical trainers and a 5-person steering group. A modified Copenhagen Academy for Medical Education and Simulation Needs Assessment Formula was applied to the generated data to produce ranked procedural lists, which were returned to participants for re-prioritization.

RESULTS

Prioritized lists were generated for simulation-based operative competence assessments at 2 key stages of training; the end of 'phase 2' prior to the development of a sub-specialty interest, and the end of 'phase 3', that is, end-of-training certification. A total of 21 and 16 procedures were deemed suitable for assessments at each of these stages, respectively.

CONCLUSIONS

This study describes a national needs assessment approach to content generation for simulation-based assessments of operative competence in general surgery using Delphi consensus methodology. The prioritized procedural lists generated by this study can be used to further develop operative skill assessments for use in high-stakes scenarios, such as trainee progression, entrustment, and end-of-training certification, before subsequent validity testing.

An update on simulation training in rhinology: a systematic review of evidence

BACKGROUND

Rhinological procedures demand a high degree of technical expertise and anatomical knowledge. Because of limited surgical opportunities, ethical considerations and the complexity of these procedures, simulation-based training has become increasingly important. This review aimed to evaluate the effectiveness of simulation models used in rhinology training.

METHODS

Searches were conducted on PubMed, Embase, Cochrane and Google Scholar for studies conducted between July 2012 and July 2022. The Preferred Reporting Items for Systematic Reviews and Meta-Analysis ('PRISMA') protocol defined a final list of articles. Each validated study was assigned a level of evidence and a level of recommendation based on the Oxford Centre of Evidence-Based Medicine classification.

RESULTS

Following exclusions, 42 articles were identified which encompassed six types of simulation models and 26 studies evaluated validity. The rhinological skills assessed included endoscopic sinus surgery (n = 28), skull base/cerebrospinal fluid leak repair (n = 14), management of epistaxis and/or sphenopalatine artery ligation (n = 8), and septoplasty and septorhinoplasty (n = 6). All studies reported the beneficial impact of their simulation models on trainee development.

CONCLUSION

Simulation training in rhinology is a valuable adjunct to traditional surgical education. Although evidence is of moderate quality, the findings highlight the importance of simulation-based training in rhinology training.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

Simulation-based assessment of robotic cardiac surgery skills: An international multicenter, cross-specialty trial

Objective

This study aimed to investigate the validity of simulation-based assessment of robotic-assisted cardiac surgery skills using a wet lab model, focusing on the use of a time-based score (TBS) and modified Global Evaluative Assessment of Robotic Skills (mGEARS) score.

Methods

We tested 3 wet lab tasks (atrial closure, mitral annular stitches, and internal thoracic artery [ITA] dissection) with both experienced robotic cardiac surgeons and novices from multiple European centers. The tasks were assessed using 2 tools: TBS and mGEARS score. Reliability, internal consistency, and the ability to discriminate between different levels of competence were evaluated.

Results

The results demonstrated a high internal consistency for all 3 tasks using mGEARS assessment tool. The mGEARS score and TBS could reliably discriminate between different levels of competence for the atrial closure and mitral stitches tasks but not for the ITA harvesting task. A generalizability study also revealed that it was feasible to assess competency of the atrial closure and mitral stitches tasks using mGEARS but not the ITA dissection task. Pass/fail scores were established for each task using both TBS and mGEARS assessment tools.

Conclusions

The study provides sufficient evidence for using TBS and mGEARS scores in evaluating robotic-assisted cardiac surgery skills in wet lab settings for intracardiac tasks. Combining both assessment tools enhances the evaluation of proficiency in robotic cardiac surgery, paving the way for standardized, evidence-based preclinical training and credentialing.

Clinical trial registry number

NCT05043064.

Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

BACKGROUND

Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP).

METHODS

We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS).

RESULTS

One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034).

CONCLUSION

We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.

Transfer of skills between laparoscopic and robot-assisted surgery: a systematic review

BACKGROUND

Robot-assisted surgery is today well-implemented in many surgical specialties, but requires another skill set than laparoscopy. Most often, robot-assisted surgery is considered add-on to laparoscopic skills but very little is known about the transfer of skills. The aim of the study was to examine to what extent surgical skills are transferable between laparoscopic and robot-assisted surgery.

METHODS

A systematic search was conducted in three databases (Ovid Medline, Embase, and Web of Science). Studies investigating transfer of skills between laparoscopy and robot-assisted surgery in either a phantom-based, simulation-based, animal model, or clinical setting were eligible for inclusion. Quality assessment was done using the Medical education research study quality instrument and educational New Ottawa Scale.

RESULTS

Of 15,610 studies identified, 89 studies continued to full-text reading, and 37 studies were included. Four studies were found non-comparable and were left out of the results for the primary outcome. All 33 studies explored transfer from laparoscopy to robot-assisted surgery and 17 found a positive transfer whereas 15 did not. Only 11 studies explored transfer from robot-assisted surgery to laparoscopy, of which only three found a positive transfer.

CONCLUSION

An almost equal number of publications found a positive transfer and no transfer from laparoscopic to robot-assisted surgery. Fewer studies explored the transfer from robot-assisted surgery to laparoscopy. Very little evidence supports that surgeons trained solely in robot-assisted surgery can perform laparoscopy. This must be considered in future training programs as robot-assisted surgery is expected to become the first-in-line modality for many future surgeons.

Importance and potential of simulation training in interventional radiology

BACKGROUND

Simulation training is a common method in many medical disciplines and is used to teach content knowledge, manual skills, and team skills without potential patient danger.

METHODS

Simulation models and methods in interventional radiology are explained. Strengths and weaknesses of both simulators for non-vascular and vascular radiological interventions are highlighted and necessary future developments are addressed.

RESULTS

Both custom-made and commercially available phantoms are available for non-vascular interventions. Interventions are performed under ultrasound guidance, with computed tomography assistance, or using mixed-reality methods. The wear and tear of physical phantoms can be countered with in-house production of 3D-printed models. Vascular interventions can be trained on silicone models or hightech simulators. Increasingly, patient-specific anatomies are replicated and simulated pre-intervention. The level of evidence of all procedures is low.

CONCLUSION

Numerous simulation methods are available in interventional radiology. Training on silicone models and hightech simulators for vascular interventions has the potential to reduce procedural time. This is associated with reduced radiation dose for both patient and physician, which can also contribute to improved patient outcome, at least in endovascular stroke treatment. Although a higher level of evidence should be achieved, simulation training should already be integrated into the guidelines of the professional societies and accordingly into the curricula of the radiology departments.

KEY POINTS

· There are numerous simulation methods for nonvascular and vascular radiologic interventions.. · Puncture models can be purchased commercially or made using 3D printing.. · Silicone models and hightech simulators allow patient-specific training.. · Simulation training reduces intervention time, benefiting both the patient and the physician.. · A higher level of evidence is possible via proof of reduced procedural times..

CITATION FORMAT

· Kreiser K, Sollmann N, Renz M. Importance and potential of simulation training in interventional radiology. Fortschr Röntgenstr 2023; 195: 883 - 889.

Gathering validity evidence for a 3D-printed simulator for training of myringotomy and ventilation tube insertion

Objectives

This study aimed to gather validity evidence according to Messick's framework for a novel 3D-printed simulator for myringotomy with ventilation tube insertion for use in technical skills training of otorhinolaryngology (ORL) residents.

Methods

The study included 15 junior ORL residents (trainees) and 13 experienced teaching otolaryngologists (experts). Experts and trainees first received an identically structured introduction to the procedure, simulator, and simulation setup. Five procedures performed by each participant were video-recorded and ordered randomly for blinded rating by two independent raters. The rating tools used were a global rating scale (GBRS) and a task-specific checklist. Validity evidence was collected according to Messick's framework. Differences in time consumption and performance scores were analyzed. Finally, a pass/fail standard was established using the contrasting groups' method.

Results

Trainees used significantly more time per procedure (109 s, 95% CI: 99-120) than experts (82 s, 95% CI: 71-93; p < .001). Adjusted for repetition and rater leniency, experts achieved an average GBRS score of 18.8 (95% CI: 18.3-19.2) out of 20 points, whereas trainees achieved an average of 17.1 points (95% CI: 16.6-17.5; p < .001). In contrast to the task-specific checklist, the GBRS score discriminated between repetition number and participant experience. The pass/fail standard for the GBRS was established at 18.4 points.

Conclusion

We established educational validity evidence for a novel 3D-printed model for simulation-based training of ventilation tube insertion and established a reliable pass/fail standard.

Level of Evidence

1b.

Insight into the history and trends of surgical simulation training in education: a bibliometric analysis

BACKGROUND

Surgical simulation training enables surgeons to acquire clinical experience or skills from the operating room to the simulation environment. Historically, it has changed with advances in science and technology. Moreover, no previous study has analyzed this field from the bibliometric analysis dimension. The study aimed to review changes in surgical simulation training worldwide using bibliometric software.

MATERIALS AND METHODS

Two searches were performed on the core collection database, Web of Science, regarding data from 1991 to the end of 2020 using three topic words (surgery, training, and simulation). From 1 January 2000, to 15 May 2022, the keyword 'robotic' was added for the hotspot exploration. The data were chiefly analyzed by publication date, country, author(s), and keywords using bibliometric software.

RESULTS

A total of 5285 articles were initially analyzed, from which it was clear that laparoscopic skill, three-dimensional printing, and virtual reality were the main focuses during those study periods. Subsequently, 348 publications on robotic surgery training were identified.

CONCLUSION

This study systematically summarizes the current status in the field of surgical simulation training and provides insights into the research focuses and future hotspot in a global context.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

A systematic review of the application of 3D-printed models to colorectal surgical training

BACKGROUND

The aim of this review was to explore the role of three-dimensional (3D) printing in colorectal surgical education and procedural simulation, and to assess the effectiveness of 3D-printed models in anatomic and operative education in colorectal surgery.

METHODS

A systematic review of the literature was performed following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to identify relevant publications relating to the use of 3D-printed models in colorectal surgery in an educational context. The search encompassed OVID Medline, Web of Science and EMBASE including papers in English published from 1 January 1995 to 1 January 2023. A total of 1018 publications were screened, and 5 met the criteria for inclusion in this review.

RESULTS

Four distinct 3D models were described across five studies. Two models demonstrated objective benefits in the use of 3D-printed models in anatomical education in academic outcomes at all levels of learner medical experience and were well accepted by learners. One model utilised for preoperative visualisation demonstrated improved operative outcomes in complete mesocolic excision compared with preoperative imaging review, with a 22.1% reduction in operative time (p < 0.001), 9.2% reduction in surgical duration (p = 0.035) and 37.3% reduction in intraoperative bleeding volume amongst novice surgeons (p < 0.01). Technical simulation has been demonstrated in a feasibility context in one model but remains limited in scope and application on account of the characteristics of available printing materials.

CONCLUSIONS

3D printing is well accepted and effective for anatomic education and preoperative procedural planning amongst colorectal surgeons, trainees and medical students but remains a technology in the early stages of its possible application. Technological advancements are required to improve the tissue realism of 3D-printed organ models to achieve greater fidelity and provide realistic colorectal surgical simulations.

National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients

Background

Objective

Design, setting, and participants

Outcome measurements and statistical analyses

Results and limitations

Conclusions

Patient summary