Gathering Validity Evidence for Surgical Simulation: A Systematic Review

The impact of simulator training of vascular anastomosis and video assessment for surgical residents

INTRODUCTION

The rapid increase of minimally invasive surgery and the shortened training period for surgical residents has resulted in limited opportunities to acquire proficiency in open surgical techniques, such as vascular anastomosis. However, vascular anastomosis remains an essential skill in every surgery for bleeding control. This study aimed to validate the effectiveness of surgical education model for vascular anastomosis and assess the impact on the comprehension, skill, and confidence of surgical residents in performing vascular anastomosis.

METHODS

A total of 21 surgical residents with 1st to 3rd years of experience at Seoul National University Hospital participated in a four-week vascular anastomosis training program. The program included an educational lecture and the performance of an end-to-side anastomosis on a procedural model, with evaluations being conducted using the Objective Structured Assessment of Technical Skills (OSATS) and the End Product Rating Score (EPRS) in pre- and post-training surveys.

RESULTS

Significant improvement was observed in the OSATS score (from 9.22 ± 2.4 in week 1 to 12.87 ± 3.1 in week 4; P < 0.001) and the EPRS score (from 12.47 ± 4.1 in week 1 to 17.57 ± 2.2 in week 4; P < 0.001). Additionally, the surgical performance time significantly decreased from 20.99 ± 4.6 minutes to 16.33 ± 4.2 minutes (P = 0.019) CONCLUSIONS: Simulator training of in vitro vascular anastomosis, when accompanied by expert-led instruction, can effectively enhance the surgical proficiency, confidence, and overall surgical outcomes of residents, as inferred from the observed improvements in OSATS and EPRS scores. The results suggest that integration of this training model into surgical curricula could be a promising strategy for enhancing vascular surgical training.

Computer-aided quality assessment of endoscopist competence during colonoscopy: a systematic review

BACKGROUND AND AIMS

Endoscopists' competence can vary widely, as shown in the variation in the adenoma detection rate (ADR). Computer-aided quality assessment (CAQ) can automatically assess performance during individual procedures. In this review we identified and described different CAQ systems for colonoscopy.

METHODS

A systematic review of the literature was done using MEDLINE, EMBASE, and Scopus based on 3 blocks of terms according to the inclusion criteria: colonoscopy, competence assessment, and automatic evaluation. Articles were systematically reviewed by 2 reviewers, first by abstract and then in full text. The methodological quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

Of 12,575 identified studies, 6831 remained after removal of duplicates and 6806 did not pass the eligibility criteria and were excluded, leaving 25 studies, of which 13 studies were included in the final analysis. Five categories of CAQ systems were identified: withdrawal speedometer (7 studies), endoscope movement analysis (3 studies), effective withdrawal time (1 study), fold examination quality (1 study), and visual gaze pattern (1 study). The withdrawal speedometer was the only CAQ system that tested its feedback by examining changes in ADR. Three studies observed an improvement in ADR, and 2 studies did not. The methodological quality of the studies was high (mean MERSQI, 15.2 points; maximum, 18 points).

CONCLUSIONS

Thirteen studies developed or tested CAQ systems, most frequently by correlating it to the ADR. Only 5 studies tested feedback by implementing the CAQ system. A meta-analysis was impossible because of the heterogeneous study designs, and more studies are warranted.

Validity of A Novel Simulator for Percutaneous Transforaminal Endoscopic Discectomy

BACKGROUND

Percutaneous transforaminal endoscopic discectomy (PTED) has steep learning curves and a high incidence of complications, but currently, efficient and economical training methods are lacking. This study aimed to validate a novel simulator for PTED.

METHODS

The simulated PTED included puncturing and establishing the working channel (PEWC) and endoscopic discectomy, with the PEWC being the tested module. Eleven experts and 21 novices were included and introduced to the simulator and tasks; all participants completed the PEWC. Outcomes included: total operation time, number of fluoroscopy for positioning the working sheath, number of spinal risk region invasion, Global Rating Scale (GRS) and a modified GRS, etc. The Mann-Whitney U test was used to compare 2 groups. Spearman's correlation coefficient analyzed continuous variables.

RESULTS

Experts outperformed novices in total operation time (P = 0.001), requiring fewer number of fluoroscopies for positioning the working sheath (P = 0.003). Additionally, experts had a lower number of spinal risk region invasions (P = 0.016) and higher scores on both the GRS (P < 0.001) and modified GRS (P < 0.001). PTED experience correlated with GRS scores (P = 0.001) and modified GRS (P < 0.001). The overall realism scored a median of 4 (3.75-5), and educational value had a median of 4 (range 3-5).

CONCLUSIONS

This study demonstrates the validity of the novel simulator, revealing significant associations between PTED experience and performance metrics in a simulated PEWC setting. Furthermore, the PEWC module also offers a good realistic design and high education value according to experts.

Development and Preliminary Evaluation of A Soft Tissue Microtia Simulator

Surgical simulation has been used extensively for learning microtia reconstruction and has almost exclusively involved framework creation. However, soft tissue reconstruction in microtia is equally challenging and would benefit from a simulation platform. This study aimed to describe the development and preliminary evaluation of a high-fidelity soft tissue microtia simulator. Three-dimensional modeling software, fused deposition 3-dimensional printing, adhesive techniques, silicones, and polyurethane rubbers were utilized to create a right lobular-type microtia simulator that comprises skin, subcutaneous tissue, and cartilage. Two expert microtia surgeons performed a microtia reconstruction on the simulator and evaluated its value and realism using a Likert-type questionnaire. The surgeons utilized a previously developed synthetic framework and successfully performed the critical steps of the soft tissue reconstruction, including marking, incising, dissection, removal of the cartilage remnant, drain insertion, insertion of the framework, closing of the skin, and demonstration of the soft tissue conforming over the framework using suction. A preliminary assessment of the simulator demonstrated that the simulator is anatomically accurate, realistic, and highly valuable as a training tool. A high-fidelity soft tissue microtia simulator was successfully developed and tested. The simulator provides a valuable training platform for learning a critical component of microtia reconstruction.

Comparing the impact of surgical expert versus non-ophthalmologist instructors on virtual-reality surgical performance: A randomized controlled trial

PURPOSE

To compare Manual Small Incision Cataract Surgery (MSICS) microsurgical performance in course participants who received virtual reality simulation-based training by either a surgical expert or a non-ophthalmologist instructor.

SETTING

Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark.

DESIGN

Randomized controlled trial.

METHODS

Residents and specialists in ophthalmology with no prior MSICS experience were included to receive virtual reality simulation training in MSICS using the HelpMeSee simulator. The participants were randomly allocated to receive training from either an experienced MSICS surgeon or a non-ophthalmologist, also known as near-peer teaching. The performances of the participants were evaluated at baseline and post-training using a MSICS proficiency-based test with evidence of validity.

RESULTS

Thirty participants were included in the study and 29 completed the course. There was no significant difference in final test score between the two groups (p = 0.13). The performance score of both groups of participants increased significantly after receiving the training (p < 0.001). All participants passed the proficiency-based test after receiving the training.

CONCLUSION

We found no significant difference in surgical proficiency-level whether the participants were trained by a surgical expert or a non-ophthalmologist instructor for MSICS in a virtual-reality based setting. The findings of this study suggest that near-peer teaching within microsurgical performance potentially could be applied with teaching outcomes comparable to a surgical expert-instructor.

Generating a Prioritized List of Operative Procedures for Simulation-based Assessment of General Surgery Trainees Through Consensus

OBJECTIVE

To develop appropriate content for high-stakes simulation-based assessments of operative competence in general surgery training through consensus.

BACKGROUND

Valid methods of summative operative competence assessment are required by competency-based training programs in surgery.

METHOD

An online Delphi consensus study was conducted. Procedures were derived from the competency expectations outlined by the Joint Committee on Surgical Training Curriculum 2021, and subsequent brainstorming. Procedures were rated according to their perceived importance, perceived procedural risk, how frequently they are performed, and simualtion feasibility by a purposive sample of 30 surgical trainers and a 5-person steering group. A modified Copenhagen Academy for Medical Education and Simulation Needs Assessment Formula was applied to the generated data to produce ranked procedural lists, which were returned to participants for re-prioritization.

RESULTS

Prioritized lists were generated for simulation-based operative competence assessments at 2 key stages of training; the end of 'phase 2' prior to the development of a sub-specialty interest, and the end of 'phase 3', that is, end-of-training certification. A total of 21 and 16 procedures were deemed suitable for assessments at each of these stages, respectively.

CONCLUSIONS

This study describes a national needs assessment approach to content generation for simulation-based assessments of operative competence in general surgery using Delphi consensus methodology. The prioritized procedural lists generated by this study can be used to further develop operative skill assessments for use in high-stakes scenarios, such as trainee progression, entrustment, and end-of-training certification, before subsequent validity testing.

Training and assessment for colorectal surgery and appendicectomy- a systematic review

AIM

There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach.

METHOD

A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported.

RESULTS

Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery.

CONCLUSION

This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence.

Randomized controlled trial of the CMR immersive virtual reality (IVR) headset training compared to e-learning for operating room configuration of the CMR versius robot

Robotic surgery offers potential advantages over laparoscopic procedures, but the training for configuring robotic systems in the operating room remains underexplored. This study seeks to validate immersive virtual reality (IVR) headset training for setting up the CMR Versius in the operating room. This single-blinded randomized control trial randomised medical students with no prior robotic experience using an online randomiser. The intervention group received IVR headset training, and the control group, e-learning modules. Assessors were blinded to participant group. Primary endpoint was overall score (OS): Likert-scale 1-5: 1 reflecting independent performance, with increasing verbal prompts to a maximum score of 5, requiring physical assistance to complete the task. Secondary endpoints included task scores, time, inter-rater reliability, and concordance with participant confidence scores. Statistical analysis was performed using IBM SPSS Version 27. Of 23 participants analysed, 11 received IVR and 12 received e-learning. The median OS was lower in the IVR group than the e-learning group 53.5 vs 84.5 (p < 0.001). VR recipients performed tasks independently more frequently and required less physical assistance than e-learning participants (p < 0.001). There was no significant difference in time to completion (p = 0.880). Self-assessed confidence scores and assessor scores differed for e-learning participants (p = 0.008), though not IVR participants (p = 0.607). IVR learning is more effective than e-learning for preparing robot-naïve individuals in operating room set-up of the CMR Versius. It offers a feasible, realistic, and accessible option in resource-limited settings and changing dynamics of operating theatre teams. Ongoing deliberate practice, however, is still necessary for achieving optimal performance. ISCRTN Number 10064213.

Developing a phantom for simulating robotic-assisted complete mesocolic excision using 3D printing and medical imaging

BACKGROUND

Robotic-assisted complete mesocolic excision is an advanced procedure mainly because of the great variability in anatomy. Phantoms can be used for simulation-based training and assessment of competency when learning new surgical procedures. However, no phantoms for robotic complete mesocolic excision have previously been described. This study aimed to develop an anatomically true-to-life phantom, which can be used for training with a robotic system situated in the clinical setting and can be used for the assessment of surgical competency.

METHODS

Established pathology and surgical assessment tools for complete mesocolic excision and specimens were used for the phantom development. Each assessment item was translated into an engineering development task and evaluated for relevance. Anatomical realism was obtained by extracting relevant organs from preoperative patient scans and 3D printing casting moulds for each organ. Each element of the phantom was evaluated by two experienced complete mesocolic excision surgeons without influencing each other's answers and their feedback was used in an iterative process of prototype development and testing.

RESULTS

It was possible to integrate 35 out of 48 procedure-specific items from the surgical assessment tool and all elements from the pathological evaluation tool. By adding fluorophores to the mesocolic tissue, we developed an easy way to assess the integrity of the mesocolon using ultraviolet light. The phantom was built using silicone, is easy to store, and can be used in robotic systems designated for patient procedures as it does not contain animal-derived parts.

CONCLUSIONS

The newly developed phantom could be used for training and competency assessment for robotic-assisted complete mesocolic excision surgery in a simulated setting.

Surgical Artificial Intelligence in Urology: Educational Applications

Surgical education has seen immense change recently. Increased demand for iterative evaluation of trainees from medical school to independent practice has led to the generation of an overwhelming amount of data related to an individual's competency. Artificial intelligence has been proposed as a solution to automate and standardize the ability of stakeholders to assess the technical and nontechnical abilities of a surgical trainee. In both the simulation and clinical environments, evidence supports the use of machine learning algorithms to both evaluate trainee skill and provide real-time and automated feedback, enabling a shortened learning curve for many key procedural skills and ensuring patient safety.

Development of a continuously perfused ex vivo kidney training model for laparoscopic partial nephrectomy: validity and efficiency

BACKGROUND

Suture hemostasis is essential for laparoscopic partial nephrectomy (LPN). This study aimed to develop, validate, and test the efficacy of a continuously perfused training model (CPTM) in LPN with high-level simulated bleeding.

MATERIALS AND METHODS

The CPTM was constructed using fresh porcine kidneys with renal arteries continuously perfused with red-dyed liquid gelatin. Twenty-nine participants with expert, intermediate, or novice laparoscopic experience levels were recruited. The expert and intermediate participants evaluated the CPTM, and the novice participants were randomly assigned to one of two groups to complete training on a CPTM or dry box training model (DBTM). Messick's framework criteria were utilized to assess the validity and training efficacy of the model. The data were analyzed using the Mann-Whitney U , Kruskal-Wallis, and Friedman tests. A value of P< 0.05 was considered statistically significant.

RESULTS

Positive comments were provided by all experts and intermediates for the Content . The Relationships with other variables demonstrated significant differences among novices, intermediates, and experts in all metrics ( P< 0.05). The Consequences showed that the CPTM helped novices acquire LPN skills. The training efficacy was significantly better than that of the DBTM ( P< 0.05). There were no significant differences between the final performances of the novices and the initial performances of the experts ( P >0.05). Synthesizing all metrics, the LPN skills learned using CPTMs were significantly improved in the 12th round of training.

CONCLUSION

The CPTM offered a high-level simulation of bleeding with realistic tissue texture for acquiring LPN skills. Training of no fewer than 12 rounds is recommended for a novice's LPN training on the CPTM.

Transfer of skills between laparoscopic and robot-assisted surgery: a systematic review

BACKGROUND

Robot-assisted surgery is today well-implemented in many surgical specialties, but requires another skill set than laparoscopy. Most often, robot-assisted surgery is considered add-on to laparoscopic skills but very little is known about the transfer of skills. The aim of the study was to examine to what extent surgical skills are transferable between laparoscopic and robot-assisted surgery.

METHODS

A systematic search was conducted in three databases (Ovid Medline, Embase, and Web of Science). Studies investigating transfer of skills between laparoscopy and robot-assisted surgery in either a phantom-based, simulation-based, animal model, or clinical setting were eligible for inclusion. Quality assessment was done using the Medical education research study quality instrument and educational New Ottawa Scale.

RESULTS

Of 15,610 studies identified, 89 studies continued to full-text reading, and 37 studies were included. Four studies were found non-comparable and were left out of the results for the primary outcome. All 33 studies explored transfer from laparoscopy to robot-assisted surgery and 17 found a positive transfer whereas 15 did not. Only 11 studies explored transfer from robot-assisted surgery to laparoscopy, of which only three found a positive transfer.

CONCLUSION

An almost equal number of publications found a positive transfer and no transfer from laparoscopic to robot-assisted surgery. Fewer studies explored the transfer from robot-assisted surgery to laparoscopy. Very little evidence supports that surgeons trained solely in robot-assisted surgery can perform laparoscopy. This must be considered in future training programs as robot-assisted surgery is expected to become the first-in-line modality for many future surgeons.

Current Status of Simulation in Thoracic Surgical Training

BACKGROUND

Simulation is playing an increasingly important role in surgical training but is not yet a mandatory part of most surgical curricula. A simulator must undergo rigorous validation to verify it as a reliable tool. The aim of this study was to review the literature to identify simulators that are currently available to augment thoracic surgical training and to analyze any evidence supporting or validating them.

METHODS

A literature search of the MEDLINE (1946 to November 2022) and Embase (1947 to November 2022) databases was performed to identify simulators for basic skills and procedures in thoracic surgery. A selection of keywords were used to perform the literature search. After identification of appropriate articles, data were extracted and analyzed.

RESULTS

Thirty-three simulators were found in 31 articles. Simulators for basic skills (n = 13) and thoracic lobectomy (n = 13) were most commonly described, followed by miscellaneous (n = 7). Most models were of a hybrid modality (n = 18). Evidence of validity was established in 48.5% (n = 16) of simulators. In total, 15.2% (n = 5) of simulators had 3 or more elements of validity demonstrated, and only 3.0% (n = 1) accomplished full validation.

CONCLUSIONS

Numerous simulators of varying modality and fidelity exist for a variety of thoracic surgical skills and procedures, although validation evidence is frequently inadequate. Simulation models may be able to provide training in basic surgical and procedural skills; however, further assessment of validity needs to be undertaken before consideration of their integration into training programs.

Benchtop simulation of the retrosigmoid approach: Validation of a surgical simulator and development of a task-specific outcome measure score

Background

Neurosurgical training is changing globally. Reduced working hours and training opportunities, increased patient safety expectations, and the impact of COVID-19 have reduced operative exposure. Benchtop simulators enable trainees to develop surgical skills in a controlled environment. We aim to validate a high-fidelity simulator model (RetrosigmoidBox, UpSurgeOn) for the retrosigmoid approach to the cerebellopontine angle (CPA).

Methods

Novice and expert Neurosurgeons and Ear, Nose, and Throat surgeons performed a surgical task using the model - identification of the trigeminal nerve. Experts completed a post-task questionnaire examining face and content validity. Construct validity was assessed through scoring of operative videos employing Objective Structured Assessment of Technical Skills (OSATS) and a novel Task-Specific Outcome Measure score.

Results

Fifteen novice and five expert participants were recruited. Forty percent of experts agreed or strongly agreed that the brain tissue looked real. Experts unanimously agreed that the RetrosigmoidBox was appropriate for teaching. Statistically significant differences were noted in task performance between novices and experts, demonstrating construct validity. Median total OSATS score was 14/25 (IQR 10-19) for novices and 22/25 (IQR 20-22) for experts (p < 0.05). Median Task-Specific Outcome Measure score was 10/20 (IQR 7-17) for novices compared to 19/20 (IQR 18.5-19.5) for experts (p < 0.05).

Conclusion

The RetrosigmoidBox benchtop simulator has a high degree of content and construct validity and moderate face validity. The changing landscape of neurosurgical training mean that simulators are likely to become increasingly important in the delivery of high-quality education. We demonstrate the validity of a Task-Specific Outcome Measure score for performance assessment of a simulated approach to the CPA.

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Surgical skills assessment of pancreaticojejunostomy using a simulator may predict patient outcomes: A multicenter prospective observational study

BACKGROUND

Pancreatoduodenectomy, an advanced surgical procedure with a high complication rate, requires surgical skill in performing pancreaticojejunostomy, which correlates with operative outcomes. We aimed to analyze the correlation between pancreaticojejunostomy assessment conducted in a simulator environment and the operating room and patient clinical outcomes.

METHODS

We recruited 30 surgeons (with different experience levels in pancreatoduodenectomy) from 11 institutes. Three trained blinded raters assessed the videos of the pancreaticojejunostomy procedure performed in the operating room using a simulator according to an objective structured assessment of technical skill and a newly developed pancreaticojejunostomy assessment scale. The correlations between the assessment score of the pancreaticojejunostomy performed in the operating room and using the simulator and between each assessment score and patient outcomes were calculated. The participants were also surveyed regarding various aspects of the simulator as a training tool.

RESULTS

There was no correlation between the average score of the pancreaticojejunostomy performed in the operating room and that in the simulator environment (r = 0.047). Pancreaticojejunostomy scores using the simulator were significantly lower in patients with postoperative pancreatic fistula than in those without postoperative pancreatic fistula (P = .05). Multivariate analysis showed that pancreaticojejunostomy assessment scores were independent factors in postoperative pancreatic fistula (P = .09). The participants highly rated the simulator and considered that it had the potential to be used for training.

CONCLUSION

There was no correlation between pancreaticojejunostomy surgical performance in the operating room and the simulation environment. Surgical skills evaluated in the simulation setting could predict patient surgical outcomes.

A synthetic model simulator for intracranial aneurysm clipping: validation of the UpSurgeOn AneurysmBox

Background and objectives

In recent decades, the rise of endovascular management of aneurysms has led to a significant decline in operative training for surgical aneurysm clipping. Simulation has the potential to bridge this gap and benchtop synthetic simulators aim to combine the best of both anatomical realism and haptic feedback. The aim of this study was to validate a synthetic benchtop simulator for aneurysm clipping (AneurysmBox, UpSurgeOn).

Methods

Expert and novice surgeons from multiple neurosurgical centres were asked to clip a terminal internal carotid artery aneurysm using the AneurysmBox. Face and content validity were evaluated using Likert scales by asking experts to complete a post-task questionnaire. Construct validity was evaluated by comparing expert and novice performance using the modified Objective Structured Assessment of Technical Skills (mOSATS), developing a curriculum-derived assessment of Specific Technical Skills (STS), and measuring the forces exerted using a force-sensitive glove.

Results

Ten experts and eighteen novices completed the task. Most experts agreed that the brain looked realistic (8/10), but far fewer agreed that the brain felt realistic (2/10). Half the expert participants (5/10) agreed that the aneurysm clip application task was realistic. When compared to novices, experts had a significantly higher median mOSATS (27 vs. 14.5; p < 0.01) and STS score (18 vs. 9; p < 0.01); the STS score was strongly correlated with the previously validated mOSATS score (p < 0.01). Overall, there was a trend towards experts exerting a lower median force than novices, however, these differences were not statistically significant (3.8 N vs. 4.0 N; p = 0.77). Suggested improvements for the model included reduced stiffness and the addition of cerebrospinal fluid (CSF) and arachnoid mater.

Conclusion

At present, the AneurysmBox has equivocal face and content validity, and future versions may benefit from materials that allow for improved haptic feedback. Nonetheless, it has good construct validity, suggesting it is a promising adjunct to training.

Knowledge Attainment and Engagement Among Medical Students: A Comparison of Three Forms of Online Learning

OBJECTIVE

This study compared knowledge attainment and student enjoyment and engagement between clinical case vignette, patient-testimony videos and mixed reality (MR) teaching via the Microsoft HoloLens 2, all delivered remotely to third year medical students. The feasibility of conducting MR teaching on a large scale was also assessed.

SETTING & PARTICIPANTS

Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend these scheduled teaching sessions and to complete the formative assessment. Inclusion of their data used as part of the research trial was optional.

PRIMARY AND SECONDARY OUTCOME MEASURES

The primary outcome measure was performance on a formative assessment, which served to compare knowledge attainment between three forms of online learning. Moreover, we aimed to explore student engagement with each form of learning via a questionnaire, and also feasibility of applying MR as a teaching tool on a large scale. Comparisons between performances on the formative assessment between the three groups were investigated using a repeated measures two-way ANOVA. Engagement and enjoyment were also analysed in the same manner.

RESULTS

A total of 252 students participated in the study. Knowledge attainment of students using MR was comparable with the other two methods. Participants reported higher enjoyment and engagement (p<0.001) for the case vignette method, compared with MR and video-based teaching. There was no difference in enjoyment or engagement ratings between MR and the video-based methods.

CONCLUSION

This study demonstrated that the implementation of MR is an effective, acceptable, and feasible way of teaching clinical medicine to undergraduate students on a large scale. However, case-based tutorials were found to be favoured most by students. Future work could further explore the best uses for MR teaching within the medical curriculum.

3D-printing a cost-effective model for mastoidectomy training

BACKGROUND

3D-printed temporal bone models can potentially provide a cost-effective alternative to cadaver surgery that can be manufactured locally at the training department. The objective of this study was to create a cost-effective 3D-printed model suitable for mastoidectomy training using entry level and commercially available print technologies, enabling individuals, without prior experience on 3D-printing, to manufacture their own models for basic temporal bone training.

METHODS

Expert technical professionals and an experienced otosurgeon identified the best material for replicating the temporal bone and created a cost-effective printing routine for the model using entry-level print technologies. Eleven participants at a temporal bone dissection course evaluated the model using a questionnaire.

RESULTS

The 3D-printed temporal bone model was printed using a material extrusion 3D-printer with a heat resistant filament, reducing melting during drilling. After printing, a few simple post-processing steps were designed to replicate the dura, sigmoid sinus and facial nerve. Modifying the 3D-printer by installing a direct-drive and ruby nozzle resulted in more successful prints and less need for maintenance. Upon evaluation by otorhinolaryngology trainees, unanimous feedback was that the model provided a good introduction to the mastoidectomy procedure, and supplementing practice to cadaveric temporal bones.

CONCLUSION

In-house production of a cost-effective 3D-printed model for temporal bone training is feasible and enables training institutions to manufacture their own models. Further, this work demonstrates the feasibility of creating new temporal bone models with anatomical variation to provide ample training opportunity.

A Validated 3D Printed Laryngeal Suturing Simulator for Endoscopic Laryngeal Cleft Repair

OBJECTIVES

Endoscopic laryngeal cleft repair (ELCR) with endolaryngeal suturing is an advanced surgical skill. This study objective was to assess the validity of 3-dimensionally (3D) printed laryngeal suturing simulator for ELCR.

STUDY DESIGN

Development and validation of a simulator for ELCR.

METHODS

An ELCR model was developed using 3D printed and readily available materials. Participants were surveyed before and after a simulation session using five-point Likert scale questions. Performance data was assessed using blinded expert video review and rated using a novel objective structured assessment of technical skills (OSATS) for endoscopic laryngeal suturing.

RESULTS

Twenty-one participants ranging from residents to attendings completed the simulation session. Survey respondents reported on a five-point Likert scale that the model was "easy to use" and "quite realistic" (both mean of 4). Confidence improved significantly in 86% of participants (p < 0.01). Overall OSATS scores (out of a total of 55) showed a median improvement in technical skills of 11.7 points (p = 0.004). OSATS demonstrated good intra-rater (κ = 0.689 and 0.677) and moderate inter-rater (κ = 0.573) reliability. Completion times improved from the first to the last suture by a median time of 512 to 350 s (decrease of 202 s, p = 0.002). Participants with no prior ELCR experience improved more than those with in vivo experience.

CONCLUSION

This study demonstrates the validity of a simulator utilizing 3D printed larynges for ELCR. A novel OSATS for endoscopic laryngeal suturing was successfully implemented. Confidence, technical skills, and completion times improved with the use of the model across a variety of participants. Laryngoscope, 133:785-791, 2023.

Objective structured assessment ultrasound skill scale for hyomental distance competence - psychometric study

BACKGROUND

Ultrasound assessment of the airway recently integrates the point-of-care approach to patient evaluation since ultrasound measurements can predict a difficult laryngoscopy and tracheal intubation. Because ultrasonography is performer-dependent, a proper training and assessment tool is needed to increase diagnostic accuracy. An objective, structured assessment ultrasound skill (OSAUS) scale was recently developed to guide training and assess competence. This work aims to study the psychometric properties of OSAUS Scale when used to evaluate competence in ultrasound hyomental distance (HMD) measurement.

METHODS

Prospective and experimental study. Volunteers were recruited and enrolled in groups with different expertise. Each participant performed three ultrasonographic HMD evaluation. The performance was videorecorded and anonymized. Five assessors blindly rated participants' performance using OSAUS scale and a Global Rating Scale (GRS). A psychometric study of OSAUS scale as assessment tool for ultrasound HMD competence was done.

RESULTS

Fifteen voluntaries participated on the study. Psychometric analysis of OSAUS showed strong internal consistency (Cronbach's alpha 0.916) and inter-rater reliability (ICC 0.720; p < 0.001). The novice group scored 15.4±0.18 (mean±SD), the intermediate 14.3±0.75 and expert 13.6±0.1.25, with a significant difference between novice and expert groups (p = 0.036). The time in seconds to complete the task was evaluated: novice (90±34) (mean±SD), intermediate (84±23) and experts (83±15), with no significant differences between groups. A strong correlation was observed between OSAUS and global rating scale (r = 0.970, p < 0.001).

CONCLUSION

The study demonstrated evidence of validity and reliability. Further studies are needed to implement OSAUS scale in the clinical setting for training and assessment of airway ultrasound competence.

Comparing Simulator Metrics and Rater Assessment of Laparoscopic Suturing Skills

BACKGROUND

Laparoscopic intracorporeal suturing is important to master and competence should be ensured using an optimal method in a simulated environment before proceeding to real operations. The objectives of this study were to gather validity evidence for two tools for assessing laparoscopic intracorporeal knot tying and compare the rater-based assessment of laparoscopic intracorporeal suturing with the assessment based on simulator metrics.

METHODS

Twenty-eight novices and 19 experienced surgeons performed four laparoscopic sutures on a Simball Box simulator twice. Two surgeons used the Intracorporeal Suturing Assessment Tool (ISAT) for blinded video rating.

RESULTS

Composite Simulator Score (CSS) had higher test-retest reliability than the ISAT. The correlation between the number performed procedures including suturing and ISAT score was 0.51, p<0.001, and 0.59 p<0.001 for CSS. We found an inter-rater reliability (0.72, p<0.001 for test 1 and 0.53 p<0.001 for test 2). The pass/fail rates for ISAT and CSS were similar.

CONCLUSION

CSS and ISAT provide similar results for assessing laparoscopic suturing but assess different aspects of performance. Using simulator metrics and raters' assessments in combination should be considered for a more comprehensive evaluation of laparoscopic knot-tying competency.

Computer-assisted simulated workplace-based assessment in surgery: application of the universal framework of intraoperative performance within a mixed-reality simulation

Objectives

Workplace-based assessment (WBA) is a key requirement of competency-based medical education in postgraduate surgical education. Although simulated workplace-based assessment (SWBA) has been proposed to complement WBA, it is insufficiently adopted in surgical education. In particular, approaches to criterion-referenced and automated assessment of intraoperative surgical competency in contextualized SWBA settings are missing.Main objectives were (1) application of the universal framework of intraoperative performance and exemplary adaptation to spine surgery (vertebroplasty); (2) development of computer-assisted assessment based on criterion-referenced metrics; and (3) implementation in contextualized, team-based operating room (OR) simulation, and evaluation of validity.

Design

Multistage development and assessment study: (1) expert-based definition of performance indicators based on framework's performance domains; (2) development of respective assessment metrics based on preoperative planning and intraoperative performance data; (3) implementation in mixed-reality OR simulation and assessment of surgeons operating in a confederate team. Statistical analyses included internal consistency and interdomain associations, correlations with experience, and technical and non-technical performances.

Setting

Surgical simulation center. Full surgical team set-up within mixed-reality OR simulation.

Participants

Eleven surgeons were recruited from two teaching hospitals. Eligibility criteria included surgical specialists in orthopedic, trauma, or neurosurgery with prior VP or kyphoplasty experience.

Main outcome measures

Computer-assisted assessment of surgeons' intraoperative performance.

Results

Performance scores were associated with surgeons' experience, observational assessment (Objective Structured Assessment of Technical Skill) scores and overall pass/fail ratings. Results provide strong evidence for validity of our computer-assisted SWBA approach. Diverse indicators of surgeons' technical and non-technical performances could be quantified and captured.

Conclusions

This study is the first to investigate computer-assisted assessment based on a competency framework in authentic, contextualized team-based OR simulation. Our approach discriminates surgical competency across the domains of intraoperative performance. It advances previous automated assessment based on the use of current surgical simulators in decontextualized settings. Our findings inform future use of computer-assisted multidomain competency assessments of surgeons using SWBA approaches.

The Sequence of Steps: A Key Concept Missing in Surgical Training-A Systematic Review and Recommendations to Include It

BACKGROUND

Surgical procedures have an inherent feature, which is the sequence of steps. Moreover, studies have shown variability in surgeons' performances, which is valuable to expose residents to different ways to perform a procedure. However, it is unclear how to include the sequence of steps in training programs.

METHODS

We conducted a systematic review, including studies reporting explicit teaching of a standard sequence of steps, where assessment considered adherence to a standard sequence, and where faculty or students at any level participated. We searched for articles on PubMed, EMBASE, CINAHL, Web of Science, and Google Scholar databases.

RESULTS

We selected nine articles that met the inclusion criteria. The main strategy to teach the sequence was to use videos to demonstrate the procedure. The simulation was the main strategy to assess the learning of the sequence of steps. Non-standardized scoring protocols and written tests with variable validity evidence were the instruments used to assess the learning, and were focused on adherence to a standard sequence and the omission of steps.

CONCLUSIONS

Teaching and learning assessment of a standard sequence of steps is scarcely reported in procedural skills training literature. More research is needed to evaluate whether the new strategies to teach and assess the order of steps work. We recommend the use of Surgical Process Models and Surgical Data Science to incorporate the sequence of steps when teaching and assessing procedural skills.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Stress in surgical educational environments: a systematic review

BACKGROUND

The effects of stress on surgical residents and how stress management training can prepare residents to effectively manage stressful situations is a relevant topic. This systematic review aimed to analyze the literature regarding (1) the current stress monitoring tools and their use in surgical environments, (2) the current methods in surgical stress management training, and (3) how stress affects surgical performance.

METHODS

A search strategy was implemented to retrieve relevant articles from Web of Science, Scopus, and PubMed. The 787 initially retrieved articles were reviewed for further evaluation according to the inclusion/exclusion criteria (Prospero registration number CRD42021252682).

RESULTS

Sixty-one articles were included in the review. The stress monitoring methods found in the articles showed heart rate analysis as the most used monitoring tool for physiological parameters while the STAI-6 scale was preferred for psychological parameters. The stress management methods found in the articles were mental-, simulation- and feedback-based training, with the mental-based training showing clear positive effects on participants. The studies analyzing the effects of stress on surgical performance showed both negative and positive effects on technical and non-technical performance.

CONCLUSIONS

The impact of stress responses presents an important factor in surgical environments, affecting residents' training and performance. This study identified the main methods used for monitoring stress parameters in surgical educational environments. The applied surgical stress management training methods were diverse and demonstrated positive effects on surgeons' stress levels and performance. There were negative and positive effects of stress on surgical performance, although a collective pattern on their effects was not clear.

Assessing VATS competence based on simulated lobectomies of all five lung lobes

OBJECTIVES

To determine the number of procedures and expert raters necessary to provide a reliable assessment of competence in Video-Assisted Thoracoscopic Surgery (VATS) lobectomy.

METHODS

Three randomly selected VATS lobectomies were performed on a virtual reality simulator by participants with varying experience in VATS. Video recordings of the procedures were independently rated by three blinded VATS experts using a modified VATS lobectomy assessment tool (VATSAT). The unitary framework of validity was used to describe validity evidence, and generalizability theory was used to explore the reliability of different assessment options.

RESULTS

Forty-one participants (22 novices, 10 intermediates, and 9 experienced) performed a total of 123 lobectomies. Internal consistency reliability, inter-rater reliability, and test-retest reliability were 0.94, 0.85, and 0.90, respectively. Generalizability theory found that a minimum of two procedures and four raters or three procedures and three raters were needed to ensure the overall reliability of 0.8. ANOVA showed significant differences in test scores between the three groups (P < 0.001). A pass/fail level of 19 out of 25 points was established using the contrasting groups' standard setting method, leaving one false positive (one novice passed) and zero false negatives (all experienced passed).

CONCLUSION

We demonstrated validity evidence for a VR simulator test with different lung lobes, and a credible pass/fail level was identified. Our results can be used to implement a standardized mastery learning training program for trainees in VATS lobectomies that ensures that everyone reaches basic competency before performing supervised operations on patients.

A new approach for the acquisition of trauma surgical skills: an OSCE type of simulation training program

BACKGROUND

Worldwide, trauma-related deaths are one of the main causes of mortality. Appropriate surgical treatment is crucial to prevent mortality, however, in the past decade, general surgery residents' exposure to trauma cases has decreased, particularly since the COVID-19 pandemic. In this context, accessible simulation-based training scenarios are essential.

METHODS

A low-cost, previously tested OSCE scenario for the evaluation of surgical skills in trauma was implemented as part of a short training boot camp for residents and recently graduated surgeons. The following stations were included bowel anastomosis, vascular anastomosis, penetrating lung injury, penetrating cardiac injury, and gastric perforation (laparoscopic suturing). A total of 75 participants from 15 different programs were recruited. Each station was videotaped in high definition and assessed in a remote and asynchronous manner. The level of competency was assessed through global and specific rating scales alongside procedural times. Self-confidence to perform the procedure as the leading surgeon was evaluated before and after training.

RESULTS

Statistically significant differences were found in pre-training scores between groups for all stations. The lowest scores were obtained in the cardiac and lung injury stations. After training, participants significantly increased their level of competence in both grading systems. Procedural times for the pulmonary tractotomy, bowel anastomosis, and vascular anastomosis stations increased after training. A significant improvement in self-confidence was shown in all stations.

CONCLUSION

An OSCE scenario for training surgical skills in trauma was effective in improving proficiency level and self-confidence. Low pre-training scores and level of confidence in the cardiac and lung injury stations represent a deficit in residency programs that should be addressed. The incorporation of simulation-based teaching tools at early stages in residency would be beneficial when future surgeons face extremely severe trauma scenarios.

Development and validation of a composed canine simulator for advanced veterinary laparoscopic training

The development of innovative simulation models for veterinary laparoscopic surgery training is a priority today. This study aimed to describe a didactic simulation tool for the training of total laparoscopic gastropexy (TLG) with intracorporeal sutures in dogs. CALMA Veterinary Lap-trainer composite simulator (CLVTS) was developed from a plaster cast of 2 Great Dane canines mimicking the space and the correct position to carry out a TLG. After video instruction, 16 veterinarians with different degrees of experience in minimally invasive surgery (Experts, n = 6 and intermediates, n = 10) evaluated four sequential simulating TLG with intracorporeal suturing in the CLVTS. Subsequently, they completed an anonymous questionnaire analyzing the realism, usefulness, and educational quality of the simulator. The CLVTS showed a good preliminary acceptance (4.7/5) in terms of the usefulness and adequacy of the exercises that, in the participants' opinion, are appropriate and are related to the difficulty of the TLG. In addition, both experienced and intermediate surgeons gave high marks (4.5/5) to the feeling of realism, design, and practicality. There were no significant differences between the responses of the two groups. The results suggest that the CVLTS has both face and content validity. Where it can be practiced in a structured environment for the development of a total laparoscopic gastropexy with intracorporeal suture and without compromising patient safety, but still has some limitations of the scope of the study. Further studies are needed to establish the ability to assess or measure technical skills, including the degree of transferability to the actual surgical environment.

Modern Educational Simulation-Based Tools Among Residents of Ophthalmology: A Narrative Review

INTRODUCTION

With no specific overview on simulation-based training for educational purposes among residents in ophthalmology, this narrative review attempts to highlight the current literature on modern educational simulation-based tools used to educate residents in ophthalmology.

METHODS

We searched databases Web of Science and PubMed between March 15 and July 21, 2022. Relevant and accessible articles and abstracts published after 2006 and in English only were included.

RESULTS

Simulation-based cataract surgery is associated with better outcomes in the operating room and faster surgeries. Construct validity has been established across different procedures and levels in simulation-based cataract surgery and simulation-based vitreoretinal surgery. Other simulation-based procedures indicate promising results but in general lack evidence-based validity.

DISCUSSION

This narrative review highlights and evaluates the current and relevant literature of modern educational simulation-based tools to train ophthalmology residents in different fundamental skills like simulation-based ophthalmoscopy and complex surgical procedures like simulation-based cataract surgery and vitreoretinal surgery. Some studies attempt to develop simulators for the use in education of ophthalmology residents. Other studies strive to establish validity of the respective procedures or modern education tools and some studies investigate the effect of simulation-based training. The most validated modern educational simulation-based tool is the Eyesi Surgical Simulator (VRmagic, Germany). However, other modern educational simulation-based tools have also been evaluated, including the HelpMeSee Eye Surgery Simulator (HelpMeSee Inc., New York, USA) and the MicroVisTouch Surgical Simulator (ImmersiveTouch, USA).

CONCLUSION

Simulation-based training has already been established for residents in ophthalmology to benefit the most from skill-demanding procedures resulting in better learning and better patient handling. Future studies should aim to validate more simulation-based procedures for the teaching of ophthalmology residents so that the evidence is kept at a high standard.

A Systematic Review of Simulation-Based Training in Vascular Surgery

Introduction

Recent advancements in surgical technology, reduced working hours, and training opportunities exacerbated by the COVID-19 pandemic have led to an increase in simulation-based training. Furthermore, a rise in endovascular procedures has led to a requirement for high-fidelity simulators that offer comprehensive feedback. This review aims to identify vascular surgery simulation models and assess their validity and levels of effectiveness (LoE) for each model in order to successfully implement them into current training curricula.

Methods

PubMed and EMBASE were searched on January 1, 2021, for full-text English studies on vascular surgery simulators. Eligible articles were given validity ratings based on Messick’s modern concept of validity alongside an LoE score according to McGaghie’s translational outcomes.

Results

Overall 76 eligible articles validated 34 vascular surgery simulators and training courses for open and endovascular procedures. High validity ratings were achieved across studies for: content (35), response processes (12), the internal structure (5), relations to other variables (57), and consequences (2). Only seven studies achieved an LoE greater than 3/5. Overall, ANGIO Mentor was the most highly validated and effective simulator and was the only simulator to achieve an LoE of 5/5.

Conclusions

Simulation-based training in vascular surgery is a continuously developing field with exciting future prospects, demonstrated by the vast number of models and training courses. To effectively integrate simulation models into current vascular surgery curricula and assessments, there is a need for studies to look at trainee skill retention over a longer period of time. A more detailed discussion on cost-effectiveness is also needed.

National Implementation of Simulator Training Improves Transurethral Resection of Bladder Tumours in Patients

Background

Objective

Design, setting, and participants

Outcome measurements and statistical analyses

Results and limitations

Conclusions

Patient summary

Using Immersive Virtual Reality Simulation to Ensure Competence in Contrast-Enhanced Ultrasound

Contrast-enhanced ultrasound (CEUS) is used in various medical specialties as a diagnostic imaging tool and for procedural guidance. Experience in the procedure is currently attained via supervised clinical practice that is challenged by patient availability and risks. Prior simulation-based training and subsequent assessment could improve and ensure competence before performance on patients, but no simulator currently exists. Immersive virtual reality (IVR) is a new promising simulation tool that can replicate complex interactions and environments that are unfeasible to achieve by traditional simulators. This study was aimed at developing an IVR simulation-based test for core CEUS competencies and gathering validity evidence for the test in accordance with Messick's framework. The test was developed by IVR software specialists and clinical experts in CEUS and medical education and imitated a CEUS examination of a patient with a focal liver lesion with emphasis on the pre-contrast preparations. Twenty-five medical doctors with varying CEUS experience were recruited as test participants, and their results were used to analyze test quality and to establish a pass/fail standard. The final test of 23 test items had good internal reliability (Cronbach's α = 0.85) and discriminatory abilities. The risks of false positives and negatives (9.1% and 23.6%, respectively) were acceptable for the test to be used as a certification tool prior to supervised clinical training in CEUS.

Overcoming Gender Bias in Assessment of Surgical Skills

OBJECTIVE

Female and male surgical residents are treated differently based on their gender. The use of assessment tools can help obtain objectivity in surgical skills assessment, avoid gender bias, and promote equal learning opportunities. The objective of the study was to explore whether knowledge of gender causes bias in the rating of surgical skills, and whether the raters' gender, surgical specialty, or experience affect ratings.

DESIGN

The study is designed as a cross-sectional study, where an anonymous video showing surgical performance was rated by surgeons from different surgical specialties and different levels of surgical experience. The same video was presented to two randomized groups as either a male or female surgical trainee performing the procedure.

SETTING

The participants used Objective Structured Assessment of Technical Skills (OSATS) to rate a video-recorded exam from a surgical skills course at Copenhagen Academy for Medical Education and Simulation (CAMES). The video rating was done through an internet-based platform and could thus be performed at home or at another location of choice.

PARTICIPANTS

Novice and experienced surgeons from departments of gastrointestinal surgery, gynaecology, and urology in Denmark.

RESULTS

There was no difference in OSATS rating score of the perceived female and male trainee (female trainee (n = 50):12.26 (SD = 3.08), male trainee (n = 57): 12.00 (SD = 3.28), p-value = 0.67). Rater characteristics: gender, age, surgical experience, and surgical specialty did not affect ratings.

CONCLUSIONS

When OSATS a systematic assessment tool was used in rating surgical trainees we did not find any significant implicit gender bias. The rating was unaffected by the raters' gender, age, surgical specialty, or experience.

Training and assessment of musculoskeletal ultrasound and injection skills-a systematic review

OBJECTIVES

To examine how residents are trained and assessed in musculoskeletal ultrasound (MSUS), MSUS-guided and landmark-guided joint aspiration and injection. Additionally, to present the available assessment tools and examine their supporting validity evidence.

METHODS

A systematic search of PubMed, Cochrane Library, and Embase was conducted in accordance with the PRISMA guidelines and studies published from January 1, 2000 to May 31, 2021 were included. Two independent reviewers performed the search and data extraction. The studies were evaluated using the Medical Education Research Quality Instrument (MERSQI).

RESULTS

9,884 articles were screened, and 43 were included; 3 were randomized studies, 21 pre- and post-test studies, 16 descriptive studies, and 3 studies developing assessment tools. The studies used various theoretical training modalities e.g. lectures, anatomical quizzes and e-learning. The practical training models varied from mannequins and cadavers to healthy volunteers and patients. The majority of studies used subjective "comfort level" as assessment, others used practical examination and/or theoretical examination. All training programs increased trainees' self-confidence, theoretical knowledge, and/or practical performance, however few used validated assessment tools to measure the effect. Only one study met the MERSQI high methodical quality cut-off score of 14.

CONCLUSION

The included studies were heterogeneous, and most were of poor methodological quality and not based on contemporary educational theories. This review highlights the need for educational studies using validated theoretical and practical assessment tools to ensure optimal MSUS training and assessment in rheumatology.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

Step-by-step development and evaluation of a 3D printed home-made low-cost pediatric tracheobronchial tree for foreign body aspiration extractions

OBJECTIVES

The management of foreign body aspirations (FBA) is dreaded by pediatric physicians due to the high risk of respiratory distress and a potential fatal outcome, favored by a lack of experience of young specialists. Furthermore, there has been an increasing requirement for low-cost simulation. The aim was to describe the step-by-step manufacturing process and to validate a low-cost, easily home-made training model of pediatric tracheo-bronchial tree (pTBT) for simulation-based training in order to teach young physicians to practice foreign body (FBA) extractions.

METHODS

A simulator was designed in order to reproduce the physical and esthetic properties of a pTBT. The production cost of a single simulator was estimated. The simulator was then tested by experienced physicians using a rigid bronchoscope. A manufacturing manual of the simulator is hereby presented. A group of 7 experienced pediatric otolaryngologists performed a FBA extraction in the conditions of installation of an operating room.

RESULTS

The result of the survey showed a high fidelity of the simulator in mimicking the biological esthetics and physical properties of a pTBT during a FBA extraction (mean 4.3 ± 0.8). The total cost of the custom-made simulator is about 20.5 € ($23.4) for the production of the first simulator.

CONCLUSIONS

A highly realistic and easily reproducible pediatric tracheo-bronchial tree simulator is presented and can therefore be used during simulation-based training.

Simulation in Upper and Lower Limb Trauma Skill Acquisition: A Review

SUMMARY STATEMENT

This review aimed to explore the published evidence with regard to the types and composition of both full- and part-task trainers to teach surgeons extremity exploration procedures in limb trauma management. Studies were included if they reported the development and/or validation of synthetic or virtual task trainers. Studies were evaluated to determine their derivation, usability, and clinical utility.A total of 638 citations were identified and 63 satisfied the inclusion criteria. Twenty-five articles addressed simulator validation and 36 addressed level of learning achieved with simulator engagement. Two studies described a dedicated limb simulator. Simulators were developed to repair limb structures including skin (n = 15), tendon (n = 7), nerve (n = 1), fascia (n = 1), muscle (n = 1), vascular (n = 24), and bone (n = 11). Considerations such as material fidelity, learning outcomes, cost or reusability, validity, and effectiveness are inconsistently reported. Future studies should address design standards for the effective production of synthetic or virtual simulators for limb trauma management.

Comparative Effectiveness of Training with Simulators Versus Traditional Instruction in Veterinary Education: Meta-Analysis and Systematic Review

One of the greatest challenges in veterinary education is adequately preparing students with the clinical skills they need to be successful health care providers. Integration of simulators, devices used to represent real world phenomena under test conditions, into the veterinary curriculum can help address challenges to clinical instruction. The use of simulators in veterinary education is increasing; however, their effectiveness remains unclear. This meta-analysis seeks to synthesize the evidence for simulator training in veterinary education to provide a consensus of effect, guide the integration of simulators into the curriculum, and direct the development of future simulation-based research in veterinary medicine. A systematic search identified 416 potential manuscripts from which 60 articles were included after application of inclusion criteria. Information was extracted from 71 independent experiments. The overall weighted mean effect size for simulator training was g = 0.49 for the random-effects model. The outcome measures of knowledge, time, process, and product all produced statistically significant mean effect sizes favoring simulation (d = 0.41, 0.35, 0.70, 0.53, respectively). A moderator analysis revealed that study characteristics and instructional design features moderated the effectiveness of simulator training. Overall, the results indicate that simulator training in veterinary education can be effective for knowledge and clinical skill outcomes.

Merged virtual reality teaching of the fundamentals of laparoscopic surgery: a randomized controlled trial

Background

The COVID-19 pandemic challenges our ability to provide surgical education, as our ability to gather and train together has been restricted due to safety concerns. However, the importance of quality surgical education has remained. High-fidelity simulation platforms have been developed that merge virtual reality video streams to allow for remote instruction and collaboration. This study sought to validate the use of a merged virtual reality (MVR) platform for the instruction and assessment of the fundamentals of laparoscopic surgery (FLS) skills.

Methods

This was a prospective randomized controlled non-inferiority study. Thirty participants were randomized between three groups: The standard group received in-person instruction and expert feedback, the experimental group received identical training via the MVR platform, and the control group practiced on their own, but received no feedback. All participants were pre-tested for baseline performance at the beginning of the study. Change in performance was evaluated immediately after training and one month later for retention. Ordinary one-way analysis of variance was used to evaluate the effects of time, group, and time-on-group.

Results

The pre-test confirmed baseline homogeneity between the groups. MVR was non-inferior to standard in-person training for total FLS times on either the post-test (p = 0.632) or the retention test (p = 0.829). Performance was also identical between MVR and standard training groups for each of the individual FLS tasks. Each group improved significantly in nearly all tasks after practice; however, the standard and MVR training groups both improved significantly more than controls for the ligating loop, extracorporeal suturing, intracorporeal suturing, and total FLS task training but did not reach statistical significance for peg transfer and pattern cut tasks.

Conclusion

This randomized, controlled trial has demonstrated the use of an MVR platform as non-inferior to in-person instruction for the FLS program, forming the foundation for future work on remote instruction and collaboration.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-021-08939-4.

The Resource Costs of Maintaining Learner Utilization of a Simulation Center During the COVID-19 Pandemic

BACKGROUND

Despite advances in online education during the COVID-19 pandemic, its impact on surgical simulation remains unclear. The aim of this study was to compare the costs and resources required to maintain simulation training in the pandemic and to evaluate how it affected exposure of medical students to simulation during their surgical clerkship.

METHODS

The number of learners, contact hours, staff hours, and costs were collected from a multi-departmental simulation center of a single academic institution in a retrospective fashion. Utilization and expenditure metrics were compared between the first quarter of academic years 2018-2020. Statistical analysis was performed to evaluate potential differences between overall resource utilization before and during the pandemic, and subgroup analysis was performed for the resources required for the training of the third-year medical students.

RESULTS

The overall number of learners and contact hours decreased during the first quarter of the academic year 2020 in comparison with 2019 and 2018. However, the staff hours increased. In addition, the costs for PPE increased for the same periods of time. In the subgroup analysis of the third-year medical students, there was an increase in the number of learners, as well as in the staff hours and in the space required to perform the simulation training.

DISCUSSION

Despite an increase in costs and resources spent on surgical simulation during the pandemic, the utilization by academic entities has remained unaffected. Further studies are required to identify potential solutions to lower simulation resources without a negative impact on the quality of surgical simulation.

Validation of a novel assessment tool identifying proficiency in Transurethral Bladder Tumour Resection: The OSATURBS assessment tool

BACKGROUND

Competence in transurethral bladder tumour resection (TURB) is critical in bladder cancer management and should be ensured before independent practice.

OBJECTIVE

Develop an assessment tool for TURB and explore validity evidence in a clinical context.

DESIGN, SETTING, AND PARTICIPANTS

July 2019-March 2021, a total of 33 volunteer doctors from three hospitals were included. Participants performed two TURB procedures on patients with bladder tumours. A newly developed assessment tool (OSATURBS) was used for direct observation assessment, self-assessment, and blinded video-assessment. Outcome measurements and statistical analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t-test and independent samples t-test.

RESULTS AND LIMITATIONS

The internal consistency reliability across items Cronbach's alpha was 0.94 (n = 260, p < 0.001). Inter-rater reliability = 0.80 (n = 64, p < 0.001). Test-retest correlation was high, r = 0.71 (n = 32, p < 0.001). Relation to TURB experience was high, r = 0.71 (n = 32, p < 0.001). Pass/fail score = 19 points. Direct observation assessments were strongly correlated with video ratings (r = 0.85, p < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances.

CONCLUSIONS

OSATURBS assessment tool for TURB can be used for assessment of surgical proficiency in the clinical setting. Direct observation assessment and self-assessment are biased, and blinded video-assessment of TURB performances is advised.

Design and Validity Evidence for a Unique Endoscopy Simulator Using a Commercial Video Game

Background Procedural simulation enhances early endoscopy training. Multiple commercial simulators are available; however, their application is limited by cost and poor user compliance. First-person "shooter" (FPS) video games are popular. In this study, we aimed to show that a novel in-house designed colonoscope controller used to play an FPS video game shares similar constructs with real-life endoscopy. Methodology Participants completed the first three levels on an FPS video game, Portal (Valve Corporation, Bellevue, WA), first using a conventional controller and then the modified endoscope controller. A total of 12 expert endoscopists and 12 surgical residents with minimal endoscopy experience were evaluated based on completion time, button presses, and hand motion analyses. Results Experts outperformed novices for completion time (expert: 944 seconds; novice: 1,515 seconds; p = 0.006) and hand movements (expert: 1,263.1; novice: 2,052.6; p = 0.004) in using the novel colonoscope controller. There was no difference in button presses or total path length traveled. Furthermore, performance did not differ using conventional game controls. Conclusions Experts outperformed novices using the endoscope but not the conventional controller with respect to the economy of movement and completion time. This result confirms that our endoscope-controlled video game shares similar paradigms with real-life endoscopy and serves as a first step toward creating a more enjoyable and cheaper alternative to commercially available endoscopy simulators.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.