Performance metrics in mastoidectomy training: a systematic review

Virtual temporal bone simulators and their use in surgical training: a narrative review

OBJECTIVE

Temporal bone dissection is a difficult skill to acquire, and the challenge has recently been further compounded by a reduction in conventional surgical training opportunities during the coronavirus disease 2019 pandemic. Consequently, there has been renewed interest in ear simulation as an adjunct to surgical training for trainees. We review the state-of-the-art virtual temporal bone simulators for surgical training.

MATERIALS AND METHODS

A narrative review of the current literature was performed following a Medline search using a pre-determined search strategy.

RESULTS AND ANALYSIS

Sixty-one studies were included. There are five validated temporal bone simulators: Voxel-Man, CardinalSim, Ohio State University Simulator, Melbourne University's Virtual Reality Surgical Simulation and Visible Ear Simulator. The merits of each have been reviewed, alongside their role in surgical training.

CONCLUSION

Temporal bone simulators have been demonstrated to be useful adjuncts to conventional surgical training methods and are likely to play an increasing role in the future.

Further Validity Evidence for Patient-Specific Virtual Reality Temporal Bone Surgical Simulation

OBJECTIVE

Patient-specific virtual reality (VR) simulation of cochlear implant (CI) surgery potentially enables preoperative rehearsal and planning. We aim to gather supporting validity evidence for patient-specific simulation through the analysis of virtual performance and comparison with postoperative imaging.

METHODS

Prospective, multi-institutional study. Pre- and postoperative cone-beam CT scans of CI surgical patients were obtained and processed for patient-specific VR simulation. The virtual performances of five trainees and four attendings were recorded and (1) compared with volumes removed during actual surgery as determined in postoperative imaging, and (2) assessed using the Copenhagen Cochlear Implant Surgery Assessment Tool (CISAT) by two blinded raters. The volumes compared were cortical mastoidectomy, facial recess, and round window (RW) cochleostomy as well as violation of the facial nerve and chorda.

RESULTS

Trainees drilled more volume in the cortical mastoidectomy and facial recess, whereas attendings drilled more volume for the RW cochleostomy and made more violations. Except for the cochleostomy, attendings removed volumes closer to that determined in postoperative imaging. Trainees achieved a higher CISAT performance score compared with attendings (22.0 vs. 18.4 points) most likely due to lack of certain visual cues.

CONCLUSION

We found that there were differences in performance of trainees and attendings in patient-specific VR simulation of CI surgery as assessed by raters and in comparison with actual drilled volumes. The presented approach of volume comparison is novel and might be used for further validation of patient-specific VR simulation before clinical implementation for preoperative rehearsal in temporal bone surgery.

LEVEL OF EVIDENCE

n/a Laryngoscope, 134:1403-1409, 2024.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Video Analysis of Otologic Instrument Movement During Resident Mastoidectomies

OBJECTIVE

To measure surgical instrument movement during resident mastoidectomies and identify metrics that correlate with experience.

STUDY DESIGN

Retrospective case series.

SETTING

Tertiary care center.

SUBJECTS

Ten postgraduate year (PGY) 2, 6 PGY3, 7 PGY4, and 19 PGY5 recordings of mastoidectomy performed by otolaryngology residents.

INTERVENTIONS

One-minute intraoperative recordings of mastoidectomies performed during cochlear implantation were collected. Drill and suction-irrigator motion were analyzed with sports motion tracking software.

MAIN OUTCOME MEASURES

Mean instrument speed, angle, and angular velocity were calculated. Mann-Whitney U tests compared mean instrument metrics between PGY levels. Change in drill speed for seven residents between their PGY2 to PGY5 years was individually analyzed.

RESULTS

Mean drill speed was significantly greater for PGY5 residents compared with PGY2s (2.9 versus 1.8 cm/s, p = 0.001). Compared with PGY2 residents, suction speed was greater as a PGY5 (1.2 versus 0.9 cm/s; p = 0.201) and significantly greater as a PGY4 (1.5 versus 0.9 cm/s, p = 0.039). Of the seven residents individually analyzed, group mean drill speed increased by 0.4 cm/s, yearly.

CONCLUSIONS

Drill and suction-irrigator movement during the second minute of drilling of a cortical mastoidectomy seems to increase with resident level. Objective video analysis is a potential adjunct for differentiating novices from more experienced surgeons and monitoring surgical skills progress.

Automated summative feedback improves performance and retention in simulation training of mastoidectomy: a randomised controlled trial

OBJECTIVE

This study aimed to investigate the effects of automated metrics-based summative feedback on performance, retention and cognitive load in distributed virtual reality simulation training of mastoidectomy.

METHOD

Twenty-four medical students were randomised in two groups and performed 15 mastoidectomies on a distributed virtual reality simulator as practice. The intervention group received additional summative metrics-based feedback; the control group followed standard instructions. Two to three months after training, participants performed a retention test without learning supports.

RESULTS

The intervention group had a better final-product score (mean difference = 1.0 points; p = 0.001) and metrics-based score (mean difference = 12.7; p < 0.001). At retention, the metrics-based score for the intervention group remained superior (mean difference = 6.9 per cent; p = 0.02). Also at the retention, cognitive load was higher in the intervention group (mean difference = 10.0 per cent; p < 0.001).

CONCLUSION

Summative metrics-based feedback improved performance and lead to a safer and faster performance compared with standard instructions and seems a valuable educational tool in the early acquisition of temporal bone skills.

Automated assessment of cortical mastoidectomy performance in virtual reality

INTRODUCTION

Cortical mastoidectomy is a core skill that Otolaryngology trainees must gain competency in. Automated competency assessments have the potential to reduce assessment subjectivity and bias, as well as reducing the workload for surgical trainers.

OBJECTIVES

This study aimed to develop and validate an automated competency assessment system for cortical mastoidectomy.

PARTICIPANTS

Data from 60 participants (Group 1) were used to develop and validate an automated competency assessment system for cortical mastoidectomy. Data from 14 other participants (Group 2) were used to test the generalisability of the automated assessment.

DESIGN

Participants drilled cortical mastoidectomies on a virtual reality temporal bone simulator. Procedures were graded by a blinded expert using the previously validated Melbourne Mastoidectomy Scale: a different expert assessed procedures by Groups 1 and 2. Using data from Group 1, simulator metrics were developed to map directly to the individual items of this scale. Metric value thresholds were calculated by comparing automated simulator metric values to expert scores. Binary scores per item were allocated using these thresholds. Validation was performed using random sub-sampling. The generalisability of the method was investigated by performing the automated assessment on mastoidectomies performed by Group 2, and correlating these with scores of a second blinded expert.

RESULTS

The automated binary score compared with the expert score per item had an accuracy, sensitivity and specificity of 0.9450, 0.9547 and 0.9343, respectively, for Group 1; and 0.8614, 0.8579 and 0.8654, respectively, for Group 2. There was a strong correlation between the total scores per participant assigned by the expert and calculated by the automatic assessment method for both Group 1 (r = .9144, P < .0001) and Group 2 (r = .7224, P < .0001).

CONCLUSION

This study outlines a virtual reality-based method of automated assessment of competency in cortical mastoidectomy, which proved comparable to the assessment provided by human experts.

Assessing competence in cochlear implant surgery using the newly developed Cochlear Implant Surgery Assessment Tool

PURPOSE

To develop and gather validity evidence for a novel tool for assessment of cochlear implant (CI) surgery, including virtual reality CI surgery training.

METHODS

Prospective study gathering validity evidence according to Messick's framework. Four experts developed the CI Surgery Assessment Tool (CISAT). A total of 35 true novices (medical students), trained novices (residents) and CI surgeons performed two CI-procedures each in the Visible Ear Simulator, which were rated by three blinded experts. Classical test theory and generalizability theory were used for reliability analysis.

RESULTS

The CISAT significantly discriminated between the three groups (p < 0.001). The generalizability coefficient was 0.76 and most of the score variance (53.3%) was attributable to the participant and only 6.8% to the raters. When exploring a standard setting for CI surgery, the contrasting groups method suggested a pass/fail score of 36.0 points (out of 55), but since the trained novices performed above this, we propose using the mean CI surgeon performance score (45.3 points).

CONCLUSION

Validity evidence for simulation-based assessment of CI performance supports the CISAT. Together with the standard setting, the CISAT might be used to monitor progress in competency-based training of CI surgery and to determine when the trainee can advance to further training.

Objective and automated assessment of surgical technical skills with IoT systems: A systematic literature review

The assessment of surgical technical skills to be acquired by novice surgeons has been traditionally done by an expert surgeon and is therefore of a subjective nature. Nevertheless, the recent advances on IoT (Internet of Things), the possibility of incorporating sensors into objects and environments in order to collect large amounts of data, and the progress on machine learning are facilitating a more objective and automated assessment of surgical technical skills. This paper presents a systematic literature review of papers published after 2013 discussing the objective and automated assessment of surgical technical skills. 101 out of an initial list of 537 papers were analyzed to identify: 1) the sensors used; 2) the data collected by these sensors and the relationship between these data, surgical technical skills and surgeons' levels of expertise; 3) the statistical methods and algorithms used to process these data; and 4) the feedback provided based on the outputs of these statistical methods and algorithms. Particularly, 1) mechanical and electromagnetic sensors are widely used for tool tracking, while inertial measurement units are widely used for body tracking; 2) path length, number of sub-movements, smoothness, fixation, saccade and total time are the main indicators obtained from raw data and serve to assess surgical technical skills such as economy, efficiency, hand tremor, or mind control, and distinguish between two or three levels of expertise (novice/intermediate/advanced surgeons); 3) SVM (Support Vector Machines) and Neural Networks are the preferred statistical methods and algorithms for processing the data collected, while new opportunities are opened up to combine various algorithms and use deep learning; and 4) feedback is provided by matching performance indicators and a lexicon of words and visualizations, although there is considerable room for research in the context of feedback and visualizations, taking, for example, ideas from learning analytics.

The Impact of Virtual Reality Training on the Quality of Real Antromastoidectomy Performance

BACKGROUND

The aim of this paper is to analyze the results of virtual reality (VR) antromastoidectomy simulation training and the transferability of the obtained skills to real temporal bone surgery.

METHODS

The study was conducted prospectively on a group of 10 physicians, and was composed of five VR simulation training sessions followed by live temporal bone surgery. The quality of performance was evaluated with a Task-Based Checklist (TBC) prepared by John Hopkins Hospital. Additionally, during every VR session, the number and type of mistakes (complications) were noted.

RESULTS

The quality of performance measured by the TBC increased significantly during consecutive VR sessions. The mean scores for the first and fifth sessions were 1.84 and 4.27, respectively (p < 0.001). Furthermore, the number of mistakes in consecutive VR sessions was gradually reduced from 11 to 0. During supervised surgery, all the participants were able to perform at least part of an antromastoidectomy, and the mean TBC score was 3.57. There was a significant strong positive correlation between the individual results of the fifth VR session and the individual results of supervised surgery in the operating room (r_p = 0.89, p = 0.001).

CONCLUSIONS

Virtual reality for temporal bone training makes it possible to acquire surgical skills in a safe environment before performing supervised surgery. Furthermore, the individual final score of virtual antromastoidectomy training allows a prediction of the quality of performance in real surgery.

The Effect of Simulator-Integrated Tutoring for Guidance in Virtual Reality Simulation Training

INTRODUCTION

Simulation-integrated tutoring in virtual reality (VR) simulation training by green lighting is a common learning support in simulation-based temporal bone surgical training. However, tutoring overreliance can negatively affect learning. We therefore wanted to investigate the effects of simulator-integrated tutoring on performance and learning.

METHODS

A prospective, educational cohort study of a learning intervention (simulator-integrated tutoring) during repeated and distributed VR simulation training for directed, self-regulated learning of the mastoidectomy procedure. Two cohorts of novices (medical students) were recruited: 16 participants were trained using the intervention program (intermittent simulator-integrated tutoring) and 14 participants constituted a nontutored reference cohort. Outcomes were final-product performance assessed by 2 blinded raters and simulator-recorded metrics.

RESULTS

Simulator-integrated tutoring had a large and positive effect on the final-product performance while turned on (mean difference = 3.8 points, P < 0.0001). However, this did not translate to a better final-product performance in subsequent nontutored procedures. The tutored cohort had a better metrics-based score, reflecting higher efficiency of drilling (mean difference = 3.6%, P = 0.001). For the individual metrics, simulator-integrated tutoring had mixed effects both during procedures and on the tutored cohort in general (learning effect).

CONCLUSIONS

Simulator-integrated tutoring by green lighting did not induce a better final-product performance but increased efficiency. The mixed effects on learning could be caused by tutoring overreliance, resulting from a lack of cognitive engagement when the tutor function is on. Further learning strategies such as feedback should be explored to support novice learning and cognitive engagement.