Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool

Automatic Final-Product Assessment of Virtual Reality Mastoidectomy Performance: A Validity and Reliability Study

OBJECTIVE

Assessment is key in modern surgical education to monitor progress and document sufficient skills. Virtual reality (VR) temporal bone simulators allow automated tracking of basic metrics such as time, volume removed, and collisions. However, adequate performance assessment further includes compound rating of the stepwise bony excavation, and exposure and preservation of soft tissue structures. Such complex assessment requires further development of automated assessment routines in the VR simulation environment. In this study, we present the integration of automated mastoidectomy final-product assessment with validation against manual rating.

METHODS

At two international temporal bone courses, 33 ORL trainees performed anatomical mastoidectomies in the Visible Ear (VR) Simulator with automatic performance assessment using a newly implemented rating routine based on the modified Welling Scale. Automated assessment was compared with manual ratings by experts using absolute agreement, intraclass correlation, and generalizability analysis to establish validity and reliability.

RESULTS

The overall average agreement between manual and automatic assessment was 83.9% compared with the inter-rater agreement of 88.9%. A majority of items (15 out of 26) showed high agreement between automated and manual rating (>85%). Intraclass correlation coefficients were found to be high. Generalizability analysis with D-studies found that five repetitions per participant are needed for a G coefficient >0.8, which is considered necessary for high-stakes assessments.

CONCLUSION

We have demonstrated the feasibility, validity, and reliability of an automatic assessment system integrated into a VR temporal bone simulator. This can prove to be an important tool for future self-directed training with skills certification.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.