Standard Setting of Competency in Mastoidectomy for the Cross-Institutional Mastoidectomy Assessment Tool

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.