Mastery learning: how is it helpful? An analytical review

Are Video Recordings Reliable for Assessing Surgical Performance? A Prospective Reliability Study Using Generalizability Theory

INTRODUCTION

Reliability is pivotal in surgical skills assessment. Video-based assessment can be used for objective assessment without physical presence of assessors. However, its reliability for surgical assessments remains largely unexplored. In this study, we evaluated the reliability of video-based versus physical assessments of novices' surgical performances on human cadavers and 3D-printed models-an emerging simulation modality.

METHODS

Eighteen otorhinolaryngology residents performed 2 to 3 mastoidectomies on a 3D-printed model and 1 procedure on a human cadaver. Performances were rated by 3 experts evaluating the final surgical result using a well-known assessment tool. Performances were rated both hands-on/physically and by video recordings. Interrater reliability and intrarater reliability were explored using κ statistics and the optimal number of raters and performances required in either assessment modality was determined using generalizability theory.

RESULTS

Interrater reliability was moderate with a mean κ score of 0.58 (range 0.53-0.62) for video-based assessment and 0.60 (range, 0.55-0.69) for physical assessment. Video-based and physical assessments were equally reliable (G coefficient 0.85 vs. 0.80 for 3D-printed models and 0.86 vs 0.87 for cadaver dissections). The interaction between rater and assessment modality contributed to 8.1% to 9.1% of the estimated variance. For the 3D-printed models, 2 raters evaluating 2 video-recorded performances or 3 raters physically assessing 2 performances yielded sufficient reliability for high-stakes assessment (G coefficient >0.8).

CONCLUSIONS

Video-based and physical assessments were equally reliable. Some raters were affected by changing from physical to video-based assessment; consequently, assessment should be either physical or video based, not a combination.

3-D-Printed Models for Temporal Bone Training: A Validity Study

OBJECTIVE

3-D printing offers convenient and low-cost mastoidectomy training; nonetheless, training benefits using 3-D-printed temporal bones remain largely unexplored. In this study, we have collected validity evidence for a low-cost, 3-D-printed temporal bone for mastoidectomy training and established a credible pass/fail score for performance on the model.

STUDY DESIGN

A prospective educational study gathering validity evidence using Messick's validity framework.

SETTING

Seven Danish otorhinolaryngology training institutions.

PARTICIPANTS

Eighteen otorhinolaryngology residents (novices) and 11 experienced otosurgeons (experts).

INTERVENTION

Residents and experienced otosurgeons each performed two to three anatomical mastoidectomies on a low-cost, 3-D-printed temporal bone model produced in-house. After drilling, mastoidectomy performances were rated by three blinded experts using a 25-item modified Welling scale (WS).

MAIN OUTCOME MEASURE

Validity evidence using Messick's framework including reliability assessment applying both classical test theory and Generalizability theory.

RESULTS

Novices achieved a mean score of 13.9 points; experienced otosurgeons achieved 23.2 points. Using the contrasting groups method, we established a 21/25-point pass/fail level. The Generalizability coefficient was 0.91, and 75% of the score variance was attributable to participant performance, indicating a high level of assessment reliability. Subsequent D studies revealed that two raters rating one performance or one rater rating two performances were sufficiently reliable for high-stakes assessment.

CONCLUSION

Validity evidence supports using a low-cost, 3-D-printed model for mastoidectomy training. The model can be printed in-house using consumer-grade 3-D printers and serves as an additional training tool in the temporal bone curriculum. For competency-based training, we established a cut-off score of 21 of 25 WS points using the contrasting groups method.

Prehospital Surgical Airway Management Skills in a Rural Emergency Medical Service System

BACKGROUND

The objective of this study is to describe the education, training, and use of prehospital surgical airways in a rural Emergency Medical Service (EMS) system.

MATERIALS AND METHODS

We conducted an internet-based survey instrument of all advanced life support (ALS) EMS agencies in a seven-county rural EMS system in Pennsylvania. ALS agencies were queried regarding basic demographic information as well as the number of surgical airways performed in the previous 10 years as well as the education and training of EMS providers in surgical airways.

RESULTS

The survey was completed by 11 of 20 ALS EMS agencies in our region (55% rate of return). The content and frequency of training varied considerably among EMS agencies. Only four prehospital surgical airways were performed during the study period. One patient survived to hospital discharge to home.

CONCLUSION

Surgical airways are an infrequently performed procedure in the rural prehospital setting. There is no universally accepted standard for teaching or evaluating the competency of this potentially life-saving procedure. Further efforts to establish a core educational curriculum appear warranted.

Becoming virtually real: using the Virtual Environment for Radiotherapy Training (VERT™) platform for the summative assessment of performance in a palliative radiotherapy treatment technique

Introduction:

Direct observation in the radiotherapy clinic remains the primary method for the assessment of procedural skills. But with increasing automation and reductions in clinical placement time during the recent pandemic, the role of summative, simulation-based assessment is being revisited. The aim of this article is to share early experiences of using Virtual Environment for Radiotherapy Training (VERT) in the assessment of a palliative, parallel pair, external beam treatment delivery technique in a new pre-registration undergraduate programme.

Methods:

Eight first-year students completed a campus-based individual virtual assessment using virtual patient plans representing metastatic disease in the brain and pelvis and late-stage primary lung tumours. Performance was logged on a 25-item checklist for pre and post-procedure checks and treatment delivery tasks.

Results:

All eight students participated in the assessment with seven students (87·5%) achieving a pass grade at first attempt. The course team observed that participants demonstrated a range of skills and abilities and were able to compare and contrast individual approaches to patient positioning. Specific feed-forward action points were also highlighted as areas for students’ to focus on during their second placement. The project also identified logistical benefits for assessment teams.

Conclusions:

While these experiences are a single snapshot, a VERT simulation-based summative assessment is feasible and identified benefits included controlled observation and structured feedback on individual performance and scheduling pressure reductions for clinical teams. However, more work is needed to determine the psychometric qualities and predictability of performance in more complex techniques.

Automated summative feedback improves performance and retention in simulation training of mastoidectomy: a randomised controlled trial

OBJECTIVE

This study aimed to investigate the effects of automated metrics-based summative feedback on performance, retention and cognitive load in distributed virtual reality simulation training of mastoidectomy.

METHOD

Twenty-four medical students were randomised in two groups and performed 15 mastoidectomies on a distributed virtual reality simulator as practice. The intervention group received additional summative metrics-based feedback; the control group followed standard instructions. Two to three months after training, participants performed a retention test without learning supports.

RESULTS

The intervention group had a better final-product score (mean difference = 1.0 points; p = 0.001) and metrics-based score (mean difference = 12.7; p < 0.001). At retention, the metrics-based score for the intervention group remained superior (mean difference = 6.9 per cent; p = 0.02). Also at the retention, cognitive load was higher in the intervention group (mean difference = 10.0 per cent; p < 0.001).

CONCLUSION

Summative metrics-based feedback improved performance and lead to a safer and faster performance compared with standard instructions and seems a valuable educational tool in the early acquisition of temporal bone skills.

Automatic and Objective Assessment of Motor Skills Performance in Flexible Bronchoscopy

BACKGROUND

Motor skills have been identified as a useful measure to evaluate competency in bronchoscopy. However, no automatic assessment system of motor skills with a clear pass/fail criterion in flexible bronchoscopy exists.

OBJECTIVES

The objective of the study was to develop an objective and automatic measure of motor skills in bronchoscopy and set a pass/fail criterion.

METHODS

Participants conducted 3 bronchoscopies each in a simulated setting. They were equipped with a Myo Armband that measured lower arm movements through an inertial measurement unit, and hand and finger motions through electromyography sensors. These measures were composed into an objective and automatic composite score of motor skills, the motor bronchoscopy skills score (MoBSS).

RESULTS

Twelve novices, eleven intermediates, and ten expert bronchoscopy operators participated, resulting in 99 procedures available for assessment. MoBSS was correlated with a higher diagnostic completeness (Pearson's correlation, r = 0.43, p < 0.001) and a lower procedure time (Pearson's correlation, r = -0.90, p < 0.001). MoBSS was able to differentiate operator performance based on the experience level (one-way ANOVA, p < 0.001). Using the contrasting groups' method, a passing score of -0.08 MoBSS was defined that failed 30/36 (83%) novice, 5/33 (15%) intermediate, and 1/30 (3%) expert procedures.

CONCLUSIONS

MoBSS can be used as an automatic and unbiased assessment tool for motor skills performance in flexible bronchoscopy. MoBSS has the potential to generate automatic feedback to help guide trainees toward expert performance.

Improving diagnosis by feedback and deliberate practice: one-on-one coaching for diagnostic maturation

Reflective practice is essential for the ongoing maturation of clinicians and requires regular self-evaluation in association with ongoing mentoring and feedback. Currently, most resident physicians do not have access to educational experiences that fulfill these needs. We present a novel model for structured one-on-one longitudinal coaching using the principles of deliberate practice to improve diagnostic skills. This is an easily implementable educational model that can be replicated in residencies across the country to improve clinical reasoning. Skills learned through this program have the potential not only to bolster the academic approach to patients but to also directly improve the clinical assessment and care of patients under the trainee's care.

Using structured progress to measure competence in flexible bronchoscopy

Background

Flexible bronchoscopy is a core invasive procedure in pulmonary medicine and training in the procedure is mandatory. Diagnostic completeness and procedure time have been identified as useful measures of competence. No outcome measures have been developed regarding navigational path in bronchoscopy to assess whether the bronchial segments have been identified in an arbitrary or structured order. We investigated whether a new outcome measure for structured progression could be used to assess competency in flexible bronchoscopy.

Methods

The study was designed as a prospective comparative study. Twelve novices, eleven intermediates, and ten expert bronchoscopy operators completed three full bronchoscopies in a simulated setting on a phantom. The following outcome measures were collected through a checklist evaluation by a trained rater: Diagnostic Completeness as amount of visualized bronchial segments, Structured Progress between the bronchial segments in ascending order, and average intersegmental time (AIT).

Results

The ability to follow a structured ascending path through the bronchial tree correlated with a higher amount of identified bronchial segments (Pearson’s correlation, r=0.62, P<0.001) and a lower AIT (Pearson’s correlation, r=−0.52, P<0.001).

Conclusions

Operators should advance through the bronchial tree in a structured ascending order to ensure systematic progress with the highest level of diagnostic yield and the lowest procedure time. Structured progression is a useful measure to evaluate competency in flexible bronchoscopy.

A Functional Contextualist Approach to Mastery Learning in Vocational Education and Training

Along with technological progress, vocational education and training (VET) is consistently changing. Workforce disruption has serious consequences for workers and international economies, often requiring adults to transition into different occupations or to upskill to maintain employment. We review recent literature covering VET trends, theoretical considerations for the 21st century, and present an approach to workforce training to help workers not only learn necessary skills but also become adaptable to constant change. We suggest a functional contextualist approach to mastery learning achieves this aim. Specifically, we offer suggestions for pedagogy that not only develop skills but also encourage higher order thinking. Within a novice to expert continuum, we suggest deliberate practice, mental simulation, and reflective meaning making as methods to achieve efficiency and transfer-learning outcomes relevant to a changing workforce. This approach recognizes that learning is context bound and should promote broader human capabilities that support both employability and the continuing development of life literacies.

Perceptions of foundation dentists on minor oral surgery teaching in dental foundation training

AIMS

The aim of this service evaluation was to explore the experiences of foundation dentists (FDs) during their dental foundation year (DFT) in the field of oral surgery and surgical extractions.

METHODS

A phenomenological approach was taken aiming to explore the FDs' experiences through a qualitative design. All FDs in the region deemed eligible for the evaluation were invited to attend a semi-structured interview or answer a qualitative questionnaire when attending for their exit interview at the completion of DFT. The interviews were then transcribed and thematic analysis was undertaken.

RESULTS

79 FDs took part in the evaluation, representing 86% of trainees eligible for inclusion. Findings showed that not all FDs had the opportunity to undertake surgical extractions independently during the year. Data was analysed and 4 main themes generated when evaluating FDs' experiences; specific learning encounters, influence of the educational supervisor, supportive learning environment and ready for independent practice.

CONCLUSIONS

This evaluation shows that there are a number of factors associated with FDs' experience during DFT, and not all FDs are leaving the training with a positive experience, with a query as to whether they are fulfilling the aims of training to become an "independent practitioner." It poses questions about what can be done to improve future training and what exactly does DFT need to achieve in terms of experience in surgical extractions.

A Systematic Review of the Educational Effectiveness of Simulation Used in Open Surgery

STATEMENT

The role of simulation to teach and access open surgical skills has become more prevalent in recent years. This systematic review synthesizes the totality of evidence with respect to the educational effectiveness of simulators used in open surgical training. A systematic literature search was conducted in PubMed, Embase, CINAHL, Scopus, and Web of Science. Only randomized controlled trials were included that explored the educational efficacy of theses simulators. Six randomized controlled trials were included from the 9934 studies found. The methodological quality of the included studies was variable. Overall, the use of the simulators was more educationally effective compared with standard teaching of the skill without a simulator (P < 0.05). Two studies showed that the simulator was as good as an animal model of much higher fidelity. Further studies are needed to secure higher evidence for the educational value, validity, and transferability of the skills to the hospital setting for all simulators.

Simulation-Based Mastery Learning of Flexible Bronchoscopy: Deciding Factors for Completion

BACKGROUND

Several studies have shown the beneficial effects of mastery learning of a simulation-based course, but not all trainees complete it.

OBJECTIVES

The purpose of this study was to find deciding factors for the completion of a simulation-based mastery learning course with distributive practice in flexible bronchoscopy.

METHODS

Seventy-seven trainees who signed up for the course were invited to a survey for deciding factors of completing the course. Sixty-two (81%) trainees answered the survey.

RESULTS

Male trainees were more likely to complete the course. The most important factor for completion was clinical relevance, and the most important factor for not completing the course was being "too busy."

CONCLUSION

Several deciding factors for completing the course were identified. Successful simulation-based mastery learning courses should be clinically relevant, and the trainees should be provided protected time to complete the training. The instructional design should also be adapted systematically for male and female trainees to achieve the necessary competencies.

Standardised assessment of competence in Focused Assessment with Sonography for Trauma

PURPOSE

The aim of this study was to develop and gather validity evidence for a standardised test of competence in Focused Assessment with Sonography for Trauma (FAST) and to define the appropriate cut-off point in simulation-based learning of the FAST protocol.

METHODS

A 20-item simulation-based test for assessing competence in FAST was created. The test was administered to thirteen novices and twelve radiologists experienced in abdominal ultrasound diagnostics. The Contrasting Groups' method was used to establish a credible passing score.

RESULTS

The internal consistency was high (Cronbach's α = 0.90) and the test had good discriminatory ability (P < .001). The mean score was 16.9 (95% CI: 15.5-18.3) in the experienced group and 8.0 (95% CI: 5.8-10.2) in the novice group, corresponding to 85% and 40% of the total score, respectively. A pass/fail standard of 14 points was established using the Contrasting Groups' method.

CONCLUSIONS

The FAST simulation-based test provided valid assessment of competence in FAST. The FAST test could be used to guide training and ensure basic competence of physicians using FAST.

Simulation in bronchoscopy: current and future perspectives

OBJECTIVE

To provide an overview of current literature that informs how to approach simulation practice of bronchoscopy and discuss how findings from other simulation research can help inform the use of simulation in bronchoscopy training.

SUMMARY

We conducted a literature search on simulation training of bronchoscopy and divided relevant studies in three categories: 1) structuring simulation training in bronchoscopy, 2) assessment of competence in bronchoscopy training, and 3) development of cheap alternatives for bronchoscopy simulation.

CONCLUSION

Bronchoscopy simulation is effective, and the training should be structured as distributed practice with mastery learning criteria (ie, training until a certain level of competence is achieved). Dyad practice (training in pairs) is possible and may increase utility of available simulators. Trainee performance should be assessed with assessment tools with established validity. Three-dimensional printing is a promising new technology opening possibilities for developing cheap simulators with innovative features.