Simulation-based assessment in anesthesia: an international multicentre validation study

Deep Learning Model for Automated Trainee Assessment During High-Fidelity Simulation

PROBLEM

Implementation of competency-based medical education has necessitated more frequent trainee assessments. Use of simulation as an assessment tool is limited by access to trained examiners, cost, and concerns with interrater reliability. Developing an automated tool for pass/fail assessment of trainees in simulation could improve accessibility and quality assurance of assessments. This study aimed to develop an automated assessment model using deep learning techniques to assess performance of anesthesiology trainees in a simulated critical event.

APPROACH

The authors retrospectively analyzed anaphylaxis simulation videos to train and validate a deep learning model. They used an anaphylactic shock simulation video database from an established simulation curriculum, integrating a convenience sample of 52 usable videos. The core part of the model, developed between July 2019 and July 2020, is a bidirectional transformer encoder.

OUTCOMES

The main outcome was the F1 score, accuracy, recall, and precision of the automated assessment model in analyzing pass/fail of trainees in simulation videos. Five models were developed and evaluated. The strongest model was model 1 with an accuracy of 71% and an F1 score of 0.68.

NEXT STEPS

The authors demonstrated the feasibility of developing a deep learning model from a simulation database that can be used for automated assessment of medical trainees in a simulated anaphylaxis scenario. The important next steps are to (1) integrate a larger simulation dataset to improve the accuracy of the model; (2) assess the accuracy of the model on alternative anaphylaxis simulations, additional medical disciplines, and alternative medical education evaluation modalities; and (3) gather feedback from education leadership and clinician educators surrounding the perceived strengths and weaknesses of deep learning models for simulation assessment. Overall, this novel approach for performance prediction has broad implications in medical education and assessment.

Development of pediatric simulation-based education - a systematic review

BACKGROUND

This systematic literature review explored the general characteristics, validation, and reliability of pediatric simulation-based education (P-SBE).

METHODS

A literature search was conducted between May 23 and 28 using the PRISMA guidelines, which covered databases such as MEDLINE, EMBASE, CINAHL, and Cochrane Library. In the third selection process, the original texts of 142 studies were selected, and 98 documents were included in the final content analysis.

RESULTS

A total of 109 papers have been published in the ten years since 2011. Most of the study designs were experimental studies, including RCT with 76 articles. Among the typologies of simulation, advanced patient simulation was the most common (92), and high-fidelity simulation was the second most common (75). There were 29 compatibility levels and professional levels, with 59 scenarios related to emergency interventions and 19 scenarios related to communication feasibility and decision making. Regarding the effect variable, 65 studies confirmed that skills were the most common. However, validity of the scenarios and effect variables was not verified in 56.1% and 67.3% of studies, respectively.

CONCLUSION

Based on these findings, simulation based-education (SBE) is an effective educational method that can improve the proficiency and competence of medical professionals dealing with child. Learning through simulation provides an immersive environment in which learners interact with the presented patient scenario and make decisions, actively learning the attitudes, knowledge, and skills necessary for medical providers. In the future, it is expected that such research on SBE will be actively followed up and verified for its validity and reliability.

Objective structured assessment ultrasound skill scale for hyomental distance competence - psychometric study

BACKGROUND

Ultrasound assessment of the airway recently integrates the point-of-care approach to patient evaluation since ultrasound measurements can predict a difficult laryngoscopy and tracheal intubation. Because ultrasonography is performer-dependent, a proper training and assessment tool is needed to increase diagnostic accuracy. An objective, structured assessment ultrasound skill (OSAUS) scale was recently developed to guide training and assess competence. This work aims to study the psychometric properties of OSAUS Scale when used to evaluate competence in ultrasound hyomental distance (HMD) measurement.

METHODS

Prospective and experimental study. Volunteers were recruited and enrolled in groups with different expertise. Each participant performed three ultrasonographic HMD evaluation. The performance was videorecorded and anonymized. Five assessors blindly rated participants' performance using OSAUS scale and a Global Rating Scale (GRS). A psychometric study of OSAUS scale as assessment tool for ultrasound HMD competence was done.

RESULTS

Fifteen voluntaries participated on the study. Psychometric analysis of OSAUS showed strong internal consistency (Cronbach's alpha 0.916) and inter-rater reliability (ICC 0.720; p < 0.001). The novice group scored 15.4±0.18 (mean±SD), the intermediate 14.3±0.75 and expert 13.6±0.1.25, with a significant difference between novice and expert groups (p = 0.036). The time in seconds to complete the task was evaluated: novice (90±34) (mean±SD), intermediate (84±23) and experts (83±15), with no significant differences between groups. A strong correlation was observed between OSAUS and global rating scale (r = 0.970, p < 0.001).

CONCLUSION

The study demonstrated evidence of validity and reliability. Further studies are needed to implement OSAUS scale in the clinical setting for training and assessment of airway ultrasound competence.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.