Beyond the “E” in OSCE

The ultrasound use of simulators, current view, and perspectives: Requirements and technical aspects (WFUMB state of the art paper)

Simulation has been shown to improve clinical learning outcomes, speed up the learning process and improve learner confidence, whilst initially taking pressure off busy clinical lists. The World Federation for Ultrasound in Medicine and Biology (WFUMB) state of the art paper on the use of simulators in ultrasound education introduces ultrasound simulation, its advantages and challenges. It describes different simulator types, including low and high-fidelity simulators, the requirements and technical aspects of simulators, followed by the clinical applications of ultrasound simulation. The paper discusses the role of ultrasound simulation in ultrasound clinical training, referencing established literature. Requirements for successful ultrasound simulation acceptance into educational structures are explored. Despite being in its infancy, ultrasound simulation already offers a wide range of training opportunities and likely holds the key to a broader point of care ultrasound education for medical students, practicing doctors, and other health care professionals. Despite the drawbacks of simulation, there are also many advantages, which are expanding rapidly as the technology evolves.

Educating Anesthesiologists During the Coronavirus Disease 2019 Pandemic and Beyond

The coronavirus disease 2019 (COVID-19) pandemic has altered approaches to anesthesiology education by shifting educational paradigms. This vision article discusses pre-COVID-19 educational methodologies and best evidence, adaptations required under COVID-19, and evidence for these modifications, and suggests future directions for anesthesiology education. Learning management systems provide structure to online learning. They have been increasingly utilized to improve access to didactic materials asynchronously. Despite some historic reservations, the pandemic has necessitated a rapid uptake across programs. Commercially available systems offer a wide range of peer-reviewed curricular options. The flipped classroom promotes learning foundational knowledge before teaching sessions with a focus on application during structured didactics. There is growing evidence that this approach is preferred by learners and may increase knowledge gain. The flipped classroom works well with learning management systems to disseminate focused preclass work. Care must be taken to keep virtual sessions interactive. Simulation, already used in anesthesiology, has been critical in preparation for the care of COVID-19 patients. Multidisciplinary, in situ simulations allow for rapid dissemination of new team workflows. Physical distancing and reduced availability of providers have required more sessions. Early pandemic decreases in operating volumes have allowed for this; future planning will have to incorporate smaller groups, sanitizing of equipment, and attention to use of personal protective equipment. Effective technical skills training requires instruction to mastery levels, use of deliberate practice, and high-quality feedback. Reduced sizes of skill-training workshops and approaches for feedback that are not in-person will be required. Mock oral and objective structured clinical examination (OSCE) allow for training and assessment of competencies often not addressed otherwise. They provide formative and summative data and objective measurements of Accreditation Council for Graduate Medical Education (ACGME) milestones. They also allow for preparation for the American Board of Anesthesiology (ABA) APPLIED examination. Adaptations to teleconferencing or videoconferencing can allow for continued use. Benefits of teaching in this new era include enhanced availability of asynchronous learning and opportunities to apply universal, expert-driven curricula. Burdens include decreased social interactions and potential need for an increased amount of smaller, live sessions. Acquiring learning management systems and holding more frequent simulation and skills sessions with fewer learners may increase cost. With the increasing dependency on multimedia and technology support for teaching and learning, one important focus of educational research is on the development and evaluation of strategies that reduce extraneous processing and manage essential and generative processing in virtual learning environments. Collaboration to identify and implement best practices has the potential to improve education for all learners.

Assessment Scores of a Mock Objective Structured Clinical Examination Administered to 99 Anesthesiology Residents at 8 Institutions

BACKGROUND

Objective Structured Clinical Examinations (OSCEs) are used in a variety of high-stakes examinations. The primary goal of this study was to examine factors influencing the variability of assessment scores for mock OSCEs administered to senior anesthesiology residents.

METHODS

Using the American Board of Anesthesiology (ABA) OSCE Content Outline as a blueprint, scenarios were developed for 4 of the ABA skill types: (1) informed consent, (2) treatment options, (3) interpretation of echocardiograms, and (4) application of ultrasonography. Eight residency programs administered these 4 OSCEs to CA3 residents during a 1-day formative session. A global score and checklist items were used for scoring by faculty raters. We used a statistical framework called generalizability theory, or G-theory, to estimate the sources of variation (or facets), and to estimate the reliability (ie, reproducibility) of the OSCE performance scores. Reliability provides a metric on the consistency or reproducibility of learner performance as measured through the assessment.

RESULTS

Of the 115 total eligible senior residents, 99 participated in the OSCE because the other residents were unavailable. Overall, residents correctly performed 84% (standard deviation [SD] 16%, range 38%-100%) of the 36 total checklist items for the 4 OSCEs. On global scoring, the pass rate for the informed consent station was 71%, for treatment options was 97%, for interpretation of echocardiograms was 66%, and for application of ultrasound was 72%. The estimate of reliability expressing the reproducibility of examinee rankings equaled 0.56 (95% confidence interval [CI], 0.49-0.63), which is reasonable for normative assessments that aim to compare a resident's performance relative to other residents because over half of the observed variation in total scores is due to variation in examinee ability. Phi coefficient reliability of 0.42 (95% CI, 0.35-0.50) indicates that criterion-based judgments (eg, pass-fail status) cannot be made. Phi expresses the absolute consistency of a score and reflects how closely the assessment is likely to reproduce an examinee's final score. Overall, the greatest (14.6%) variance was due to the person by item by station interaction (3-way interaction) indicating that specific residents did well on some items but poorly on other items. The variance (11.2%) due to residency programs across case items was high suggesting moderate variability in performance from residents during the OSCEs among residency programs.

CONCLUSIONS

Since many residency programs aim to develop their own mock OSCEs, this study provides evidence that it is possible for programs to create a meaningful mock OSCE experience that is statistically reliable for separating resident performance.

The escape room as evaluation method: A qualitative study of nursing students' experiences

Purpose: Gamification or learning using game elements is a process that seeks to capture students' interest. One of the most popular games in recent years is the Escape Room, but the study of its use in education and with health professionals is still limited. The aim of this study was to find out the perceptions and experiences of final year nursing students in an Objective Structured Clinical Examination (OSCE) by means of an Escape Room.Methods: In January 2019, 9 focus groups (FG) were held, with a total of 95 final year nursing students. We applied a qualitative content analysis approach and ATLAS.ti version 8 was used for data analysis.Results: The data revealed 3 main themes and 8 sub-themes. The three main themes, which were mapped to the conceptual framework, were student learning outcomes, emotional impact on students and conclusions on the serious games experience. Both the main themes and the sub-themes were illustrated using representative quotes from the participants.Conclusions: These results can help to apply these methodologies, such as the Escape Room, alongside other pre-existing ones, complementing the way in which students are assessed and the development of important nursing skills, such as teamwork and communication.

Learners and Luddites in the Twenty-first Century: Bringing Evidence-based Education to Anesthesiology

Anesthesiologists are both teachers and learners and alternate between these roles throughout their careers. However, few anesthesiologists have formal training in the methodologies and theories of education. Many anesthesiology educators often teach as they were taught and may not be taking advantage of current evidence in education to guide and optimize the way they teach and learn. This review describes the most up-to-date evidence in education for teaching knowledge, procedural skills, and professionalism. Methods such as active learning, spaced learning, interleaving, retrieval practice, e-learning, experiential learning, and the use of cognitive aids will be described. We made an effort to illustrate the best available evidence supporting educational practices while recognizing the inherent challenges in medical education research. Similar to implementing evidence in clinical practice in an attempt to improve patient outcomes, implementing an evidence-based approach to anesthesiology education may improve learning outcomes.

Should Objective Structured Clinical Examinations Assist the Clinical Competency Committee in Assigning Anesthesiology Milestones Competency?

BACKGROUND

With the integration of Objective Structured Clinical Examinations into the Anesthesiology primary board certification process, residency programs may choose to implement Objective Structured Clinical Examinations for resident skill assessment. The aim of this study was to evaluate Objective Structured Clinical Examination-based milestone assessment and compare with Clinical Competency Committee milestone assessment that is based purely on clinical evaluations.

METHODS

An annual Objective Structured Clinical Examination event was used to obtain milestone assessment of clinical anesthesia year 0-clinical anesthesia year 3 residents for selected milestones in patient care, professionalism, and interpersonal/communication skills. The Objective Structured Clinical Examination scenarios were different for each training level. The Clinical Competency Committee evaluated each resident semiannually based on clinical evaluations of resident performance. The Clinical Competency Committee milestone assessments from 2014 to 2016 that were recorded closest to the Objective Structured Clinical Examination event (±3 months) were compared to the Objective Structured Clinical Examination milestone assessments. A total of 35 residents were included in this analysis in 3 different training cohorts: A (graduates 2016, n = 12); B (graduates 2017, n = 10); and C (graduates 2018, n = 13). All residents participated in Objective Structured Clinical Examinations because their clinical anesthesia year 0 year and Clinical Competency Committee milestone data had been reported since December 2014.

RESULTS

Both assessment techniques indicated a competency growth proportional to the length in training. Despite limited cumulative statistics in this study, average trends in the Objective Structured Clinical Examination-Clinical Competency Committee relationship indicated: (1) a good proportionality in reflecting competency growth; (2) a grade enhancement associated with Clinical Competency Committee assessment, dominated by evaluations of junior residents (clinical anesthesia year 0-clinical anesthesia year 1); and (3) an expectation bias in Clinical Competency Committee assessment, dominated by evaluation of senior residents (clinical anesthesia year 2-clinical anesthesia year 3).

CONCLUSIONS

Our analysis confirms the compatibility of the 2 evaluation methods in reflecting longitudinal growth. The deviation of Objective Structured Clinical Examination assessments versus Clinical Competency Committee assessments suggests that Objective Structured Clinical Examinations may be providing additional or different information on resident performance. Educators might consider using both assessment methods to provide the most reliable and valid competency assessments during residency.