Simulation-based assessment to identify critical gaps in safe anesthesia resident performance

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study

OBJECTIVES

The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.

DESIGN

Observational multicenter study.

SETTING

Thirteen Spanish ICU Departments.

PARTICIPANTS

Thirty six R3.

INTERVENTION

The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.

MAIN VARIABLES OF INTEREST

Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.

RESULTS

A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64-0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65-0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.

CONCLUSION

The expected level of competency after one year in the ICU was achieved only in half of the performances. A more evidence-based educational approach is needed. Multiple center simulation-based assessment showed feasibility and reliability as an evaluation method of competency.

TRIAL REGISTRATION

COBALIDATION. NCT04278976. (https://register.

CLINICALTRIALS

gov).

Identifying patient safety competences among anesthesiology residents: systematic review

Introduction and objective

Patient safety is a concept of great importance to managers, health professionals, and patients and their families, given patient safety promotes more effective care and reduces costs. Moreover, while analyzing the area of anesthesiology, one can realize the epidemiological changes, increased complexity and number of procedures, and the adoption of a new matrix of essential skills mandatory for residents of anesthesiology in Brazil. Thus, it is relevant to identify current patient safety competences among anesthesiology residents.

Methods

A systematic review was elaborated using PubMed, SciELO, BVS, Cochrane Library, LILACS and CAPES databases with the descriptors “anesthesiology”, “patient safety”, “residency” and “competence”.

Results and conclusions

Thirteen articles published in the past 10 years were analyzed. The articles depicted competences grouped into three categories: knowledge (identification, prevention and management of adverse events; use of correct and up-to-date information; understanding of human factors; and continuous learning), skills (efficient communication; teamwork; leadership; decision-making; and self-confidence), and attitude (management of stress and fatigue; and infection control). All these skills can be developed and assessed through simulation and active learning methods, profiting from a multidisciplinary approach. Studies also reveal that residents perform poorly in certain patient safety domains due to lack of effective in-depth understanding, appreciation of the topic and ineffective teaching. As a result, greater investment in the topic is needed by teaching and health institutions and researchers.

Multimodal In-training Examination in an Emergency Medicine Residency Training Program: A Longitudinal Observational Study

Background

In-training examination (ITE) has been widely adopted as an assessment tool to measure residents' competency. We incorporated different formats of assessments into the emergency medicine (EM) residency training program to form a multimodal, multistation ITE. This study was conducted to examine the cost and effectiveness of its different testing formats.

Methods

We conducted a longitudinal study in a tertiary teaching hospital in Taiwan. Nine EM residents were enrolled and followed for 4 years, and the biannual ITE scores were recorded and analyzed. Each ITE consisted of 8–10 stations and was categorized into four formats: multiple-choice question (MCQ), question and answer (QA), oral examination (OE), and high-fidelity simulation (HFS) formats. The learner satisfaction, validity, reliability, and costs were analyzed.

Results

486 station scores were recorded during the 4 years. The numbers of MCQ, OE, QA, and HFS stations were 45 (9.26%), 90 (18.5%), 198 (40.7%), and 135 (27.8%), respectively. The overall Cronbach's alpha reached 0.968, indicating good overall internal consistency. The correlation with EM board examination was highest for HFS (ρ = 0.657). The average costs of an MCQ station, an OE station, and an HFS station were ~3, 14, and 21 times that of a QA station.

Conclusions

Multi-dimensional assessment contributes to good reliability. HFS correlates best with the final training exam score but is also the most expensive format among ITEs. Increased testing domains with various formats improve ITE's overall reliability. Program directors must understand each test format's strengths and limitations to bring forth the best combination of exams under the local context.

High-fidelity simulation is associated with good discriminability in emergency medicine residents' in-training examinations

In-training examinations (ITEs), arranged during residency training, evaluate the residents' performances periodically. There is limited literature focusing on the effectiveness of resident ITEs in the format of simulation-based examinations, as compared to traditional oral or written tests. Our primary objective is to investigate the effectiveness and discriminative ability of high-fidelity simulation compared with other measurement formats in emergency medicine (EM) residency training program.This is a retrospective cohort study. During the 5-year study period, 8 ITEs were administered to 68 EM residents, and 253 ITE measurements were collected. Different ITE scores were calculated and presented as mean and standard deviation. The ITEs were categorized into written, oral, or high-fidelity simulation test forms. Discrimination of ITE scores between different training years of residency was examined using a one-way analysis of variance test.The high-fidelity simulation scores correlated to the progression of EM training, and residents in their fourth training year (R4) had the highest scores consistently, followed by R3, R2, and then R1. The oral test scores had similar results but not as consistent as the high-fidelity simulation tests. The written test scores distribution failed to discriminate the residents' seniority. The high-fidelity simulation test had the best discriminative ability and better correlation between different EM residency training years comparing to other forms.High-fidelity simulation tests had the good discriminative ability and were well correlated to the EM training year. We suggest high-fidelity simulation should be a part of ITE in training programs associated with critical or emergency patient cares.

Creation of Simulation-Based Curriculum of Perioperative Emergencies for Residents in Anesthesiology

Introduction: Crisis management is difficult to practice and evaluate for resident learners and leadership given the rarity of these events in clinical practice. However, simulation provides a medium to bridge this gap. We identified a need for simulation in our anesthesiology residency program to help residents learn to treat perioperative emergencies.

Objective: To describe the process of curriculum development, content, and early outcomes of a simulation-based curriculum for the management of perioperative emergencies for all levels of anesthesiology learners.

Materials and methods: Curriculum development began in the Spring of 2019 and simulations began in August 2019. All anesthesiology residents rotating at a single center through December 2020 were eligible. Each resident was given their own simulation scenario detailing a specific perioperative emergency and then debriefed as a group afterward. All residents participating in the scenario were given a post-simulation survey assessing the value of the educational experience, relevance to their level of training, and quality of learning environment.

Results: Out of 90 eligible residents, 79 participated in the study (87%). Overall, 100% of participants completed the post-simulation survey; 100% of residents reported that the curriculum was useful to their education; 98% of residents reported that the curriculum was relevant to their training level; 99% of residents reported that the simulation was an engaging learning experience.

Conclusion: A simulation-based curriculum of perioperative emergencies for anesthesiology residents is feasible to implement, viewed as worthwhile by trainees, and can foster education in a different learning environment.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

The effect of simulation-based training in non-physician anesthetists in Tigray region, Ethiopia

Objective

In Ethiopia simulation-based anesthesia education is a new way of teaching method which started in Mekelle University as of January, 2019. Hence, the purpose of this study is to evaluate whether simulation-based training improves non-physician anesthetists’ knowledge and attitude on maternal and neonatal anesthesia cares or not.

Results

Out of 50 study subjects, 66% had a working experience of less than 5 years. Knowledge score improved significantly from 49.78 to 66.22% in pretest and posttest results respectively. The posttest result was significantly improved (P < 0.001) for all knowledge questions. The respondents were asked about a negative statement and a positive statement about the need to have effective closed lope communication, maternal resuscitation and neonatal resuscitation. The attitude score improved from 72.45 to 79.11% in pretest and posttest respectively. From the 9 questions the attitudinal mean score for pretest was 6.52 and posttest 7.12. The null hypothesis of equal knowledge and attitude was rejected, t (49) = − 5.54, P < 0.001 and t (49) = − 2.25, P < 0.03 respectively.

Maintenance of certification for practicing physicians: a review of current challenges and considerations

Maintenance of certification (MOC) has become increasingly important in medicine to ensure maintenance of competence throughout a physician's career. This paper reviews current issues and challenges associated with MOC in medicine, including how to define medical competencies for practicing physicians, assessment, and how best to support physicians' lifelong learning in a continuous and self-motivated way. We explore how the combination of self-monitoring, regular feedback, and peer support could improve self-assessment. Effective MOC programs are learner-driven, focused on every day practice, and incorporate educational principles. We discuss the importance of MOC to the physicians' actual practice to improve acceptability. We review the benefits of tailored programs as well as decentralization of MOC programs to better characterize the physician's practice. Lastly, we discuss the value of simulation-based medical education in MOC programs. Simulation-based education could be used to practice uncommon complications, life-threatening scenarios, non-technical skills improvement, and become proficient with new technology. As learners find simulation experiences educationally valuable, clinically relevant, and positive, simulation could be a way of increasing physicians' participation in MOC programs.

Simulation-based evaluation of anaesthesia residents: optimising resource use in a competency-based assessment framework

Introduction

Simulation training in anaesthesiology bridges the gap between theory and practice by allowing trainees to engage in high-stakes clinical training without jeopardising patient safety. However, implementing simulation-based assessments within an academic programme is highly resource intensive, and the optimal number of scenarios and faculty required for accurate competency-based assessment remains to be determined. Using a generalisability study methodology, we examine the structure of simulation-based assessment in regard to the minimal number of scenarios and faculty assessors required for optimal competency-based assessments.

Methods

Seventeen anaesthesiology residents each performed four simulations which were assessed by two expert raters. Generalisability analysis (G-analysis) was used to estimate the extent of variance attributable to (1) the scenarios, (2) the assessors and (3) the participants. The D-coefficient and the G-coefficient were used to determine accuracy targets and to predict the impact of adjusting the number of scenarios or faculty assessors.

Results

We showed that multivariate G-analysis can be used to estimate the number of simulations and raters required to optimise assessment. In this study, the optimal balance was obtained when four scenarios were assessed by two simulation experts.

Conclusion

Simulation-based assessment is becoming an increasingly important tool for assessing the competency of medical residents in conjunction with other assessment methods. G-analysis can be used to assist in planning for optimal resource use and cost-efficacy.

Simulation-based assessment in anesthesia: an international multicentre validation study

PURPOSE

Simulated clinical events provide a means to evaluate a practitioner's performance in a standardized manner for all candidates that are tested. We sought to provide evidence for the validity of simulation-based assessment tools in simulated pediatric anesthesia emergencies.

METHODS

Nine centres in two countries recruited subjects to participate in simulated operating room events. Participants ranged in anesthesia experience from junior residents to staff anesthesiologists. Performances were video recorded for review and scored by specially trained, blinded, expert raters. The rating tools consisted of scenario-specific checklists and a global rating scale that allowed the rater to make a judgement about the subject's performance, and by extension, preparedness for independent practice. The reliability of the tools was classified as "substantial" (intraclass correlation coefficients ranged from 0.84 to 0.96 for the checklists and from 0.85 to 0.94 for the global rating scale).

RESULTS

Three-hundred and ninety-one simulation encounters were analysed. Senior trainees and staff significantly out-performed junior trainees (P = 0.04 and P < 0.001 respectively). The effect size of grade (junior vs senior trainee vs staff) on performance was classified as "medium" (partial η² = 0.06). Performance deficits were observed across all grades of anesthesiologist, particularly in two of the scenarios.

CONCLUSIONS

This study supports the validity of our simulation-based anesthesiologist assessment tools in several domains of validity. We also describe some residual challenges regarding the validity of our tools, some notes of caution in terms of the intended consequences of their use, and identify opportunities for further research.

Quantitative Research Methods in Medical Education

There has been a dramatic growth of scholarly articles in medical education in recent years. Evaluating medical education research requires specific orientation to issues related to format and content. Our goal is to review the quantitative aspects of research in medical education so that clinicians may understand these articles with respect to framing the study, recognizing methodologic issues, and utilizing instruments for evaluating the quality of medical education research. This review can be used both as a tool when appraising medical education research articles and as a primer for clinicians interested in pursuing scholarship in medical education.

Simulation-based Remediation in Emergency Medicine Residency Training: A Consensus Study

INTRODUCTION

Resident remediation is a pressing topic in emergency medicine (EM) training programs. Simulation has become a prominent educational tool in EM training and been recommended for identification of learning gaps and resident remediation. Despite the ubiquitous need for formalized remediation, there is a dearth of literature regarding best practices for simulation-based remediation (SBR).

METHODS

We conducted a literature search on SBR practices using the terms "simulation," "remediation," and "simulation based remediation." We identified relevant themes and used them to develop an open-ended questionnaire that was distributed to EM programs with experience in SBR. Thematic analysis was performed on all subsequent responses and used to develop survey instruments, which were then used in a modified two-round Delphi panel to derive a set of consensus statements on the use of SBR from an aggregate of 41 experts in simulation and remediation in EM.

RESULTS

Faculty representing 30 programs across North America composed the consensus group with 66% of participants identifying themselves as simulation faculty, 32% as program directors, and 2% as core faculty. The results from our study highlight a strong agreement across many areas of SBR in EM training. SBR is appropriate for a range of deficits, including procedural, medical knowledge application, clinical reasoning/decision-making, communication, teamwork, and crisis resource management. Simulation can be used both diagnostically and therapeutically in remediation, although SBR should be part of a larger remediation plan constructed by the residency leadership team or a faculty expert in remediation, and not the only component. Although summative assessment can have a role in SBR, it needs to be very clearly delineated and transparent to everyone involved.

CONCLUSION

Simulation may be used for remediation purposes for certain specific kinds of competencies as long as it is carried out in a transparent manner to all those involved.

Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

BACKGROUND

Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

METHODS

Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

RESULTS

The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

CONCLUSIONS

Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

Recent advances of simulation in obstetric anesthesia

PURPOSE OF REVIEW

Simulation training in obstetric anesthesia has become widespread in recent years. Simulations are used to train staff and trainees, assess and improve team performance, and evaluate the work environment. This review summarizes current research in these categories.

RECENT FINDINGS

Simulation to improve individual technical skills has focused on induction of general anesthesia for emergent cesarean delivery, an infrequently encountered scenario by anesthesia trainees. Low- and high-fidelity simulation devices for the learning and practicing neuraxial and non-neuraxial procedures have been described, and both are equally effective. The use of checklists in obstetric emergencies has become common as and post-scenario debriefing techniques have improved. Although participant task performance improves, whether participants retain learned skills or whether simulation improves patient outcomes has not yet been established. Tools to assess teamwork during simulation have been developed, but none have been rigorously validated. In-situ vs. offsite simulations do not differ in effectiveness.

SUMMARY

Simulation allows for practice of tasks and teamwork in a controlled manner. There is little data whether simulation improves patient outcomes and metrics to predict the long-term retention of skills by simulation participants have not been developed.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes

Background

Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.

Methods

Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.

Results

Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P &lt; 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.

Conclusions

Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Comparing Real-time Versus Delayed Video Assessments for Evaluating ACGME Sub-competency Milestones in Simulated Patient Care Environments

Background

Simulation is an effective method for creating objective summative assessments of resident trainees. Real-time assessment (RTA) in simulated patient care environments is logistically challenging, especially when evaluating a large group of residents in multiple simulation scenarios. To date, there is very little data comparing RTA with delayed (hours, days, or weeks later) video-based assessment (DA) for simulation-based assessments of Accreditation Council for Graduate Medical Education (ACGME) sub-competency milestones. We hypothesized that sub-competency milestone evaluation scores obtained from DA, via audio-video recordings, are equivalent to the scores obtained from RTA.

Methods

Forty-one anesthesiology residents were evaluated in three separate simulated scenarios, representing different ACGME sub-competency milestones. All scenarios had one faculty member perform RTA and two additional faculty members perform DA. Subsequently, the scores generated by RTA were compared with the average scores generated by DA. Variance component analysis was conducted to assess the amount of variation in scores attributable to residents and raters.

Results

Paired t-tests showed no significant difference in scores between RTA and averaged DA for all cases. Cases 1, 2, and 3 showed an intraclass correlation coefficient (ICC) of 0.67, 0.85, and 0.50 for agreement between RTA scores and averaged DA scores, respectively. Analysis of variance of the scores assigned by the three raters showed a small proportion of variance attributable to raters (4% to 15%).

Conclusions

The results demonstrate that video-based delayed assessment is as reliable as real-time assessment, as both assessment methods yielded comparable scores. Based on a department’s needs or logistical constraints, our findings support the use of either real-time or delayed video evaluation for assessing milestones in a simulated patient care environment.

Anesthesiology Resident Induction Month: a pilot study showing an effective and safe way to train novice residents through simulation

BACKGROUND

The transition of new residents from medical school to the post-graduate clinical environment remains challenging. We hypothesized that an introductory simulation course could improve new residents' performance in anesthesiology.

METHODS

The Anesthesiology Residents Induction Month (ARIM) program was designed as a non-clinical simulation training program aiming at providing the theoretical and practical skills to safely approach, as junior anesthesiologists, the operating rooms. For each participant, specific knowledge, procedural skills and non-technical performance were assessed with a pre and post-test approach, before and immediately after the participation in the study.

RESULTS

Fifteen first-month residents participated in the study. As compared to pre-test, residents significantly improved in all three evaluated areas. Pre-test knowledge assessment mean improved from 56% to 73% in the post-test (P<0.001). In the procedural skills assessment, pre-test mean improved from 43% to 77% (P<0.001) and non-technical skills assessment improved from 3.17 to 4.61 (in a scale out of seven points) in the post-test (P<0.001).

CONCLUSIONS

Data suggest that an intensive simulation-based program can be an effective way for first-year residents to rapidly acquire and develop basic skills specific to anesthesiology. There might be benefits to begin residency with a training program aiming at developing and standardizing technical and non-technical skills.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists

Background

We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods.

Methods

A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant’s technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist.

Results

Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance.

Conclusions

Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

Assessing anesthesiology residents' out-of-the-operating-room (OOOR) emergent airway management

BACKGROUND

At many academic institutions, anesthesiology residents are responsible for managing emergent intubations outside of the operating room (OOOR), with complications estimated to be as high as 39%. In order to create an OOOR training curriculum, we evaluated residents' familiarity with the content and correct adherence to the American Society of Anesthesiologists' Difficult Airway Algorithm (ASA DAA).

METHODS

Residents completed a pre-simulation multiple-choice survey measuring their understanding and use of the DAA. Residents then managed an emergent, difficult OOOR intubation in the simulation center, where two trained reviewers assessed performance using checklists. Post-simulation, the residents completed a survey rating their behaviors during the simulation. The primary outcome was comprehension and adherence to the DAA as assessed by survey responses and behavior in the simulation.

RESULTS

Sixty-three residents completed both surveys and the simulation. Post-survey responses indicated a shift toward decreased self-perceived familiarity with the DAA content compared to pre-survey responses. During the simulation, 22 (35%) residents were unsuccessful with intubation. Of these, 46% placed an LMA and 46% prepared for cricothyroidotomy. Nineteen residents did not attempt intubation. Of these, only 31% considered LMA placement, and 26% initiated cricothyroidotomy.

CONCLUSIONS

Many anesthesiology residency training programs permit resident autonomy in managing emergent intubations OOOR. Residents self-reported familiarity with the content of and adherence to the DAA was higher than that observed during the simulation. Curriculum focused on comprehension of the DAA, as well as improving communication with higher-level physicians and specialists, may improve outcomes during OOORs.

Effect of a simulation-based workshop on breaking bad news for anesthesiology residents: an intervention study

Background

Breaking bad news (BBN) to patients and their relatives is a complex and stressful task. The ideal structure, training methods and assessment instruments best used to teach and assess BBN for anesthesiology residents remain unclear. The purpose of this study is to evaluate the effectiveness of an education intervention for BBN based on immersive experiences with a high fidelity simulator and role-play with standardized patients (SPs). A secondary purpose is to gather validity evidence to support the use of a GRIEV_ING instrument to assess BBN skills.

Methods

The communication skills for BBN of 16 residents were assessed via videotaped SP encounters at baseline and immediately post-intervention. Residents’ perceptions about their ability and comfort for BBN were collected using pre and post workshop surveys.

Results

Posttest scores were significantly higher than the pretest scores for the GRIEV_ING checklist, as well as on the communication global rating. The GRIEV_ING checklist had acceptable inter-rater and internal-consistency reliabilities. Performance was not related to years of training, or previous BBN experience.

Conclusion

Anesthesiology residents’ communication skills when BBN in relation to a critical incident may be improved with educational interventions based on immersive experiences with a high fidelity simulator and role-play with SPs.

Electronic supplementary material

The online version of this article (doi:10.1186/s12871-017-0374-7) contains supplementary material, which is available to authorized users.

Integration of simulation in postgraduate studies in Saudi Arabia: The current practice in anesthesia training program

The educational programs in the Saudi Commission for Health Specialties are developing rapidly in the fields of technical development. Such development is witnessed, particularly in the scientific areas related to what is commonly known as evidence-based medicine. This review highlights the critical need and importance of integrating simulation into anesthesia training and assessment. Furthermore, it describes the current utilization of simulation in anesthesia and critical care assessment process.

Anaesthesia and Intensive Care Residents' Perception of Simulation Training in Four Romanian Centres

Introduction

Simulation training offers an opportunity to educate anaesthesia and intensive care (AIC) residents safely. At present, it is not yet a mandatory part of residency curriculum.The aim of the study was to evaluate the residents' perception of the inclusion of simulation-based training in the Romanian AIC residency program.

Material and methods

Romanian AlCs in anaesthesia and intensive care from four training centres completed a twenty-question survey regarding their views on simulation during their residency training. Residents were divided into junior, in the first three years, or senior groups, in the last two years of residency training. The questionnaire included Likert-scale multiple-choice, open-ended, and "yes" or "no"questions regarding simulator learning, the frequency of simulation sessions, and the value of the simulation sessions in improving practice, skills or teamwork. The open-ended questions, asked which were the respondents' preferred topics to be included in simulation sessions.

Results

Fifty-six percent of residents completed and returned the questionnaire. Ninety-eight percent of them considered simulation-learning useful once a month or every three months especially in the first two years of training. All residents thought simulation sessions would improve their skills, communication abilities, and teamwork. Senior residents paid more attention to clinical scenarios (p=0.007), haemodynamic monitoring (p=0.017) and mechanical ventilation (p=0.004) as compared to juniors. All residents considered difficult airway management, and cardiac life support to be very important issues to be included in simulation sessions.

Conclusion

The survey demonstrated that simulation-based training should play a greater role and eventually became compulsory in training program in AIC academic centres.

Performance gaps and improvement plans from a 5-hospital simulation programme for anaesthesiology providers: a retrospective study

Background

Simulation is increasingly employed in healthcare provider education, but usage as a means of identifying system-wide practitioner gaps has been limited. We sought to determine whether practice gaps could be identified, and if meaningful improvement plans could result from a simulation course for anaesthesiology providers.

Methods

Over a 2-year cycle, 288 anaesthesiologists and 67 certified registered nurse anaesthetists (CRNAs) participated in a 3.5 hour, malpractice insurer-mandated simulation course, encountering 4 scenarios. 5 anaesthesiology departments within 3 urban academic healthcare systems were represented. A real-time rater scored each individual on 12 critical performance items (CPIs) representing learning objectives for a given scenario. Participants completed a course satisfaction survey, a 1-month postcourse practice improvement plan (PIP) and a 6-month follow-up survey.

Results

All recorded course data were retrospectively reviewed. Course satisfaction was generally positive (88–97% positive rating by item). 4231 individual CPIs were recorded (of a possible 4260 rateable), with a majority of participants demonstrating remediable gaps in medical/technical and non-technical skills (97% of groups had at least one instance of a remediable gap in communication/non-technical skills during at least one of the scenarios). 6 months following the course, 91% of respondents reported successfully implementing 1 or more of their PIPs. Improvements in equipment/environmental resources or personal knowledge domains were most often successful, and several individual reports demonstrated a positive impact on actual practice.

Conclusions

This professional liability insurer-initiated simulation course for 5 anaesthesiology departments was feasible to deliver and well received. Practice gaps were identified during the course and remediation of gaps, and/or application of new knowledge, skills and resources was reported by participants.

Simulation With PARTS (Phase-Augmented Research and Training Scenarios): A Structure Facilitating Research and Assessment in Simulation

INTRODUCTION

Assessment in simulation is gaining importance, as are scenario design methods increasing opportunity for assessment. We present our approach to improving measurement in complex scenarios using PARTS [Phase-Augmented Research and Training Scenarios], essentially separating cases into clearly delineated phases.

METHODS

We created 7 PARTS with real-time rating instruments and tested these in 63 cases during 4 weeks of simulation. Reliability was tested by comparing real-time rating with postsimulation video-based rating using the same instrument. Validity was tested by comparing preintervention and postintervention total results, by examining the difference in improvement when focusing on the phase-specific results addressed by the intervention, and further explored by trying to demonstrate the discrete improvement expected from proficiency in the rare occurrence of leader inclusive behavior.

RESULTS

Intraclass correlations [3,1] between real-time and postsimulation ratings were 0.951 (95% confidence interval [CI], 0.794-0.990), 1.00 (95% CI, --to--), 0.948 (95% CI, 0.783-0.989), and 0.995 (95% CI, 0.977-0.999) for 3 phase-specific scores and total scenario score, respectively. Paired t tests of prelecture-postlecture performance showed an improvement of 14.26% (bias-corrected and accelerated bootstrap [BCa] 95% CI, 4.71-23.82; P = 0.009) for total performance but of 28.57% (BCa 95% CI, 13.84-43.30; P = 0.002) for performance in the respective phase. The correlation of total scenario performance with leader inclusiveness was not significant (rs = 0.228; BCa 95% CI. -0.082 to 0.520; P = 0.119) but significant for specific phase performance (rs = 0.392; BCa 95% CI, 0.118-0.632; P = 0.006).

CONCLUSIONS

The PARTS allowed for improved reliability and validity of measurements in complex scenarios.

Improving Patient Safety through Simulation Training in Anesthesiology: Where Are We?

There have been colossal technological advances in the use of simulation in anesthesiology in the past 2 decades. Over the years, the use of simulation has gone from low fidelity to high fidelity models that mimic human responses in a startlingly realistic manner, extremely life-like mannequin that breathes, generates E.K.G, and has pulses, heart sounds, and an airway that can be programmed for different degrees of obstruction. Simulation in anesthesiology is no longer a research fascination but an integral part of resident education and one of ACGME requirements for resident graduation. Simulation training has been objectively shown to increase the skill-set of anesthesiologists. Anesthesiology is leading the movement in patient safety. It is rational to assume a relationship between simulation training and patient safety. Nevertheless there has not been a demonstrable improvement in patient outcomes with simulation training. Larger prospective studies that evaluate the improvement in patient outcomes are needed to justify the integration of simulation training in resident education but ample number of studies in the past 5 years do show a definite benefit of using simulation in anesthesiology training. This paper gives a brief overview of the history and evolution of use of simulation in anesthesiology and highlights some of the more recent studies that have advanced simulation-based training.

Comparison of 2 resident learning tools-interactive screen-based simulated case scenarios versus problem-based learning discussions: a prospective quasi-crossover cohort study

STUDY OBJECTIVE

Simulation-based learning is emerging as an alternative educational tool in this era of a relative shortfall of teaching anesthesiologists. The objective of the study is to assess whether screen-based (interactive computer simulated) case scenarios are more effective than problem-based learning discussions (PBLDs) in improving test scores 4 and 8 weeks after these interventions in anesthesia residents during their first neuroanesthesia rotation.

DESIGN

Prospective, nonblinded quasi-crossover study.

SETTING

Cleveland Clinic.

PATIENTS

Anesthesiology residents.

INTERVENTIONS

Two case scenarios were delivered from the Anesoft software as screen-based sessions, and parallel scripts were developed for 2 PBLDs. Each resident underwent both types of training sessions, starting with the PBLD session, and the 2 cases were alternated each month (ie, in 1 month, the screen-based intervention used case 1 and the PBLD used case 2, and vice versa for the next month).

MEASUREMENTS

Test scores before the rotation (baseline), immediately after the rotation (4 weeks after the start of the rotation), and 8 weeks after the start of rotation were collected on each topic from each resident. The effect of training method on improvement in test scores was assessed using a linear mixed-effects model.

MAIN RESULTS

Compared to the departmental standard of PBLD, the simulation method did not improve either the 4- or 8-week mean test scores (P = .41 and P = .40 for training method effect on 4- and 8-week scores, respectively). Resident satisfaction with the simulation module on a 5-point Likert scale showed subjective evidence of a positive impact on resident education.

CONCLUSIONS

Screen-based simulators were not more effective than PBLD for education during the neuroanesthesia rotation in anesthesia residency.

Progress in simulation education: developing an anesthesia curriculum

PURPOSE OF REVIEW

Simulation's role in anesthesia education is expanding to include more advanced skills and training for subspecialty practice. This review will provide an overview of many recent studies that expand the simulation curriculum for anesthesia education.

RECENT FINDINGS

Recent studies describe a curriculum that uses a range of simulation modalities, including part-task trainers, mannequin-based simulation, virtual reality, in-situ techniques, screen-based simulations as well as encounters with 'standardized' patients, nurses or physician colleagues. A variety of studies describe the use of task-training devises to more effectively acquire skills, such as fibre-optic intubation, ultrasound-guided regional anesthesia and transthoracic echocardiography as well as expand on a variety of teamwork skills particularly in subspecialty anesthesia practice.

SUMMARY

A curriculum is emerging that utilizes a variety of simulation modalities as part of a more comprehensive educational strategy for anesthesia specialty training.

Simulation in the operating room

Simulation has become a significant training tool in the operating room (OR). It can be used in both simple task training and complex scenarios. The challenge for simulation in the OR is how to translate that which is learned, and perceived to beneficial, into behavioral change and improved patient outcomes. Simulation in the developing world is progressing, but is still hampered by a shortage of material, personnel funding.