Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists

Simulation in pediatric anesthesiology: current state and visions for the future

PURPOSE OF REVIEW

Simulation is a well established practice in medicine. This review reflects upon the role of simulation in pediatric anesthesiology in three parts: training anesthesiologists to care for pediatric patients safely and effectively; evaluating and improving systems of care for children; and visions for the future.

RECENT FINDINGS

Simulation continues to prove a useful modality to educate both novice and experienced clinicians in the perioperative care of infants and children. It is also a powerful tool to help analyze and improve upon how care is provided to infants and children. Advances in technology and computational power now allow for a greater than ever degree of innovation, accessibility, and focused reflection and debriefing, with an exciting outlook for promising advances in the near future.

SUMMARY

Simulation plays a key role in developing and achieving peak performance in the perioperative care of infants and children. Although simulation already has a great impact, its full potential is yet to be harnessed.

Effect of Polypectomy Simulation-Based Mastery Learning on Skill Retention Among Practicing Endoscopists

PURPOSE

Practicing endoscopists frequently perform and teach screening colonoscopies and polypectomies, but there is no standardized method to train and assess physicians who perform polypectomy procedures. The authors created a polypectomy simulation-based mastery learning (SBML) curriculum and hypothesized that completion of the curriculum would lead to immediate improvement in polypectomy skills and skill retention at 6 and 12 months after training.

METHOD

The authors performed a pretest-posttest cohort study with endoscopists who completed SBML and were randomized to follow-up at 6 or 12 months from May 2021 to August 2022. Participants underwent SBML training, including a pretest, a video lecture, deliberate practice, and a posttest. All learners were required to meet or exceed a minimum passing standard on a 17-item skills checklist before completing training and were randomized to follow-up at 6 or 12 months. The authors compared simulated polypectomy skills performance on the checklist from pretest to posttest and posttest to 6- or 12-month follow-up test.

RESULTS

Twenty-four of 30 eligible participants (80.0%) completed the SBML intervention, and 20 of 24 (83.3%) completed follow-up testing. The minimum passing standard was set at 93% of checklist items correct. The pretest passing rate was 4 of 24 participants (16.7%) compared with 24 of 24 participants (100%) at posttest ( P < .001). There were no significant differences in passing rates from posttest to combined 6- and 12-month posttest in which 18 of 20 participants (90.0%) passed.

CONCLUSIONS

Before training and despite years of clinical experience, practicing endoscopists demonstrated poor performance of polypectomy skills. SBML was an effective method for practicing endoscopists to acquire and maintain polypectomy skills during a 6- to 12-month period.

Age and its impact on crisis management performance and learning after simulation-based education by acute care physicians: a multicentre prospective cohort study

BACKGROUND

Physiological changes associated with ageing could negatively impact the crisis resource management skills of acute care physicians. This study was designed to determine whether physician age impacts crisis resource management skills, and crisis resource management skills learning and retention using full-body manikin simulation training in acute care physicians.

METHODS

Acute care physicians at two Canadian universities participated in three 8-min simulated crisis (pulseless electrical activity) scenarios. An initial crisis scenario (pre-test) was followed by debriefing with a trained facilitator and a second crisis scenario (immediate post-test). Participants returned for a third crisis scenario 3-6 months later (retention post-test).

RESULTS

For the 48 participants included in the final analysis, age negatively correlated with baseline Global Rating Scale (GRS; r=-0.30, P<0.05) and technical checklist scores (r=-0.44, P<0.01). However, only years in practice and prior simulation experience, but not age, were significant in a subsequent stepwise regression analysis. Learning from simulation-based education was shown with a mean difference in scores from pre-test to immediate post-test of 2.28 for GRS score (P<0.001) and 1.69 for technical checklist correct score (P<0.001); learning was retained for 3-6 months. Only prior simulation experience was significantly correlated with a decreased change in learning (r=-0.30, P<0.05).

CONCLUSIONS

A reduced amount of prior simulation training and increased years in practice, but not age on its own, were significant predictors of low baseline crisis resource management performance. Simulation-based education leads to crisis resource management learning that is well retained for 3-6 months, regardless of age or years in practice.

Continuing professional development (CPD) for anesthetists: A systematic review

BACKGROUND

In accordance with the focus on patient safety and quality in healthcare, continuing professional development (CPD) has received increasing levels of attention as a means to ensure physicians maintain their clinical competencies and are fit to practice. There is some evidence of a beneficial effect of CPD, though few studies have evaluated its effect within anesthesia. The primary aim of this systematic review was to establish which CPD activities anesthetists are engaged in and their effectiveness. The secondary aim was to explore which methods are employed to evaluate anesthetists' clinical performance.

METHODS

Databases searched: Medline, Embase and Web of Science, in May 2023. Additional papers were identified through searching the references of included studies. Eligible studies included anesthetists, either exclusively or combined with other healthcare professionals, who underwent a learning activity or assessment method as part of a formalized CPD program or a stand-alone activity. Non-English language studies, non-peer reviewed studies and studies published prior to 2000 were excluded. Eligible studies were quality assessed and narratively synthesized, with results presented as descriptive summaries.

RESULTS

A total of 2112 studies were identified, of which 63 were eligible for inclusion, encompassing more than 137,518 participants. Studies were primarily of quantitative design and medium quality. Forty-one studies reported outcomes of single learning activities, whilst 12 studies investigated different roles of assessment methods in CPD and ten studies evaluated CPD programs or combined CPD activities. A 36 of the 41 studies reported positive effects of single learning activities. Investigations of assessment methods revealed evidence of inadequate performance amongst anesthetists and a mixed effect of feedback. Positive attitudes and high levels of engagement were identified for CPD programs, with some evidence of a positive impact on patient/organizational outcomes.

DISCUSSION

Anesthetists are engaged in a variety of CPD activities, with evidence of high levels of satisfaction and a positive learning effect. However, the impact on clinical practice and patient outcomes remains unclear and the role of assessment is less well-defined. There is a need for further, high-quality studies, evaluating a broader range of outcomes, in order to identify which methods are most effective to train and assess specialists in anesthesia.

Clinical triggers and vital signs influencing crisis acknowledgment and calls for help by anesthesiologists: A simulation-based observational study

STUDY OBJECTIVE

In a perioperative emergency, anesthesiologists must acknowledge the unfolding crisis promptly, call for timely assistance, and avert patient harm. We aimed to identify vital signs and qualitative factors prompting crisis acknowledgment and to compare responses between observers and participants in simulation.

DESIGN

Prospective, simulation-based, observational study.

SETTING

An anesthesia crisis resource management course at a freestanding simulation center.

SUBJECTS

Sixty attending anesthesiologists from a variety of practice settings.

INTERVENTIONS

In each case, a primary anesthesiologist in charge (PAIC) managed a simulated patient undergoing a uniformly scripted sequence of perioperative anaphylaxis and called for help from another anesthesiologist when a crisis began. Anesthesiologist observers (AOs) viewed the case separately and recorded times of crisis onset.

MEASUREMENTS

Simulation footage was reviewed by investigators for patient vital signs and participant behaviors at times of crisis acknowledgment, with the call for help as an explicit proxy for PAIC crisis acknowledgment. These factors were categorized, and group-level data were compared.

RESULTS

Nineteen PAICs and 41 AOs were included. Clinicians acknowledged crises around a mean arterial pressure (MAP) of 65 mmHg and oxygen saturation of 94% as anaphylactic shock progressed. PAICs acknowledged crises at a higher respiratory rate than AOs (20 vs. 18 breaths/min, p = 0.038). Other vitals and timing of crisis acknowledgment did not differ between PAICs and AOs. Nearly half of all participants (45%) identified crises at MAP <65 mmHg. Timing of crisis acknowledgment varied widely (range: 421 s).

CONCLUSIONS

Despite overall heterogeneity in clinical performance, anesthesiologists acknowledged crises per standard definitions of hypotension. Thresholds for crisis acknowledgment did not significantly differ between PAICs and AOs, suggesting minimal effect from active care responsibility. Many indicated crises at MAP <65 mmHg or after significant deterioration, risking failure-to-rescue events. We suggest that crisis management instruction should address triggers for requesting help.

Assessment Practices in Continuing Professional Development Activities in Health Professions: A Scoping Review

INTRODUCTION

In continuing professional development (CPD), educators face the need to develop and implement innovative assessment strategies to adhere to accreditation standards and support lifelong learning. However, little is known about the development and validation of these assessment practices. We aimed to document the breadth and depth of what is known about the development and implementation of assessment practices within CPD activities.

METHODS

We conducted a scoping review using the framework proposed by Arksey and O'Malley (2005) and updated in 2020. We examined five databases and identified 1733 abstracts. Two team members screened titles and abstracts for inclusion/exclusion. After data extraction, we conducted a descriptive analysis of quantitative data and a thematic analysis of qualitative data.

RESULTS

A total of 130 studies were retained for the full review. Most reported assessments are written assessments (n = 100), such as multiple-choice items (n = 79). In 99 studies, authors developed an assessment for research purpose rather than for the CPD activity itself. The assessment validation process was detailed in 105 articles. In most cases, the authors examined the content with experts (n = 57) or pilot-tested the assessment (n = 50). We identified three themes: 1-satisfaction with assessment choices; 2-difficulties experienced during the administration of the assessment; and 3-complexity of the validation process.

DISCUSSION

Building on the adage "assessment drives learning," it is imperative that the CPD practices contribute to the intended learning and limit the unintended negative consequences of assessment. Our results suggest that validation processes must be considered and adapted within CPD contexts.

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study

OBJECTIVES

The current official model of training in Intensive Care Medicine (ICM) in Spain is based on exposure to experiences through clinical rotations. The main objective was to determine the level of competency (I novice to V independent practitioner) achieved by the residents at the end of the 3rd year of training (R3) in ICM through a simulation-based OSCE. Secondary objectives were: (1) To identify gaps in performance, and (2) To investigate the reliability and feasibility of conducting simulation-based assessment at multiple sites.

DESIGN

Observational multicenter study.

SETTING

Thirteen Spanish ICU Departments.

PARTICIPANTS

Thirty six R3.

INTERVENTION

The participants performed on five, 15-min, high-fidelity crisis scenarios in four simulation centers. The performances were video recorded for later scoring by trained raters.

MAIN VARIABLES OF INTEREST

Via a Delphi technique, an independent panel of expert intensivists identified critical essential performance elements (CEPE) for each scenario to define the levels of competency.

RESULTS

A total of 176 performances were analyzed. The internal consistency of the check-lists were adequate (KR-20 range 0.64-0.79). Inter-rater reliability was strong [median Intraclass Correlation Coefficient across scenarios: 0.89 (0.65-0.97)]. Competency levels achieved by R3 were: Level I (18.8%), II (35.2%), III (42.6%), IV/V (3.4%). Overall, a great heterogeneity in performance was observed.

CONCLUSION

The expected level of competency after one year in the ICU was achieved only in half of the performances. A more evidence-based educational approach is needed. Multiple center simulation-based assessment showed feasibility and reliability as an evaluation method of competency.

TRIAL REGISTRATION

COBALIDATION. NCT04278976. (https://register.

CLINICALTRIALS

gov).

Emergency cricothyroidotomy in difficult airway simulation – a national observational study of Air Ambulance crew performance

Background

Advanced prehospital airway management includes complex procedures carried out in challenging environments, necessitating a high level of technical and non-technical skills. We aimed to describe Norwegian Air Ambulance-crews’ performance in a difficult airway scenario simulation, ending with a “cannot intubate, cannot oxygenate”-situation.

Methods

The study describes Air Ambulance crews’ management of a simulated difficult airway scenario. We used video-observation to assess time expenditure according to pre-defined time intervals and technical and non-technical performance was evaluated according to a structured evaluation-form.

Results

Thirty-six crews successfully completed the emergency cricothyroidotomy with mean procedural time 118 (SD: ±70) seconds. There was variation among the crews in terms of completed procedural steps, including preparation of equipment, patient- monitoring and management. The participants demonstrated uniform and appropriate situational awareness, and effective communication and resource utilization within the crews was evident.

Conclusions

We found that Norwegian Air Ambulance crews managed a prehospital “cannot intubate, cannot oxygenate”-situation with an emergency cricothyroidotomy under stressful conditions with effective communication and resource utilization, and within a reasonable timeframe. Some discrepancies between standard operating procedures and performance are observed. Further studies to assess the impact of check lists on procedural aspects of airway management in the prehospital environment are warranted.

Importance of non-technical skills in anaesthesia education

Rising concern about patient safety has resulted in growing interest in non-technical skills (NTS) among anaesthesiologists. Growing evidence suggesting the use of good NTS training in patient safety in simulated as well as real-world environment made them important in medical education. Both technical skills (TS) and NTS are interdependent. Successful task performance depends on effective integration of both TS and NTS for any given situation. Development of tools for assessing the NTS of an anaesthesiologist in improving health care outcomes is challenging. Teaching, understanding and evaluating NTS among anaesthesiologists in improving health care outcomes is a domain which is supposed to be a rich seam for future studies.

A Mixed Methods Study of Senior Medical Student Help Calling in an Individual, Acute Care Simulation Experience

INTRODUCTION

The ability to appropriately seek help is a key skill for medical students transitioning to residency. We designed a simulation activity for senior medical students and became interested in studying help calling behaviors and attitudes.

METHODS

We collected quantitative data regarding help calling for 2 simulation cases and qualitative data using a survey. We undertook a structured thematic analysis based on grounded theory methodology. We identified and compared groups who did and did not call for help.

RESULTS

One hundred thirty-four students participated and 122 (91%) completed an evaluation. More students called for help without prompting in the second case (34/134, 25% in first case; 110/134, 82% in second case, P < 0.001). Most students did not call for help in the first case but called in the second case (81, 60%). Our qualitative analysis identified 5 themes: (1) students seek to avoid shame and burdening their team, (2) prior institutional simulation and testing experience may imprint help calling behaviors, (3) students view help calling primarily through an individual lens, (4) students overestimate the complexity of the help calling process, and (5) the simulation environment and intentional experimentation make it difficult to observe natural behavior. When compared with help callers, more non-help callers had phrases coded as "perception of expectations" (37% vs. 10%, P = 0.03).

CONCLUSIONS

Learners participating in simulation exercises designed to promote help-calling behaviors face training-based barriers related to shame and the desire for autonomy and simulation-based challenges related to assessment.

Honesty and Transparency, Indispensable to the Clinical Mission-Part I: How Tiered Professionalism Interventions Support Teamwork and Prevent Adverse Events

At the foundation of clinical medicine is the relationship among patients, families, and health care professionals. Implicit to that social contract, professionals pledge to bring clinical excellence to advance their patients' wellness and healing-and to prevent harm. Patients trust that those privileged to deliver care will do so unwaveringly in service of patients' best interests; however, the incentives and infrastructure surrounding health care delivery can promote or undermine individual performance, teamwork, and patient safety. Modeling professionalism and identifying slips and lapses supports pursuit of high reliability. Part 1, Promoting Professionalism, introduces the first of 3 pillars of advancing the clinical mission.

Simulation in the Continuing Professional Development of Academic Emergency Physicians: A Canadian National Survey

INTRODUCTION

Simulation is becoming a popular educational modality for physician continuing professional development (CPD). This study sought to characterize how simulation-based CPD (SBCPD) is being used in Canada and what academic emergency physicians (AEPs) desire in an SBCPD program.

METHODS

Two national surveys were conducted from March to June 2018. First, the SBCPD Needs Assessment Survey was administered online to all full-time AEPs across 9 Canadian academic emergency medicine (EM) sites. Second, the SBCPD Status Survey was administered by telephone to the department representatives (DRs)-simulation directors or equivalent-at 20 Canadian academic EM sites.

RESULTS

Response rates for the SBCPD Needs Assessment and the SBCPD Status Survey were 40% (252/635) and 100% (20/20) respectively. Sixty percent of Canadian academic EM sites reported using SBCPD, although only 30% reported dedicated funding support. Academic emergency physician responses demonstrated a median annual SBCPD of 3 hours. Reported incentivization for SBCPD participation varied with AEPs reporting less incentivization than DRs. Academic emergency physicians identified time commitments outside of shift, lack of opportunities, and lack of departmental funding as their top barriers to participation, whereas DRs thought AEPs fear of peer judgment and inexperience with simulation were substantial barriers. Content areas of interest for SBCPD were as follows: rare procedures, pediatric resuscitation, and neonatal resuscitation. Lastly, interprofessional involvement in SBCPD was valued by both DRs and AEPs.

CONCLUSIONS

Simulation-based CPD programs are becoming common in Canadian academic EM sites. Our findings will guide program coordinators in addressing barriers to participation, selecting content, and determining the frequency of SBCPD events.

Competence in Decision Making: Setting Performance Standards for Critical Care

BACKGROUND

Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees.

METHODS

Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards.

RESULTS

For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario.

CONCLUSIONS

Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains.

Faculty Sim: Implementation of an Innovative, Simulation-based Continuing Professional Development Curriculum for Academic Emergency Physicians

BACKGROUND

Maintaining and enhancing competence in the breadth of emergency medicine (EM) is an ongoing challenge. In particular, resuscitative care in EM involves high-risk clinical encounters that demand strong procedural skills, effective team leadership, and up-to-date clinical knowledge. Simulation-based medical education is an effective modality for enhancing technical and nontechnical skills in crisis situations and has been effectively embedded in undergraduate and postgraduate medical curricula worldwide. To our knowledge, there are few existing systematic department-wide simulation programs to address continuing professional development (CPD) for practicing academic EM faculty.

DEVELOPMENT PROCESS

We developed our novel, simulation-based CPD program following Kern's six-step model. Based on the results of a multimodal needs assessment, a longitudinal curriculum was mapped and tailored to the available resources. Institutional support was provided in the form of a departmental grant to fund a physician program lead, monthly session instructors, and operating costs.

OUTCOMES

CPD simulation sessions commenced in January 2017. Our needs assessment identified two key types of educational needs: 1) crisis resource management skills and 2) frequent practice of high-stakes critical care procedures (e.g., surgical airways). Simulation sessions involve two high-fidelity simulated resuscitations and one skills lab per day. To date, 21 sessions have been delivered, reaching 161 practicing EPs. Feedback from our faculty has been positive.

REFLECTIVE DISCUSSION

We have successfully introduced a curriculum of monthly simulation-based CPD based on the educational needs of our EPs. Future work will include more detailed program evaluation linked to clinical outcomes and program expansion to support nearby institutions.

A correlation between anaesthesia-related cardiac arrest outcomes and country human development index: A narrative review

Studies have demonstrated gaps between developed and developing countries in the quality of surgical and anaesthesia care. The aim of this review was to provide a critical overview of documented outcomes from the 2010s of anaesthesia-related cardiac arrest events in countries with largely differing Human Development Indexes (HDIs). The HDI ranges from 0 to 1, representing the lowest and highest levels of development, respectively. Most related studies conducted between 2011 and 2020 showed low rates (from 0 to 215 per million anaesthetics) of anaesthesia-related mortality up to the 30th postoperative day in very high-HDI countries (HDI ≥ 0.800) and higher rates (from 0 to 915.4 per million anaesthetics) in high-HDI countries (HDI: 0.700-0.799). Low-HDI countries (HDI < 0.550) showed higher anaesthesia-related mortality rates, which were greater than 1500 per million anaesthetics. The anaesthesia-related mortality rates per quartile demonstrated a gap in the anaesthesia-related safety between very high- and high-HDI countries, and especially between very high- and low-HDI countries. Anaesthesia-related cardiac arrest showed similarly high survival proportions in very high-HDI countries (45.9% to 100%) and high-HDI countries (62.9% to 100%), while in a low-HDI country, the anaesthesia-related cardiac arrest survival was lower (22.2%). Our review demonstrates large gaps among countries with largely differing HDIs regarding anaesthesia-related cardiac arrest outcomes in the last decade. This finding highlights the need to improve patient safety care in low-HDI countries. Anaesthesia patient safety has improved in high-HDI countries, but there is still a persistent gap in the health care systems of these countries and those of very high-HDI countries. Our review also found a consistent improvement in anaesthesia patient safety in very high-HDI countries.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Tools for Assessing the Performance of Pediatric Perioperative Teams During Simulated Crises: A Psychometric Analysis of Clinician Raters' Scores

INTRODUCTION

The pediatric perioperative setting is a dynamic clinical environment where multidisciplinary interprofessional teams interact to deliver complex care to patients. This environment requires clinical teams to possess high levels of complex technical and nontechnical skills. For perioperative teams to identify and maintain clinical competency, well-developed and easy-to-use measures of competency are needed.

METHODS

Tools for measuring the technical and nontechnical performance of perioperative teams were developed and/or identified, and a group of raters were trained to use the instruments. The trained raters used the tools to assess pediatric teams managing simulated emergencies. A psychometric analysis of the trained raters' scores using the different instruments was performed and the agreement between the trained raters' scores and a reference score was determined.

RESULTS

Five raters were trained and scored 96 recordings of perioperative teams managing simulated emergencies. Scores from both technical skills assessment tools demonstrated significant reliability within and between ratings with the scenario-specific performance checklist tool demonstrating greater interrater agreement than scores from the global rating scale. Scores from both technical skills assessment tools correlated well with the other and with the reference standard scores. Scores from the Team Emergency Assessment Measure nontechnical assessment tool were more reliable within and between raters and correlated better with the reference standard than scores from the BARS tool.

CONCLUSIONS

The clinicians trained in this study were able to use the technical performance assessment tools with reliable results that correlated well with reference scores. There was more variability between the raters' scores and less correlation with the reference standard when the raters used the nontechnical assessment tools. The global rating scale used in this study was able to measure the performance of teams across a variety of scenarios and may be generalizable for assessing teams in other clinical scenarios. The Team Emergency Assessment Measure tool demonstrated reliable measures when used to assess interprofessional perioperative teams in this study.

High-Fidelity Simulation Education and Crisis Resource Management

This article explores high-fidelity simulation in anesthesiology education and provides strategies for its use to improve management of critical events. Educational theories that underlie the use of simulation are described. High-fidelity simulation is useful in teaching technical (diagnostic and procedural) and nontechnical (communication and professionalism) skills, including crisis resource management (CRM) skills. The practice of CRM is fundamental to ensuring patient safety during critical events and to the safe practice of anesthesiology, and its critical elements are presented. A discussion of the use of high-fidelity simulation to learn to combine highly complex procedural skills and CRM is also provided.

Relationship Between "Simulated Patient Scenarios and Role-Playing" Method and OSCE Performance in Senior Anesthesiology Residents: A Correlation Assessment Study

BACKGROUND

"Simulated-patient scenarios and role-playing" and OSCE are among the many non-traditional education methods with variable results in different clinical settings.

OBJECTIVES

This cross-sectional study was performed to assess the correlation between the results of these two methods in senior anesthesiology residents, with a special focus on four of the six ACGME core competencies.

METHODS

During two years, senior anesthesiology residents were subject to "simulated patient scenario and role-playing" sessions. Two faculty members took the role of the patient and one of the relatives. An objective checklist with 15 items was prepared to be rated by other department faculty members. Meanwhile, an ordered pattern of OSCE was prepared to cover four core competencies that were more related to this academic process (from a total of six core competencies). The mean and standard deviation of the score of each of the 15 items in the checklist were calculated. The correlation between cumulative checklist scoring results and OSCE exam results was assessed. A P value of less than 0.05 was considered significant.

RESULTS

A total of 40 senior anesthesiology residents, with 344 assessments by faculty members in 40 sessions, were enrolled in the study. The questionnaire's Cronbach's alpha reliability was 0.74. No statistically significant disparity was detected between the results of the two assessment methods, while the results of the two assessments had a significant correlation (two-tailed correlation coefficient = 0.886; P value < 0.001).

CONCLUSIONS

There was an objective relationship between the results of "simulated patient scenario and role-playing" strategies and the results of OSCE exams using an observer-based rating method. Thus, they could be used as surrogates in the assessment of core clinical competencies of senior anesthesiology residents.

Data and debriefing observations on healthcare simulation to prepare for the COVID-19 pandemic

We report on data and debriefing observations in the context of an immersive simulation conducted to (a) train clinicians and (b) test new protocols and kits, developed in table-top exercises without prior clinical experience to fit anticipated clinical encounters in the setting of the rapidly expanding COVID-19 pandemic. We simulated scenarios with particular relevance for anesthesiology, perioperative and critical care, including (1) cardiac arrest, (2) emergency airway management, (3) tele-instruction for remote guidance and supervision, and (4) transporting an intubated patient. Using a grounded theory approach, three authors (MHA, DLR, EHS) developed emergent themes. First alone and then together, we sought consensus in uncovering overarching themes and constructs from the debriefings. We thus performed an informal qualitative thematic analysis based in a critical realist epistemological position - the understanding that our findings, while real, are affected by situational variables and the observer's perspective[1,2]. We compared data from videos and triangulated the data by member checking. All participants and course instructors volunteered to participate in this educational project and contributed as co-authors to this manuscript. During debriefing, we applied crisis resource management concepts including situation awareness, prioritization of tasks, and clear communication practices, conducting the debriefing with emphasis on current TeamStepps 2.0 terminology and concepts. [3,4] In addition, we re-evaluated formerly familiar processes, as shortcomings of protocols, kits, and interdisciplinary cooperation became apparent. The data provide detailed observations on how immersive simulation and debriefing among peers mitigated the unfamiliarity of individual clinicians and the organization at large with the demands of an unprecedented healthcare crisis. We also observed and report on the anxiety caused by resource constraints, risk to clinicians in the face of limited personal equipment, and the overall uncertainty surrounding COVID-19. We began to summarize, interpret, critique, and discuss our data and debriefing observations in a rapid co-publication in the Journal of Clinical Anesthesia. [Healthcare Simulation to Prepare for the COVID-19 Pandemic][5].

Cautionary study on the effects of pay for performance on quality of care: a pilot randomised controlled trial using standardised patients

BACKGROUND

Due to the difficulty of studying incentives in practice, there is limited empirical evidence of the full-impact pay-for-performance (P4P) incentive systems.

OBJECTIVE

To evaluate the impact of P4P in a controlled, simulated environment.

DESIGN

We employed a simulation-based randomised controlled trial with three standardised patients to assess advanced practice providers' performance. Each patient reflected one of the following: (A) indicated for P4P screenings, (B) too young for P4P screenings, or (C) indicated for P4P screenings, but screenings are unrelated to the reason for the visit. Indication was determined by the 2016 Centers for Medicare and Medicaid Services quality measures.

INTERVENTION

The P4P group was paid $150 and received a bonus of $10 for meeting each of five outcome measures (breast cancer, colorectal cancer, pneumococcal, tobacco use and depression screenings) for each of the three cases (max $300). The control group received $200.

SETTING

Learning resource centre.

PARTICIPANTS

35 advanced practice primary care providers (physician assistants and nurse practitioners) and 105 standardised patient encounters.

MEASUREMENTS

Adherence to incentivised outcome measures, interpersonal communication skills, standards of care, and misuse.

RESULTS

patient was more likely to be prescribed screenings not indicated, but highlighted by P4P: breast cancer screening (47% P4P vs 0% control, p<0.01) and colorectal cancer screening (24% P4P vs 0% control, p=0.03). The P4P group over-reported completion of incentivised measures resulting in overpayment (average of $9.02 per patient).

LIMITATIONS

A small sample size and limited variability in patient panel limit the generalisability of findings.

CONCLUSIONS

Our findings caution the adoption of P4P by highlighting the unintended consequences of the incentive system.

Development and Usability Testing of the Society for Pediatric Anesthesia Pedi Crisis Mobile Application

When life-threatening, critical events occur in the operating room, the fast-paced, high-distraction atmosphere often leaves little time to think or deliberate about management options. Success depends on applying a team approach to quickly implement well-rehearsed, systematic, evidence-based assessment and treatment protocols. Mobile devices offer resources for readily accessible, easily updatable information that can be invaluable during perioperative critical events. We developed a mobile device version of the Society for Pediatric Anesthesia 26 Pediatric Crisis paper checklists-the Pedi Crisis 2.0 application-as a resource to support clinician responses to pediatric perioperative life-threatening critical events. Human factors expertise and principles were applied to maximize usability, such as by clustering information into themes that clinicians utilize when accessing cognitive aids during critical events. The electronic environment allowed us to feature optional diagnostic support, optimized navigation, weight-based dosing, critical institution-specific phone numbers pertinent to emergency response, and accessibility for those who want larger font sizes. The design and functionality of the application were optimized for clinician use in real time during actual critical events, and it can also be used for self-study or review. Beta usability testing of the application was conducted with a convenience sample of clinicians at 9 institutions in 2 countries and showed that participants were able to find information quickly and as expected. In addition, clinicians rated the application as slightly above "excellent" overall on an established measure, the Systems Usability Scale, which is a 10-item, widely used and validated Likert scale created to assess usability for a variety of situations. The application can be downloaded, at no cost, for iOS devices from the Apple App Store and for Android devices from the Google Play Store. The processes and principles used in its development are readily applicable to the development of future mobile and electronic applications for the field of anesthesiology.

Roads less traveled: understanding the "why" in simulation as an integrated continuing professional development activity

Background

The simulation community has experienced significant advances, strengthening the case for the use of simulation in medical education toward improving patient outcomes. However, an underlying assumption particularly regarding utilization of simulation by those who are in practice, is that simulation will be selected as a continuing professional development (CPD) strategy. Exploring reasons for choices of educational formats, particularly regarding simulation, is poorly integrated into CPD research.

Discussion

Despite significant advances the scientific simulation community has been slow to produce evidence regarding why practitioners may be reserved in engaging in simulation or not. Using examples from related education contexts the author attempts to bridge simulation science, CPD and less commonly used theoretical frameworks to address this issue. The author argues that theoretical perspectives that recognize the use of simulation for CPD as a socio-personal process and/or a personal or group issue (e.g., theories of intelligence, self-determination theory, theory of planned behavior, social identity theory) and that are conceptually distinct from educational mechanisms/ provision are necessary to advance simulation use in CPD contexts.

Conclusion

Given the close relationship practicing clinicians have to patient outcomes a new imperative may be to focus on the theoretical and practical links informing simulation use for CPD at the level of the individual and individual-among-professional groups. The simulation community may therefore need to engage in research that attempts to further uncover and address underlying issues of “why” clinicians integrate simulation as CPD activities or not.

A Prospective Observational Cohort Study of Calls for Help in a Tertiary Care Academic Operating Room Suite

While significant literature exists on hospital-based "code calls," there is a lack of research on calls for help in the operating room (OR). The purpose of this study was to quantify the rate and nature of calls for help in the OR of a tertiary care hospital. For a 1-year period, all calls were recorded in the main OR at The University of California, Irvine Medical Center. The average rate of calls per 1000 anesthesia hours was 1.4 (95% CI, 1.1-1.8), corresponding to a rate of 5.0 (3.8-6.5) calls per 1000 cases. Airway (44%), cardiac (32%), and hemorrhagic (11%) emergencies were the most common etiologies. Thirty-day mortality approached 11% for patients who required a call for help in the OR.

Exposure to incivility hinders clinical performance in a simulated operative crisis

Background

Effective communication is critical for patient safety. One potential threat to communication in the operating room is incivility. Although examined in other industries, little has been done to examine how incivility impacts the ability to deliver safe care in a crisis. We therefore sought to determine how incivility influenced anaesthesiology resident performance during a standardised simulation scenario of occult haemorrhage.

Methods

This is a multicentre, prospective, randomised control trial from three academic centres. Anaesthesiology residents were randomly assigned to either a normal or ‘rude’ environment and subjected to a validated simulated operating room crisis. Technical and non-technical performance domains including vigilance, diagnosis, communication and patient management were graded on survey with Likert scales by blinded raters and compared between groups.

Results

76 participants underwent randomisation with 67 encounters included for analysis (34 control, 33 intervention). Those exposed to incivility scored lower on every performance metric, including a binary measurement of overall performance with 91.2% (control) versus 63.6% (rude) obtaining a passing score (p=0.009). Binary logistic regression to predict this outcome was performed to assess impact of confounders. Only the presence of incivility reached statistical significance (OR 0.110, 95% CI 0.022 to 0.544, p=0.007). 65% of the rude group believed the surgical environment negatively impacted performance; however, self-reported performance assessment on a Likert scale was similar between groups (p=0.112).

Conclusion

Although self-assessment scores were similar, incivility had a negative impact on performance. Multiple areas were impacted including vigilance, diagnosis, communication and patient management even though participants were not aware of these effects. It is imperative that these behaviours be eliminated from operating room culture and that interpersonal communication in high-stress environments be incorporated into medical training.

Operating Room Codes Redefined: A Highly Reliable Model Integrating the Core Hospital Code Team

Introduction:

Typically, multidisciplinary teams manage cardiac arrests occurring outside of the operating room (OR). This approach results in reduced morbidity. However, arrests that occur in the OR are usually managed by OR personnel alone, missing the benefits of out-of-OR hospital code teams. At our institution, there were multiple pathways to activate codes, each having different respondents, depending on time and day of the week. This improvement initiative aimed to create a reliable intraoperative emergency response system with standardized respondents and predefined roles.

Methods:

A multidisciplinary improvement team led this project at an academic pediatric hospital in California. After simulations performed in the OR (in situ), the team identified a valuable key driver—a consistent activation process that initiated standard respondents, 24 hours a day, 7 days a week. By utilizing core hospital code members routinely available outside of the OR during days, nights, and weekends, respondents were identified to augment OR personnel. Code roles were preassigned. After education, we conducted in situ simulations that included the perioperative and out-of-OR code team members. We administered a knowledge assessment to perioperative staff.

Results:

The knowledge assessment for perioperative staff (n = 52) had an average score of 96%. Review of subsequent OR codes reflects an improved initiation process and management.

Conclusions:

The process for activating the emergency response system and roles for intraoperative code respondents were standardized to ensure a predictable code response, regardless of time or day of the week. Ongoing simulations with perioperative personnel continue to optimize the process.

Use of a Portable Functional Near-Infrared Spectroscopy (fNIRS) System to Examine Team Experience During Crisis Event Management in Clinical Simulations

Objective: The aim of this study was to investigate the utilization of a portable functional near-infrared spectroscopy (fNIRS) system, the fNIRS Pioneer^TM, to examine team experience in high-fidelity simulation-based crisis event management (CEM) training for anesthesiologists in operating rooms.

Background: Effective evaluation of team performance and experience in CEM simulations is essential for healthcare training and research. Neurophysiological measures with wearable devices can provide useful indicators of team experience to compliment traditional self-report, observer ratings, and behavioral performance measures. fNIRS measured brain blood oxygenation levels and neural synchrony can be used as indicators of workload and team engagement, which is vital for optimal team performance.

Methods: Thirty-three anesthesiologists, who were attending CEM training in two-person teams, participated in this study. The participants varied in their expertise level and the simulation scenarios varied in difficulty level. The oxygenated and de-oxygenated hemoglobin (HbO and HbR) levels in the participants’ prefrontal cortex were derived from data recorded by a portable one-channel fNIRS system worn by all participants throughout CEM training. Team neural synchrony was measured by HbO/HbR wavelet transformation coherence (WTC). Observer-rated workload and self-reported workload and mood were also collected.

Results: At the individual level, the pattern of HbR level corresponded to changes of workload for the individuals in different roles during different phases of a scenario; but this was not the case for HbO level. Thus, HbR level may be a better indicator for individual workload in the studied setting. However, HbR level was insensitive to differences in scenario difficulty and did not correlate with observer-rated or self-reported workload. At the team level, high levels of HbO and HbR WTC were observed during active teamwork. Furthermore, HbO WTC was sensitive to levels of scenario difficulty.

Conclusion: This study showed that it was feasible to use a portable fNIRS system to study workload and team engagement in high-fidelity clinical simulations. However, more work is needed to establish the sensitivity, reliability, and validity of fNIRS measures as indicators of team experience.

Translating evidence into practice: Lessons for CPD

BACKGROUND

Failure to translate best evidence into practice often generates inappropriate, unsafe, and costly healthcare. The continuing professional development (CPD) of physicians and other health professionals represents a widely underutilized strategy to improve both clinician performance and healthcare quality and safety. The evidence: Despite the clear evidence of the potential impact of CPD based in learning theory and science, some CPD providers, health systems, and clinicians themselves implement less-than-effective effective learning strategies. This phenomenon is the product of several factors: within health systems, a lack of recognition of the importance of ongoing, system-linked professional education; among CPD providers, an adherence to old but easy-to-deliver "one-and-done" methods CPD; and even among clinicians themselves, choosing less engaging learning activities, uninformed by objective performance data.

RECOMMENDATION

Suggestions to improve this lack of translation of best evidence into practice fall into four groups. Academic medical institutions, employers and educators need to embrace principles and practices of self-directed learning; health systems must share responsibility for the physician learning and the performance data and feedback on which such learning is best-based; physician specialty societies and licensing boards must undertake meaningful re-licensure and re-certification processes; and CPD planners must seek out partnerships with health system leadership and quality improvement managers as they create engaging, integrated, and impactful CPD activities.

Preventable Anesthesia-Related Adverse Events at a Large Tertiary Care Center: A Nine-Year Retrospective Analysis

BACKGROUND

Anesthesiologists have studied adverse events during anesthesia dating back to the original critical incident studies of the 1970s. Despite improvements, adverse events continue to occur. The purpose of this study was to characterize anesthesia-related adverse events within a single large tertiary care institution and to distinguish preventable adverse events from those that are not preventable.

METHODS

A retrospective review of all cases referred to the Performance Improvement (PI) Committee at a large academic medical center from 2007 to 2015 was performed. The primary adverse event and underlying cause of the event were determined using a two-reviewer system for each case. Univariate analysis was performed to determine overall characteristics of cases, the underlying causes of adverse events, and whether the event was preventable; p < 0.05 was considered significant.

RESULTS

A total of 1,424 records were referred to the PI Committee during the study period. After exclusions, 747 cases were included in the final analysis. Respiratory complications (n = 245) were the most frequently reported adverse event type. The most common respiratory events included unplanned reintubations, aspirations, and respiratory arrests. A large proportion of the adverse events (42.8%) may have been preventable. In particular, respiratory, trauma, and medication adverse events were often preventable.

CONCLUSION

Anesthesia-related adverse events continue to occur even though the field is considered at the forefront of patient safety. Respiratory, trauma, and medication events were often preventable, and these represent areas to allocate resources to improve patient safety and perioperative outcomes.

Simulation-based Assessment to Reliably Identify Key Resident Performance Attributes

Background

Obtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.

Methods

Seven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.

Results

Interrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P &lt; 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.

Conclusions

Sensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.

Use of Simulation in Performance Improvement

Human error and system failures continue to play a substantial role in preventable errors that lead to adverse patient outcomes or death. Many of these deaths are not the result of inadequate medical knowledge and skill, but occur because of problems involving communication and team management. Anesthesiologists pioneered the use of simulation for medical education in an effort to improve physician performance and patient safety. This article explores the use of simulation for performance improvement. Educational theories that underlie effective simulation programs are described as driving forces behind the advancement of simulation in performance improvement.

Relaxation before Debriefing during High-fidelity Simulation Improves Memory Retention of Residents at Three Months

Background

High-fidelity simulation is known to improve participant learning and behavioral performance. Simulation scenarios generate stress that affects memory retention and may impact future performance. The authors hypothesized that more participants would recall three or more critical key messages at three months when a relaxation break was performed before debriefing of critical event scenarios.

Methods

Each resident actively participated in one scenario and observed another. Residents were randomized in two parallel-arms. The intervention was a 5-min standardized relaxation break immediately before debriefing; controls had no break before debriefing. Five scenario-specific messages were read aloud by instructors during debriefings. Residents were asked by telephone three months later to recall the five messages from their two scenarios, and were scored for each scenario by blinded investigators. The primary endpoint was the number of residents participating actively who recalled three or more messages. Secondary endpoints included: number of residents observing who recalled three or more messages, anxiety level, and debriefing quality.

Results

In total, 149 residents were randomized and included. There were 52 of 73 (71%) residents participating actively who recalled three or more messages at three months in the intervention group versus 35 of 76 (46%) among controls (difference: 25% [95% CI, 10 to 40%], P = 0.004). No significant difference was found between groups for observers, anxiety or debriefing quality.

Conclusions

There was an additional 25% of active participants who recalled the critical messages at three months when a relaxation break was performed before debriefing of scenarios. Benefits of relaxation to enhance learning should be considered for medical education.