Evaluation of a Simpler Tool to Assess Nontechnical Skills During Simulated Critical Events

Spanish Society of Anesthesiology, Reanimation and Pain Therapy (SEDAR), Spanish Society of Emergency and Emergency Medicine (SEMES) and Spanish Society of Otolaryngology, Head and Neck Surgery (SEORL-CCC) Guideline for difficult airway management. Part II

The Airway Management section of the Spanish Society of Anesthesiology, Resuscitation, and Pain Therapy (SEDAR), the Spanish Society of Emergency Medicine (SEMES), and the Spanish Society of Otorhinolaryngology and Head and Neck Surgery (SEORL-CCC) present the Guide for the comprehensive management of difficult airway in adult patients. Its principles are focused on the human factors, cognitive processes for decision-making in critical situations, and optimization in the progression of strategies application to preserve adequate alveolar oxygenation in order to enhance safety and the quality of care. The document provides evidence-based recommendations, theoretical-educational tools, and implementation tools, mainly cognitive aids, applicable to airway management in the fields of anesthesiology, critical care, emergencies, and prehospital medicine. For this purpose, an extensive literature search was conducted following PRISMA-R guidelines and was analyzed using the GRADE methodology. Recommendations were formulated according to the GRADE methodology. Recommendations for sections with low-quality evidence were based on expert opinion through consensus reached via a Delphi questionnaire.

Development of pediatric simulation-based education - a systematic review

BACKGROUND

This systematic literature review explored the general characteristics, validation, and reliability of pediatric simulation-based education (P-SBE).

METHODS

A literature search was conducted between May 23 and 28 using the PRISMA guidelines, which covered databases such as MEDLINE, EMBASE, CINAHL, and Cochrane Library. In the third selection process, the original texts of 142 studies were selected, and 98 documents were included in the final content analysis.

RESULTS

A total of 109 papers have been published in the ten years since 2011. Most of the study designs were experimental studies, including RCT with 76 articles. Among the typologies of simulation, advanced patient simulation was the most common (92), and high-fidelity simulation was the second most common (75). There were 29 compatibility levels and professional levels, with 59 scenarios related to emergency interventions and 19 scenarios related to communication feasibility and decision making. Regarding the effect variable, 65 studies confirmed that skills were the most common. However, validity of the scenarios and effect variables was not verified in 56.1% and 67.3% of studies, respectively.

CONCLUSION

Based on these findings, simulation based-education (SBE) is an effective educational method that can improve the proficiency and competence of medical professionals dealing with child. Learning through simulation provides an immersive environment in which learners interact with the presented patient scenario and make decisions, actively learning the attitudes, knowledge, and skills necessary for medical providers. In the future, it is expected that such research on SBE will be actively followed up and verified for its validity and reliability.

A Feasible Web-Conference-Style Remote Simulation using Demonstration Video Clips in Anaesthesia under the COVID-19 Outbreaks: A Preliminary Survey Study

Objective

The Coronavirus disease-2019 (COVID-19) outbreak has deprived simulation-based medical education for health care workers. Attendees are strictly prohibited to converge at a simulation training venue because of the COVID-19 outbreaks. To address this issue, we developed a web-conference-style remote simulation program using demonstration video clips. This report introduced the method and described participants' satisfaction.

Methods

This preliminary survey study evaluated learners' satisfaction in multiple institutions. The Satisfaction Scale Questionnaire with High-Fidelity Clinical Simulation (SSHF) by a 5-degree scale was used to evaluate participants' satisfaction. The survey was conducted immediately after completing the simulation sessions.

Results

Ten (100%) participants (7 anaesthesia individuals, 2 anaesthesia residents and 1 nurse anaesthetist) from nine institutions responded to the survey. All median values of the satisfaction scores were ≥4.0, whereas the median values of scores for environmental fidelity and psychological insecurity were 3.5 and 3.0, respectively (P=0.005).

Conclusion

A web-conference-style remote simulation using demonstration video clips is a feasible method for conducting simulation-based medical education under COVID-19 that showed high satisfaction scores. Further, additional studies are required to explore the internal and external validity and the effectiveness of mastery learning.

Cross-cultural adaptation and validation of the Spanish version of the Anesthetists' Non-Technical Skills (ANTS) assessment tool

STUDY OBJECTIVE

Establish the transcultural validity of Anesthetists Non-Technical Skills (ANTS) in a Spanish-speaking country.

DESIGN

Prospective cohort.

SETTING

Clinical simulation center.

SUBJECTS

Forty-two Anesthesia PY2 and PY3 residents participated in the study.

INTERVENTIONS

Four clinical scenarios simulating anesthesia crises were assessed with a Spanish version of ANTS. Every simulated scenario was run twice with a time span of 3 to 4 months between them.

MEASUREMENTS

Two anesthesiologists independently assessed all simulated sessions using ANTS. The ANTS indicators of construct validity were obtained by confirmatory factor analysis. Various goodness-of-fit indices of the factorial model were calculated: Comparative Fit Index (CFI); Tucker-Lewis Adjustment Index (TLI) and Root Mean Square Error of Approximation (RMSEA). The standardized factor loadings and the determination coefficient (R2) was also estimated.

MAIN RESULTS

A total of 212 clinical scenarios were analyzed. The specified factorial model had the same grouping of elements in four domains as the original version of ANTS. The CFI index and the TLI were 0.99 and the RMSEA reached 0.07 (95% CI 0.06-0.08). All the standardized factor loadings were found to be >0.4. Also, the elements obtained an R2 value that fluctuated between 0.54 and 0.92.

CONCLUSIONS

The Spanish version of ANTS is a valid, reliable and a useful tool to assess non-technical skills in Spanish-speaking countries. The applicability of the instrument was comparable to the original setting. The high reliability of ANTS in our setting allows us to propose its use not just in an educational and research setting; it can be used as an assessment tool of non-technical skills.

Reliability of the Behaviorally Anchored Rating Scale (BARS) for assessing non-technical skills of medical students in simulated scenarios

PURPOSE

Caring for critically ill patients requires non-technical skills such as teamwork, communication, and task management. The Behaviorally Anchored Rating Scale (BARS) is a brief tool used to assess non-technical skills. The investigators determined inter- and intra-rater reliability of the BARS when used to assess medical students in simulated scenarios.

METHOD

The investigators created simulation scenarios for medical students during their pediatric clerkship. Content experts reviewed video recordings of the simulations and assigned BARS scores for four performance components (Situational Awareness, Decision-Making, Communication, and Teamwork) for the leader and for the team as a whole. Krippendorff's alpha with ordinal difference was calculated to measure inter- and intra-rater reliability.

RESULTS

Thirty medical students had recordings available for review. Inter- and intra-rater reliability for performance components were, respectively, Individual Situational Awareness (0.488, 0.638), Individual Decision-Making (0.529, 0.691), Individual Communication (0.347, 0.473), Individual Teamwork (0.414, 0.466), Team Situational Awareness (0.450, 0.593), Team Decision Making (0.423, 0.703), Team Communication (0.256, 0.517), and Team Teamwork (0.415, 0.490).

CONCLUSIONS

The BARS demonstrated limited reliability when assessing medical students during their pediatric clerkship. Given the unique needs of this population, a modified or new objective scoring system for assessing non-technical skills may be needed for medical students.

Instruments to evaluate non-technical skills during high fidelity simulation: A systematic review

Introduction

High Fidelity Simulations (HFS) are increasingly used to develop Non-Technical Skills (NTS) in healthcare providers, medical and nursing students. Instruments to measure NTS are needed to evaluate the healthcare providers' (HCPs) performance during HFS. The aim of this systematic review is to describe the domains, items, characteristics and psychometric properties of instruments devised to evaluate the NTS of HCPs during HFS.

Methods

A systematic review of the literature was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Studies were retrieved from PubMed, Cinahl, Web of Science, Cochrane Library, ProQuest and PubPsych. Studies evaluating the measurement properties of instruments used to assess NTS during HFS training were included. Pairs of independent reviewers determined the eligibility, extracted and evaluated the data. Risk of bias and appraisal of the methodological quality of the studies was assessed using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist, and the quality of the evidence with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

Results

A total of 3,953 articles were screened. A total of 110 reports were assessed for eligibility and 26 studies were included. Studies were conducted in Europe/United Kingdom (n = 13; 50%), North America/Australia (n = 12; 46%) and Thailand (n = 1; 4%). The NTS instruments reported in this review included from 1 to 14 domains (median of 4, Q1 = 3.75, Q3 = 5) and from 3 to 63 items (median of 15, Q1 = 10, Q3 = 19.75). Out of 19 NTS assessment instruments for HFS, the Team Emergency Assessment Measure (TEAM) can be recommended for use to assess NTS. All the other instruments require further research to assess their quality in order to be recommended for use during HFS training. Eight NTS instruments had a positive overall rating of their content validity with at least a moderate quality of evidence.

Conclusion

Among a large variety of published instruments, TEAM can be recommended for use to assess NTS during HFS. Evidence is still limited on essential aspects of validity and reliability of all the other NTS instruments included in this review. Further research is warranted to establish their performance in order to be reliably used for HFS.

Importance of non-technical skills in anaesthesia education

Rising concern about patient safety has resulted in growing interest in non-technical skills (NTS) among anaesthesiologists. Growing evidence suggesting the use of good NTS training in patient safety in simulated as well as real-world environment made them important in medical education. Both technical skills (TS) and NTS are interdependent. Successful task performance depends on effective integration of both TS and NTS for any given situation. Development of tools for assessing the NTS of an anaesthesiologist in improving health care outcomes is challenging. Teaching, understanding and evaluating NTS among anaesthesiologists in improving health care outcomes is a domain which is supposed to be a rich seam for future studies.

Reliability of simulation-based assessment for practicing physicians: performance is context-specific

INTRODUCTION

Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established.

METHODS

Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters.

RESULTS

The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability.

CONCLUSIONS

The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Development and Evaluation of the Taxonomy of Trauma Leadership Skills-Shortened for Observation and Reflection in Training: A Practical Tool for Observing and Reflecting on Trauma Leadership Performance

INTRODUCTION

Trauma leadership skills are increasingly being addressed in trauma courses, but few resources are available to systematically observe and debrief trainees' performances. The authors therefore translated their previously developed, extensive Taxonomy of Trauma Leadership Skills (TTLS) into a practical observation tool that is tailored to the vocabulary of clinician instructors and their workflow and workload during simulation-based training.

METHODS

In 2016 to 2018, the TTLS was subjected to practical evaluation in an iterative process of 2 stages. In the first stage, testing panels of trauma specialists observed excerpts from videotaped simulations and indicated from the list of elements which behaviors they felt were being shown. Any ambiguities or redundancy were addressed by rephrasing or combining elements. In the second stage, iterations were used in actual scenario training to observe and debrief trainees' performances. The instructors' recommendations resulted in further improvements of clarity, ease of use, and usefulness, until no new suggestions were raised.

RESULTS

The resultant "TTLS-Shortened for Observation and Reflection in Training" was given a simpler structure and more concrete and self-explanatory benchmarks. It contains 6 skill categories for evaluation, each with 4 to 6 benchmark behaviors.

CONCLUSIONS

The TTLS-Shortened for Observation and Reflection in Training is an important addition to other trauma assessment tools because of its specific focus on leadership skills. It helps set concrete performance expectations, simplify note taking, and target observations and debriefings. One central challenge was striking a balance between its conciseness and specificity. The authors reflected on how the decisions for the resultant structure ease and leverage the conduct of observations and performance debriefing.

Tools for Assessing the Performance of Pediatric Perioperative Teams During Simulated Crises: A Psychometric Analysis of Clinician Raters' Scores

INTRODUCTION

The pediatric perioperative setting is a dynamic clinical environment where multidisciplinary interprofessional teams interact to deliver complex care to patients. This environment requires clinical teams to possess high levels of complex technical and nontechnical skills. For perioperative teams to identify and maintain clinical competency, well-developed and easy-to-use measures of competency are needed.

METHODS

Tools for measuring the technical and nontechnical performance of perioperative teams were developed and/or identified, and a group of raters were trained to use the instruments. The trained raters used the tools to assess pediatric teams managing simulated emergencies. A psychometric analysis of the trained raters' scores using the different instruments was performed and the agreement between the trained raters' scores and a reference score was determined.

RESULTS

Five raters were trained and scored 96 recordings of perioperative teams managing simulated emergencies. Scores from both technical skills assessment tools demonstrated significant reliability within and between ratings with the scenario-specific performance checklist tool demonstrating greater interrater agreement than scores from the global rating scale. Scores from both technical skills assessment tools correlated well with the other and with the reference standard scores. Scores from the Team Emergency Assessment Measure nontechnical assessment tool were more reliable within and between raters and correlated better with the reference standard than scores from the BARS tool.

CONCLUSIONS

The clinicians trained in this study were able to use the technical performance assessment tools with reliable results that correlated well with reference scores. There was more variability between the raters' scores and less correlation with the reference standard when the raters used the nontechnical assessment tools. The global rating scale used in this study was able to measure the performance of teams across a variety of scenarios and may be generalizable for assessing teams in other clinical scenarios. The Team Emergency Assessment Measure tool demonstrated reliable measures when used to assess interprofessional perioperative teams in this study.

Utilization of a Voice-Based Virtual Reality Advanced Cardiac Life Support Team Leader Refresher: Prospective Observational Study

BACKGROUND

The incidence of cardiac arrests per year in the United States continues to increase, yet in-hospital cardiac arrest survival rates significantly vary between hospitals. Current methods of training are expensive, time consuming, and difficult to scale, which necessitates improvements in advanced cardiac life support (ACLS) training. Virtual reality (VR) has been proposed as an alternative or adjunct to high-fidelity simulation (HFS) in several environments. No evaluations to date have explored the ability of a VR program to examine both technical and behavioral skills and demonstrate a cost comparison.

OBJECTIVE

This study aimed to explore the utility of a voice-based VR ACLS team leader refresher as compared with HFS.

METHODS

This prospective observational study performed at an academic institution consisted of 25 postgraduate year 2 residents. Participants were randomized to HFS or VR training and then crossed groups after a 2-week washout. Participants were graded on technical and nontechnical skills. Participants also completed self-assessments about the modules. Proctors were assessed for fatigue and task saturation, and cost analysis based on local economic data was performed.

RESULTS

A total of 23 of 25 participants were included in the scoring analysis. Fewer participants were familiar with VR compared with HFS (9/25, 36% vs 25/25, 100%; P<.001). Self-reported satisfaction and utilization scores were similar; however, significantly more participants felt HFS provided better feedback: 99 (IQR 89-100) vs 79 (IQR 71-88); P<.001. Technical scores were higher in the HFS group; however, nontechnical scores for decision making and communication were not significantly different between modalities. VR sessions were 21 (IQR 19-24) min shorter than HFS sessions, the National Aeronautics and Space Administration task load index scores for proctors were lower in each category, and VR sessions were estimated to be US $103.68 less expensive in a single-learner, single-session model.

CONCLUSIONS

Utilization of a VR-based team leader refresher for ACLS skills is comparable with HFS in several areas, including learner satisfaction. The VR module was more cost-effective and was easier to proctor; however, HFS was better at delivering feedback to participants. Optimal education strategies likely contain elements of both modalities. Further studies are needed to examine the utility of VR-based environments at scale.

Nontechnical Competency Framework for Health Professionals in All-Hazard Emergency Environment: A Systematic Review

OBJECTIVES

To summarize characteristics and commonalities of non-technical competency frameworks for health professionals in emergency and disaster.

METHODS

An electronic literature search was conducted in PubMed, MEDLINE, ERIC, Scopus, Cochrane database, and Google Scholar to identify original English-language articles related to development, evaluation or application of the nontechnical competency frameworks. Reviewers assessed identified articles for exclusion/inclusion criteria and abstracted data on study design, framework characteristics, and reliability/validity evidence.

RESULTS

Of the 9627 abstracts screened, 65 frameworks were identified from 94 studies that were eligible for result extraction. Sixty (63.8%) studies concentrated on clinical settings. Common scenarios of the studies were acute critical events in hospitals (44;46.8%) and nonspecified disasters (39;41.5%). Most of the participants (76; 80.9%) were clinical practitioners, and participants in 36 (38.3%) studies were multispecialty. Thirty-three (50.8%) and 42 (64.6%) frameworks had not reported evidence on reliability and validity, respectively. Fourteen of the most commonly involved domains were identified from the frameworks.

CONCLUSIONS

Nontechnical competency frameworks applied to multidisciplinary emergency health professionals are heterogeneous in construct and application. A fundamental framework with standardized terminology for the articulation of competency should be developed and validated so as to be accepted and adapted universally by health professionals in all-hazard emergency environment.

Optimizing assessors' mental workload in rater-based assessment: a critical narrative review

INTRODUCTION

Rater-based assessment has resulted in high cognitive demands for assessors within the education of health professionals. Rating quality may be influenced by the mental workload required of assessors to complete rating tasks. The objective of this review was to explore interventions or strategies aimed at measuring and reducing mental workload for improvement in assessment outcomes in health professions education.

METHODS

A critical narrative review was conducted for English-language articles using the databases PubMed, EMBASE, and Google Scholar from conception until November 2018. To be included, articles were eligible if they reported results of interventions aimed at measuring or reducing mental workload in rater-based assessment.

RESULTS

A total of six articles were included in the review. All studies were conducted in simulation settings (OSCEs or videotaped interactions). Of the four studies that measured mental workload, none found any reduction in mental workload as demonstrated by objective secondary task performance after interventions of assessor training or reductions in competency dimension assessment. Reductions in competency dimensions, however, did result in improvements in assessment quality across three studies.

DISCUSSION

The concept of mental workload in assessment in medical education needs further exploration, including investigation into valid measures of assessors' mental workload. It appears that adjusting raters' focus may be a valid strategy to improve assessment outcomes. Future research should be designed to inform how to best reduce load in assessments to improve quality, while balancing the type and quantity of data needed for judgments.

Development and Empirical Testing of a Novel Team Leadership Assessment Measure: A Pilot Study Using Simulated and Live Patient Encounters

BACKGROUND

Team leadership is critical to health care resuscitation team performance. There has been increased focus on competency in team leadership behaviors; however, there is still variability in how team leadership is assessed within emergency medicine. The objective of this study was to develop and pilot a novel team leadership assessment measure for emergency medicine resuscitation teams.

METHODS

Team leadership dimensions and associated behaviors were identified through a systematic literature review and expert consensus. Included behaviors were used to create behaviorally anchored rating scales, which were then revised based on subject matter expert ratings. Four raters from three different academic institutions observed 30 video-recorded resuscitations (20 simulated and 10 actual patient care resuscitations). Mean leadership scores were calculated. Intraclass coefficients (ICCs) were calculated for each item and for overall leadership scores. Leader scores for the simulation-based scenarios were compared to external variables including level of training, team process, clinical performance, and team situational awareness. The study was conducted from July 2017 through June 2018.

RESULTS

Leadership scores ranged from 2.23 to 4.30 (mean [±SD] = 3.18 [±0.50]). The ICC for the overall score was 0.79 for all observations, 0.87 for simulation-based observations, and 0.24 for the patient care observations. Team leadership scores on simulation-based observations did not correlate with available external variables.

CONCLUSIONS

We developed a novel team leadership assessment measure for emergency medicine resuscitation teams with supporting validity evidence, including content validity and response process. The measure demonstrated acceptable inter-rater reliability when applied to simulation-based medical resuscitations; however, this did not translate to trauma resuscitations in the actual patient care setting.

Use of a Portable Functional Near-Infrared Spectroscopy (fNIRS) System to Examine Team Experience During Crisis Event Management in Clinical Simulations

Objective: The aim of this study was to investigate the utilization of a portable functional near-infrared spectroscopy (fNIRS) system, the fNIRS Pioneer^TM, to examine team experience in high-fidelity simulation-based crisis event management (CEM) training for anesthesiologists in operating rooms.

Background: Effective evaluation of team performance and experience in CEM simulations is essential for healthcare training and research. Neurophysiological measures with wearable devices can provide useful indicators of team experience to compliment traditional self-report, observer ratings, and behavioral performance measures. fNIRS measured brain blood oxygenation levels and neural synchrony can be used as indicators of workload and team engagement, which is vital for optimal team performance.

Methods: Thirty-three anesthesiologists, who were attending CEM training in two-person teams, participated in this study. The participants varied in their expertise level and the simulation scenarios varied in difficulty level. The oxygenated and de-oxygenated hemoglobin (HbO and HbR) levels in the participants’ prefrontal cortex were derived from data recorded by a portable one-channel fNIRS system worn by all participants throughout CEM training. Team neural synchrony was measured by HbO/HbR wavelet transformation coherence (WTC). Observer-rated workload and self-reported workload and mood were also collected.

Results: At the individual level, the pattern of HbR level corresponded to changes of workload for the individuals in different roles during different phases of a scenario; but this was not the case for HbO level. Thus, HbR level may be a better indicator for individual workload in the studied setting. However, HbR level was insensitive to differences in scenario difficulty and did not correlate with observer-rated or self-reported workload. At the team level, high levels of HbO and HbR WTC were observed during active teamwork. Furthermore, HbO WTC was sensitive to levels of scenario difficulty.

Conclusion: This study showed that it was feasible to use a portable fNIRS system to study workload and team engagement in high-fidelity clinical simulations. However, more work is needed to establish the sensitivity, reliability, and validity of fNIRS measures as indicators of team experience.

Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

BACKGROUND

Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

METHODS

Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

RESULTS

The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

CONCLUSIONS

Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

Optimizing education in difficult airway management: meeting the challenge

PURPOSE OF REVIEW

The last 2 decades have seen a vast change in the science and technology of airway management. As a result, there is an increasing need to equip anesthesiologists with the new knowledge and skills for the safe management of a difficult airway.

RECENT FINDINGS

In addition to knowledge and expertise, human factors and nontechnical skills (NTS), including situational awareness, communication and team work, play an important role during difficult airway management and contribute to the outcome. Didactic sessions are useful to impart knowledge. Self-learning, interactive discussions, simulation and debriefing are important tools for teaching and training in difficult airway management. Manikin training and simulation enable development of technical as well as NTS without subjecting patients to risk and allow multiple training sessions of relatively uncommon scenarios. Guidelines are useful teaching tools, whereas cognitive tools such as the Vortex approach may be useful during a difficult airway.

SUMMARY

There is need for research on difficult airway management and optimized training methods. Research is also required to determine the barriers to adoption of guidelines and strategies to ensure widespread dissemination and implementation of guidelines and best practices for difficult airway management.

A new way to look at simulation-based assessment: the relationship between gaze-tracking and exam performance

Objective

A key task of the team leader in a medical emergency is effective information gathering. Studying information gathering patterns is readily accomplished with the use of gaze-tracking glasses. This technology was used to generate hypotheses about the relationship between performance scores and expert-hypothesized visual areas of interest in residents across scenarios in simulated medical resuscitation examinations.

Methods

Emergency medicine residents wore gaze-tracking glasses during two simulation-based examinations (n=29 and 13 respectively). Blinded experts assessed video-recorded performances using a simulation performance assessment tool that has validity evidence in this context. The relationships between gaze patterns and performance scores were analyzed and potential hypotheses generated. Four scenarios were assessed in this study: diabetic ketoacidosis, bradycardia secondary to beta-blocker overdose, ruptured abdominal aortic aneurysm and metabolic acidosis caused by antifreeze ingestion.

Results

Specific gaze patterns were correlated with objective performance. High performers were more likely to fixate on task-relevant stimuli and appropriately ignore task-irrelevant stimuli compared with lower performers. For example, shorter latency to fixation on the vital signs in a case of diabetic ketoacidosis was positively correlated with performance (r=0.70, p<0.05). Conversely, total time spent fixating on lab values in a case of ruptured abdominal aortic aneurysm was negatively correlated with performance (r= −0.50, p<0.05).

Conclusions

There are differences between the visual patterns of high and low-performing residents. These findings may allow for better characterization of expertise development in resuscitation medicine and provide a framework for future study of visual behaviours in resuscitation cases.

Crisis Resource Management in the Delivery Room: Development of Behavioral Markers for Team Performance in Emergency Simulation

Human factors are the most relevant issues contributing to adverse events in obstetrics. Specific training of Crisis Resource Management (CRM) skills (i.e., problem solving and team management, resource allocation, awareness of environment, and dynamic decision-making) is now widespread and is often based on High Fidelity Simulation. In order to be used as a guideline in simulated scenarios, CRM skills need to be mapped to specific and observable behavioral markers. For this purpose, we developed a set of observable behaviors related to the main elements of CRM in the delivery room. The observational tool was then adopted in a two-days seminar on obstetric hemorrhage where teams working in obstetric wards of six Italian hospitals took part in simulations. The tool was used as a guide for the debriefing and as a peer-to-peer feedback. It was then rated for its usefulness in facilitating the reflection upon one’s own behavior, its ease of use, and its usefulness for the peer-to-peer feedback. The ratings were positive, with a median of 4 on a 5-point scale. The CRM observational tool has therefore been well-received and presents a promising level of inter-rater agreement. We believe the tool could have value in facilitating debriefing and in the peer-to-peer feedback.

Simulation-based Assessment of the Management of Critical Events by Board-certified Anesthesiologists

Background

We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods.

Methods

A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant’s technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist.

Results

Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance.

Conclusions

Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.