Team performance in resuscitation teams: comparison and critique of two recently developed scoring tools

Assessing Team Performance: A Mixed-Methods Analysis Using Interprofessional in situ Simulation

Introduction

Optimizing the performance of emergency department (ED) teams impacts patient care, but the utility of current, team-based performance assessment tools to comprehensively measure this impact is underexplored. In this study we aimed to 1) evaluate ED team performance using current team-based assessment tools during an interprofessional in situ simulation and 2) identify characteristics of effective ED teams.

Methods

This mixed-methods study employed case study methodology based on a constructivist paradigm. Sixty-three eligible nurses, technicians, pharmacists, and postgraduate year 2-4 emergency medicine residents at a tertiary academic ED participated in a 10-minute in situ simulation of a critically ill patient. Participants self-rated performance using the Team Performance Observation Tool (TPOT) 2.0 and completed a brief demographic form. Two raters independently reviewed simulation videos and rated performance using the TPOT 2.0, Team Emergency Assessment Measure (TEAM), and Ottawa Crisis Resource Management Global Rating Scale (Ottawa GRS). Following simulations, we conducted semi-structured interviews and focus groups with in situ participants. Transcripts were analyzed using thematic analysis.

Results

Eighteen team-based simulations took place between January-April 2021. Raters' scores were on the upper end of the tools for the TPOT 2.0 (R1 4.90, SD 0.17; R2 4.53, SD 0.27, IRR [inter-rater reliability] 0.47), TEAM (R1 3.89, SD 0.19; R2 3.58, SD 0.39, IRR 0.73), and Ottawa GRS (R1 6.6, SD 0.56; R2 6.2, SD 0.54, IRR 0.68). We identified six themes from our interview data: team member entrustment; interdependent energy; leadership tone; optimal communication; strategic staffing; and simulation empowering team performance.

Conclusion

Current team performance assessment tools insufficiently discriminate among high performing teams in the ED. Emergency department-specific assessments that capture features of entrustability, interdependent energy, and leadership tone may offer a more comprehensive way to assess an individual's contribution to a team's performance.

Relationship Between Resuscitation Team Members' Self-Efficacy and Team Competence During In-Hospital Cardiac Arrest

OBJECTIVES

Inadequate self-efficacy of resuscitation team members may impair team performance, but high self-efficacy does not guarantee competence. We evaluated the relationship between individual self-efficacy and resuscitation team competence.

DESIGN

Secondary analysis of a randomized controlled trial.

SETTING

High-fidelity in situ in-hospital cardiac arrest simulations at seven hospitals in Utah.

SUBJECTS

Multidisciplinary cardiac arrest resuscitation team members.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

Resuscitation team members completed surveys evaluating resuscitation self-efficacy (confidence in resuscitation role, difficulty thinking clearly, and concerns about committing errors) after each simulation. The primary outcome was event-level chest compression hands-on fraction greater than 75%. Secondary outcomes included other measures of resuscitation quality, advanced cardiac life support protocol adherence, and nontechnical team performance. Analyses employed the Datta-Satten rank-sum method to account for response clustering within simulation events. Of 923 participants in 76 analyzable simulations, 612 (66%) submitted complete surveys and 33 (43%) resuscitation teams achieved hands-on fraction greater than 75%. Event-level chest compression hands-on fraction greater than 75% versus less than or equal to 75% was not associated with the percentage of resuscitation team members reporting confidence in their team role (n = 213 [74%] vs. n = 251 [77%], respectively, p = 0.18), lack of difficulty thinking clearly (n = 186 [65%] vs. n = 214 [66%], p = 0.92), or lack of worry about making errors (n = 155 [54%] vs. n = 180 [55%], p = 0.41). Team members' confidence was also not associated with secondary outcomes, except that teams with confident members had better values for composite (3.55 [interquartile range, IQR 3.00-3.82] vs. 3.18 [IQR 2.57-3.64], p = 0.024) and global (8 [7-9] vs. 8 [6-8], p = 0.029) scales measuring nontechnical team performance.

CONCLUSIONS

Team members' self-efficacy was not associated with most team-level competence metrics during simulated cardiac arrest resuscitation. These data suggest that self-efficacy should have a limited role for evaluation of resuscitation training programs and for initial certification and monitoring of individual resuscitation team members' competence.

Application of the Team Emergency Assessment Measure Scale in undergraduate medical students and interprofessional clinical teams: validity evidence of a Spanish version applied in Chile

Background

Teamwork is one of the competencies necessary for physicians to work effectively in health systems and is a competency that can be developed with simulation in professionals and medicine students. The Team Emergency Assessment Measurement (TEAM) was created to evaluate the non-technical performance of team members during resuscitation events in real teams. The TEAM scale includes items to assess leadership, teamwork, situational awareness, and task management. An objective evaluation tool in Spanish is valuable for training health professionals at all undergraduate and continuing education levels. This study aimed to generate evidence of the validity of the Team Emergency Assessment Measure (TEAM) in Spanish to measure the performance of medical students and adult, pediatric, and obstetric emergency clinical teams in simulated emergencies as a self-assessment tool.

Methods

To develop the Spanish version of the instrument, a forward and backward translation process was followed by independent translators, native and fluent in English and Spanish, and a review by a panel of Chilean experts comprising three trained simulation instructors to verify semantics and cultural equivalence. High-fidelity simulations with debriefing were conducted with 5th-year medical students, in which students and instructors applied the Spanish version of the TEAM scale. In the second stage, adult, pediatric, and obstetric emergency management simulations were conducted using the TEAM scale for real clinical teams as a self-assessment tool.

Findings

By applying the overall TEAM scale to medicine students and clinical teams, Cronbach's alpha was 0.921. For medical students' self-assessment, we obtained Cronbach's alpha of 0.869. No significant differences were found between the overall scores and the scores by dimensions evaluated by instructors and students (p > 0.05). In the case of clinical team training, Cronbach's alpha was 0.755 for adult emergency teams, 0.797 for pediatric emergency teams, and 0.853 for obstetric emergency teams.

Conclusion

The validated instrument is adequate for evaluating teamwork in medical student simulations by instructors and peers and for self-assessment in adult, pediatric, and obstetric emergency clinical teams.

Review article: Use of the Team Emergency Assessment Measure in the rating of emergency teams' non-technical skills: A mapping review

The present study aims to explore the utility of the Team Emergency Assessment Measure (TEAM) in relation to the enhancement of emergency team non-technical skills based on research conducted over the last decade. In this mapping review, a citation mining process identified 22 primary studies for inclusion, published between 2012 and 2022. It provides outcome data on emergency teams' non-technical skills following team training and/or real-life patient emergencies. Emergency team studies related to resuscitation teams (adult, paediatric, newborn and obstetric cases) and medical emergency team (MET) management of patient deterioration. Team performance ratings varied, ranging from approximately 90% for experienced clinical teams down to 38% for students. Statistically significant improvements in performance were notable following training and/or repeated practice. Validity evidence, across 11 studies that provided change data described positive learning outcomes and moderate intervention effects. However, according to Kirkpatrick's model of educational evaluation the studies were limited to professional development phases of learning and immediate post-training assessments rather than care quality improvement. The review highlights a lack of studies evidencing quality improvement or clinical impact such as change of patient care practice or health service performance. There is a need to conduct well-designed studies that explore both technical and non-technical skills of resuscitation teams and METs. Currently, non-technical skills training and repeated performance evaluations using the TEAM contribute immensely to the proficiency of emergency teams.

Team effectiveness: epidemiologists' perception of collective performance during emergency response

BACKGROUND

To describe epidemiologists' experience of team dynamics and leadership during emergency response, and explore the utility of the Team Emergency Assessment Measure (TEAM) tool during future public health emergency responses. The TEAM tool included categories for leadership, teamwork, and task management.

METHODS

We conducted a cross-sectional survey between October 2019 and February 2020 with the global applied field epidemiology workforce. To validate the TEAM tool for our context, we used exploratory and confirmatory factor analysis.

RESULTS

We analysed 166 completed surveys. Respondents included national and international emergency responders with representation of all WHO regions. We were unable to validate the TEAM tool for use with epidemiology teams involved in emergency response, however descriptive analysis provided insight into epidemiology emergency response team performance. We found female responders were less satisfied with response leadership than male counterparts, and national responders were more satisfied across all survey categories compared to international responders.

CONCLUSION

Functional teams are a core attribute of effective public health emergency response. Our findings have shown a need for a greater focus on team performance. We recommend development of a fit-for-purpose performance management tool for teams responding to public health emergencies. The importance of building and supporting the development of the national workforce is another important finding of this study.

Development and psychometric properties of teamwork scale in resuscitation: A sequential exploratory mixed-method study

BACKGROUND

Non-technical skills as a complement to technical skills have become increasingly important in cardio-pulmonary resuscitation, and teamwork is one of the most prominent of these skills. Because of the limited number of tools introduced and validated for measuring teamwork in resuscitation, this study was conducted to develop and evaluate the psychometric properties of the Teamwork Scale in Resuscitation (TSR).

MATERIALS AND METHODS

This study was a sequential exploratory mixed-method study and was conducted between March 2020 and April 2021 and in two phases. In the first phase (qualitative), semi-structured interviews were performed with resuscitation team members who were selected using a purposeful sampling method with maximum variation in terms of experience, level of education, age, and gender. Using directed qualitative content analysis, the data were analyzed and the primary item pool was developed. In the second phase (quantitative), assessing face, content, and construct validity as well as reliability via the internal consistency and stability, the psychometric properties of the instrument are evaluated.

RESULTS

The initial tool consisted of 54 items; after assessing the face validity, ten items were removed during content validity, and the final tool consisted of 44 items and three main categories including the essential prerequisites for resuscitation, leadership, and teamwork, which was scored based on a 5-point Likert scale from strongly disagree to strongly agree. The convergent validity and concurrent validity of the tool were confirmed. The reliability of the instrument was also confirmed using Cronbach's alpha test and re-test and the intra-class correlation coefficient (ICC) between raters.

CONCLUSION

The validity and reliability of TSR for measuring teamwork in resuscitation in a simulated environment are approved. Further studies are suggested for different contexts of resuscitation and with trained raters to evaluate its use in real resuscitation situations.

Challenges for Optimum Cardiopulmonary Resuscitation in the Emergency Departments of Limpopo Province: A Qualitative Study

AIM

To describe the challenges for optimum resuscitation processes in Emergency Departments in Limpopo Province, South Africa.

DESIGN

A qualitative explorative research approach was adopted to explore the resuscitation team's experiences in Emergency Departments.

METHOD

Five medical doctors and twelve professional nurses were purposively sampled to participate in the study. The depth of the information obtained from the participants determined the sample size. Data collected from semi-structured individual interviews were analyzed using thematic analysis. Data quality was ensured by applying four elements: credibility, transferability, dependability, and confirmability.

RESULTS

The study findings indicated diverse challenges for optimum resuscitation processes that include: A general shortage of emergency personnel, the lack of material resources and the unavailability of funds for payment of national and international trauma symposiums, the poor maintenance of emergency equipment, the lack of a continuous training program and the resuscitation team receiving different instructions from various team leaders about the standardized procedures and policies of the resuscitation process. The team leaders and managers often blamed, depreciated and disregarded the resuscitation team for failed resuscitation efforts. Public contribution: The study findings are a point of reference for the emergency resuscitation team and the department of health policymakers. Trained and well-equipped emergency resuscitation teams can improve the quality of life for patients with cardiac arrest.

Influence of airway trolley organization on efficiency and team performance: A randomized, crossover simulation study

BACKGROUND

Failed management of unanticipated difficult airway situations contributes to significant anesthesia-related morbidity and mortality. Optimization of design and layout of difficult airway trolleys (DATs) may influence outcomes during airway emergencies. The main objective of the current study was to evaluate whether a difficult airway algorithm-based DAT with integrated cognitive aids improves efficiency and team performance in difficult airway scenarios.

METHODS

In a crossover design, 16 teams (anesthetist, nurse anesthetist, assistant nurse) completed two high-fidelity simulated unanticipated difficult airway scenarios. Teams used both an algorithm-based DAT and a comparison, standard DAT, in the scenarios and were randomized to order of trolley type. Outcome measures included objective efficiency parameters, team performance assessment and subjective user-ratings. Linear mixed models ANOVA, including DAT type and order of condition as main factors, was utilized for the primary analyses of the team results.

RESULTS

Usage of the algorithm-based DAT was associated with fewer departures from the difficult airway algorithm (p = .010), and reduced number of unnecessary drawer openings (p = .002), but no significant differences in time to retrieval of airway devices or time to first effective ventilation, compared to the standard DAT. There were no significant differences in team performance, although participants expressed strong preference for the algorithm-based DAT (all user-rated measures p < .0001). Higher percentage of female members of the team improved adherence to the difficult airway algorithm (p = .043).

CONCLUSIONS

Algorithm-based DATs with integrated cognitive aids may improve efficiency in difficult airway situations, compared to traditional DATs. These findings have implications for improvement of anesthetic practice.

Development and use of an infant resuscitation performance tool (Infa-RePT) to improve team performance

BACKGROUND

Measurement of neonatal team resuscitation performance is critical to identify opportunities for improvement and to target education. An effective tool to measure team performance during infant resuscitations is lacking.

METHODS

We developed an in-hospital infant resuscitation performance tool (Infa-RePT) using the modified Delphi method. We employed a QI framework and targeted interventions, including the use of role responsibility checklists, mock codes, and an educational video. We tracked Infa-RePT scores, mock code team attendance, and confidence surveys. Our specific aim was to improve Infa-RePT score from a baseline of 7.4 to <5 (lower is better) over a 26-month period.

RESULTS

Twenty-five elements reached >80% consensus as essential components to include on the Infa-RePT. Independent observation showed 86% concordance on checklist items. Simulation (n = 26) and unit-based code (n = 10) Infa-RePT scores showed significant improvement after project start from 7.4 to 4.2 (p < 0.01) with special cause variation noted on control chart analysis. No significant difference was observed between simulations and in-unit codes. Staff confidence self-reports improved over the study period.

CONCLUSIONS

Use of a novel scoring tool can help monitor team progress over time and identify areas for improvement. Focused interventions can improve resuscitation team performance.

IMPACT

We developed and used a novel, comprehensive measurement tool for team infant resuscitation performance in both simulation and in-unit settings. Using QI methodology, team performance improved after the enhancement of a mock code simulation program. Review of team performance scores can highlight key areas to target interventions and monitor progress over time.

Preparing and Presenting Validation Studies: A Guide for the Perplexed

SUMMARY STATEMENT

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

Resuscitation Team Members 'Experiences of Teamwork: A Qualitative Study

BACKGROUND

Despite the obvious importance of teamwork in emergency care, the dimensions of teamwork in resuscitation remain a subject of debate among specialists and researchers. The aim of this study was to identify the dimensions of teamwork based on the experiences of members of the resuscitation team.

MATERIALS AND METHODS

This study was conducted between March 2020 and April 2021. By purposive sampling, participants were selected. Sixteen semi-structured interviews were conducted with experienced emergency nurses and emergency medicine residents individually and analysed through deductive content analysis by using the Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) model. Interviews were encoded for analysis using MAXQDA software (version 2020).

RESULTS

Three main categories were identified around teamwork in resuscitation. These categories were as follows (1) leadership, (2) teamwork, and (3) essential prerequisites for resuscitation. The main leadership category consisted of three sub-categories of time management, resource allocation, and task management. From sub-categories, situation monitoring, communication, and mutual support, the main category of teamwork emerged. Also, the emerging category of essential prerequisites for regeneration included two generic categories: professional requirements and workplace requirements and the teamwork category included situation monitoring, communication, and mutual support.

CONCLUSIONS

Based on the experiences of the resuscitation team members, leadership, teamwork, and essential prerequisites for resuscitation are the most important dimensions of teamwork in resuscitation. Recognizing the dimensions of teamwork in resuscitation is an initial step and then should be reflected in educational programs and future guidelines.

Telemedical Intensivist Consultation During In-Hospital Cardiac Arrest Resuscitation: A Simulation-Based, Randomized Controlled Trial

BACKGROUND

High-quality leadership improves resuscitation for in-hospital cardiac arrest (IHCA), but experienced resuscitation leaders are unavailable in many settings.

RESEARCH QUESTION

Does real-time telemedical intensivist consultation improve resuscitation quality for IHCA?

STUDY DESIGN AND METHODS

In this multicenter randomized controlled trial, standardized high-fidelity simulations of IHCA conducted between February 2017 and September 2018 on inpatient medicine and surgery units at seven hospitals were assigned randomly to consultation (intervention) or simulated observation (control) by a critical care physician via telemedicine. The primary outcome was the fraction of time without chest compressions (ie, no-flow fraction) during an approximately 4- to 6-min analysis window beginning with telemedicine activation. Secondary outcomes included other measures of chest compression quality, defibrillation and medication timing, resuscitation protocol adherence, nontechnical team performance, and participants' experience during resuscitation participation.

RESULTS

No-flow fraction did not differ between the 36 intervention group (0.22 ± 0.13) and the 35 control group (0.19 ± 0.10) resuscitation simulations included in the intention-to-treat analysis (P = .41). The etiology of the simulated cardiac arrest was identified more often during evaluable resuscitations supported by a telemedical intensivist consultant (22/32 [69%]) compared with control resuscitations (10/34 [29%]; P = .001), but other measures of resuscitation quality, resuscitation team performance, and participant experience did not differ between intervention groups. Problems with audio quality or the telemedicine connection affected 14 intervention group resuscitations (39%).

INTERPRETATION

Consultation by a telemedical intensivist physician did not improve resuscitation quality during simulated ward-based IHCA.

TRIAL REGISTRY

ClinicalTrials.gov; No.: NCT03000829; URL: www.

CLINICALTRIALS

gov.

Measuring non-technical skills during prehospital advanced cardiac life support: A pilot study

Aim

To analyse non-technical skills of mobile medical teams during out-of-hospital cardiac arrests (OHCA) using the validated Team Emergency Assessment Measure (TEAM) tool. To research the correlation between non-technical skills and patient outcome.

Methods

Adult patients who experienced an out-of-hospital cardiac arrest between July 2016, and June 2018, and were treated by a mobile medical team from the University Hospital Leuven, were eligible for the study. Resuscitations were video recorded from the team leader’s perspective. Video recordings were reviewed and scored by emergency physicians, using the TEAM evaluation form.

Results

In total 114 OHCAs were analysed. The mean TEAM score was 34.4/44 (SD = 5.5). The mean item score was 3.1/4 (SD = 0.8). On average, ‘effective team communication’ had the lowest score (2.4), while ‘acting with composure and control’ and ‘following of approved standards/guidelines’ scored the highest (3.4). The average non-technical skills theme scores were 2.9 (SD = 0.9) for ‘Leadership’, 3.1 (SD = 0.8) for ‘Teamwork’ and 3.3 (SD = 0.7) for ‘Task management’. ‘Leadership’ was rated significantly lower than ‘Teamwork’ (p = 0.004) and ‘Task management’ (p < 0.001). No significant correlation was found between TEAM and return of spontaneous circulation (p = 0.574) or one month survival (p = 0.225).

Conclusion

The mean overall TEAM score was categorized as good. Task management scored high, while leadership and team communication received lower scores. Future training programs should thus focus on improving leadership and communication. In this pilot study no correlation was found between non-technical skills and survival.

Factors Influencing Team and Task Performance in Intensive Care Teams in a Simulated Scenario

INTRODUCTION

Healthcare teams and their performance in a complex environment such as that of intensive care units (ICUs) are influenced by several factors. This study investigates the relationship between team background characteristics and team as well as task performance.

METHODS

This study included 105 professionals (26 teams), working at the ICUs of 2 hospitals in Northern Sweden. The team-based simulation training sessions were video recorded, and thereafter, team performance and task performance were analyzed based on ratings of the TEAM instrument and the ABCDE checklist.

RESULTS

The final analyses showed that a higher age was significantly associated with better total team performance (β = 0.35, P = 0.04), teamwork (β = 0.04, P = 0.04), and task management (β = 0.04, P = 0.05) and with a higher overall rating for global team performance (β = 0.09, P = 0.02). The same pattern was found for the association between age and task performance (β = 0.02, P = 0.04). In addition, prior team training without video-facilitated reflection was significantly associated with better task performance (β = 0.35, P = 0.04). On the other hand, prior team training in communication was significantly associated with worse (β = -1.30, P = 0.02) leadership performance.

CONCLUSIONS

This study reveals that a higher age is important for better team performance when caring for a severely ill patient in a simulation setting in the ICU. In addition, prior team training had a positive impact on task performance. Therefore, on a team level, this study indicates that age and, to some extent, prior team training without video-facilitated reflection have an impact on team performance in the care of critically ill patients.

Emergencies within hospital wards: An observational study of the non-technical skills of medical emergency teams

BACKGROUND

Medical emergency teams are essential in responding to acute deterioration of patients in hospitals, requiring both clinical and non-technical skills. This study aims to assess the non-technical skills of medical emergency teams during hospital ward emergencies and explore team members perceptions and experiences of the use non-technical skills during medical emergencies.

METHODS

A multi-methods study was conducted in two phases. During phase one observation and assessment of non-technical skills used in medical emergencies using the Team Emergency Assessment Measure (TEAM™) was conducted; and in the phase two in-depth interviews were undertaken with medical emergency team members.

RESULTS

Based on 20 observations, mean TEAM™ ratings for non-technical skill domains were: 'leadership' 5.0 out of 8 (±2.0); 'teamwork' 21.6 out of 28 (±3.6); and 'task management' 6.5 out of 8 (±1.4). The mean 'global' score was 7.5 out of 10 (±1.5). The qualitative findings identified three areas, 'individual', 'team' and 'other' contributing factors, which impacted upon the non-technical skills of medical emergency teams.

CONCLUSION

Non-technical skills of hospital medical emergency teams differ, and the impact of the skill mix on resuscitation outcomes was recognised by team members. These findings emphasize the importance non-technical skills in resuscitation training and well-developed processes for medical emergency teams.

How Prepared Are Pediatric Residents for Pediatric Emergencies: Is Pediatric Advanced Life Support Certification Every 2 Years Adequate?

Objectives. We assessed pediatric residents’ retention of knowledge and clinical skills according to the time since their last American Heart Association Pediatric Advanced Life Support (AHA PALS) certification. Methods. Sixty-four pediatric residents were recruited and divided into 3 groups based on the time since their last PALS certification, as follows: group 1, 0 to 8 months; group 2, 9 to 16 months, and group 3, 17 to 24 months. Residents’ knowledge was tested using 10 multiple-choice AHA PALS pretest questions and their clinical skills performance was assessed with simulation mock code scenarios using 2 different AHA PALS checklists, and mean scores were calculated for the 3 groups. Differences in the test scores and overall clinical skill performances among the 3 groups were analyzed using analyses of variance, χ² tests, and Jonckheere-Terpstra tests. Statistical significance was set at P < .05. Results. The pediatric residents’ mean overall clinical skills performance scores declined within the first 8 months after their last AHA PALS certification date and continued to decrease over time (87%, 82.6%, and 77.4% for groups 1, 2, and 3, respectively; P = .048). Residents’ multiple-choice test scores declined in all 3 groups, but the scores were not significantly different. Conclusions. Residents’ clinical skills performance declined within the first 8 months after PALS certification and continued to decline as the time from the last certification increased. Using mock code simulations and reinforcing AHA PALS guidelines during pediatric residency deserve further evaluation.

Reliability of instruments that measure situation awareness, team performance and task performance in a simulation setting with medical students

OBJECTIVES

The assessment of situation awareness (SA), team performance and task performance in a simulation training session requires reliable and feasible measurement techniques. The objectives of this study were to test the Airways-Breathing-Circulation-Disability-Exposure (ABCDE) checklist and the Team Emergency Assessment Measure (TEAM) for inter-rater reliability, as well as the application of Situation Awareness Global Assessment Technique (SAGAT) for feasibility and internal consistency.

DESIGN

Methodological approach.

SETTING

Data collection during team training using full-scale simulation at a university clinical training centre. The video-recorded scenarios were rated independently by four raters.

PARTICIPANTS

55 medical students aged 22-40 years in their fourth year of medical studies, during the clerkship in anaesthesiology and critical care medicine, formed 23 different teams. All students answered the SAGAT questionnaires, and of these students, 24 answered the follow-up postsimulation questionnaire (PSQ). TEAM and ABCDE were scored by four professionals.

MEASURES

The ABCDE and TEAM were tested for inter-rater reliability. The feasibility of SAGAT was tested using PSQ. SAGAT was tested for internal consistency both at an individual level (SAGAT) and a team level (Team Situation Awareness Global Assessment Technique (TSAGAT)).

RESULTS

The intraclass correlation was 0.54/0.83 (single/average measurements) for TEAM and 0.55/0.83 for ABCDE. According to the PSQ, the items in SAGAT were rated as relevant to the scenario by 96% of the participants. Cronbach's alpha for SAGAT/TSAGAT for the two scenarios was 0.80/0.83 vs 0.62/0.76, and normed χ² was 1.72 vs 1.62.

CONCLUSION

Task performance, team performance and SA could be purposefully measured, and the reliability of the measurements was good.

Rating the quality of teamwork-a comparison of novice and expert ratings using the Team Emergency Assessment Measure (TEAM) in simulated emergencies

Background

Training in teamwork behaviour improves technical resuscitation performance. However, its effect on patient outcome is less clear, partly because teamwork behaviour is difficult to measure. Furthermore, it is unknown who should evaluate it. In clinical practice, experts are obliged to participate in resuscitation efforts and are thus unavailable to assess teamwork quality. Consequently, we sought to determine if raters with little clinical experience and experts provide comparable evaluations of teamwork behaviour.

Methods

Novice and expert raters judged teamwork behaviour during 6 emergency medicine simulations using the Teamwork Emergency Assessment Measure (TEAM). Ratings of both groups were analysed descriptively and compared with U and t tests. We used a mixed effects model to identify the proportion of variance in TEAM scores attributable to rater status and other sources.

Results

Twelve raters evaluated 7 teams rotating through 6 cases, for a total of 84 observations. We found no significant difference between expert and novice ratings for 7 of the 11 items of the TEAM or in the sums of all item scores. Novices rated teamwork behaviour higher on 4 items and overall. Rater status accounted for 11.1% of the total variance in scores.

Conclusions

Experts’ and novices’ ratings were similarly distributed, implying that raters with limited experience can provide reliable data on teamwork behaviour. Novices show a consistent, but slightly more lenient rating behaviour. Clinical studies and real-life teams may thus employ novices using a structured observational tool such as TEAM to inform their performance review and improvement.

Electronic supplementary material

The online version of this article (10.1186/s13049-019-0591-9) contains supplementary material, which is available to authorized users.

Interprofessional learning in immediate life support training does effect TEAM performance during simulated resuscitation

Aim of the study

To assess performance in a simulated resuscitation after participating in either an interprofessional learning (IPL) or uniprofessional learning (UPL) immediate life support (ILS) training course.

Introduction

The Team Emergency Assessment Measure (TEAM) is routinely used in Resuscitation Council (UK) Advanced Life Support courses. This study used the psychometrically validated tool to assess if the delivery of an IPL ILS to final year medical and nursing students could improve overall behavioural performance and global TEAM score.

Methods

A randomised study of medical (n=48) and nursing (n=48) students, assessing performance in a simulated resuscitation following the IPL or UPL ILS courses. Postcourse completion participants were invited back to undertake a video-recorded simulated-resuscitation scenario. Each of these were reviewed using the TEAM tool, at the time by an experienced advanced life support instructor and subsequently by a clinician, independent to the study and blinded as to which cohort they were reviewing.

Results

Inter-rater reliability was tested using a Bland-Altman plot indicating non-proportional bias between raters. Parametric testing and analysis showed statistically significant higher global overall mean TEAM scores for those who had attended the IPL ILS courses.

Conclusion

Our results demonstrate that an IPL approach in ILS produced an increased effect on TEAM scores with raters recording a significantly more collaborative team performance. A postscenario questionnaire for students also found a significantly improved experience within the team following the IPL course compared with those completing UPL training. Although this study shows that team behaviour and performance can change and improve in the short-term, we acknowledge further studies are required to assess the long-term effects of IPL interventions. Additionally, through this type of study methodology, other outcomes in regard to resuscitation team performance may be measured, highlighting other potential benefit to patients, at level four of Kirkpatrick’s hierarchy.

Measuring the teamwork performance of teams in crisis situations: a systematic review of assessment tools and their measurement properties

BACKGROUND

Educational interventions to improve teamwork in crisis situations have proliferated in recent years with substantial variation in teamwork measurement. This systematic review aimed to synthesise available tools and their measurement properties in order to identify the most robust tool for measuring the teamwork performance of teams in crisis situations.

METHODS

Searches were conducted in Embase (via OVID), PsycINFO, Cumulative Index to Nursing and Allied Health Literature, Education Resources Information Center, Medline and Medline In-Process (via OVID) (through 12 January 2017). Studies evaluating the measurement properties of teamwork assessment tools for teams in clinical or simulated crisis situations were included. Two independent reviewers screened studies based on predetermined criteria and completed data extraction. Risk of bias was assessed using the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) checklist.

RESULTS

The search yielded 1822 references. Twenty studies were included, representing 13 assessment tools. Tools were primarily assessed in simulated resuscitation scenarios for emergency department teams. The Team Emergency Assessment Measure (TEAM) had the most validation studies (n=5), which demonstrated three sources of validity (content, construct and concurrent) and three sources of reliability (internal consistency, inter-rater reliability and test-retest reliability). Most studies of TEAM's measurement properties were at no risk of bias.

CONCLUSIONS

A number of tools are available for assessing teamwork performance of teams in crisis situations. Although selection will ultimately depend on the user's context, TEAM may be the most promising tool given its measurement evidence. Currently, there is a lack of tools to assess teamwork performance during intraoperative crisis situations. Additional research is needed in this regard.

Estimating the Reliability of Nontechnical Skills in Medical Teams

OBJECTIVE

To assess the reliability of measurement of assessments of nontechnical skills in medical teams.

BACKGROUND

There are several scales for assessing nontechnical skills in medical teams. In this critique, we describe the reliability of such assessments and discuss factors that may influence the obtained estimate of reliability for such scales.

METHOD

A review of the literature yielded 28 reliability estimates, mostly intraclass correlations. We analyzed the methods of scale development and reliability estimation for factors that are likely to influence the resulting estimate, including aspects of the judges, the targets, what is observed, and the kind of reliability coefficient.

RESULTS

Typical reliability was approximately 0.7, but estimates were quite varied in magnitude, from quite low (0.3) to quite high (0.9). Thus, highly reliable assessment is possible, but rarely achieved. Factors that are likely to influence the magnitude of reliability include the number of assessors, the participants' range in skill level, and the observational process. Most studies do not present enough information to make the estimates comparable from one study to the next.

CONCLUSIONS

Although nontechnical skills can be measured reliably under some circumstances, it is not clear from the existing literature that one scale provides more reliable data than another. We recommend several practices that would help readers to make informed decisions about which scales are most useful for their purposes.

An extended version of the Rasmussen’s Dynamic Safety Model for measuring multitasking behaviors during medical emergency

An extended version of the Rasmussen’s Dynamic Safety Model was implemented to empirically index team activity during a medical emergency. The video recordings of two emergency training sessions during the simulation of an adverse event with two paramedical teams were analyzed with a coding scheme based on the model. We show that individual’s allocation of perceptions and actions to multiple work constraints (i.e., patient’s state monitoring, information processing with team, and equipment management) can be dynamically tracked.

Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees - a Canadian multi-centred study

BACKGROUND

The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.

METHOD

EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE. Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen's Simulation Assessment Tool (QSAT). Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario. A fully crossed generalizability study was also conducted for each examination centre.

RESULTS

Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.

CONCLUSIONS

This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

Teamwork evaluation during emergency medicine residents’ high-fidelity simulation

Background

Teamwork training has been included in several emergency medicine (EM) curricula; the aim of this study was to compare different scales’ performance in teamwork evaluation during simulation for EM residents.

Methods

In the period October 2013–June 2014, we performed bimonthly high-fidelity simulation sessions, with novice (I–III year, group 1 (G1)) and senior (IV–V year, group 2 (G2)) EM residents; scenarios were designed to simulate management of critical patients. Videos were assessed by three independent raters with the following scales: Emergency Team Dynamics (ETD), Clinical Teamwork Scale (CTS) and Team Emergency Assessment Measure (TEAM). In the period March–June, after each scenario, participants completed the CTS and ETD.

Results

The analysis based on 18 sessions showed good internal consistency and good to fair inter-rater reliability for the three scales (TEAM, CTS, ETD: Cronbach's α 0.954, 0.954, 0.921; Intraclass Correlation Coefficients (ICC), 0.921, 0.917, 0.608). Single CTS items achieved highly significant ICC results, with 12 of the total 13 comparisons achieving ICC results ≥0.70; a similar result was confirmed for 4 of the total 11 TEAM items and 1 of the 8 total ETD items. Spearman's r was 0.585 between ETD and CTS, 0.694 between ETD and TEAM, and 0.634 between TEAM and CTS (scales converted to percentages, all p<0.0001). Participants gave themselves a better evaluation compared with external raters (CTS: 101±9 vs 90±9; ETD: 25±3 vs 20±5, all p<0.0001).

Conclusions

All examined scales demonstrated good internal consistency, with a slightly better inter-rater reliability for CTS compared with the other tools.

Simulation-based multiprofessional obstetric anaesthesia training conducted in situ versus off-site leads to similar individual and team outcomes: a randomised educational trial

OBJECTIVE

To investigate the effect of in situ simulation (ISS) versus off-site simulation (OSS) on knowledge, patient safety attitude, stress, motivation, perceptions of simulation, team performance and organisational impact.

DESIGN

Investigator-initiated single-centre randomised superiority educational trial.

SETTING

Obstetrics and anaesthesiology departments, Rigshospitalet, University of Copenhagen, Denmark.

PARTICIPANTS

100 participants in teams of 10, comprising midwives, specialised midwives, auxiliary nurses, nurse anaesthetists, operating theatre nurses, and consultant doctors and trainees in obstetrics and anaesthesiology.

INTERVENTIONS

Two multiprofessional simulations (clinical management of an emergency caesarean section and a postpartum haemorrhage scenario) were conducted in teams of 10 in the ISS versus the OSS setting.

PRIMARY OUTCOME

Knowledge assessed by a multiple choice question test.

EXPLORATORY OUTCOMES

Individual outcomes: scores on the Safety Attitudes Questionnaire, stress measurements (State-Trait Anxiety Inventory, cognitive appraisal and salivary cortisol), Intrinsic Motivation Inventory and perceptions of simulations. Team outcome: video assessment of team performance. Organisational impact: suggestions for organisational changes.

RESULTS

The trial was conducted from April to June 2013. No differences between the two groups were found for the multiple choice question test, patient safety attitude, stress measurements, motivation or the evaluation of the simulations. The participants in the ISS group scored the authenticity of the simulation significantly higher than did the participants in the OSS group. Expert video assessment of team performance showed no differences between the ISS versus the OSS group. The ISS group provided more ideas and suggestions for changes at the organisational level.

CONCLUSIONS

In this randomised trial, no significant differences were found regarding knowledge, patient safety attitude, motivation or stress measurements when comparing ISS versus OSS. Although participant perception of the authenticity of ISS versus OSS differed significantly, there were no differences in other outcomes between the groups except that the ISS group generated more suggestions for organisational changes.

TRIAL REGISTRATION NUMBER

NCT01792674.

Teamwork education improves trauma team performance in undergraduate health professional students

PURPOSE

Effective trauma resuscitation requires efficient and coordinated care from a team of providers; however, providers are rarely instructed on how to be effective members of trauma teams. Team-based learning using Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) has been shown to improve team dynamics among practicing professionals, including physicians and nurses. The impact of TeamSTEPPS on students being trained in trauma management in an undergraduate health professional program is currently unknown. We sought to determine the impact of TeamSTEPPS on team dynamics among undergraduate students being trained in trauma resuscitation.

METHODS

We enrolled teams of undergraduate health professional students from four programs: nursing, physician assistant, radiologic science, and respiratory care. After completing an online training on trauma resuscitation principles, the participants completed a trauma resuscitation scenario. The participants then received teamwork training using TeamSTEPPS and completed a second trauma resuscitation scenario identical to the first. All resuscitations were recorded and scored offline by two blinded research assistants using both the Team Emergency Assessment Measure (TEAM) and Trauma Team Performance Observation Tool (TPOT) scoring systems. Pre-test and post-test TEAM and TPOT scores were compared.

RESULTS

We enrolled a total of 48 students in 12 teams. Team leadership, situational monitoring, and overall communication improved with TeamSTEPPS training (P=0.04, P=0.02, and P=0.03, respectively), as assessed by the TPOT scoring system. TeamSTEPPS also improved the team's ability to prioritize tasks and work together to complete tasks in a rapid manner (P<0.01 and P=0.02, respectively) as measured by TEAM.

CONCLUSIONS

Incorporating TeamSTEPPS into trauma team education leads to improved TEAM and TPOT scores among undergraduate health professionals.

Beyond crisis resource management: new frontiers in human factors training for acute care medicine

PURPOSE OF REVIEW

Error is ubiquitous in medicine, particularly during critical events and resuscitation. A significant proportion of adverse events can be attributed to inadequate team-based skills such as communication, leadership, situation awareness and resource utilization. Aviation-based crisis resource management (CRM) training using high-fidelity simulation has been proposed as a strategy to improve team behaviours. This review will address key considerations in CRM training and outline recommendations for the future of human factors education in healthcare.

RECENT FINDINGS

A critical examination of the current literature yields several important considerations to guide the development and implementation of effective simulation-based CRM training. These include defining a priori domain-specific objectives, creating an immersive environment that encourages deliberate practice and transfer-appropriate processing, and the importance of effective team debriefing. Building on research from high-risk industry, we suggest that traditional CRM training may be augmented with new training techniques that promote the development of shared mental models for team and task processes, address the effect of acute stress on team performance, and integrate strategies to improve clinical reasoning and the detection of cognitive errors.

SUMMARY

The evolution of CRM training involves a 'Triple Threat' approach that integrates mental model theory for team and task processes, training for stressful situations and metacognition and error theory towards a more comprehensive training paradigm, with roots in high-risk industry and cognitive psychology. Further research is required to evaluate the impact of this approach on patient-oriented outcomes.

Undergraduate nursing students' performance in recognising and responding to sudden patient deterioration in high psychological fidelity simulated environments: an Australian multi-centre study

BACKGROUND

Early recognition and situation awareness of sudden patient deterioration, a timely appropriate clinical response, and teamwork are critical to patient outcomes. High fidelity simulated environments provide the opportunity for undergraduate nursing students to develop and refine recognition and response skills.

OBJECTIVES

This paper reports the quantitative findings of the first phase of a larger program of ongoing research: Feedback Incorporating Review and Simulation Techniques to Act on Clinical Trends (FIRST2ACTTM). It specifically aims to identify the characteristics that may predict primary outcome measures of clinical performance, teamwork and situation awareness in the management of deteriorating patients.

DESIGN

Mixed-method multi-centre study.

SETTING

High fidelity simulated acute clinical environment in three Australian universities.

PARTICIPANTS

A convenience sample of 97 final year nursing students enrolled in an undergraduate Bachelor of Nursing or combined Bachelor of Nursing degree were included in the study.

METHOD

In groups of three, participants proceeded through three phases: (i) pre-briefing and completion of a multi-choice question test, (ii) three video-recorded simulated clinical scenarios where actors substituted real patients with deteriorating conditions, and (iii) post-scenario debriefing. Clinical performance, teamwork and situation awareness were evaluated, using a validated standard checklist (OSCE), Team Emergency Assessment Measure (TEAM) score sheet and Situation Awareness Global Assessment Technique (SAGAT). A Modified Angoff technique was used to establish cut points for clinical performance.

RESULTS

Student teams engaged in 97 simulation experiences across the three scenarios and achieved a level of clinical performance consistent with the experts' identified pass level point in only 9 (1%) of the simulation experiences. Knowledge was significantly associated with overall teamwork (p=.034), overall situation awareness (p=.05) and clinical performance in two of the three scenarios (p=.032 cardiac and p=.006 shock). Situation awareness scores of scenario team leaders were low overall, with an average total score of 41%.

CONCLUSIONS

Final year undergraduate nursing students may have difficulty recognising and responding appropriately to patient deterioration. Improving pre-requisite knowledge, rehearsal of first response and team management strategies need to be a key component of undergraduate nursing students' education and ought to specifically address clinical performance, teamwork and situation awareness.

'In situ simulation' versus 'off site simulation' in obstetric emergencies and their effect on knowledge, safety attitudes, team performance, stress, and motivation: study protocol for a randomized controlled trial

BACKGROUND

Unexpected obstetric emergencies threaten the safety of pregnant women. As emergencies are rare, they are difficult to learn. Therefore, simulation-based medical education (SBME) seems relevant. In non-systematic reviews on SBME, medical simulation has been suggested to be associated with improved learner outcomes. However, many questions on how SBME can be optimized remain unanswered. One unresolved issue is how 'in situ simulation' (ISS) versus 'off site simulation' (OSS) impact learning. ISS means simulation-based training in the actual patient care unit (in other words, the labor room and operating room). OSS means training in facilities away from the actual patient care unit, either at a simulation centre or in hospital rooms that have been set up for this purpose.

METHODS AND DESIGN

The objective of this randomized trial is to study the effect of ISS versus OSS on individual learning outcome, safety attitude, motivation, stress, and team performance amongst multi-professional obstetric-anesthesia teams.The trial is a single-centre randomized superiority trial including 100 participants. The inclusion criteria were health-care professionals employed at the department of obstetrics or anesthesia at Rigshospitalet, Copenhagen, who were working on shifts and gave written informed consent. Exclusion criteria were managers with staff responsibilities, and staff who were actively taking part in preparation of the trial. The same obstetric multi-professional training was conducted in the two simulation settings. The experimental group was exposed to training in the ISS setting, and the control group in the OSS setting. The primary outcome is the individual score on a knowledge test. Exploratory outcomes are individual scores on a safety attitudes questionnaire, a stress inventory, salivary cortisol levels, an intrinsic motivation inventory, results from a questionnaire evaluating perceptions of the simulation and suggested changes needed in the organization, a team-based score on video-assessed team performance and on selected clinical performance.

DISCUSSION

The perspective is to provide new knowledge on contextual effects of different simulation settings.

TRIAL REGISTRATION

ClincialTrials.gov NCT01792674.

Tactical and operational response to major incidents: feasibility and reliability of skills assessment using novel virtual environments

OBJECTIVES

To determine feasibility and reliability of skills assessment in a multi-agency, triple-site major incident response exercise carried out in a virtual world environment.

METHODS

Skills assessment was carried out across three scenarios. The pre-hospital scenario required paramedics to triage and treat casualties at the site of an explosion. Technical skills assessment forms were developed using training syllabus competencies and national guidelines identified by pre-hospital response experts. Non-technical skills were assessed using a seven-point scale previously developed for use by pre-hospital paramedics. The two in-hospital scenarios, focusing on a trauma team leader and a silver/clinical major incident co-ordinator, utilised the validated Trauma-NOTECHS scale to assess five domains of performance. Technical competencies were assessed using an ATLS-style competency scale for the trauma scenario. For the silver scenario, the assessment document was developed using competencies described from a similar role description in a real-life hospital major incident plan. The technical and non-technical performance of all participants was assessed live by two experts in each of the three scenarios and inter-assessor reliability was computed. Participants also self-assessed their performance using identical proformas immediately after the scenarios were completed. Self and expert assessments were correlated (assessment cross-validation).

RESULTS

Twenty-three participants underwent all scenarios and assessments. Performance assessments were feasible for both experts as well as the participants. Non-technical performance was generally scored higher than technical performance. Very good inter-rater reliability was obtained between expert raters across all scenarios and both technical and non-technical aspects of performance (reliability range 0.59-0.90, Ps<0.01). Significant positive correlations were found between self and expert assessment in technical skills across all three scenarios (correlation range 0.52-0.84, Ps<0.05), although no such correlations were observed in non-technical skills.

CONCLUSIONS

This study establishes feasibility and reliability of virtual environment technical and non-technical skills assessment in major incident scenarios for the first time. The development for further scenarios and validated assessment scales will enable major incident planners to utilise virtual technologies for improved major incident preparation and training.

Beyond NAVMEC: competency-based veterinary education and assessment of the professional competencies

The implementation of competency-based curricula within the health sciences has been an important paradigm shift over the past 30 years. As a result, one of the five strategic goals recommended by the North American Veterinary Medical Education Consortium (NAVMEC) report was to graduate career-ready veterinarians who are proficient in, and have the confidence to use, an agreed-upon set of core competencies. Of the nine competencies identified as essential for veterinary graduates, seven could be classified as professional or non-technical competencies: communication; collaboration; management (self, team, system); lifelong learning, scholarship, value of research; leadership; diversity and multicultural awareness; and adaptation to changing environments. Traditionally, the professional competencies have received less attention in veterinary curricula and their assessment is often sporadic or inconsistent. In contrast, the same or similar competencies are being increasingly recognized in other health professions as essential skills and abilities, and their assessment is being undertaken with enhanced scrutiny and critical appraisal. Several challenges have been associated with the assessment of professional competencies, including agreement as to their definition and therefore their evaluation, the fact that they are frequently complex and require multiple integrative assessments, and the ability and/or desire of faculty to teach and assess these competencies. To provide an improved context for assessment of the seven professional competencies identified in the NAVMEC report, this article describes a broad framework for their evaluation as well as specific examples of how these or similar competencies are currently being measured in medical and veterinary curricula.

Unannounced in situ simulations: integrating training and clinical practice

Simulation-based training for healthcare providers is well established as a viable, efficacious training tool, particularly for the training of non-technical team-working skills. These skills are known to be critical to effective teamwork, and important in the prevention of error and adverse events in hospitals. However, simulation suites are costly to develop and releasing staff to attend training is often difficult. These factors may restrict access to simulation training. We discuss our experiences of 'in situ' simulation for unannounced cardiac arrest training when the training is taken to the clinical environment. This has the benefit of decreasing required resources, increasing realism and affordability, and widening multidisciplinary team participation, thus enabling assessment and training of non-technical team-working skills in real clinical teams. While there are practical considerations of delivering training in the clinical environment, we feel there are many potential benefits compared with other forms of simulation training. We are able to tailor the training to the needs of the location, enabling staff to see a scenario that is relevant to their practice. This is particularly useful for staff who have less exposure to cardiac arrest events, such as radiology staff. We also describe the important benefit of risk assessment for a clinical environment. During our simulations we have identified a number of issues that, had they occurred during a real resuscitation attempt, may have led to patient harm or patient death. For these reasons we feel in situ simulation should be considered by every hospital as part of a patient safety initiative.