Measuring non-technical skills of medical emergency teams: An update on the validity and reliability of the Team Emergency Assessment Measure (TEAM)

A Customizable Digital Cognitive Aid for Neonatal Resuscitation: A Simulation-Based Randomized Controlled Trial

OBJECTIVE

Adherence to the International Liaison Committee on Resuscitation (ILCOR) algorithm optimizes the initial management of critically ill neonates. In this randomized controlled trial, we assessed the impact of a customizable sequential digital cognitive aid (DCA), adapted from the 2020 ILCOR recommendations, compared with a poster cognitive aid (standard of care [SOC]), on technical and nontechnical performance of junior trainees during a simulated critical neonatal event at birth.

METHODS

For this prospective, bicentric video-recorded study, students were recruited on a voluntary basis, and randomized into groups of 3 composed of a pediatric resident and two midwife students. They encountered a simulated cardiac arrest at birth either (1) with DCA use and ILCOR algorithm poster displayed on the wall (intervention group) or (2) with sole ILCOR algorithm poster (poster cognitive aid [SOC]). Technical and nontechnical skills (NTS) between the two groups were assessed using a standardized scoring of videotaped performances. A neonate specific NTS score was created from the adult Team score.

RESULTS

108 students (36 groups of three) attended the study, 20 groups of 3 in the intervention group and 16 groups of 3 in the poster cognitive aid (SOC) group. The intervention group showed a significant improvement in the technical score ( P < 0.001) with an average of 24/27 points (24.0 [23.5-25.0]) versus 20.8/27 (20.8 [19.9-22.5]) in poster cognitive aid (SOC) group. No nontechnical score difference was observed. Feedback on the application was positive.

CONCLUSIONS

During a simulated critical neonatal event, use of a DCA was associated with higher technical scores in junior trainees, compared with the sole use of ILCOR poster algorithm.

Structured Callouts, the Shared Mental Model, and Teamwork: A Video-Based Study in a Pediatric Emergency Department

OBJECTIVES

The shared mental model is essential to high-quality resuscitations. A structured callout (SCO) is often performed to establish the shared mental model, but the literature on SCOs is limited. The objectives of this study are to describe performance of SCOs during pediatric medical emergencies and to determine whether a SCO is associated with better teamwork.

METHODS

This was a retrospective study in the resuscitation area of an academic pediatric emergency department, where performance of a SCO is a standard expectation. Only medical or nontrauma patients were eligible for inclusion. Data collection was performed by structured video review by 2 observers and verified by a third blinded observer. A SCO was defined as team leader (Pediatric Emergency Medicine fellow or faculty physician) verbalization of at least 1 element of the patient history/examination or an assessment of patient physiology and 1 element of the diagnostic or therapeutic plan. We independently measured teamwork using the Teamwork Emergency Assessment Measure (TEAM) tool.

RESULTS

We reviewed 60 patient encounters from the pediatric emergency department resuscitation area between April 2018 and June 2020. Median patient age was 6 years; the team leader was a Pediatric Emergency Medicine fellow in 55% of encounters. The physician team leader performed a SCO in 38 (63%) of patient encounters. The TEAM scores were collected for 46 encounters. Mean TEAM score (SD) was 42.3 (1.7) in patients with a SCO compared with 40.0 (3.0) in those without a SCO ( P = 0.007).

CONCLUSIONS

Performance of a SCO was associated with better teamwork, but the difference was of unclear clinical significance.

Application of the Team Emergency Assessment Measure Scale in undergraduate medical students and interprofessional clinical teams: validity evidence of a Spanish version applied in Chile

Background

Teamwork is one of the competencies necessary for physicians to work effectively in health systems and is a competency that can be developed with simulation in professionals and medicine students. The Team Emergency Assessment Measurement (TEAM) was created to evaluate the non-technical performance of team members during resuscitation events in real teams. The TEAM scale includes items to assess leadership, teamwork, situational awareness, and task management. An objective evaluation tool in Spanish is valuable for training health professionals at all undergraduate and continuing education levels. This study aimed to generate evidence of the validity of the Team Emergency Assessment Measure (TEAM) in Spanish to measure the performance of medical students and adult, pediatric, and obstetric emergency clinical teams in simulated emergencies as a self-assessment tool.

Methods

To develop the Spanish version of the instrument, a forward and backward translation process was followed by independent translators, native and fluent in English and Spanish, and a review by a panel of Chilean experts comprising three trained simulation instructors to verify semantics and cultural equivalence. High-fidelity simulations with debriefing were conducted with 5th-year medical students, in which students and instructors applied the Spanish version of the TEAM scale. In the second stage, adult, pediatric, and obstetric emergency management simulations were conducted using the TEAM scale for real clinical teams as a self-assessment tool.

Findings

By applying the overall TEAM scale to medicine students and clinical teams, Cronbach's alpha was 0.921. For medical students' self-assessment, we obtained Cronbach's alpha of 0.869. No significant differences were found between the overall scores and the scores by dimensions evaluated by instructors and students (p > 0.05). In the case of clinical team training, Cronbach's alpha was 0.755 for adult emergency teams, 0.797 for pediatric emergency teams, and 0.853 for obstetric emergency teams.

Conclusion

The validated instrument is adequate for evaluating teamwork in medical student simulations by instructors and peers and for self-assessment in adult, pediatric, and obstetric emergency clinical teams.

Successful implementation of a rater training program for medical students to evaluate simulated pediatric emergencies

Introduction

Simulation-based training is increasingly used in pediatrics to teach technical skills, teamwork, and team communication, and to improve potential deficiencies in pediatric emergency care. Team performance must be observed, analyzed, and evaluated by trained raters. The structured training of medical students for the assessment of simulated pediatric emergencies has not yet been investigated.

Methods

We developed a rater training program for medical students to assess guideline adherence, teamwork, and team communication in simulated pediatric emergencies. Interrater reliability was measured at each training stage using Kendall tau coefficients.

Results

In 10 out of 15 pairs of raters interrater reliability was moderate to high (tau>0.4), whereas it was low in the remaining 5 pairs of raters.

Discussion

The interrater reliability showed good agreement between medical students and expert raters at the end of the rater training program. Medical students can be successfully involved in the assessment of guideline adherence as well as teamwork and team communication in simulated pediatric emergencies.

Impact and effectiveness of a mandatory competency-based simulation program for pediatric emergency medicine faculty

Introduction

Pediatric emergency medicine physicians struggle to maintain their critical procedural and resuscitation skills. Continuing professional development programs incorporating simulation and competency-based standards may help ensure skill maintenance. Using a logic model framework, we sought to evaluate the effectiveness of a mandatory annual competency-based medical education (CBME) simulation program.

Methods

The CBME program, evaluated from 2016 to 2018, targeted procedural, point-of-care ultrasound (POCUS) and resuscitation skills. Delivery of educational content included a flipped-classroom website, deliberate practice, mastery-based learning, and stop-pause debriefing. Participants' competence was assessed using a 5-point global rating scale (GRS; 3 = competent, 5 = mastery). Statistical process control charts were used to measure the effect of the CBME program on team performance during in situ simulations (ISS), measured using the Team Emergency Assessment Measure (TEAM) scale. Faculty completed an online program evaluation survey.

Results

Forty physicians and 48 registered nurses completed at least one course over 3 years (physician mean ± SD 2.2 ± 0.92). Physicians achieved competence on 430 of 442 stations (97.3%). Mean ± SD GRS scores for procedural, POCUS, and resuscitation stations were 4.34 ± 0.43, 3.96 ± 0.35, and 4.17 ± 0.27, respectively. ISS TEAM scores for "followed standards and guidelines" improved significantly. No signals of special cause variation emerged for the other 11 TEAM items, indicating skills maintenance. Physicians rated CBME training as highly valuable (mean question scores 4.15-4.85/5). Time commitment and scheduling were identified as barriers to participation.

Conclusions

Our mandatory simulation-based CBME program had high completion rates and very low station failures. The program was highly rated and faculty improved or maintained their ISS performance across TEAM scale domains.

Development and psychometric testing of the non-technical skills scale in medical and surgical hospital units for nursing students

OBJECTIVE

To develop and validate a tool to assess the non-technical skills of medical and surgical hospital units undergraduate nursing students.

BACKGROUND

In the area of healthcare, non-technical skills complement technical ones, and contribute to patient safety. High-fidelity simulation is an ideal resource for working on these skills. Thus, evaluation instruments are needed to understand the efficiency of this methodology. Although many evaluation instruments already exist, none measure non-technical skills of undergraduate nursing students in medical and surgical hospital units.

DESIGN

An instrumental study design was employed.

METHODS

Two-phases were used to develop and validate the scale: 1) Scale development. A group of experts defined the dimensions and components. Afterwards, the content was validated by experts, and a pilot study was conducted with undergraduate Nursing students. 2) Analysis of the psychometric properties of the scale. A total of 393 students were evaluated in high-fidelity simulation scenarios by three evaluators, through the use of the Non-Technical Skills in Medical and Surgical Hospital Units (NTS-Nursing) Scale.

RESULTS

The content validity indexes were adequate for the total of the items and the total of the scale. The statistical descriptors of the items, the internal structure, and the reliability (internal consistency and inter-evaluator reliability) were analyzed, as well as the external evidence of validity, with adequate values obtained.

CONCLUSION

The NTS-Nursing scale is a valid and reliable instrument. Its structure of 10 items makes its use fast and easy.

Preparing and Presenting Validation Studies: A Guide for the Perplexed

SUMMARY STATEMENT

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

Application of the Team Emergency Assessment Measure for Prehospital Cardiopulmonary Resuscitation

Introduction: Communication and teamwork are critical for ensuring patient safety, particularly during prehospital cardiopulmonary resuscitation (CPR). The Team Emergency Assessment Measure (TEAM) is a tool applicable to such situations. This study aimed to validate the TEAM efficiency as a suitable tool even in prehospital CPR. Methods: A multi-centric observational study was conducted using the data of all non-traumatic out-of-hospital cardiac arrest patients aged over 18 years who were treated using video communication-based medical direction in 2018. From the extracted data of 1494 eligible patients, 67 sample cases were randomly selected. Two experienced raters were assigned to each case. Each rater reviewed 13 or 14 videos and scored the TEAM items for each field cardiopulmonary resuscitation performance. The internal consistency, concurrent validity, and inter-rater reliability were measured. Results: The TEAM showed high reliability with a Cronbach’s alpha value of 0.939, with a mean interitem correlation of 0.584. The mean item–total correlation was 0.789, indicating significant associations. The mean correlation coefficient between each item and the global score range was 0.682, indicating good concurrent validity. The mean intra-class correlation coefficient was 0.804, indicating excellent agreement. Discussion: The TEAM can be a valid and reliable tool to evaluate the non-technical skills of a team of paramedics performing CPR.

Improve Cardiac Emergency Preparedness by Building a Team-Based Cardiopulmonary Resuscitation Educational Plan

Objective

To design an innovative team-based cardiopulmonary resuscitation (CPR) educational plan for multiple bystanders and evaluate whether it was associated with better teamwork and higher quality of resuscitation.

Methods

The team-based CPR plan defined the process for a three-person team, emphasize task allocation, leadership, and closed-loop communication. Participants qualified for single-rescuer CPR skills were randomized into teams of 3. The teamwork performance and CPR operation skills were evaluated in one simulated cardiac arrest scenario before and after training on the team-based CPR plan. The primary outcomes were measured by the Team Emergency Assessment Measure (TEAM) scale and chest compression fraction (CCF).

Results

Forty-three teams were included in the analysis. The team-based CPR plan significantly improved the team performance (global rating 6.7 ± 1.3 vs. 9.0 ± 0.7, corrected p < 0.001 after Bonferroni's correction). After implementing the team-based CPR plan, CCF increased [median 59 (IQR 48–69) vs. 64 (IQR 57–71%)%, corrected p = 0.002], while hands-off time decreased [median 233.2 (IQR 181.0–264.0) vs. 207 (IQR 174–222.9) s, corrected p = 0.02]. We found the average compression depth was significantly improved through the team-based CPR training [median 5.1 (IQR 4.7–5.6) vs. 5.3 (IQR 4.9–5.5) cm, p = 0.03] but no more significantly after applying the Bonferroni's correction (corrected p = 0.35). The compression depths were significantly improved by collaborating and exchanging the role of compression among the participants after the 6th min.

Conclusion

The team-based CPR plan is feasible for improving bystanders teamwork performance and effective for improving resuscitation quality in prearrival care. We suggest a wide application of the team-based CPR plan in the educational program for better resuscitation performance in real rescue events.

Non-technical skills in pre-hospital care in the Czech Republic: a prospective multicentric observational study (NTS study)

Background

Non-technical skills (NTS) are important for the proper functioning of emergency medical ambulance crews but have hardly been researched in the conditions of clinical pre-hospital care.

The primary objective of this study, therefore, is to describe the use of NTS in practice. The secondary objective is to compare if the performance of NTS varies according to the type of case.

Methods

In this multicentric observational study the modified Team Emergency Assessment Measure (TEAM) score was used to assess the performed NTS of two or more crews on site. The evaluation consisted of leadership, teamwork and task management, rated by a field supervisor.

The study observations took place in real clinical pre-hospital emergency medical cases when two or more crews were dispatched between October 2019 and August 2020. The sample size was determined by researchers prior to the study to at least 100 evaluated events per each of the three participating emergency medical services.

The results are presented as median and interquartile range. The internal reliability, consistency and validity of test items and results were evaluated. The Kruskal–Wallis test and the post hoc Mann-Whitney U test with Bonferroni correction were used for multiple comparisons of three groups.

Results

A total of 359 events were evaluated. Surprisingly, the median value for all eight items was as high as 3.0 with a similar interquartile range of 1.0. There were no differences observed by case type (CPR vs. TRAUMA vs. MEDICAL) except from item 1. A post hoc analysis revealed that this difference is in favour of a higher rated performance of non-technical skills in CPR.

Conclusions

The overall result of the performance of non-technical skills can be regarded as very good and can serve for further evaluations. The crews achieved better parameters of NTS in leadership in resuscitation situations in comparison with general medical events.

Trial Registration

The study is registered at Clinical Trials under the ID: NCT04503369.

Integrative Review of Non-Technical Skills Frameworks to Apply for Air Medical Transfer of Pregnant Women

OBJECTIVE

To identify the most suitable non-technical skills framework to adapt and apply to the air medical transfer of pregnant women.

DATA SOURCES

Embase, PsycINFO, PubMed, MEDLINE, Web of Science, CINAHL, Science Direct, and Google Scholar.

STUDY SELECTION

We retrieved potentially relevant articles using a predefined combination of keywords extended with truncation and Boolean operators. Database and manual reference searches yielded 569 peer-reviewed articles. We included articles if they presented empirical data and described non-technical or cognitive competency skills frameworks for health care professionals. We discussed any ambiguities regarding inclusion, and they were resolved by consensus. We retained 71 full-text articles for final review.

DATA EXTRACTION

We coded extracted data under four criteria: non-technical skill categories, context of use, psychometric properties, and rating system. We generated descriptive summary tables of the characteristics of existing non-technical skills frameworks based on publication year, method of development, clinical setting, clinical specialty, routine/crisis-based performance, and team/individual performance.

DATA SYNTHESIS

We identified 42 non-technical skills frameworks from a variety of health care settings. We critically examined context of use and how use in various clinical settings may align with air transfers of pregnant women. Our findings illustrate the importance of team-based and routine performance rather than crisis-focused skills. Maintaining situational awareness throughout all stages of the transfer and communicating effectively with team members, the pregnant woman, and her partner are skills that are particularly important to ensure good outcomes.

CONCLUSION

We selected the Global Assessment of Obstetric Team Performance as the most suitable non-technical skills framework to adapt to the clinical setting of air medical transfer of pregnant women. We considered the clinical specialty, specific non-technical skills required in the setting, the framework's properties, and the requirement to focus on routine team performance.

Team emergency assessment measure (TEAM) of non-technical skills: The Brazilian Portuguese version of the TEAM tool

OBJECTIVES

The aim of this study was to conduct the translation and cross-cultural adaptation of the original Team Emergency Assessment Measure (TEAM) tool into the Brazilian Portuguese language and investigate the internal consistency, inter-rater reliability, and concurrent validity of this new version (bp-TEAM).

METHODS

Independent medical translators performed forward and backward translations of the TEAM tool between English and Portuguese, creating the bp-TEAM. The authors selected 23 videos from final-year medical students during in-situ emergency simulations. Three independent raters assessed all the videos using the bp-TEAM and provided a score for each of the 12 items of the tool. The authors assessed the internal consistency and the inter-rater reliability of the tool.

RESULTS

Raters assessed all 23 videos. Internal consistency was assessed among the 11 items of the bp-TEAM from one rater, yielding a Cronbach's alpha of 0.89. inter-item correlation analysis yielded a mean correlation coefficient rho of 0.46. Inter-rater reliability analysis among the three raters yielded an intraclass correlation coefficient of 0.86 (95% CI 0.83‒0.89), p < 0.001.

CONCLUSION

The Brazilian Portuguese version of the TEAM tool presented acceptable psychometric properties, similar to the original English version.

Measuring non-technical skills during prehospital advanced cardiac life support: A pilot study

Aim

To analyse non-technical skills of mobile medical teams during out-of-hospital cardiac arrests (OHCA) using the validated Team Emergency Assessment Measure (TEAM) tool. To research the correlation between non-technical skills and patient outcome.

Methods

Adult patients who experienced an out-of-hospital cardiac arrest between July 2016, and June 2018, and were treated by a mobile medical team from the University Hospital Leuven, were eligible for the study. Resuscitations were video recorded from the team leader’s perspective. Video recordings were reviewed and scored by emergency physicians, using the TEAM evaluation form.

Results

In total 114 OHCAs were analysed. The mean TEAM score was 34.4/44 (SD = 5.5). The mean item score was 3.1/4 (SD = 0.8). On average, ‘effective team communication’ had the lowest score (2.4), while ‘acting with composure and control’ and ‘following of approved standards/guidelines’ scored the highest (3.4). The average non-technical skills theme scores were 2.9 (SD = 0.9) for ‘Leadership’, 3.1 (SD = 0.8) for ‘Teamwork’ and 3.3 (SD = 0.7) for ‘Task management’. ‘Leadership’ was rated significantly lower than ‘Teamwork’ (p = 0.004) and ‘Task management’ (p < 0.001). No significant correlation was found between TEAM and return of spontaneous circulation (p = 0.574) or one month survival (p = 0.225).

Conclusion

The mean overall TEAM score was categorized as good. Task management scored high, while leadership and team communication received lower scores. Future training programs should thus focus on improving leadership and communication. In this pilot study no correlation was found between non-technical skills and survival.

Validating the Team Emergency Assessment Measure (TEAM) in obstetric and gynaecologic resuscitation teams

BACKGROUND

Simulation-based training is an effective method of enhancing the knowledge, skill, and technical abilities of individuals and teams encountering obstetric and gynaecologic emergencies. Simulation may also enhance the non-technical performance of teams resulting in improved patient outcomes. Although simulation-based training is widely recognised as an effective educational approach, issues around feasibility - the lack of simulation experts and malleable outcome measures of team performance - remain critical barriers to their implementation.

AIM

To evaluate the psychometric properties of the Team Emergency Assessment Measure (TEAM) when used by medical professionals in simulated obstetric and gynaecological emergencies.

METHODS

There were 151 participants (63% female; 60% consultants; 69% no previous simulation-based training) who observed three live high-fidelity obstetric and gynaecological resuscitation simulations and completed the TEAM.

RESULTS

Confirmatory factor analysis evaluated the construct validity of the TEAM, yielding a second-order structure identified by 'leadership', 'teamwork', and 'team management'. Convergent validity was supported by the average item-to-scale total correlation which was 0.75, P < 0.001 and the average analysis of variance extracted (AVE) 0.88. The individual factors also yielded high factor-to-scale total correlations (mean [M] = 0.87), and AVE (M = 0.89). The internal reliability was high for the whole scale (average alpha = 0.92) and across the sub-factors (average alpha = 0.80). The inter-rater reliability was excellent (inter-class correlation coefficient 1 = 0.98). Participants with differing levels of simulation training experience did not significantly differ.

CONCLUSION

The TEAM is a viable instrument for the assessment of non-technical performance during simulated obstetric and gynaecologic emergencies, thus enhancing the feasibility of simulation-based training.

Emergencies within hospital wards: An observational study of the non-technical skills of medical emergency teams

BACKGROUND

Medical emergency teams are essential in responding to acute deterioration of patients in hospitals, requiring both clinical and non-technical skills. This study aims to assess the non-technical skills of medical emergency teams during hospital ward emergencies and explore team members perceptions and experiences of the use non-technical skills during medical emergencies.

METHODS

A multi-methods study was conducted in two phases. During phase one observation and assessment of non-technical skills used in medical emergencies using the Team Emergency Assessment Measure (TEAM™) was conducted; and in the phase two in-depth interviews were undertaken with medical emergency team members.

RESULTS

Based on 20 observations, mean TEAM™ ratings for non-technical skill domains were: 'leadership' 5.0 out of 8 (±2.0); 'teamwork' 21.6 out of 28 (±3.6); and 'task management' 6.5 out of 8 (±1.4). The mean 'global' score was 7.5 out of 10 (±1.5). The qualitative findings identified three areas, 'individual', 'team' and 'other' contributing factors, which impacted upon the non-technical skills of medical emergency teams.

CONCLUSION

Non-technical skills of hospital medical emergency teams differ, and the impact of the skill mix on resuscitation outcomes was recognised by team members. These findings emphasize the importance non-technical skills in resuscitation training and well-developed processes for medical emergency teams.

Team performance training for medical students: Low vs high fidelity simulation

Objectives

The aim of this study is to evaluate a simulation-based team performance course for medical students and compare its low- and high-fidelity components.

Study design

This is a prospective crossover observational study. Groups participated in one low- and one high-fidelity session twice. Low-fidelity scenarios included management of an emergency case on a simulated-patient, whereas high-fidelity scenarios constituted of multiple-trauma cases where simulated-patients wore a hyper-realistic suit. Team performance was assessed objectively, using the TEAM™ tool, and subjectively using questionnaires. Questionnaires were also used to assess presence levels, stress levels and evaluate the course.

Results

Participants’ team performance was higher in the low-fidelity intervention as assessed by the TEAM™ tool. An overall mean increase in self-assessed confidence towards non-technical skills attitudes was noted after the course, however there was no difference in self-assessed performance between the two interventions. Both reported mean stress and presence levels were higher for the high-fidelity module. Evaluation scores for all individual items of the questionnaire were ≥4.60 in both NTS modules. Students have assessed the high-fidelity module higher (4.88 out of 5, SD = 0.29) compared to low-fidelity module (4.74 out of 5, SD = 0.67).

Conclusions

Both the low- and high-fidelity interventions demonstrated an improvement in team performance of the attending medical students. The high-fidelity intervention was more realistic, yet more stressful. Furthermore, it proved to be superior in harvesting leadership, teamwork and task management skills. Both modules were evaluated highly by the students, however, future research should address retention of the taught skills and adaptability of such interventions.

•

The high-fidelity intervention was superior in harvesting leadership, teamwork and task management skills, yet more stressful.

•

We set the group to develop a novel group of team-based OSCE performance.

•

The next step is to adapt such taught skills to different specialties and implement initiatives in the medical school curricula.

Exploring the use of high and low demand simulation for human performance assessment during multiorgan retrieval with the joint scrub practitioner

Introduction

The National Organ Retrieval Service (NORS) 2015 review recommended a Joint Scrub Practitioner for abdominal and cardiac teams during combined organ retrieval. To evaluate the feasibility of this role, and to understand the functional implications, this study explores the use of simulation and provides a novel and comprehensive approach to assess individual and team performance in simulated multiorgan retrievals.

Methods

Two high-fidelity simulations were conducted in an operating theatre with porcine organs, en bloc, placed in a mannequin. For donation after brainstem death (DBD) simulation, an anaesthetic machine provided simulated physiological output. Retrievals following donation after circulatory death (DCD) began with rapid arrival in theatre of the mannequin. Cardiothoracic (lead surgeon) and abdominal (lead and assistant surgeons; joint scrub practitioner, n=9) teams combined for the retrievals. Data collected before, during and after simulations used self-report and expert observers to assess: attitudinal expectations, mental readiness, mental effort, non-technical skills, teamwork, task workload and social validation perceptions.

Results

Attitudinal changes regarding feasibility of a joint scrub practitioner for DBD and DCD are displayed in the main body. There were no significant differences in mental readiness prior to simulations nor in mental effort indicated afterwards; however, variance was noted between simulations for individual team members. Non-technical skills were slightly lower in DCD than in DBD. Global ratings of teamwork were significantly (p<0.05) lower in DCD than in DBD. Measures of attitude indicated less support for the proposed joint scrub practitioner role for DCD than for DBD.

Discussion

The paper posits that the joint scrub practitioner role in DCD multiorgan retrieval may bring serious and unanticipated challenges. Further work to determine the feasibility of the NORS recommendation is required. Measures of team performance and individual psychological response can inform organ retrieval feasibility considerations nationally and internationally.

Videography in Pediatric Emergency Research: Establishing a Multicenter Collaborative and Resuscitation Registry

OBJECTIVES

High-quality clinical research of resuscitations in a pediatric emergency department is challenging because of the limitations of traditional methods of data collection (chart review, self-report) and the low frequency of cases in a single center. To facilitate valid and reliable research for resuscitations in the pediatric emergency department, investigators from 3 pediatric centers, each with experience completing successful single-center, video-based studies, formed the Videography In Pediatric Emergency Research (VIPER) collaborative.

METHODS

Our initial effort was the development of a multicenter, video-based registry and simulation-based testing of the feasibility and reliability of the VIPER registry. Feasibility of data collection was assessed by the frequency of an indeterminate response for all data elements in the registry. Reliability was assessed by the calculation of Cohen κ for dichotomous data elements and intraclass correlation coefficients for continuous data elements.

RESULTS

Video-based data collection was completed for 8 simulated pediatric resuscitations, with at least 2 reviewers per case. Data were labeled as indeterminate by at least 1 reviewer for 18 (3%) of 524 relevant data fields. The Cohen κ for all dichotomous data fields together was 0.81 (95% confidence interval, 0.61-1.0). For all continuous (time-based) variables combined, the intraclass correlation coefficient was 0.88 (95% confidence interval, 0.70-0.96).

CONCLUSIONS

Initial simulation-based testing suggests video-based data collection using the VIPER registry is feasible and reliable. Our next step is to assess feasibility and reliability for actual pediatric resuscitations and to complete several prospective, hypothesis-based studies of specific aspects of resuscitative care, including of cardiopulmonary resuscitation, tracheal intubation, and teamwork and communication.

Reliability of instruments that measure situation awareness, team performance and task performance in a simulation setting with medical students

OBJECTIVES

The assessment of situation awareness (SA), team performance and task performance in a simulation training session requires reliable and feasible measurement techniques. The objectives of this study were to test the Airways-Breathing-Circulation-Disability-Exposure (ABCDE) checklist and the Team Emergency Assessment Measure (TEAM) for inter-rater reliability, as well as the application of Situation Awareness Global Assessment Technique (SAGAT) for feasibility and internal consistency.

DESIGN

Methodological approach.

SETTING

Data collection during team training using full-scale simulation at a university clinical training centre. The video-recorded scenarios were rated independently by four raters.

PARTICIPANTS

55 medical students aged 22-40 years in their fourth year of medical studies, during the clerkship in anaesthesiology and critical care medicine, formed 23 different teams. All students answered the SAGAT questionnaires, and of these students, 24 answered the follow-up postsimulation questionnaire (PSQ). TEAM and ABCDE were scored by four professionals.

MEASURES

The ABCDE and TEAM were tested for inter-rater reliability. The feasibility of SAGAT was tested using PSQ. SAGAT was tested for internal consistency both at an individual level (SAGAT) and a team level (Team Situation Awareness Global Assessment Technique (TSAGAT)).

RESULTS

The intraclass correlation was 0.54/0.83 (single/average measurements) for TEAM and 0.55/0.83 for ABCDE. According to the PSQ, the items in SAGAT were rated as relevant to the scenario by 96% of the participants. Cronbach's alpha for SAGAT/TSAGAT for the two scenarios was 0.80/0.83 vs 0.62/0.76, and normed χ² was 1.72 vs 1.62.

CONCLUSION

Task performance, team performance and SA could be purposefully measured, and the reliability of the measurements was good.

Non-technical skills assessments in undergraduate medical education: A focused BEME systematic review: BEME Guide No. 54

Consensus on how to assess non-technical skills is lacking. This systematic review aimed to evaluate the evidence regarding non-technical skills assessments in undergraduate medical education, to describe the tools used, learning outcomes and the validity, reliability and psychometrics of the instruments. A standardized search of online databases was conducted and consensus reached on included studies. Data extraction, quality assessment, and content analysis were conducted per Best Evidence in Medical Education guidelines. Nine papers met the inclusion criteria. Assessment methods broadly fell into three categories: simulated clinical scenarios, objective structured clinical examinations, and questionnaires or written assessments. Tools to assess non-technical skills were often developed locally, without reference to conceptual frameworks. Consequently, the tools were rarely validated, limiting dissemination and replication. There were clear themes in content and broad categories in methods of assessments employed. The quality of this evidence was poor due to lack of theoretical underpinning, with most assessments not part of normal process, but rather produced as a specific outcome measure for a teaching-based study. While the current literature forms a good starting position for educators developing materials, there is a need for future work to address these weaknesses as such tools are required across health education.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Concepts for the Simulation Community: Development of the International Simulation Data Registry

STATEMENT

The past several decades have seen tremendous growth in our understanding of best practices in simulation-based healthcare education. At present, however, there is limited infrastructure available to assist programs in translation of these best practices into more standardized educational approaches, higher quality of care, and ultimately improved outcomes. In 2014, the International Simulation Data Registry (ISDR) was launched to address this important issue. The existence of such a registry has important implications not just for educational practice but also for research. The ISDR currently archives data related to pulseless arrest, malignant hyperthermia, and difficult airway simulations. Case metrics are designed to mirror the American Heart Association's Get With the Guidelines Registry, allowing for direct comparisons with clinical scenarios. This article describes the rationale for the ISDR, and outlines its development. Current data are presented to highlight the educational and research value of this approach. Projected future developments are also discussed.

Interprofessional learning in immediate life support training does effect TEAM performance during simulated resuscitation

Aim of the study

To assess performance in a simulated resuscitation after participating in either an interprofessional learning (IPL) or uniprofessional learning (UPL) immediate life support (ILS) training course.

Introduction

The Team Emergency Assessment Measure (TEAM) is routinely used in Resuscitation Council (UK) Advanced Life Support courses. This study used the psychometrically validated tool to assess if the delivery of an IPL ILS to final year medical and nursing students could improve overall behavioural performance and global TEAM score.

Methods

A randomised study of medical (n=48) and nursing (n=48) students, assessing performance in a simulated resuscitation following the IPL or UPL ILS courses. Postcourse completion participants were invited back to undertake a video-recorded simulated-resuscitation scenario. Each of these were reviewed using the TEAM tool, at the time by an experienced advanced life support instructor and subsequently by a clinician, independent to the study and blinded as to which cohort they were reviewing.

Results

Inter-rater reliability was tested using a Bland-Altman plot indicating non-proportional bias between raters. Parametric testing and analysis showed statistically significant higher global overall mean TEAM scores for those who had attended the IPL ILS courses.

Conclusion

Our results demonstrate that an IPL approach in ILS produced an increased effect on TEAM scores with raters recording a significantly more collaborative team performance. A postscenario questionnaire for students also found a significantly improved experience within the team following the IPL course compared with those completing UPL training. Although this study shows that team behaviour and performance can change and improve in the short-term, we acknowledge further studies are required to assess the long-term effects of IPL interventions. Additionally, through this type of study methodology, other outcomes in regard to resuscitation team performance may be measured, highlighting other potential benefit to patients, at level four of Kirkpatrick’s hierarchy.

Force of Habit: Developing Situation Awareness in Critical Care Transport

Situation awareness (SA) is a vital cognitive skill for high-stakes, high-hazard occupations, including military, aviation, and health care. The ability to maintain SA can deteriorate in stressful situations, exposing patients to dangerous errors. The literature regarding how to best teach SA techniques is sparse. This article explores specific techniques to promote and maintain SA in dynamic clinical environments using principles derived from cognitive psychology, neuroscience, and human behavioral and organizational research. The authors propose strategies to help individuals and teams to develop ingrained, subconscious behaviors that can help to maintain effective SA in high-stress environments. Situation awareness (SA) is critical in high-stakes circumstances, such as the resuscitation of critically ill or injured patients. Exploratory research in psychology, neuroscience, human factors engineering, and to a lesser extent health care has led to a deeper understanding of what SA is and how it can be measured. Unfortunately, little is known about how we can adapt training in order to more consistently create behaviors that heighten SA during dynamic, high-stakes clinical events. In this article, the prevailing theory of SA is reviewed, and the evidence for evaluating it in medicine is presented. In addition, the authors draw from the fields of neuroscience and cognitive psychology to suggest some strategies that can develop effective behaviors that promote SA in resuscitation.

Non-technical skills assessment scale in nursing: construction, development and validation1

OBJECTIVES

The introduction of non-technical skills during nursing education is crucial to prepare nurses for the clinical context and increase patient safety. We found no instrument developed for this purpose. to construct, develop and validate a non-technical skills assessment scale in nursing.

METHOD

methodological research. Based on the literature review and experience of researchers on non-technical skills in healthcare and the knowledge of the principles of crisis resource management, a list of 63 items with a five-point Likert scale was constructed. The scale was applied to 177 nursing undergraduate students. Descriptive statistics, correlations, internal consistency analysis and exploratory factor analysis were performed to evaluate the psychometric properties of the scale.

RESULTS

scale items presented similar values for mean and median. The maximum and the minimum values presented a good distribution amongst all response options. Most items presented a significant and positive relationship. Cronbach alpha presented a good value (0.94), and most correlations were significant and positive. Exploratory factor analysis using the Kaiser-Meyer-Olkin test showed a value of 0.849, and the Bartlett's test showed adequate sphericity values (χ2=6483.998; p=0.000). One-factor model explained 26% of the total variance.

CONCLUSION

non-technical skills training and its measurement could be included in undergraduate or postgraduate courses in healthcare professions, or even be used to ascertain needs and improvements in healthcare contexts.

ROSC rates and live discharge rates after cardiopulmonary resuscitation by different CPR teams - a retrospective cohort study

Background

Previous studies have reported that the quality of cardiopulmonary resuscitation (CPR) is closely associated with patient outcomes. The aim of this study was to compare patient CPR outcomes across resident, emergency medicine, and rapid response teams.

Methods

The records of patients who underwent CPR at the Seoul National University Bundang Hospital from January 1, 2013 to December 31, 2016 were analyzed retrospectively. Return of spontaneous circulation, 10- and 30-day survival, and live discharge after return of spontaneous circulation were compared across patients treated by the three CPR teams.

Results

Of the 1145 CPR cases, 444 (39%) were conducted by the resident team, 431 (38%) by the rapid response team, and 270 (23%) by the emergency medicine team. The adjusted odds ratios for the return of spontaneous circulation and subsequent 10-day survival among patients who received CPR from the resident team compared to the rapid response team were 0.59 (P = 0.001) and 0.71 (P = 0.037), respectively. There were no significant differences in the 30-day survival and rate of live discharge between patients who received CPR from the rapid response and resident teams; likewise, no significant differences were observed between patients who received CPR from the emergency medicine and rapid response teams.

Conclusions

Patients receiving CPR from the rapid response team may have higher 10-day survival and return of spontaneous circulation rates than those who receive CPR from the resident team. However, our results are limited by the differences in approach, time of CPR, and room settings between teams.

Non-technical skills assessment for prelicensure nursing students: An integrative review

BACKGROUND

In nursing, non-technical skills are recognized as playing an important role to increase patient safety and successful clinical outcomes (Pearson and McLafferty, 2011). Non-technical skills are cognitive and social resource skills that complement technical skills and contribute to safe and efficient task performance (Flin et al., 2008). In order to effectively provide non-technical skills training, it is essential to have an instrument to measure these skills.

METHODOLOGY

An online search was conducted. Articles were selected if they referred to and/or described instruments assessing non-technical skills for nurses and/or prelicensure nursing students in educational, clinical and/or simulated settings with validation evidence (inclusion criteria).

RESULTS

Of the 53 articles located, 26 met the inclusion criteria. Those referred to and/or described 16 instruments with validation evidence developed to assess non-technical skills in multidisciplinary teams including nurses.

CONCLUSION

Although articles have shown 16 valid and reliable instruments, to our knowledge, no instrument has been published or developed and validated for the assessment of non-technical skills of only nurses in general, relevant for use in high-fidelity simulation-based training for prelicensure nursing students. Therefore, there is a need for the development of such an instrument.

Development of an instrument for the evaluation of advanced life support performance

BACKGROUND

Assessing advanced life support (ALS) competence requires validated instruments. Existing instruments include aspects of technical skills (TS), non-technical skills (NTS) or both, but one instrument for detailed assessment that suits all resuscitation situations is lacking. This study aimed to develop an instrument for the evaluation of the overall ALS performance of the whole team.

METHODS

This instrument development study had four phases. First, we reviewed literature and resuscitation guidelines to explore items to include in the instrument. Thereafter, we interviewed resuscitation team professionals (n = 66), using the critical incident technique, to determine possible additional aspects associated with the performance of ALS. Second, we developed an instrument based on the findings. Third, we used an expert panel (n = 20) to assess the validity of the developed instrument. Finally, we revised the instrument based on the experts' comments and tested it with six experts who evaluated 22 video recorded resuscitations.

RESULTS

The final version of the developed instrument had 69 items divided into adherence to guidelines (28 items), clinical decision-making (5 items), workload management (12 items), team behaviour (8 items), information management (6 items), patient integrity and consideration of laymen (4 items) and work routines (6 items). The Cronbach's α values were good, and strong correlations between the overall performance and the instrument were observed.

CONCLUSION

The instrument may be useful for detailed assessment of the team's overall performance, but the numerous items make the use demanding. The instrument is still under development, and more research is needed to determine its psychometric properties.

Evaluating trauma team performance in a Level I trauma center: Validation of the trauma team communication assessment (TTCA-24)

BACKGROUND

Nontechnical skills (NTS), such as team communication, are well-recognized determinants of trauma team performance and good patient care. Measuring these competencies during trauma resuscitations is essential, yet few valid and reliable tools are available. We aimed to demonstrate that the Trauma Team Communication Assessment (TTCA-24) is a valid and reliable instrument that measures communication effectiveness during activations.

METHODS

Two tools with adequate psychometric strength (Trauma Nontechnical Skills Scale [T-NOTECHS], Team Emergency Assessment Measure [TEAM]) were identified during a systematic review of medical literature and compared with TTCA-24. Three coders used each tool to evaluate 35 stable and 35 unstable patient activations (defined according to Advanced Trauma Life Support criteria). Interrater reliability was calculated between coders using the intraclass correlation coefficient. Spearman rank correlation coefficient was used to establish concurrent validity between TTCA-24 and the other two validated tools.

RESULTS

Coders achieved an intraclass correlation coefficient of 0.87 for stable patient activations and 0.78 for unstable activations scoring excellent on the interrater agreement guidelines. The median score for each assessment showed good team communication for all 70 videos (TEAM, 39.8 of 54; T-NOTECHS, 17.4 of 25; and TTCA-24, 87.4 of 96). A significant correlation between TTTC-24 and T-NOTECHS was revealed (p = 0.029), but no significant correlation between TTCA-24 and TEAM (p = 0.77). Team communication was rated slightly better across all assessments for stable versus unstable patient activations, but not statistically significant.

CONCLUSION

TTCA-24 correlated with T-NOTECHS, an instrument measuring nontechnical skills for trauma teams, but not TEAM, a tool that assesses communication in generic emergency settings. TTCA-24 is a reliable and valid assessment that can be a useful adjunct when evaluating interpersonal and team communication during trauma activations.

LEVEL OF EVIDENCE

Diagnostic tests or criteria, level II.

Adaptive leadership curriculum for Indian paramedic trainees

BACKGROUND

Paramedic trainees in developing countries face complex and chaotic clinical environments that demand effective leadership, communication, and teamwork. Providers must rely on non-technical skills (NTS) to manage bystanders and attendees, collaborate with other emergency professionals, and safely and appropriately treat patients. The authors designed a NTS curriculum for paramedic trainees focused on adaptive leadership, teamwork, and communication skills critical to the Indian prehospital environment.

METHODS

Forty paramedic trainees in the first academic year of the 2-year Advanced Post-Graduate Degree in Emergency Care (EMT-paramedic equivalent) program at the GVK-Emergency Management and Research Institute campus in Hyderabad, India, participated in the 6-day leadership course. Trainees completed self-assessments and delivered two brief video-recorded presentations before and after completion of the curriculum.

RESULTS

Independent blinded observers scored the pre- and post-intervention presentations delivered by 10 randomly selected paramedic trainees. The third-party judges reported significant improvement in both confidence (25 %, p < 0.01) and body language of paramedic trainees (13 %, p < 0.04). Self-reported competency surveys indicated significant increases in leadership (2.6 vs. 4.6, p < 0.001, d = 1.8), public speaking (2.9 vs. 4.6, p < 0.001, d = 1.4), self-reflection (2.7 vs. 4.6, p < 0.001, d = 1.6), and self-confidence (3.0 vs. 4.8, p < 0.001, d = 1.5).

CONCLUSIONS

Participants in a 1-week leadership curriculum for prehospital providers demonstrated significant improvement in self-reported NTS commonly required of paramedics in the field. The authors recommend integrating focused NTS development curriculum into Indian paramedic education and further evaluation of the long term impacts of this adaptive leadership training.

Teamwork evaluation during emergency medicine residents’ high-fidelity simulation

Background

Teamwork training has been included in several emergency medicine (EM) curricula; the aim of this study was to compare different scales’ performance in teamwork evaluation during simulation for EM residents.

Methods

In the period October 2013–June 2014, we performed bimonthly high-fidelity simulation sessions, with novice (I–III year, group 1 (G1)) and senior (IV–V year, group 2 (G2)) EM residents; scenarios were designed to simulate management of critical patients. Videos were assessed by three independent raters with the following scales: Emergency Team Dynamics (ETD), Clinical Teamwork Scale (CTS) and Team Emergency Assessment Measure (TEAM). In the period March–June, after each scenario, participants completed the CTS and ETD.

Results

The analysis based on 18 sessions showed good internal consistency and good to fair inter-rater reliability for the three scales (TEAM, CTS, ETD: Cronbach's α 0.954, 0.954, 0.921; Intraclass Correlation Coefficients (ICC), 0.921, 0.917, 0.608). Single CTS items achieved highly significant ICC results, with 12 of the total 13 comparisons achieving ICC results ≥0.70; a similar result was confirmed for 4 of the total 11 TEAM items and 1 of the 8 total ETD items. Spearman's r was 0.585 between ETD and CTS, 0.694 between ETD and TEAM, and 0.634 between TEAM and CTS (scales converted to percentages, all p<0.0001). Participants gave themselves a better evaluation compared with external raters (CTS: 101±9 vs 90±9; ETD: 25±3 vs 20±5, all p<0.0001).

Conclusions

All examined scales demonstrated good internal consistency, with a slightly better inter-rater reliability for CTS compared with the other tools.

Teamwork skills in actual, in situ, and in-center pediatric emergencies: performance levels across settings and perceptions of comparative educational impact

INTRODUCTION

Pediatric emergencies require effective teamwork. These skills are developed and demonstrated in actual emergencies and in simulated environments, including simulation centers (in center) and the real care environment (in situ). Our aims were to compare teamwork performance across these settings and to identify perceived educational strengths and weaknesses between simulated settings. We hypothesized that teamwork performance in actual emergencies and in situ simulations would be higher than for in-center simulations.

METHODS

A retrospective, video-based assessment of teamwork was performed in an academic, pediatric level 1 trauma center, using the Team Emergency Assessment Measure (TEAM) tool (range, 0-44) among emergency department providers (physicians, nurses, respiratory therapists, paramedics, patient care assistants, and pharmacists). A survey-based, cross-sectional assessment was conducted to determine provider perceptions regarding simulation training.

RESULTS

One hundred thirty-two videos, 44 from each setting, were reviewed. Mean total TEAM scores were similar and high in all settings (31.2 actual, 31.1 in situ, and 32.3 in-center, P = 0.39). Of 236 providers, 154 (65%) responded to the survey. For teamwork training, in situ simulation was considered more realistic (59% vs. 10%) and more effective (45% vs. 15%) than in-center simulation.

DISCUSSION

In a video-based study in an academic pediatric institution, ratings of teamwork were relatively high among actual resuscitations and 2 simulation settings, substantiating the influence of simulation-based training on instilling a culture of communication and teamwork. On the basis of survey results, providers favored the in situ setting for teamwork training and suggested an expansion of our existing in situ program.

Effect of a simulation-based workshop on multidisplinary teamwork of newborn emergencies: an intervention study

Background

Video analyses of real-life newborn resuscitations have shown that Neonatal Resuscitation Program (NRP) guidelines are followed in fewer than 50 % of cases. Multidisciplinary simulation is used as a first-rate tool for the improvement of teamwork among health professionals. In the study we evaluated the impact of the crisis resource management (CRM) and anesthesia non-technical skills instruction on teamwork during simulated newborn emergencies.

Methods

Ninety-nine participants of two delivery units (17 pediatricians, 16 anesthesiologists, 14 obstetricians, 31 midwives, and 21 neonatal nurses) were divided to an intervention group (I-group, 9 teams) and a control group (C-group, 6 teams). The I-group attended a CRM and ANTS instruction before the first scenario. After each scenario the I-group performed either self- or peer-assessment depending on whether they had acted or observed in the scenario. All the teams participated in two and observed another two scenarios. All the scenarios were video-recorded and scored by three experts with Team Emergency Assessment Measure (TEAM). SPSS software and nlme package were used for the statistical analyses.

Results

The total TEAM scores of the first scenario between the I- and C-group did not differ from each other. Neither there was an increase in the TEAM scoring between the first and second scenario between the groups. The CRM instruction did not improve the I-group’s teamwork performance. Unfortunately the teams were not comparable because the teams had been allowed to self-select their members in the study design. The total TEAM scores varied a lot between the teams. Mixed-model linear regression revealed that the background of the team leader had an impact on differences of the total teamwork scores (D = 6.50, p = 0.039). When an anesthesia consultant was the team leader the mean teamwork improved by 6.41 points in comparison to specialists of other disciplines (p = 0.043).

Conclusion

The instruction of non-technical skills before simulation training did not enhance the acquisition of teamwork skills of the intervention groups over the corresponding set of skills of the control groups. The teams led by an anesthesiologist scored the best. Experience of team leaders improved teamwork over the CRM instruction.

A rater training protocol to assess team performance

INTRODUCTION

Simulation-based methodologies are increasingly used to assess teamwork and communication skills and provide team training. Formative feedback regarding team performance is an essential component. While effective use of simulation for assessment or training requires accurate rating of team performance, examples of rater-training programs in health care are scarce. We describe our rater training program and report interrater reliability during phases of training and independent rating.

METHODS

We selected an assessment tool shown to yield valid and reliable results and developed a rater training protocol with an accompanying rater training handbook. The rater training program was modeled after previously described high-stakes assessments in the setting of 3 facilitated training sessions. Adjacent agreement was used to measure interrater reliability between raters.

RESULTS

Nine raters with a background in health care and/or patient safety evaluated team performance of 42 in-situ simulations using post-hoc video review. Adjacent agreement increased from the second training session (83.6%) to the third training session (85.6%) when evaluating the same video segments. Adjacent agreement for the rating of overall team performance was 78.3%, which was added for the third training session. Adjacent agreement was 97% 4 weeks posttraining and 90.6% at the end of independent rating of all simulation videos.

DISCUSSION

Rater training is an important element in team performance assessment, and providing examples of rater training programs is essential. Articulating key rating anchors promotes adequate interrater reliability. In addition, using adjacent agreement as a measure allows differentiation between high- and low-performing teams on video review.