A scoping review of nontechnical skill assessment tools to evaluate trauma team performance

Study protocol for a Prospective Observational study of Safety Threats and Adverse events in Trauma (PrO-STAT): a pilot study at a level-1 trauma centre in Canada

INTRODUCTION

Traumatic injuries are a significant public health concern globally, resulting in substantial mortality, hospitalisation and healthcare burden. Despite the establishment of specialised trauma centres, there remains considerable variability in trauma-care practices and outcomes, particularly in the initial phase of trauma resuscitation in the trauma bay. This stage is prone to preventable errors leading to adverse events (AEs) that can impact patient outcomes. Prior studies have identified common causes of these errors, including delayed diagnostics, disorganisation of staff, equipment issues and communication breakdowns, which collectively contribute to AEs. This study addresses gaps in understanding the root causes of these errors by evaluating the most frequent AEs in trauma care through real-time video reviews of resuscitations in the trauma bay. Insights from this evaluation will inform targeted interventions to improve procedural adherence, communication and overall team performance, ultimately reducing preventable errors and improving patient safety.

METHODS AND ANALYSIS

A prospective observational study will be conducted at St. Michael's Hospital, a level-1 trauma centre, to evaluate resuscitations in the trauma bay. All consecutive trauma team activations over 12 months will be included, with data collected using audio-visual recordings and physiological monitoring. A synchronised data capture and analysis platform will comprehensively assess AEs, errors and human and environmental factors during trauma resuscitations. The study aims to detect recurring error patterns, evaluate practice variations and correlate trauma team performance with in-hospital outcomes. Statistical analyses will include descriptive statistics, logistic regression models and multivariable analyses to identify associations and predictors of AEs and patient outcomes.

ETHICS AND DISSEMINATION

Institutional research ethics approval was obtained (SMH REB # 21-009). A modified consent model will be employed for participants. Staff, physicians and learners will be provided with information regarding the study and will have the option to opt-out or withdraw consent. Similarly, trauma patients and their next of kin will be informed about the study, with provisions for opting out or withdrawing consent within 48 hours of recording. Measures will be implemented to ensure data confidentiality, anonymity and respect for participants' autonomy and privacy. The study results will be shared through peer-reviewed journal publications and conference presentations, and key institutional stakeholders will be informed about developing strategies to improve patient safety in trauma care.

Assessing Team Performance in a Longitudinal Neonatal Resuscitation Simulation Training Program: Comparing Validity Evidence to Select the Best Tool

Introduction Neonatal resuscitation is a high-acuity, low-occurrence event that requires ongoing practice by interprofessional teams to maintain proficiency. Simulation provides an ideal platform for team training and evaluation of team performance. Our simulation center supports a longitudinal in situ simulation training program for delivery room teams. In addition to adherence to the Neonatal Resuscitation Program standards, team performance assessment is an essential component of program evaluation and participant feedback. Multiple published teamwork assessment tools exist. Our objective was to select the tool with the best validity evidence for our program's needs. Methods We used Messick's framework to assess the validity of evidence for potential teamwork assessment tools. Four possible tools were identified from the literature: the Mayo High Performance Teamwork Scale (Mayo), Team Performance Observation Tool (TPOT), Clinical Teamwork Scale (CTS), and Team Emergency Assessment Measure (TEAM). Relevant context included team versus individual focus, external evaluator versus self-evaluation, and ease of use (which included efficiency, clarity of interpretation, and overall assessment). Three simulation experts identified consensus anchors for each tool and independently reviewed and scored 10 pre-recorded neonatal resuscitation simulations. Raters assigned each tool a rating according to efficiency, ease of interpretation, and completeness of teamwork assessment. Interrater reliability (IRR) was calculated using intraclass correlation for each tool across the three raters. Average team performance scores for each tool were correlated with neonatal resuscitation adherence scores for each video using Spearman's rank coefficient. Results There was a range of IRR between the tools, with Mayo having the best (single 0.55 and multi 0.78). Each of the three raters ranked Mayo optimally in terms of efficiency (mean 4.66 + 0.577) and ease of use (4+1). However, TPOT and CTS scored highest (mean 4.66 ± 0.577) for overall completeness of teamwork assessment. There was no significant correlation to NRP adherence scores for any teamwork tool. Conclusion Of the four tools assessed, Mayo demonstrated moderate IRR and scored highest for its ease of use and efficiency, though not completeness of assessment. The remaining three tools had poor IRR, which is not an uncommon problem with teamwork assessment tools. Our process emphasizes the fact that assessment tool validity is contextual. Factors such as a relatively narrow (and high) performance distribution and clinical context may have contributed to reliability challenges for tools that offered a more complete teamwork assessment.

Enhancing patient safety in trauma: Understanding adverse events, assessment tools, and the role of trauma video review

OBJECTIVES

This study aimed to investigate adverse events (AEs) in trauma resuscitation, evaluate contributing factors, and assess methods, such as trauma video review (TVR), to mitigate AEs.

BACKGROUND

Trauma remains a leading cause of global mortality and morbidity, necessitating effective trauma care. Despite progress, AEs during trauma resuscitation persist, impacting patient outcomes and the healthcare system. Identifying and analyzing AEs and their determinants are crucial for improving trauma care.

METHODS

This narrative review explored the definition, identification, and assessment of AEs associated with trauma resuscitation within the trauma system. It includes various studies and assessment tools such as STAT Taxonomy and T-NOTECHs. Additionally, it assessed the role of TVR in detecting AEs and strategies to enhance patient safety.

CONCLUSION

Integrated with standardized tools, TVR shows promise for identifying AEs. Challenges include ensuring reporting consistency and integrating approaches into existing protocols. Future research should prioritize linking trauma team performance to patient outcomes, and develop sustainable TVR programs to enhance patient safety.

Training for a mass casualty incident: Conception, development, and implementation of a crew-resource management course for forward surgical teams

BACKGROUND

Since 2021, the predeployment training of French FSTs has included a simulation-based curriculum consisting of organizational and human factors. The purpose of this article is to describe the development of a crew-resource management (CRM) training course dedicated for the forward surgical teams (FSTs) of the French Military Health Service.

METHODS

The approach was based on three steps: (1) establishment of a conceptual framework of FSTs deployment; (2) development of an aircrew-like CRM training combining lectures, laboratory exercises, and situational training exercises to consider four fundamental "nontechnical" (cognitive and social) skills for effective and safe combat casualty care: (a) leadership, (b) decision-making, (c) coordination, and (d) situational awareness; (3) Implementation of teamwork evaluation tools.

RESULTS

A multidisciplinary team designed a conceptual framework for FST preparedness, 24 French FSTs completed a high-quality training that takes into account both technical and nontechnical skills to maintain quality of combat care during mass-casualty incidents, FSTs' CRM skills were assessed using an audio/video recording of a simulated mass-casualty incident.

Impact of video recordings review with structured debriefings on trauma team performance: a prospective observational cohort study

PURPOSE

To determine the impact of structured debriefings (SD) with audio/video review of trauma patients' resuscitation events on trauma team (TT) technical and non-technical skills.

METHODS

Single-center prospective observational cohort study. The study included all emergency department patients aged 18 years or older who received resuscitation from the TT. Virtual meeting was held with the TT using SD to review one trauma patient resuscitation video. Technical skills improvement was based on adherence to the ATLS protocol and non-technical skills based on T-NOTECHS scale.

RESULTS

There was statistically significant improvement in adherence to the ATLS protocol: 73% [55-82%] vs 91% [82-100%] (p < 0.001); and improvement in T-NOTECHS scale: 12 [10-14] vs 16 [14-19] points (p < 0.001).

CONCLUSION

In this study, we found that structured debriefings with review of patients' resuscitation video recordings can have a significant positive impact on trauma team performance in the emergency department in both technical and non-technical skills.

Reliability of the safety threats and adverse events in trauma (STAT) taxonomy using trauma video review

PURPOSE

The STAT (Safety Threats and Adverse Events in Trauma) taxonomy was developed through expert consensus, and groups 65 identified trauma resuscitation adverse events (AEs) into nine distinct categories. It provides a framework for standardized analysis of trauma resuscitations and creates a foundation for targeted quality improvement and patient safety initiatives. This study aims to evaluate the reliability of the STAT taxonomy in identifying AEs during video-recorded trauma resuscitations.

METHODS

High-definition audiovisual data from 30 trauma resuscitations were reviewed. Videos were assessed and scored by four independent reviewers (two trainees and two staff). The STAT taxonomy was used to identify AEs based on binary responses: yes and no. Inter-rater reliability was calculated using Gwet's AC1. The frequencies of AEs were tallied and reported as counts and percentages.

RESULTS

The most common AEs identified in the videos were failure to measure temperature (86.7%) and inadequate personal protective equipment (86.7%), followed by inability to use closed-loop communication (76.7%). The agreement on all AEs between reviewers was 0.94 (95% CI: 0.93-0.95). The Gwet's AC1 agreement across the 9 AE categories was paramedic handover (0.82), airway and breathing (0.99), circulation (0.95), assessment of injuries (0.91), management of injuries (0.96), procedure-related (0.97), patient monitoring and IV access (0.99), disposition (0.98), team communication and dynamics (0.87).

CONCLUSION

The STAT taxonomy demonstrated excellent inter-rater reliability between reviewers and can be used to identify AEs in video-recorded trauma resuscitations. These results provide a foundation for adapting video review to objectively quantify and assess AEs in the trauma bay.

[In situ simulation training of medical-surgical teams in the management of severe trauma patients]

In the management of severe trauma, the aim is to assess the patient's clinical stability as quickly as possible, enabling referral to imaging (whole-body CT scan, embolization if necessary) or the operating room, or even the decision to perform in situ surgery (resuscitation thoracotomy). To cope with these critical situations, team training is essential, with the aim of ensuring the reproducibility of the difficulties encountered. High-fidelity in situ simulation is the ideal tool for meeting this training challenge.

Outcomes and potential for improvement in the prehospital treatment of penetrating chest injuries in a European metropolitan area: A retrospective analysis of 2009 - 2017

BACKGROUND

Trauma is the leading cause of death in patients <45 years living in high-resource settings. However, penetrating chest injuries are still relatively rare in Europe - with an upwards trend. These cases are of particular interest to emergency medical services (EMS) due to available invasive treatment options like chest tube placement or resuscitative thoracotomy. To date, there is no sufficient data from Austria regarding penetrating chest trauma in a metropolitan area, and no reliable source to base decisions regarding further skill proficiency training on.

METHODS

For this retrospective observational study, we screened all trauma emergency responses of the Viennese EMS between 01/2009 and 12/2017 and included all those with a National Advisory Committee for Aeronautics (NACA) score ≥ IV (= potentially life-threatening). Data were derived from EMS mission documentations and hospital files, and for those cases with the injuries leading to cardiopulmonary resuscitation (CPR), we assessed the EMS cardiac arrest registry and consulted a forensic physician.

RESULTS

We included 480 cases of penetrating chest injuries of NACA IV-VII (83% male, 64% > 30 years old, 74% stab wounds, 16% cuts, 8% gunshot wounds, 56% inflicted by another party, 26% self-inflicted, 18% unknown). In the study period, the incidence rose from 1.4/100,000 to 3.5/100,000 capita, and overall, about one case was treated per week. In the cases with especially severe injury patterns (= NACA V-VII, 43% of total), (tension-)pneumothorax was the most common injury (29%). The highest mortality was seen in injuries to pulmonary vessels (100%) or the heart (94%). Fifty-eight patients (12% of total) deceased, whereas in 15 cases, the forensic physician stated survival could theoretically have been possible. However, only five of these CPR patients received at least unilateral thoracostomy. Regarding all penetrating chest injuries, thoracostomy had only been performed in eight patients.

CONCLUSIONS

Severe cases of penetrating chest trauma are rare in Vienna and happened about once a week between 2009 and 2017. Both incidence and case load increased over the years, and potentially life-saving invasive procedures were only reluctantly applied. Therefore, a structured educational and skill retention approach aimed at both paramedics and emergency physicians should be implemented.

TRIAL REGISTRATION

Retrospective analysis without intervention.

Evaluation of video review tools for assessing non-technical skills in emergency department resuscitation teams: a systematic review

BACKGROUND AND IMPORTANCE

Use of video review in medicine is established in contexts such as surgery. Although not widely used in the emergency department (ED), some centres use it to evaluate non-technical skills (NTS) to support teaching and quality improvement.

OBJECTIVE

There is no consensus on assessment of NTS using video review in the ED and the purpose of this review was to identify tools used in this context.

DESIGN, SETTING AND PARTICIPANTS

Studies were identified using Embase, Medline, CINAHL and Google Scholar. Inclusion criterion for the review was NTS of resuscitation teams working within the ED were assessed using video review. A systematic search method was used, and results were synthesised after search criteria was checked by two independent reviewers. Authors settled on the same 9 studies eligible for inclusion.

OUTCOME MEASURES AND ANALYSIS

Reliability and validity of tools identified for use in this context. Due to the heterogeneity of studies, no meta-analysis occurred.

MAIN RESULTS

There are 9 studies included in the review. The review was registered with PROSPERO (Ref No: CRD42022306129). Four unique tools were identified - 6 studies used T-NOTECHS, 1 used TTCA-24, 1 used CALM and 1 used the Communication tool. T-NOTECHS is validated in the literature for use in this context.

CONCLUSION

T-NOTECHS is the tool of choice for assessing ED teams in this context.

Reliability and validity testing of team emergency assessment measure in a distributed team context

Medical multi-professional teams are increasingly collaborating via telemedicine. In distributed team settings, members are geographically separated and collaborate through technology. Developing improved training strategies for distributed teams and finding appropriate instruments to assess team performance is necessary. The Team Emergency Assessment Measure (TEAM), an instrument validated in traditional collocated acute-care settings, was tested for validity and reliability in this study when used for distributed teams. Three raters assessed video recordings of simulated team training scenarios (n = 18) among teams with varying levels of proficiency working with a remotely located physician via telemedicine. Inter-rater reliability, determined by intraclass correlation, was 0.74-0.92 on the TEAM instrument's three domains of leadership, teamwork, and task management. Internal consistency (Cronbach's alpha) ranged between 0.89-0.97 for the various domains. Predictive validity was established by comparing scores with proficiency levels. Finally, concurrent validity was established by high correlations, >0.92, between scores in the three TEAM domains and the teams' overall performance. Our results indicate that TEAM can be used in distributed acute-care team settings and consequently applied in future-directed learning and research on distributed healthcare teams.

Closed loop communication in the trauma bay: identifying opportunities for team performance improvement through a video review analysis

Objectives

Communication among trauma team members in the trauma bay is vulnerable to errors, which may impact patient outcomes. We used the previously validated trauma-non-technical skills (T-NOTECHS) tool to identify communication gaps during patient management in the trauma bay and to inform development strategies to improve team performance.

Methods

Two reviewers independently assessed non-technical skills of team members through video footage at Sunnybrook Health Sciences Centre. Team performance was measured using T-NOTECHS across five domains using a five-point Likert scale (lower score indicating worse performance): (1) leadership; (2) cooperation and resource management; (3) communication and interaction; (4) assessment and decision making; (5) situation awareness/coping with stress. Secondary outcomes assessed the number of callouts, closed loop communications and parallel conversations.

Results

The study included 55 trauma activations. Injury severity score (ISS) was used as a measure of trauma severity. A case with an ISS score ≥ 16 was considered severe. ISS was ≥ 16 in 37% of cases. Communication and interaction scored significantly lower compared to all other domains (p < 0.0001). There were significantly more callouts and completed closed loop communications in more severe cases compared to less severe cases (p = 0.017 for both). Incomplete closed loop communications and parallel conversations were identified, irrespective of case severity.

Conclusion

A lower communication score was identified using T-NOTECHS, attributed to incomplete closed loop communications and parallel conversations. Through video review of trauma team activations, opportunities for improvement in communication can be identified by the T-NOTECHS tool, as well as specifically identifying callouts and closed loop communication. This process may be useful for trauma programs as part of a quality improvement program on communication skills and team performance.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43678-022-00295-z.