Assessment of non-technical skills: why aren’t we there yet?

Nontechnical Skills for Intraoperative Team Members

Nontechnical skills, defined as the set of cognitive and social skills used by individuals and teams to reduce error and improve performance in complex systems, have become increasingly recognized as a key contributor to patient safety. Efforts to characterize, quantify, and teach nontechnical skills in the context of perioperative care continue to evolve. This review article summarizes the essential behaviors for safety, described in taxonomies for nontechnical skills assessments developed for intraoperative clinical team members (eg, surgeons, anesthesiologists, scrub practitioners, perfusionists). Furthermore, the authors describe emerging methods to advance understanding of the impact of nontechnical skills on perioperative outcomes.

Which, how, and what? Using digital tools to train surgical skills; a systematic review and meta-analysis

Background

Digital tools like digital box trainers and VR seem promising in delivering safe and tailored practice opportunities outside of the surgical clinic, yet understanding their efficacy and limitations is essential. This study investigated Which digital tools are available to train surgical skills, How these tools are used, How effective they are, and What skills they are intended to teach.

Methods

Medline, Embase, and Cochrane libraries were systematically reviewed for randomized trials, evaluating digital skill-training tools based on objective outcomes (skills scores and completion time) in surgical residents. Digital tools effectiveness were compared against controls, wet/dry lab training, and other digital tools. Tool and training factors subgroups were analysed, and studies were assessed on their primary outcomes: technical and/or non-technical.

Results

The 33 included studies involved 927 residents and six digital tools; digital box trainers, (immersive) virtual reality (VR) trainers, robot surgery trainers, coaching and feedback, and serious games. Digital tools outperformed controls in skill scores (SMD 1.66 [1.06, 2.25], P < 0.00001, I2 = 83 %) and completion time (SMD -1.05 [-1.72, -0.38], P = 0.0001, I2 = 71 %). There were no significant differences between digital tools and lab training, between tools, or in other subgroups. Only two studies focussed on non-technical skills.

Conclusion

While the efficacy of digital tools in enhancing technical surgical skills is evident - especially for VR-trainers -, there is a lack of evidence regarding non-technical skills, and need to improve methodological robustness of research on new (digital) tools before they are implemented in curricula.

Key message

This study provides critical insight into the increasing presence of digital tools in surgical training, demonstrating their usefulness while identifying current challenges, especially regarding methodological robustness and inattention to non-technical skills.

A tool to assess nontechnical skills of perfusionists in the cardiac operating room

OBJECTIVES

This study aimed to develop the Perfusionists' Intraoperative Non-Technical Skills tool, specifically to the perfusionists' context, and test its inter-rater reliability.

METHODS

An expert panel was convened to review existing surgical nontechnical skills taxonomies and develop the Perfusionists' Intraoperative Non-Technical Skills tool. During a workshop held at a national meeting, perfusionists completed the Perfusionists' Intraoperative Non-Technical Skills ratings after watching 4 videos displaying simulated cardiac operations. Two videos showed "good performance," and 2 videos showed "poor performance." Inter-rater reliability analysis was performed and intraclass correlation coefficient was reported.

RESULTS

The final version of the Perfusionists' Intraoperative Non-Technical Skills taxonomy contains 4 behavioral categories (decision making, situation awareness, task management and leadership, teamwork and communication) with 4 behavioral elements each. Categories and elements are rated using an 8-point Likert scale ranging from 0.5 to 4.0. A total of 60 perfusionist raters were included and the comparison between rating distribution on "poor performance" and "good performance" videos yielded a statistically significant difference between groups, with a P value less than .001. A similar difference was found in all behavioral categories and elements. Reliability analysis showed moderate inter-rater reliability across overall ratings (intraclass correlation coefficient, 0.735; 95% confidence interval, 0.674-0.796; P < .001). Similar inter-rater reliability was found when raters were stratified by experience level.

CONCLUSIONS

The Perfusionists' Intraoperative Non-Technical Skills tool presented moderate inter-rater reliability among perfusionists with varied levels of experience. This tool can be used to train and assess perfusionists in relevant nontechnical skills, with the potential to enhance safety and improve surgical outcomes.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

Interprofessional teamwork before and after organizational change in a tertiary emergency department: An observational study

In healthcare settings, suboptimal interprofessional teamwork and communication contribute to unsafe care and avoidable harm. Interprofessional teamwork is essential in high-risk clinical areas such as the emergency department (ED). The aims of this study were to describe interprofessional teamwork in a hospital ED and to evaluate factors influencing interprofessional communication before and after implementation of a department-wide multifaceted intervention. Structured observations were undertaken during 2015/16 and 2019. Differences in interprofessional communication practices, teamwork, and sources of interruptions were compared before and after the intervention. The following domains were surveilled: (a) healthcare professionals (HCPs) communication initiatives, (b) HCPs' contribution to patient assessment, (c) interprofessional communication processes, and (d) team interruptions. The intervention included strategies to enable use of communication tools, changes to team structures, changes in work environment, ethical principles, and establishment of a code of professional conduct during interprofessional communication. Team interruptions significantly decreased post-intervention, and our findings suggest that organizational changes affect domains of teamwork. Statistically significant differences were observed in the initiated communication pre-intervention and contribution to patient assessment significantly increased post-intervention. Multifaceted organizational interventions can positively affect interprofessional team communication and work-flow in the ED, thus patient safety and quality of care can be improved.

Knowledge and Awareness of Non-Technical Skills Over the Course of an Educational Program in Nursing - A Repeated Cross-Sectional Study

BACKGROUND

Non-technical skills (NTS) play an important role in preventing adverse events during hospitalization. Knowledge, awareness and mastery of NTS becomes important key factors in preventing errors. Current status of students and supervisor's knowledge and awareness of NTS are needed in order to construct an educational plan for improvement.

PURPOSE

To examine knowledge and awareness of NTS over the course of continuing education of nurse anaesthetists, emergency care nurses, critical care nurses, and operating room nurses.

METHODS

A descriptive, repetitive cross-sectional design with a questionnaire was used to evaluate knowledge and awareness in students and their supervisors about NTS at two different time points during the educational program. Cross tabulations were used in comparisons across specialties and between students and supervisors, frequencies to identify the levels of self-reported knowledge/importance/focus in clinical practice/ impact on adverse events.

RESULTS

The results showed that there was a numeric difference between the reported knowledge/focus in clinical practice on the one hand and importance/ impact on adverse events on the other, and that this gap was reduced after 12 months of education with special focus on NTS. There was no difference across specialties. Supervisors had higher focus on NTS in clinical practice and on the impact on adverse events, than students at both measurements.

CONCLUSION

These data suggest that NTS may have important potential for improvement if included into learning programs both in education and clinical practice. Integration of NTS in various learning activities seems to strengthen students' competence about NTS.

Perceived quality of surgical care in association with patient-related factors and correlation to reported postoperative complications in Finland: a cross-sectional study

OBJECTIVE

To study if patient-related factors are associated with patient-evaluated quality of care in surgery. To examine if there is an association with postoperative complications and patient-evaluated low quality of care.

DESIGN

A correlation cross-sectional study, in addition, a phone call interview at 30 days postoperatively to examine complications.

SETTING

The data on patients admitted for non-cardiac general and orthopaedic surgery at a central hospital in Southwestern Finland were collected in two phases during an 8-month period.

PARTICIPANTS

436 consecutive consenting and eligible in-ward non-cardiac general surgery and orthopaedic surgery adult patients. Ambulatory, paediatric and memory disorder patients were excluded. 378 patients completed the questionnaire (Good Nursing Care Scale for Patients (GNCS-P)).

METHODS

Perceived quality of care was examined by the GNCS-P questionnaire. Patient-related factors were obtained from electronic patient records and questionnaire. A telephone interview related to postdischarge complications was conducted 30 days after discharge.

MAIN OUTCOME MEASURES

Patient evaluation of quality of care at discharge, its association with patient-related factors and patient-reported postdischarge complications.

RESULTS

The overall quality was evaluated high or very high by the patients. The lowest overall quality of care rate was assessed by surgical patients living alone (p=0.0088) and patients who evaluated their state of health moderate or poor (p=0.0047). Surgical patients reporting postoperative complications after discharge evaluated lower overall quality of care (p=0.0105) than patients with no complications.

CONCLUSION

Patient demographic factors do not seem to influence the perceptions of the quality of care. Instead, subjective state of health and living conditions (living alone) may have an influence on the patient experience of quality of care. The perceived quality of care in healthcare staff technical and communication skills may have an association with reported postoperative complications.

Simulation past, present and future—a decade of progress in simulation-based education in the UK

This review will present developments in simulation-based education (SBE) over the past decade with a focus on activity in the UK’s National Health Service and the role of the national society (the Association for Simulation Practice in Healthcare). The article covers the evolution of strategic changes for the use of SBE in the UK and the operational challenges faced by clinicians and other faculty in SBE. The expansion of the evidence base to support SBE in healthcare both in technical skills and, more broadly, in interprofessional team training is explored. Finally, the wider role of simulation in patient safety and healthcare systems, including testing pathways and the development of cognitive aids and involvement of patients in SBE is considered.

Exploring transformative learning when developing medical students' non-technical skills

CONTEXT

Non-technical skills (NTS) training should be incorporated into medical students' education and simulation-based approaches are often utilised to facilitate this. Such experiences have the potential to foster transformative learning by facilitating a reassessment of one's prior assumptions and a significant shift in one's outlook, referred to as the process of perspective transformation. The aim of this research was to explore how NTS training might facilitate transformative learning in final-year medical students.

METHODS

Following ethical approval, medical student volunteers from four medical schools (Aberdeen, Dundee, Edinburgh and Glasgow) participated in simulation sessions, were debriefed with an emphasis on NTS using a behavioural marker system and then took part in focus groups. Focus group discussions were semi-structured and questions were based on the phases of perspective transformation identified by Jack Mezirow. Focus group discussions were audiorecorded, transcribed verbatim, anonymised and analysed using template analysis.

RESULTS

A total of 33 medical students took part in five focus groups. There was evidence of the following stages of perspective transformation: Phase 2 (self-examination with emotional disturbance, including fear, anxiety, guilt, shame and frustration); Phase 3 (critical assessment of assumptions, including the undervaluing of NTS, recognising that technical skills alone are insufficient, and recognising that it is possible to improve one's NTS); Phase 5 (exploring options for new roles, relationships and actions), and Phase 6 (planning a course of action for future simulations, as a medical student and as a doctor).

CONCLUSIONS

This study deepens our understanding of how exposure to NTS training in simulation-based education influences the learning of medical students and shows that such exposure can result in the cognitive phases of transformative learning. It provides us with valuable insights into medical students' perspectives on their learning of NTS at a pivotal stage in training and represents an interesting way of assessing the educational impact of such sessions.

Smartphone to-do list application to improve workflow in an intensive care unit: A superiority quasi-experimental study

BACKGROUND

Smartphone to-do list app was hypothesized to be more efficient than a paper-based list in the management of workflow and to provide additional benefits.

PURPOSE

To analyze the impact of a mobile task-management application on the workflow of an ICU medical staff.

METHODS

Superiority by a margin test, quasi-experimental study comparing the use of a smartphone application versus standard practice regarding tasks management in an academic ICU. Superiority margin was set at 8 % based on a pilot study. During two periods of 20 working days each (October 2018 and January 2019), medical staff managed tasks with both methods on a weekly rotation basis. Primary outcome was the proportion of daily tasks completed. Secondary outcomes assessed users' satisfaction and the impact of the app in terms of changes in clinical practice.

RESULTS

25 ICU physicians were enrolled. A total of 1983 tasks were recorded. The proportion of completed tasks per day was higher when using the smartphone app (99 % [96-100] versus 95 % [93-98] for the standard group, p = 0.006), but did not reach the superiority margin. Smartphone application was perceived as positive experience, as participants felt that they forgot fewer tasks (p = 0.02), were more aware of their progress on ongoing or remaining tasks (p = 0.03) and observed an improvement in communication among the medical staff (p = 0.03).

CONCLUSION

This study failed to demonstrate the superiority of a smartphone app over paper-based lists regarding the proportion of daily tasks completed. However, positive feedback regarding the application was received from the medical staff.