Evaluating structured assessment of anaesthesiologists' non-technical skills

Social and Cognitive Skills (SCOPE)-a generic model for multi-professional work and education in healthcare

In this article, we present a generic model for social and cognitive skills that can be used in work and (simulation-based) education in healthcare. We combined existing non-technical skills tools into a tool that we call SCOPE. SCOPE is a model that comprises the three social categories of "teamwork", "leading", and "task management" as well as the two cognitive categories of "situation awareness" and "decision making". Each category comprises between three and six elements. We formulated guiding questions for each category in an attempt to emphasize its core meaning. We developed a dynamic graphical representation of the categories that emphasize the constant changes in the relative importance of the categories over the course of a clinical or educational situation. Anecdotal evidence supports the value of the model for aligning language around social and cognitive skills across specialties and professions.

An interview study about how nurses and physicians talk about the same concepts differently

BACKGROUND

How healthcare professionals understand and use concepts of social and cognitive capabilities will influence their behaviour and their understanding of others' behaviour. Differing understandings of concepts might lead to healthcare professionals not acting in accordance with other healthcare professionals' expectations. Therefore, part of the problem concerning errors and adverse incidents concerning social and cognitive capabilities might be due to varying understandings of concepts among different healthcare professionals. This study aimed to examine the variations in how educators at the Copenhagen Academy for Medical Education and Simulation talk about social and cognitive capabilities.

METHODS

The study was conducted using semi-structured interviews and directed content analysis. The codes for the analysis process were derived from existing non-technical skills models and used to show variations in how the participants talk about the same concepts.

RESULTS

Educators with a background as nurses and physicians, talked differently about leadership and decision-making, with the nurses paying greater attention to group dynamics and external factors when describing both leadership and decision-making, whereas physicians focus on their individual efforts.

CONCLUSION

We found patterned differences in how the participants described leadership and decision-making that may be related to participants' professional training/background. As it can create misunderstandings and unsafe situations if nurses and physicians disagree on the meaning of leadership and decision-making (without necessarily recognising this difference), it could be beneficial to educate healthcare professionals to be aware of the specificity of their own concepts, and to communicate what exactly they mean by using a particular concept, e.g. "I want you to coordinate tasks" instead of "I want better leadership".

Collecting evidence of validity for an assessment tool for Norwegian medical students' non-technical skills (NorMS-NTS): usability and reliability when used by novice raters

BACKGROUND

The NorMS-NTS tool is an assessment tool for assessing Norwegian medical students' non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters.

METHODS

This study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools' internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool's usability.

RESULTS

The ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77-0.91. Cronbach's alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching.

CONCLUSIONS

The observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students' non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

An observational study to assess circulating nurses' non-technical skills

BACKGROUND

Non-technical skills are cognitive and social skills that are necessary for safe and efficient practices in operating theatre. These skills are inseparable parts of circulating nurses' duties and help them have a good performance during surgical processes. Circulating nurses' non-technical skills have not been closely measured, and much uncertainty still exists about the quality of their skills. This study aimed to assess circulating nurses' non-technical skills.

METHODS

A cross-sectional study was conducted on 300 circulating nurses who worked in four public hospitals in Iran, during 2020. The data of demographics of the studied population were collected using a demographic questionnaire. Kalantari et al's Circulating Practitioner's List of Non-Technical Skills was used to assess the circulating nurses' non-technical skills.

RESULTS

The situational awareness domain had the highest score, while leadership was the domain with the lowest mean score. There was a moderate positive relationship between the mean score of non-technical skills and the number of working years as a circulating nurse.

CONCLUSION

Although the circulating nurses had a moderate level of non-technical skills, they had low scores in several behaviours. However, they did well in some other behaviours. Educational interventions and policymaking solutions can help improve circulating nurses' non-technical skills.

Instruments to evaluate non-technical skills during high fidelity simulation: A systematic review

INTRODUCTION

High Fidelity Simulations (HFS) are increasingly used to develop Non-Technical Skills (NTS) in healthcare providers, medical and nursing students. Instruments to measure NTS are needed to evaluate the healthcare providers' (HCPs) performance during HFS. The aim of this systematic review is to describe the domains, items, characteristics and psychometric properties of instruments devised to evaluate the NTS of HCPs during HFS.

METHODS

A systematic review of the literature was performed according to the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA). Studies were retrieved from PubMed, Cinahl, Web of Science, Cochrane Library, ProQuest and PubPsych. Studies evaluating the measurement properties of instruments used to assess NTS during HFS training were included. Pairs of independent reviewers determined the eligibility, extracted and evaluated the data. Risk of bias and appraisal of the methodological quality of the studies was assessed using the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) checklist, and the quality of the evidence with the Grading of Recommendations, Assessment, Development and Evaluation (GRADE).

RESULTS

A total of 3,953 articles were screened. A total of 110 reports were assessed for eligibility and 26 studies were included. Studies were conducted in Europe/United Kingdom (n = 13; 50%), North America/Australia (n = 12; 46%) and Thailand (n = 1; 4%). The NTS instruments reported in this review included from 1 to 14 domains (median of 4, Q1 = 3.75, Q3 = 5) and from 3 to 63 items (median of 15, Q1 = 10, Q3 = 19.75). Out of 19 NTS assessment instruments for HFS, the Team Emergency Assessment Measure (TEAM) can be recommended for use to assess NTS. All the other instruments require further research to assess their quality in order to be recommended for use during HFS training. Eight NTS instruments had a positive overall rating of their content validity with at least a moderate quality of evidence.

CONCLUSION

Among a large variety of published instruments, TEAM can be recommended for use to assess NTS during HFS. Evidence is still limited on essential aspects of validity and reliability of all the other NTS instruments included in this review. Further research is warranted to establish their performance in order to be reliably used for HFS.

Impact of interprofessional in situ simulations on acute pediatric burn management: Combining technical and non-technical burn team skills

OBJECTIVE

This study aimed to evaluate the impact of interprofessional in situ simulations on the technical and non-technical skills of pediatric burn teams in acute burn management.

METHODS

This quasi-experimental study consisted of a one-group pre- and post-test design conducted in a pediatric burn center in Turkey. The sample consisted of nine interprofessional burn team members. Data collection tools consisted of the following: descriptive data form, burn technical skills checklists, simulation evaluation form, and Anesthesiologists' non-technical skills in Denmark rating form.

RESULTS

We found no statistically significant difference between the pre- and post-test scores for technical (p = 0.285) and non-technical skill (p = 0.180) scores. Burn team members evaluated the highest score in almost all criteria for in situ simulations.

CONCLUSION

The interprofessional in situ simulations did not improve the burn teams' acute burn management; however, according to a self-report, burn team members were satisfied with the interprofessional in situ simulation experiences and achieved their own gains.

Psychometric Testing of a Structured Assessment Instrument for Non-technical Skills (NANTS-no) for Use in Clinical Supervision of Student Nurse Anesthetists

BACKGROUND

This study evaluated psychometric properties of a structured behavioral assessment instrument, Nurse Anaesthetists' Non-Technical Skills-Norway (NANTS-no). It estimated whether reliable assessments of nontechnical skills (NTS) could be made after taking part in a workshop. An additional objective was to evaluate the instrument's acceptability and usability.

METHODS

An explorative design was used. Nurse anesthetists (n = 46) involved in clinical supervision attended a 6-hour workshop on NTS, then rated NTS in video-recorded simulated scenarios and completed a questionnaire.

RESULTS

High reliability and dependability were estimated in this setting. Participants regarded the instrument as useful for clinical supervision of student nurse anesthetists (SNAs).

CONCLUSIONS

Findings suggest that NANTS-no may be reliable for performing clinical assessments of SNAs and encouraging critical reflection. However, further research is needed to explore its use in clinical settings.

Priming healthcare students on the importance of non-technical skills in healthcare: How to set up a medical escape room game experience

Background: Non-technical skills (NTS) are essential for healthcare professionals. Earlier the students are made aware of this, the more time they have to acquire these skills. Escape rooms have been introduced in the medical literature but a detailed published manual on setting up and running such a learning modality is lacking. The purpose of this paper is to describe the use of an escape room to create learning opportunities, including detailed instructions, as well as an evaluation from two settings. Methods: A medical escape room focusing on NTS was developed and run in two different settings: a university summer school - EMSS17, and an international healthcare congress - EMS2018. Questionnaire data investigating entertainment value, self-evaluated use of NTS and ideas for further use of the escape room concept was obtained. Further, video analyses were conducted to triangulate self-rated analyses. Results: Majority of the participants found the escape room psychologically safe and enjoyable and would recommend the concept to other students and healthcare professionals. Video analyses showed the same tendencies regarding the use of NTS as reported by participants. Conclusions: This paper presents a fully applicable escape room manual, ready to implement, adapt, and modify. Evaluation data support the proof of concept.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Measuring non-technical skills of anaesthesiologists in the operating room: a systematic review of assessment tools and their measurement properties

BACKGROUND

Non-technical skills, such as communication or leadership, are integral to clinical competence in anaesthesia. There is a need for valid and reliable tools to measure anaesthetists' non-technical performance for both initial and continuing professional development. This systematic review aims to summarise the measurement properties of existing assessment tools to determine which tool is most robust.

METHODS

Embase (via OVID), Medline and Medline in Process (via OVID), and reference lists of included studies and previously published relevant systematic reviews were searched (through August 2017). Quantitative studies investigating the measurement properties of tools used to assess anaesthetists' intraoperative non-technical skills, either in a clinical or simulated environment, were included. Pairs of independent reviewers determined eligibility and extracted data. Risk of bias was assessed using the COSMIN checklist.

RESULTS

The search yielded 978 studies, of which 14 studies describing seven tools met the inclusion criteria. Of these, 12 involved simulated crisis settings only. The measurement properties of the Anaesthetists' Non-Technical Skills (ANTS) tool were most commonly assessed (n=9 studies), with studies of two types of validity (content, concurrent) and two types of reliability (internal consistency, interrater). Most of these studies, however, were at serious risk of bias.

CONCLUSIONS

Though there are seven tools for assessing the non-technical skills of anaesthetists, only ANTS has been extensively investigated with regard to its measurement properties. ANTS appears to have acceptable validity and reliability for assessing non-technical skills of anaesthetists in both simulated and clinical settings. Future research should consider additional clinical contexts and types of measurement properties.

The impact of fatigue on the non-technical skills performance of critical care air ambulance clinicians

BACKGROUND

The relationship between fatigue-related risk and impaired clinical performance is not entirely clear. Non-technical factors represent an important component of clinical performance and may be sensitive to the effects of fatigue. The hypothesis was that the sum score of overall non-technical performance is degraded by fatigue.

METHODS

Nineteen physicians undertook two different simulated air ambulance missions, once when rested, and once when fatigued (randomised crossover design). Trained assessors blinded to participants' fatigue status performed detailed structured assessments based on expected behaviours in four non-technical skills domains: teamwork, situational awareness, task management, and decision making. Participants also provided self-ratings of their performance. The primary endpoint was the sum score of overall non-technical performance.

RESULTS

The main finding, the overall non-technical skills performance rating of the clinicians, was better in rested than fatigued states (mean difference with 95% CI, 2.8 [2.2-3.4]). The findings remained consistent across individual non-technical skills domains; also when controlling for an order effect and examining the impact of a number of possible covariates. There was no difference in self-ratings of clinical performance between rested and fatigued states.

CONCLUSION

Non-technical performance of critical care air transfer clinicians is degraded when they are fatigued. Fatigued clinicians may fail to recognise the degree to which their performance is compromised. These findings represent risk to clinical care quality and patient safety in the dynamic and isolated environment of air ambulance transfer.