Evaluation of the prototype Anaesthetic Non-technical Skills for Anaesthetic Practitioners (ANTS-AP) system: a behavioural rating system to assess the non-technical skills used by staff assisting the anaesthetist

Social and Cognitive Skills (SCOPE)-a generic model for multi-professional work and education in healthcare

In this article, we present a generic model for social and cognitive skills that can be used in work and (simulation-based) education in healthcare. We combined existing non-technical skills tools into a tool that we call SCOPE. SCOPE is a model that comprises the three social categories of "teamwork", "leading", and "task management" as well as the two cognitive categories of "situation awareness" and "decision making". Each category comprises between three and six elements. We formulated guiding questions for each category in an attempt to emphasize its core meaning. We developed a dynamic graphical representation of the categories that emphasize the constant changes in the relative importance of the categories over the course of a clinical or educational situation. Anecdotal evidence supports the value of the model for aligning language around social and cognitive skills across specialties and professions.

Collecting evidence of validity for an assessment tool for Norwegian medical students' non-technical skills (NorMS-NTS): usability and reliability when used by novice raters

BACKGROUND

The NorMS-NTS tool is an assessment tool for assessing Norwegian medical students' non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters.

METHODS

This study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools' internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool's usability.

RESULTS

The ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77-0.91. Cronbach's alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching.

CONCLUSIONS

The observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students' non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

Developing a Prototype Behavioural Marker System for Farmer Non-Technical Skills (FLINTS)

OBJECTIVE

Non-technical skills, the social and cognitive skills thought necessary for safe and effective working, have been studied within the farming context over the past six years. However, these skills are not yet taught as part of a safety curriculum for farmers, due, in part, to a lack of defined framework and assessment system. The current paper describes the development of the FLINTS behavioral marker system for discussion, observation, evaluation and feedback on non-technical skills for farmers.

METHOD

The development of the behavioral marker system proceeded through three key stages. First, the current research knowledge on non-technical skills was synthesized to compile a list of non-technical skill categories and elements. Second, a series of discussion groups with subject matter experts was conducted to develop behavioral markers for each element. Lastly, refinement and review of the system was undertaken by academics and experts.

RESULTS

The prototype FLINTS taxonomy containing five non-technical skill categories and 16 elements was produced. The non-technical skill categories comprised situation awareness, teamwork & communication, leadership, task management and decision-making each with specific elements and behavioral markers.

CONCLUSION

FLINTS represents the first behavioral marker system for farmer non-technical skills, constructed through expert knowledge and advice via discussion and review groups, combined with underpinning research findings. This represents the first step towards the development of non-technical training and assessment for farmers. The current version of the FLINTS system is freely available to all potential users (https://research.abdn.ac.uk/nts-farming/flints/).

Human factors in anaesthesia: a narrative review

Healthcare relies on high levels of human performance, as described by the 'human as the hero' concept. However, human performance varies and is recognised to fall in high-pressure situations, meaning that it is not a reliable method of ensuring safety. Other safety-critical industries embed human factors principles into all aspects of their organisations to improve safety and reduce reliance on exceptional human performance; there is potential to do the same in anaesthesia. Human factors is a broad-based scientific discipline which aims to make it as easy as possible for workers to do things correctly. The human factors strategies most likely to be effective are those which 'design out' the chance of an error or adverse event occurring. When errors or adverse events do happen, barriers are in place to trap them and reduce the risk of progression to patient and/or worker harm. If errors or adverse events are not trapped by these barriers, mitigations are in place to minimise the consequences. Non-technical skills form an important part of human factors barriers and mitigation strategies and include: situation awareness; decision-making; task management; and team working. Human factors principles are not a substitute for proper investment and appropriate staffing levels. Although applying human factors science has the potential to save money in the long term, its proper implementation may require investment before reward can be reaped. This narrative review describes what is known about human factors in anaesthesia to date.

Implementing human factors in anaesthesia: guidance for clinicians, departments and hospitals: Guidelines from the Difficult Airway Society and the Association of Anaesthetists: Guidelines from the Difficult Airway Society and the Association of Anaesthetists

Human factors is an evidence-based scientific discipline used in safety critical industries to improve safety and worker well-being. The implementation of human factors strategies in anaesthesia has the potential to reduce the reliance on exceptional personal and team performance to provide safe and high-quality patient care. To encourage the adoption of human factors science in anaesthesia, the Difficult Airway Society and the Association of Anaesthetists established a Working Party, including anaesthetists and operating theatre team members with human factors expertise and/or interest, plus a human factors scientist, an industrial psychologist and an experimental psychologist/implementation scientist. A three-stage Delphi process was used to formulate a set of 12 recommendations: these are described using a 'hierarchy of controls' model and classified into design, barriers, mitigations and education and training strategies. Although most anaesthetic knowledge of human factors concerns non-technical skills, such as teamwork and communication, human factors is a broad-based scientific discipline with many other additional aspects that are just as important. Indeed, the human factors strategies most likely to have the greatest impact are those related to the design of safe working environments, equipment and systems. While our recommendations are primarily provided for anaesthetists and the teams they work with, there are likely to be lessons for others working in healthcare beyond the speciality of anaesthesia.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

The impact of unsuccessful resuscitation and manikin death during simulation on nursing student's resuscitation self-efficacy: A quasi-experimental study

BACKGROUND

There has been considerable debate about whether it is appropriate to let the manikin die during simulation teaching. Simulations are used in high-risk industries to recreate rare and potentially catastrophic events. In healthcare, there has been a reluctance to allow scenarios to progress to a catastrophe because of the potential impact on the individual if they were to then encounter the situation in real life. In healthcare, witnessed resuscitation has an overall success rate of around 23.9 %, therefore making every simulation situation successful results in an altered perception of reality.

OBJECTIVE

The researchers aimed to examine whether the manikin's death during a simulation adversely affects the resuscitation self-efficacy of nursing students.

DESIGN

Quasi-experimental design.

SETTING

Univesity of Sunderland, United Kingdom.

PARTICIPANTS

Students were invited to participate (n = 120) and 106 consented to take part in the study.

METHODS

A pre-and post-test of the nursing student's self-efficacy during a resuscitation scenario. The scenario related to a patient admitted to the emergency room with chest pain who then went into cardiac arrest. The experimental group's resuscitation was unsuccessful, and the control group's resuscitation was successful. Self-efficacy was measured using the validated Basic Resuscitation Skills Self-efficacy scale (BRS-SES). The data were analysed using a paired sample t-test.

RESULTS

Overall, both groups showed improved self-efficacy as a result of the simulation session and the death of the manikin in the experimental group did not result in a reduced level of self-efficacy related to resuscitation.

CONCLUSION

The death of the manikin during the simulation involving resuscitation had no impact on student resuscitation self-efficacy.

Integrative Review of Non-Technical Skills Frameworks to Apply for Air Medical Transfer of Pregnant Women

OBJECTIVE

To identify the most suitable non-technical skills framework to adapt and apply to the air medical transfer of pregnant women.

DATA SOURCES

Embase, PsycINFO, PubMed, MEDLINE, Web of Science, CINAHL, Science Direct, and Google Scholar.

STUDY SELECTION

We retrieved potentially relevant articles using a predefined combination of keywords extended with truncation and Boolean operators. Database and manual reference searches yielded 569 peer-reviewed articles. We included articles if they presented empirical data and described non-technical or cognitive competency skills frameworks for health care professionals. We discussed any ambiguities regarding inclusion, and they were resolved by consensus. We retained 71 full-text articles for final review.

DATA EXTRACTION

We coded extracted data under four criteria: non-technical skill categories, context of use, psychometric properties, and rating system. We generated descriptive summary tables of the characteristics of existing non-technical skills frameworks based on publication year, method of development, clinical setting, clinical specialty, routine/crisis-based performance, and team/individual performance.

DATA SYNTHESIS

We identified 42 non-technical skills frameworks from a variety of health care settings. We critically examined context of use and how use in various clinical settings may align with air transfers of pregnant women. Our findings illustrate the importance of team-based and routine performance rather than crisis-focused skills. Maintaining situational awareness throughout all stages of the transfer and communicating effectively with team members, the pregnant woman, and her partner are skills that are particularly important to ensure good outcomes.

CONCLUSION

We selected the Global Assessment of Obstetric Team Performance as the most suitable non-technical skills framework to adapt to the clinical setting of air medical transfer of pregnant women. We considered the clinical specialty, specific non-technical skills required in the setting, the framework's properties, and the requirement to focus on routine team performance.

Non-technical skill assessments across levels of US surgical training

BACKGROUND

To ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool.

METHODS

We undertook an exploratory, prospective cohort study of 32 residents-interns (postgraduate year 1), junior residents (postgraduate years 2-3), and senior residents (postgraduate years 4-5)-across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident's intraoperative performance. Mean NOTSS-US ratings (total range 4-20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1-5, with 1=poor, 3=average, 5=excellent) were stratified by residents' training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression.

RESULTS

For 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3-8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5-6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3-3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents' training levels persisted for individual NOTSS-US behavior categories.

CONCLUSION

These data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees' non-technical skills.

Psychometric Testing of a Structured Assessment Instrument for Non-technical Skills (NANTS-no) for Use in Clinical Supervision of Student Nurse Anesthetists

BACKGROUND

This study evaluated psychometric properties of a structured behavioral assessment instrument, Nurse Anaesthetists' Non-Technical Skills-Norway (NANTS-no). It estimated whether reliable assessments of nontechnical skills (NTS) could be made after taking part in a workshop. An additional objective was to evaluate the instrument's acceptability and usability.

METHODS

An explorative design was used. Nurse anesthetists (n = 46) involved in clinical supervision attended a 6-hour workshop on NTS, then rated NTS in video-recorded simulated scenarios and completed a questionnaire.

RESULTS

High reliability and dependability were estimated in this setting. Participants regarded the instrument as useful for clinical supervision of student nurse anesthetists (SNAs).

CONCLUSIONS

Findings suggest that NANTS-no may be reliable for performing clinical assessments of SNAs and encouraging critical reflection. However, further research is needed to explore its use in clinical settings.

Validity of the Medi-StuNTS behavioural marker system: assessing the non-technical skills of medical students during immersive simulation

Background

The Medical Students' Non-Technical Skills (Medi-StuNTS) behavioural marker system (BMS) is the first BMS to be developed specifically for medical students to facilitate training in non-technical skills (NTS) within immersive simulated acute care scenarios. In order to begin implementing the tool in practice, validity evidence must be sought. We aimed to assess the validity of the Medi-StuNTS system with reference to Messick's contemporary validity framework.

Methods

Two raters marked video-recorded performances of acute care simulation scenarios using the Medi-StuNTS system. Three groups were marked: third-year and fourth-year medical students (novices), final-year medical students (intermediates) and core medical trainees (experts). The scores were used to make assessments of relationships to the variable of clinical experience through expert-novice comparisons, inter-rater reliability, observability, exploratory factor analysis, inter-rater disagreements and differential item functioning.

Results

A significant difference was found between the three groups (p<0.005), with experts scoring significantly better than intermediates (p<0.005) and intermediates scoring significantly better than novices (p=0.001). There was a strong positive correlation between the two raters' scores (r=0.79), and an inter-rater disagreement of more than one point in less than one-fifth of cases. Across all scenarios, 99.7% of skill categories and 84% of skill elements were observable. Factor analysis demonstrated appropriate grouping of skill elements. Inconsistencies in test performance across learner groups were shown specifically in the skill categories of situation awareness and decision making and prioritisation.

Conclusion

We have demonstrated evidence for several aspects of validity of the Medi-StuNTS system when assessing medical students' NTS during immersive simulation. We can now begin to introduce this system into simulation-based education to maximise NTS training in this group.

The perioperative Team Brief: A patient safety initiative or another tick-box exercise?

The World Health Organization Surgical Safety Checklist has been the subject of many professional discussions following its introduction in 2008. Since the addition of the Team Brief and Debrief in 2010 and the acceptance of the Five Steps to Safer Surgery as the gold standard, compliance has steadily improved (as audited by Care Quality Commission Inspections). This review of the literature therefore examined the perioperative Team Brief and identified gaps in knowledge. Evidence appears to suggest that whilst compliance is good in quantitative terms, there may be inconsistencies within the quality of Team Briefs from organisation to organisation and surgeon to surgeon. Concluding further research is required to establish what an effective Team Brief looks, sounds and feels like to all involved.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Monitoring teamwork: a narrative review

A narrative review was carried out to identify articles on monitoring of teamwork, with particular relevance to anaesthetists. The papers reviewed showed that team monitoring takes place both implicitly and explicitly in the anaesthetic environment. No single optimal model of teamwork monitoring for all situations was identified. Most of the studies identified were of a pre-intervention, post-intervention design, without randomisation or control group. Information shared during a formal briefing is more likely to be recalled, and provides a basis for a shared team mental model. A number of studies appeared to show that targeted teamwork training has a positive impact on both teamwork and patient safety.

Factors that affect scrub practitioner non-technical skills: a qualitative analysis

Context

Non-technical skills are the cognitive and affective behaviours that underpin clinical practice in surgery. The purpose of this study was to explore the factors that affect how scrub practitioners use non-technical skills during surgery and their behavioural responses to those factors.

Methods

Eight scrub practitioners participated in a qualitative semi-structured interview study.

Results

Five major categories were found to affect scrub practitioner non-technical skills. These categories were democratises surgery, learns from experiences, comprehends situations, comprehends behaviours and insidious threats.

Key findings and implications:

A scrub practitioner’s ability to use non-technical skills results from the interdependent relationship of intrinsic and extrinsic motivators that affect their own behaviours, and those of their colleagues, during surgery.

Estimating the Reliability of Nontechnical Skills in Medical Teams

OBJECTIVE

To assess the reliability of measurement of assessments of nontechnical skills in medical teams.

BACKGROUND

There are several scales for assessing nontechnical skills in medical teams. In this critique, we describe the reliability of such assessments and discuss factors that may influence the obtained estimate of reliability for such scales.

METHOD

A review of the literature yielded 28 reliability estimates, mostly intraclass correlations. We analyzed the methods of scale development and reliability estimation for factors that are likely to influence the resulting estimate, including aspects of the judges, the targets, what is observed, and the kind of reliability coefficient.

RESULTS

Typical reliability was approximately 0.7, but estimates were quite varied in magnitude, from quite low (0.3) to quite high (0.9). Thus, highly reliable assessment is possible, but rarely achieved. Factors that are likely to influence the magnitude of reliability include the number of assessors, the participants' range in skill level, and the observational process. Most studies do not present enough information to make the estimates comparable from one study to the next.

CONCLUSIONS

Although nontechnical skills can be measured reliably under some circumstances, it is not clear from the existing literature that one scale provides more reliable data than another. We recommend several practices that would help readers to make informed decisions about which scales are most useful for their purposes.

Exploring anesthesiologists' understanding of situational awareness: a qualitative study

PURPOSE

This study explored how anesthesiologists understand situational awareness (SA) and how they think SA is learned, taught, and assessed.

METHODS

Semi-structured interviews were performed with practicing anesthesiologists involved in teaching. This qualitative study was conducted using constructivist grounded theory techniques (i.e., line-by-line coding, memoing, and constant comparison) in a thematic analysis of interview transcripts. Group meetings were held to develop and review themes emerging from the data.

RESULTS

Eighteen anesthesiologists were interviewed. Respondents displayed an understanding of SA using a mixture of examples from clinical experience and everyday life. Despite agreeing on the importance of SA, formal definitions of SA were lacking, and the participants did not explicate the topic of SA in either their practice or their teaching activities. Situational awareness had been learned informally through increasing independence in the clinical context, role modelling, reflection on errors, and formally through simulation. Respondents taught SA through modelling and discussing scanning behaviour, checklists, verbalization of thought processes, and debriefings. Although trainees' understanding of SA was assessed as part of the decision-making process for granting clinical independence, respondents found it difficult to give meaningful feedback on SA to their trainees.

CONCLUSION

Although SA is an essential concept in anesthesiology, its use remains rather tacit, primarily due to the lack of a common operational definition of the term. Faculty development is required to help anesthesiologists teach and assess SA more explicitly in the clinical environment.

Evaluating Situation Awareness: An Integrative Review

Situation awareness (SA) refers to the conscious awareness of the current situation in relation to one’s environment. In nursing, loss or failure to achieve high levels of SA is linked with adverse patient outcomes. The purpose of this integrative review is to examine various instruments and techniques used to measure SA among nurses across academic and clinical settings. Computerized database and ancestry search strategies resulted in 40 empirical research reports. Of the reports included in the review, 24 measured SA among teams that included nurses and 16 measured SA solely in nurses. Methods used to evaluate SA included direct and indirect methods. Direct methods included the Situation Awareness Global Assessment Technique and questionnaires. Indirect methods included observer rating instruments and performance outcome measures. To have a better understanding of how nurses’ make decisions in complex work environments, reliable and valid measures of SA is crucial.

Non-technical skills evaluation in the critical care air ambulance environment: introduction of an adapted rating instrument--an observational study

BACKGROUND

In the isolated and dynamic health-care setting of critical care air ambulance transport, the quality of clinical care is strongly influenced by non-technical skills such as anticipating, recognising and understanding, decision making, and teamwork. However there are no published reports identifying or applying a non-technical skills framework specific to an intensive care air ambulance setting. The objective of this study was to adapt and evaluate a non-technical skills rating framework for the air ambulance clinical environment.

METHODS

In the first phase of the project the anaesthetists' non-technical skills (ANTS) framework was adapted to the air ambulance setting, using data collected directly from clinician groups, published literature, and field observation. In the second phase experienced and inexperienced inter-hospital transport clinicians completed a simulated critical care air transport scenario, and their non-technical skills performance was independently rated by two blinded assessors. Observed and self-rated general clinical performance ratings were also collected. Rank-based statistical tests were used to examine differences in the performance of experienced and inexperienced clinicians, and relationships between different assessment approaches and assessors.

RESULTS

The framework developed during phase one was referred to as an aeromedical non-technical skills framework, or AeroNOTS. During phase two 16 physicians from speciality training programmes in intensive care, emergency medicine and anaesthesia took part in the clinical simulation study. Clinicians with inter-hospital transport experience performed more highly than those without experience, according to both AeroNOTS non-technical skills ratings (p = 0.001) and general performance ratings (p = 0.003). Self-ratings did not distinguish experienced from inexperienced transport clinicians (p = 0.32) and were not strongly associated with either observed general performance (r(s) = 0.4, p = 0.11) or observed non-technical skills performance (r(s) = 0.4, p = 0.1).

DISCUSSION

This study describes a framework which characterises the non-technical skills required by critical care air ambulance clinicians, and distinguishes higher and lower levels of performance.

CONCLUSION

The AeroNOTS framework could be used to facilitate education and training in non-technical skills for air ambulance clinicians, and further evaluation of this rating system is merited.