Nontechnical Skill Assessment of the Collective Surgical Team Using the Non-Technical Skills for Surgeons (NOTSS) System

Enhancing patient safety in trauma: Understanding adverse events, assessment tools, and the role of trauma video review

OBJECTIVES

This study aimed to investigate adverse events (AEs) in trauma resuscitation, evaluate contributing factors, and assess methods, such as trauma video review (TVR), to mitigate AEs.

BACKGROUND

Trauma remains a leading cause of global mortality and morbidity, necessitating effective trauma care. Despite progress, AEs during trauma resuscitation persist, impacting patient outcomes and the healthcare system. Identifying and analyzing AEs and their determinants are crucial for improving trauma care.

METHODS

This narrative review explored the definition, identification, and assessment of AEs associated with trauma resuscitation within the trauma system. It includes various studies and assessment tools such as STAT Taxonomy and T-NOTECHs. Additionally, it assessed the role of TVR in detecting AEs and strategies to enhance patient safety.

CONCLUSION

Integrated with standardized tools, TVR shows promise for identifying AEs. Challenges include ensuring reporting consistency and integrating approaches into existing protocols. Future research should prioritize linking trauma team performance to patient outcomes, and develop sustainable TVR programs to enhance patient safety.

Deep Learning Analysis of Surgical Video Recordings to Assess Nontechnical Skills

Importance

Assessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging.

Objective

To explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures.

Design, Setting, and Participants

This cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team's nontechnical skills by 3 expert raters.

Main Outcomes and Measures

NOTSS overall score, with motion features extracted from surgical videos as measures.

Results

A total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51, P = .005), acceleration (r = 0.48, P = .008), and entropy (r = -0.52, P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjusted R2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21]; P = .004) and team displacement entropy (adjusted R2 = 0.304; coefficient, -12.64 [95% CI, -20.54 to -4.74]; P = .003) were associated with NOTSS scores.

Conclusions and Relevance

This study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.

Remote Assessment of Real-World Surgical Safety Checklist Performance Using the OR Black Box: A Multi-Institutional Evaluation

BACKGROUND

Large-scale evaluation of surgical safety checklist performance has been limited by the need for direct observation. The operating room (OR) Black Box is a multichannel surgical data capture platform that may allow for the holistic evaluation of checklist performance at scale.

STUDY DESIGN

In this retrospective cohort study, data from 7 North American academic medical centers using the OR Black Box were collected between August 2020 and January 2022. All cases captured during this period were analyzed. Measures of checklist compliance, team engagement, and quality of checklist content review were investigated.

RESULTS

Data from 7,243 surgical procedures were evaluated. A time-out was performed during most surgical procedures (98.4%, n = 7,127), whereas a debrief was performed during 62.3% (n = 4,510) of procedures. The mean percentage of OR staff who paused and participated during the time-out and debrief was 75.5% (SD 25.1%) and 54.6% (SD 36.4%), respectively. A team introduction (performed 42.6% of the time) was associated with more prompts completed (31.3% vs 18.7%, p < 0.001), a higher engagement score (0.90 vs 0.86, p < 0.001), and a higher percentage of team members who ceased other activities (80.3% vs 72%, p < 0.001) during the time-out.

CONCLUSIONS

Remote assessment using OR Black Box data provides useful insight into surgical safety checklist performance. Many items included in the time-out and debrief were not routinely discussed. Completion of a team introduction was associated with improved time-out performance. There is potential to use OR Black Box metrics to improve intraoperative process measures.

Assessing endovascular team performances in a hybrid room using the Black Box system: a prospective cohort study

BACKGROUND

The hybrid room (HR) is a complex, high-risk environment, requiring teams (surgeons, anesthesiologists, nurses, technologists) to master various skills, including the 'As Low As Reasonably Achievable' principle of radiation safety. This prospective single center cohort reports the first use of the Operating Room Black Box (ORBB) in a HR. This medical data recording system captures procedural and audio-visual data to facilitate structured team performance analysis.

METHODS

Patients planned for endovascular repair of an infrarenal abdominal aortic aneurysm (EVAR) or treatment of symptomatic iliac-femoral-popliteal atherosclerotic disease (Peripheral Vascular Interventions or PVI) were included. Validated measures and established assessment tools were used to assess (non-)technical skills, radiation safety performance and environmental distractions.

RESULTS

Six EVAR and sixteen PVI procedures were captured. Technical performance for one EVAR was rated 19/35 on the procedure-specific scale, below the 'acceptable' score of 21. Technical skills were rated above acceptable in all PVI procedures. Shared decision making and leadership were rated highly in 12/22 cases, whereas surgical communication and nurses' task management were rated low in 14/22 cases. Team members rarely stepped back from the C-arm during digital subtraction angiography. Radiation safety behavior was scored below 'acceptable' in 14/22 cases. A median (interquartile range) number of 12 (6-23) auditory distractions was observed per procedure.

CONCLUSIONS

The ORBB facilitates holistic workplace-based assessment of endovascular performance in a HR by combining objective assessment parameters and rating scale-based evaluations. Strengths and weaknesses were identified in team members' (non-)technical and radiation safety practices. This technology has the potential to improve vascular surgical practice, though human input remains crucial. (NCT04854278).

Senior surgeons as role models in the operating theatre: a thematic analysis through the lens of aristotelian ethics

BACKGROUND

Surgeons are commonly evaluated by surgical skills and outcomes rather than their character traits. We sought to examine role model behaviours of senior surgeons through the lens of Aristotelian (virtue) ethics.

METHODS

Semi-structured focus group interviews were undertaken of anaesthetic trainees at a large university hospital NHS Foundation Trust and transcripts were subjected to thematic analysis to yield themes and subthemes. Participation of the trainees was entirely voluntary and focus groups were conducted using Zoom™.

RESULTS

The overarching themes identified were 'Teamwork makes the dream work', 'Captain of the ship' and 'Strong foundations'.

CONCLUSION

We hope to take lessons learnt in conjunction with our previous work to help refocus surgical training towards a process of character reformation, rather than simply imparting technical skills to trainees.

Intraoperative Assessment of Non-Technical Skills for Surgeons (NOTSS) and Qualitative Description of their Effects on Intraoperative Performance

OBJECTIVE

The aims of this study were (1) to evaluate the feasibility of using the Non-Technical Skills for Surgeons (NOTSS) rating tool in assessing surgeons' non-technical skills behaviors in live operations, and (2) to describe the effect of NOTSS on intraoperative performance.

SETTING DESIGN

This study was conducted in an academic hospital in North America. Two observers independently conducted direct non-participant observations using the NOTSS rating tool to assess non-technical skills, and to document examples of effective or ineffective non-technical skills behaviors. Observers took field notes to document non-technical skill gaps that were not captured by the NOTSS rating tool, and situations or scenarios that presented challenges for accurate assessment. Interclass correlation estimates and 95% confidence intervals were calculated to assess the validity of the NOTSS rating tool. Deductive thematic qualitative data analysis was used for field notes and NOTSS behavior descriptions.

PARTICIPANTS

Participants were general surgeons performing either minimally invasive (robotic assisted or laparoscopic surgery), or open procedures.

RESULTS

We observed 18 surgeries, involving 6 surgeons, 11 residents and one fellow resulting in 37 hours of direct observations. The mean NOTSS score was 3.8 (SD 0.41) for situation awareness, 3.75 (SD 0.47) for decision-making, 3.71 (SD 0.39) for communication and teamwork, and 3.76 (SD 0.38) for leadership. The inter-rater reliability ranged between 0.65 and 0.80 for each NOTSS categories. The observers documented examples of effective non-technical skills behaviors and examples of behaviors that need improvement. Furthermore, we described the effect of each observed behavior on intraoperative performance. One challenge to NOTSS use in live surgery was that observers had to infer situation awareness, decision-making, and coping with pressure as these were not easily observed without attending surgeons articulating their underlying thought process.

CONCLUSION

The use of the NOTSS tool in live surgery is a valid and practical tool to document observed behaviors and their effect on intraoperative performance in order to provide constructive feedback to surgeons. One notable limitation is that without specific articulation by the surgeon of their underlying thought process the observer must infer specific elements. By documenting specific real-world events with high inter-rater reliability and adequate surgeon score variation the process can be used to provide useful feedback for improvement.

Validation in Spanish of the Ottawa scale for non-technical skills in health personnel in crisis situations

INTRODUCTION

Non-technical skills are related to morbi-mortality in medicine; it has been proposed that 46% of fatal outcomes are explained to limitations in non-technical skills and only 5% to technical skills deficiencies, however, there is no validated instrument or scale in spanish that allows its evaluation in the management of medical crisis.

OBJECTIVE

To evaluate the psychometric properties of a Spanish-adapted version of the "Ottawa crisis resource management (CRM) global rating scale (GRS)" in medical staff involved in critical decision-making based in high-fidelity simulation, which could be beneficial to impact patient safety and improve clinical outcomes.

METHODS

Transversal cultural instrument validation and adaptation study, included 91 participants who simulated a medical crisis between 2018 and 2019, and to whom the Spanish version of the CRM-GRS was applied in order to evaluate its psychometric properties.

RESULTS

A cultural adaptation with translation into Spanish of the CRM-GRS was made. Subsequently, the scale was applied to 91 participants. An internal consistency (Cronbach's alpha) greater than 0.9 was found in each dimension. The level of inter-rater reliability, evaluated by the interclass coefficient was 0.59 to 0.69, and test-retest reliability with an interclass coefficient greater than 0.7. The validity of the convergent construct was moderate (interclass coefficient between 0.6 and 0.7 for all domains) and the validity of the divergent construct between 0.4 and 0.5 was found adequate.

CONCLUSION

The translated and adapted Spanish version of the CRM-GRS in crisis had adequate internal consistency, reliability, and construct validity.

Validation in spanish of the Ottawa scale for non-techical skills in health personnel in crisis situations

INTRODUCTION

Non-technical skills are related to morbi-mortality in medicine; it has been proposed that 46% of fatal outcomes are explained to limitations in non-technical skills and only 5% to technical skills deficiencies, however, there is no validated instrument or scale in spanish that allows its evaluation in the management of medical crisis.

OBJECTIVE

To evaluate the psychometric properties of a Spanish-adapted version of the "Ottawa crisis resource management (CRM) global rating scale (GRS)" in medical staff involved in critical decision-making based in high-fidelity simulation, which could be beneficial to impact patient safety and improve clinical outcomes.

METHODS

Transversal cultural instrument validation and adaptation study, included 91 participants who simulated a medical crisis between 2018 and 2019, and to whom the Spanish version of the CRM-GRS was applied in order to evaluate its psychometric properties.

RESULTS

A cultural adaptation with translation into Spanish of the CRM-GRS was made. Subsequently, the scale was applied to 91 participants. An internal consistency (Cronbach's alpha) greater than 0.9 was found in each dimension. The level of inter-rater reliability, evaluated by the interclass coefficient was 0.59 to 0.69, and test-retest reliability with an interclass coefficient greater than 0.7. The validity of the convergent construct was moderate (interclass coefficient between 0.6 and 0.7 for all domains) and the validity of the divergent construct between 0.4 and 0.5 was found adequate.

CONCLUSION

The translated and adapted Spanish version of the CRM-GRS in crisis had adequate internal consistency, reliability, and construct validity.

Surgeons Are Not Pilots: Is the Aviation Safety Paradigm Relevant to Modern Surgical Practice?

Error in surgery is common, although not always consequential. Surgical outcomes are often compared to safety data from commercial aviation. This industry's performance is frequently referenced as an example of high-reliability that should be reproduced in clinical practice. Consequently, the aviation-surgery analogy forms the conceptual framework for much patient safety research, advocating for the translation of aviation safety tools to the healthcare setting. Nevertheless, overuse or incorrect application of this paradigm can be misleading and may result in ineffective quality improvement interventions. This article discusses the validity and relevance of the aviation-surgery comparison, providing the necessary context to improve its application at the bedside. It addresses technical and human factors training, as well as more novel performance domains such as professional culture and optimization of operators' condition. These are used to determine whether the aviation-surgery analogy is a valuable source of cross-professional learning or simply another safety cliché.

Teaching and Evaluating Nontechnical Skills for General Surgery

Surgical training programs have long used quantitative measures of knowledge, as well as subjective evaluation of technical skills, to define the competence of trainees. However, a growing body of literature has shown the importance of nontechnical surgical skills as vital components of quality surgical care. Institutions must train nontechnical surgical skills, including leadership, communication, teamwork, situational awareness, and decision making, and incorporate these attributes into their evaluative processes to maximally enhance surgical performance at every career stage.

Development and content validation of the Safe Surgery Organizational Readiness Tool: A quality improvement study

BACKGROUND

Recent efforts to increase access to safe and high-quality surgical care in low- and middle-income countries have proven successful. However, multiple facilities implementing the same safety and quality improvement interventions may not all achieve successful outcomes. This heterogeneity could be explained, in part, by pre-intervention organizational characteristics and lack of readiness of surgical facilities. In this study, we describe the process of developing and content validating the Safe Surgery Organizational Readiness Tool.

MATERIALS AND METHODS

The new tool was developed in two stages. First, qualitative results from a Safe Surgery 2020 intervention were combined with findings from a literature review of organizational readiness and change. Second, through iterative discussions and expert review, the Safe Surgery Organizational Readiness Tool was content validated.

RESULTS

The Safe Surgery Organizational Readiness Tool includes 14 domains and 56 items measuring the readiness of surgical facilities in low- and middle-income countries to implement surgical safety and quality improvement interventions. This multi-dimensional and multi-level tool offers insights into facility members' beliefs and attitudes at the individual, team, and facility levels. A panel review affirmed the content validity of the Safe Surgery Organizational Readiness Tool.

CONCLUSION

The Safe Surgery Organizational Readiness Tool is a theory- and evidence-based tool that can be used by change agents and facility leaders in low- and middle-income countries to assess the baseline readiness of surgical facilities to implement surgical safety and quality improvement interventions. Next steps include assessing the reliability and validity of the Safe Surgery Organizational Readiness Tool, likely resulting in refinements.

Non-technical skill assessments across levels of US surgical training

BACKGROUND

To ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool.

METHODS

We undertook an exploratory, prospective cohort study of 32 residents-interns (postgraduate year 1), junior residents (postgraduate years 2-3), and senior residents (postgraduate years 4-5)-across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident's intraoperative performance. Mean NOTSS-US ratings (total range 4-20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1-5, with 1=poor, 3=average, 5=excellent) were stratified by residents' training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression.

RESULTS

For 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3-8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5-6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3-3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents' training levels persisted for individual NOTSS-US behavior categories.

CONCLUSION

These data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees' non-technical skills.

Non-technical skills and device-related interruptions in minimally invasive surgery

BACKGROUND

Device-related interruptions in the operating room (OR) may create stress among health care providers and delays. Although non-technical skills (NTS) of the OR teams, such as situational awareness and communication, are expected to influence device-related interruptions, empirical data on this relationship are limited.

METHODS

We performed a prospective cohort study of 144 consecutive elective laparoscopic operations during 13 months. A data capture system called the OR Black Box® was used to characterize device-related interruptions, NTS, and distractions. Device-related interruptions were classified according to a priori established categories. Positive and negative NTS instances were identified according to validated measurement tools specific for nurses and surgeons. We assessed the relationship between NTS and device-related interruptions after adjusting for potential confounders.

RESULTS

A total of 86 device-related interruptions occurred in 48 of 144 operations (33%). They were most frequently classified as device failure (54%) followed by improper assembly (19%) and disconnection (14%). Medians of 1 [interquartile range (IQR) 0-3] and 1 (IQR 0-2) negative NTS instance per operation were demonstrated by nurses and surgeons, respectively. Medians of 28 (IQR 15-38) and 40 (IQR 28-118) positive NTS instances per operation were demonstrated by nurses and surgeons. In a multivariable analysis, a higher frequency of negative NTS instances demonstrated by nurses was associated with device-related interruptions after risk adjustment (Odds Ratio 1.33, p = 0.02).

CONCLUSIONS

In elective laparoscopic operations, an increased likelihood of device-related interruptions in the OR was associated with more frequent negative NTS demonstrations by nursing teams.

Measuring the teamwork performance of operating room teams: a systematic review of assessment tools and their measurement properties

Teamwork is fundamental to surgical patient safety but is inconsistently measured. While many tools have been developed for elective intraoperative situations, it is unclear which is the most robust. This systematic review aimed to identify tools to measure the teamwork of operating room teams. Studies were included if they examined the measurement properties of these tools. PsycINFO, Embase (via OVID), CINAHL, ERIC, Medline and Medline in Process (via OVID) were searched through to May 3, 2019, as were reference lists of included studies and previously published relevant reviews. Retrieved articles were screened and data extracted in duplicate by two independent reviewers. Quality was assessed using the COSMIN checklist. Of the 2121 references identified, 14 studies of six assessment tools were included. Tools were validated across various specialties, mostly in clinical rather than simulated settings. The Observational Teamwork Assessment for Surgery (OTAS) and Operating Theater Team Non-Technical Skills Assessment Tool (NOTECHS) were the most frequently investigated tools. Though acceptable for assessing teamwork, both NOTECHS and OTAS rely on the questionable assumption that the teamwork of a team is equivalent to the sum of individual performances. Future studies may investigate other assessment tools that assess the whole team as the unit of analysis along with the potential of these tools to provide healthcare providers with meaningful feedback in clinical practice.

Advancing spinal fellowship training: an international multi-centre educational perspective

PURPOSE

The purpose of this article is to review the importance of contemporary spine surgery fellowships and educational strategies to assist with fellowship design and delivery.

METHODS

Spine surgery fellowship includes trainees from orthopaedic and neurosurgical backgrounds and is increasingly indicated for individuals wishing to pursue spine surgery as a career, recognizing how spinal surgery evolved significantly in scope and complexity. We combine expert opinion with a review of the literature and international experience to expound spine fellowship training.

RESULTS

Contemporary learning techniques include boot camps at the start of fellowship which may reinforce previous clinical learning and help prepare fellows for their new clinical roles. There is good evidence that surgical specialty training boot camps improve clinical skills, knowledge and trainee confidence prior to embarking upon new clinical roles with increasing levels of responsibility. Furthermore, as simulation techniques and technologies take on an increasing role in medical and surgical training, we found evidence that trainees' operative skills and knowledge can improve with simulated operations, even if just carried out briefly. Finally, we found evidence to suggest a role for establishing competence-based objectives for training in specific operative and technical procedures. Competence-based objectives are helpful for trainees and trainers to highlight gaps in a trainee's skill set that may then be addressed during training.

CONCLUSIONS

Spinal fellowships may benefit from certain contemporary strategies that assist design and delivery of training in a safe environment. Interpersonal factors that promote healthy teamwork may contribute to an environment conducive to learning. These slides can be retrieved under Electronic Supplementary Material.