Training Tools for Nontechnical Skills for Surgeons-A Systematic Review

Eye tracking and audio sensors to evaluate surgeon's non-technical skills: An empirical study

Non-Technical Skills (NTS) of medical teams are currently measured using subjective and resource-intensive ratings given by experts. This study explores if objective NTS assessment approaches with eye-tracking and audio sensors can measure teamwork and communication skills in surgery. Eight surgeons participated in a simulated two-phase surgical scenario developed to assess their NTS. Sensor-based audio, eye tracking and video data were collected and analyzed along with rating from the NOTSS scale. Different levels of communication were detected by the sensor data during the two phases of the simulated surgery. Sensor data detected leadership qualities among surgeons based on speech metrics, and eye tracking offered additional evidence about gaze patterns related to NTS. This objective approach to NTS measurement captured differences in communication in greater detail as opposed to a single collective rating obtained using current assessment tools.

Efficacy of educational stepwise robot-assisted radical prostatectomy procedure for urology residents

OBJECTIVE

To evaluate the effectiveness of an educational stepwise robot-assisted radical prostatectomy (RARP) procedure for urology residents.

METHODS

We performed a detailed evaluation of 42 RARP procedures performed by a single urology resident from July 2019 to February 2022. The RARP procedures were divided into the following nine steps: (1) bladder dissection, (2) endopelvic fascia dissection, (3) bladder neck dissection, (4) seminal vesicle dissection, (5) Denonvilliers' fascia dissection, (6) dorsal vascular complex ligation, (7) dissection of the prostatic apex, (8) posterior anastomosis, and (9) urethro-vesical anastomosis. The procedures were further subcategorized as anatomical understanding, spatial recognition, and technical skills for evaluation of resident training. The surgeries were divided into first and second halves, and patient characteristics and operative outcomes were statistically analyzed. The operative time of each of the nine steps and the reasons for proctor intervention were compared.

RESULTS

Among 42 patients, there were no significant differences in operative outcomes between the two groups. The median operative time was 169 min (164 vs. 179 min, p = .12), and the median console time was 128 min (127 vs. 130 min, p = .74). Although there were no significant differences in the time of the nine steps, the resident significantly overcame (7) dissection of the prostatic apex and (8) posterior anastomosis based on the evaluation of the proctored reasons for intervention.

CONCLUSIONS

Urology residents can safely perform and efficiently learn RARP with this stepwise educational system. This educational stepwise RARP procedure can effectively help residents to develop their skills.

Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room

OBJECTIVE

To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance.

BACKGROUND

Surgeons need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons.

METHODS

A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context.

RESULTS

From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making.

CONCLUSIONS

The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.

Non-technical skills training for Nigerian interprofessional surgical teams: a cross-sectional survey

INTRODUCTION

Non-technical skills (NTS) including communication, teamwork, leadership, situational awareness, and decision making, are essential for enhancing surgical safety. Often perceived as tangential soft skills, NTS are many times not included in formal medical education curricula or continuing medical professional development. We aimed to explore exposure of interprofessional teams in North-Central Nigeria to NTS and ascertain perceived facilitators and barriers to interprofessional training in these skills to enhance surgical safety and inform design of a relevant contextualized curriculum.

METHODS

Six health facilities characterised by high surgical volumes in Nigeria's North-Central geopolitical zone were purposively identified. Federal, state, and private university teaching hospitals, non-teaching public and private hospitals, and a not-for-profit health facility were included. A nineteen-item, web-based, cross-sectional survey was distributed to 71 surgical providers, operating room nurses, and anaesthesia providers by snowball sampling through interprofessional surgical team leads from August to November 2021. Data were analysed using Fisher's exact test, proportions, and constant comparative methods for free text responses.

RESULTS

Respondents included 17 anaesthesia providers, 21 perioperative nurses, and 29 surgeons and surgical trainees, with a 95.7% survey completion rate. Over 96% had never heard of any NTS for surgery framework useful for variable resource contexts and only 8% had ever received any form of NTS training. Interprofessional teams identified communication and teamwork as the most deficient personal skills (38, 57%), and as the most needed for surgical team improvement (45, 67%). There was a very high demand for NTS training by all surgical team members (64, 96%). The main motivations for training were expectations of resultant improved patient safety and improved interprofessional team dynamics. Week-long, hybrid training courses (with combined in-person and online components) were the preferred format for delivery of NTS education. Factors that would facilitate attendance included a desire for patient safety and self-improvement, while barriers to attendance were conflicts of time, and training costs.

CONCLUSIONS

Interprofessional surgical teams in the Nigerian context have a high degree of interest in NTS training, and believe it can improve team dynamics, personal performance, and ultimately patient safety. Implementation of NTS training programs should emphasize interprofessional communication and teamworking.

Data-driven coaching to improve statewide outcomes in CABG: before and after interventional study

BACKGROUND

The impact of quality improvement initiatives program (QIP) on coronary artery bypass grafting surgery (CABG) remains scarce, despite improved outcomes in other surgical areas. This study aims to evaluate the impact of a package of QIP on mortality rates among patients undergoing CABG.

MATERIALS AND METHODS

This prospective cohort study utilized data from the multicenter database Registro Paulista de Cirurgia Cardiovascular II (REPLICCAR II), spanning from July 2017 to June 2019. Data from 4018 isolated CABG adult patients were collected and analyzed in three phases: before-implementation, implementation, and after-implementation of the intervention (which comprised QIP training for the hospital team). Propensity Score Matching was used to balance the groups of 2170 patients each for a comparative analysis of the following outcomes: reoperation, deep sternal wound infection/mediastinitis ≤30 days, cerebrovascular accident, acute kidney injury, ventilation time >24 h, length of stay <6 days, length of stay >14 days, morbidity and mortality, and operative mortality. A multiple regression model was constructed to predict mortality outcomes.

RESULTS

Following implementation, there was a significant reduction of operative mortality (61.7%, P =0.046), as well as deep sternal wound infection/mediastinitis ( P <0.001), sepsis ( P =0.002), ventilation time in hours ( P <0.001), prolonged ventilation time ( P =0.009), postoperative peak blood glucose ( P <0.001), total length of hospital stay ( P <0.001). Additionally, there was a greater use of arterial grafts, including internal thoracic ( P <0.001) and radial ( P =0.038), along with a higher rate of skeletonized dissection of the internal thoracic artery.

CONCLUSIONS

QIP was associated with a 61.7% reduction in operative mortality following CABG. Although not all complications exhibited a decline, the reduction in mortality suggests a possible decrease in failure to rescue during the after-implementation period.

Simulation-based training for intraoperative posterior capsule rupture management: an analysis of nontechnical skills development and the relationship between technical and nontechnical skills

OBJECTIVES

To analyse development of individual nontechnical skills (NTS) domains after undertaking a previously developed simulation-based training model and analyse the relationship between technical skills (TS) and NTS in ophthalmic surgery.

METHODS

The simulation-based training model involved a cataract surgery case complicated by intraoperative posterior capsule rupture. Cataract surgeons underwent the simulation twice, separated by a training intervention. Two blinded independent experts assessed participants' NTS using HUFOES, NOn-Technical Skills for Surgeons (NOTSS), and the OSATS global rating scale for TS. Paired t-tests assessed differences in individual NTS domains, with p < 0.05 indicating significance. The Pearson Product Moment Correlation Coefficient was used to assess the correlation between scores from each scoring system.

RESULTS

All NTS domains within HUFOES and NOTSS demonstrated statistically significant improvements secondary to the training intervention. Positive correlations were demonstrated between HUFOES and OSATS scores in the pre- and post-training simulations, r = 0.870 (p < 0.001) and r = 0.861 (p < 0.001), respectively. Positive correlations were also demonstrated between NOTSS and OSATS scores in pre- and post-training simulations, r = 0.849 (p < 0.001) and r = 0.757 (p = 0.001), respectively. Positive correlations were demonstrated between HUFOES and NOTSS scores; r = 0.979 (p < 0.001) (n = 17) and r = 0.959 (p < 0.001) for pre- and post-training simulations, respectively.

CONCLUSION

All NTS domains contained within HUFOES and NOTSS demonstrated significant increases following the completion of the simulation-based training model. Positive correlations exist between an ophthalmic surgeon's TS and NTS. This is the first study to report these findings within ophthalmic surgery.

European society for trauma and emergency surgery member-identified research priorities in emergency surgery: a roadmap for future clinical research opportunities

BACKGROUND

European Society for Trauma and Emergency Surgery (ESTES) is the European community of clinicians providing care to the injured and critically ill surgical patient. ESTES has several interlinked missions - (1) the promotion of optimal emergency surgical care through networked advocacy, (2) promulgation of relevant clinical cognitive and technical skills, and (3) the advancement of scientific inquiry that closes knowledge gaps, iteratively improves upon surgical and perioperative practice, and guides decision-making rooted in scientific evidence. Faced with multitudinous opportunities for clinical research, ESTES undertook an exercise to determine member priorities for surgical research in the short-to-medium term; these research priorities were presented to a panel of experts to inform a 'road map' narrative review which anchored these research priorities in the contemporary surgical literature.

METHODS

Individual ESTES members in active emergency surgery practice were polled as a representative sample of end-users and were asked to rank potential areas of future research according to their personal perceptions of priority. Using the modified eDelphi method, an invited panel of ESTES-associated experts in academic emergency surgery then crafted a narrative review highlighting potential research priorities for the Society.

RESULTS

Seventy-two responding ESTES members from 23 countries provided feedback to guide the modified eDelphi expert consensus narrative review. Experts then crafted evidence-based mini-reviews highlighting knowledge gaps and areas of interest for future clinical research in emergency surgery: timing of surgery, inter-hospital transfer, diagnostic imaging in emergency surgery, the role of minimally-invasive surgical techniques and Enhanced Recovery After Surgery (ERAS) protocols, patient-reported outcome measures, risk-stratification methods, disparities in access to care, geriatric outcomes, data registry and snapshot audit evaluations, emerging technologies interrogation, and the delivery and benchmarking of emergency surgical training.

CONCLUSIONS

This manuscript presents the priorities for future clinical research in academic emergency surgery as determined by a sample of the membership of ESTES. While the precise basis for prioritization was not evident, it may be anchored in disease prevalence, controversy around aspects of current patient care, or indeed the identification of a knowledge gap. These expert-crafted evidence-based mini-reviews provide useful insights that may guide the direction of future academic emergency surgery research efforts.

Analyzing interprofessional teamwork in the operating room: An exploratory observational study using conventional and alternative approaches

Intraoperative teamwork is vital for patient safety. Conventional tools for studying intraoperative teamwork typically rely on behaviorally anchored rating scales applied at the individual or team level, while others capture narrative information across several units of analysis. This prospective observational study characterizes teamwork using two conventional tools (Operating Theatre Team Non-Technical Skills Assessment Tool [NOTECHS]; Team Emergency Assessment Measure [TEAM]), and one alternative approach (modified-Systems Engineering Initiative for Patient Safety [SEIPS] model). We aimed to explore the advantages and disadvantages of each for providing feedback to improve teamwork practice. Fifty consecutive surgical cases at a Canadian academic hospital were recorded with the OR Black Box®, analyzed by trained raters, and summarized descriptively. Teamwork performance was consistently high within and across cases rated with NOTECHS and TEAMS. For cases analyzed with the modified-SEIPS tool, both optimal and suboptimal teamwork behaviors were identified, and team resilience was frequently observed. NOTECHS and TEAM provided summative assessments and overall pattern descriptions, while SEIPS facilitated a deeper understanding of teamwork processes. As healthcare organizations continue to prioritize teamwork improvement, SEIPS may provide valuable insights regarding teamwork behavior and the broader context influencing performance. This may ultimately enhance the development and effectiveness of multi-level teamwork interventions.

Crew Autonomy During Simulated Medical Event Management on Long Duration Space Exploration Missions

OBJECTIVE

Our primary aim was to investigate crew performance during medical emergencies with and without ground-support from a flight surgeon located at mission control.

BACKGROUND

There are gaps in knowledge regarding the potential for unanticipated in-flight medical events to affect crew health and capacity, and potentially compromise mission success. Additionally, ground support may be impaired or periodically absent during long duration missions.

METHOD

We reviewed video recordings of 16 three-person flight crews each managing four unique medical events in a fully immersive spacecraft simulator. Crews were randomized to two conditions: with and without telemedical flight surgeon (FS) support. We assessed differences in technical performance, behavioral skills, and cognitive load between groups.

RESULTS

Crews with FS support performed better clinically, were rated higher on technical skills, and completed more clinical tasks from the medical checklists than crews without FS support. Crews with FS support also had better behavioral/non-technical skills (information exchange) and reported significantly lower cognitive demand during the medical event scenarios on the NASA-TLX scale, particularly in mental demand and temporal demand. There was no significant difference between groups in time to treat or in objective measures of cognitive demand derived from heart rate variability and electroencephalography.

CONCLUSION

Medical checklists are necessary but not sufficient to support high levels of autonomous crew performance in the absence of real-time flight surgeon support.

APPLICATION

Potential applications of this research include developing ground-based and in-flight training countermeasures; informing policy regarding autonomous spaceflight, and design of autonomous clinical decision support systems.

Measuring Quality at the Surgeon Level

Patients expect high-quality surgical care and increasingly are looking for ways to assess the quality of the surgeon they are seeing, but quality measurement is often more complicated than one might expect. Measurement of individual surgeon quality in a manner that allows for comparison among surgeons is particularly difficult. While the concept of measuring individual surgeon quality has a long history, technology now allows for new and innovative ways to measure and achieve surgical excellence. However, some recent efforts to make surgeon-level quality data publicly available have highlighted the challenges of this work. Through this chapter, the reader will be introduced to a brief history of surgical quality measurement, learn about the current state of quality measurement, and get a glimpse into what the future holds.

Nontechnical Skills for Surgeons as a Framework to Evaluate Cardiopulmonary Bypass Management Skills of Resident Trainees

BACKGROUND

Nontechnical skills are critical in cardiac surgery but currently there is no formal paradigm to teach these in residency training. We investigated the use of the Nontechnical skills for surgeons (NOTSS) system as a framework to assess and teach nontechnical skills related to cardiopulmonary bypass (CPB) management.

METHODS

Single-center retrospective analysis of Integrated and Independent pathway thoracic surgery residents who participated in dedicated nontechnical skills evaluation and training. Two CPB management simulation scenarios were utilized. All residents received a lecture on CPB fundamentals and then individually participated in the first simulation ("Pre-NOTSS"). Immediately following this, nontechnical skills were rated by self-assessment and by a NOTSS trainer. All residents then underwent group NOTSS training followed by the second individual simulation ("Post-NOTSS"). Nontechnical skills were rated as before. NOTSS categories assessed included Situation Awareness, Decision Making, Communication and Teamwork, and Leadership.

RESULTS

Nine residents were divided into 2 groups: Junior (n = 4, PGY1-4) and Senior (n = 5, PGY5-8). Pre-NOTSS resident self-ratings were higher for Senior than Junior in the categories of Decision Making, Communication and Teamwork, and Leadership while trainer ratings were similar between the groups. Post-NOTSS, resident self-ratings were higher for Senior than Junior in Situation Awareness and Decision Making while trainer scores were higher for both groups in Communication and Teamwork and Leadership.

CONCLUSIONS

The NOTSS framework in conjunction with simulation scenarios provides a practical framework to evaluate and teach nontechnical skills related to CPB management. NOTSS training can lead to improvements in both subjective and objective ratings of nontechnical skills for all PGY levels.

The silver lining of the pandemic in surgical education: virtual surgical education and recommendations for best practices

Virtual education is an evolving field within the realm of surgical training. Since the onset of the COVID-19 pandemic, the application of virtual technologies in surgical education has undergone significant exploration and advancement. While originally developed to supplement in-person curricula for the development of clinical decision-making, virtual surgical education has expanded into the realms of clinical decision-making, surgical, and non-surgical skills acquisition. This manuscript aims to discuss the various applications of virtual surgical education as well as the advantages and disadvantages associated with each education modality, while offering recommendations on best practices and future directions.

Cognitive Load Management: An Invaluable Tool for Safe and Effective Surgical Training

This article highlights the importance of considering Cognitive Load (CL) and Cognitive Load Theory (CLT) during surgical training, focusing on the acquisition of intra-operative skills. It describes the basis of CLT with the overarching aim of describing CLT-based techniques to enhance current training strategies and surgical performance, many of which are instinctively already employed in surgical practice. Currently, methods of feedback and assessment are imperfect - typically subjective, unsystematic, opportunistic, or retrospective, and at risk of human bias. Surgical Sabermetrics, the advanced analytics of surgical and audio-visual data, aims to enhance this feedback by providing objective, real-time, digital-based feedback. This article introduces the benefit of real-time measurement of CL to enhance feedback and its applications to surgical performance that follow the ethos of Surgical Sabermetrics.¹ The 2022 theme for ICOSET was "Making it Better." Cognitive Load and Surgical Sabermetrics principles provide tools to make Surgical training better, with the goal of higher quality care for patients.

Surgical Adaptation of the Situation Awareness Rating Technique (S-SART): Assessing Situational Awareness Among Medical Students

OBJECTIVE

Situational awareness (SA) impacts team dynamics and patient care. However, few formal curricula and tools exist to teach and assess longitudinal SA in surgical environments. We sought to evaluate medical students' SA in the operating room (OR) during surgical clerkships over time.

DESIGN

We prospectively administered pre- and post-rotation surveys using a surgical adaptation of the previously validated Situation Awareness Rating Technique (SART). Surgical SART (S-SART) scores were calculated as: Situational Awareness = Understanding - (Attentional Demand - Attentional Supply) + 14. Each of 10 questions was scored from 1 to 7, with a maximum possible score of 60. Two-tailed two-sample and paired t-tests were conducted to determine differences in S-SART scores.

PARTICIPANTS/SETTING

Sixty-nine of 75 second- and third-year medical students rotating in Surgery and Obstetrics/Gynecology clerkships at a tertiary academic medical center from September 2018 to September 2019 completed the surveys for a response rate of 92.0%.

RESULTS

The mean baseline pre-rotation baseline S-SART score was 29.5 (SD=8.3), which increased by 20.9% to 35.9 (SD=6.9) post-rotation (p<0.001). Significant predictors of higher baseline S-SART scores included age ≥25 (p=0.02), surgical career interest (p=0.01), completion of prior surgical rotations (p<0.001), confidence about pre-procedural OR steps (p<0.001), and knowledge of OR item locations (p<0.001). There were increases in each of the subdomains of Understanding-including OR familiarity (3.2 to 4.3, p<0.001), knowledge of the medical student OR role (3.5 to 5.7, p<0.001), and knowledge of others' OR roles (4.1 to 5.9, p<0.001)-along with increases in alertness within Attentional Supply (5.1 to 5.5, p=0.001).

CONCLUSIONS

SA is a learnable skill with increases in S-SART scores from pre- to post-rotation, with the greatest gains in the domain of Understanding. More advanced aspects of SA such as Attentional Demand may require further surgical training or curricular innovations to address these needs.

Perceived effectiveness of an innovative mobile-based serious game on the improvement of soft skills in minimally invasive surgical training

INTRODUCTION

Soft skills have a pertinent role for the quality and efficient outcomes in minimally invasive surgical procedures that are difficult to practice due to time constraints, limited resources, and other logistical reasons. The literature says that serious games serve as better resources for learning soft skills but needs evaluation from stakeholders. This study explores the perceived effectiveness of serious gaming intervention as a learning tool to improve communication, collaborative skills among the residents in surgery, house officers and junior doctors.

METHOD

A total of nine participants volunteered to take part in the exploratory study. During the study, the participants responded to a pre-test quiz, followed by exploration of the game and a post-test quiz. The perceived perceptions on the effectiveness of the gaming intervention were collected on a five-point Likert scale questionnaire with open-ended questions.

RESULTS

There was no significant difference in pre-test and post-test scores on communication and collaborative skills. Among the three themes, there was significant effect of usefulness of the application of improvement of soft skills (90% response); however, technicalities should be addressed (50% of responses). The qualitative feedback renders that, better graphics, tutorial run, user-friendly interface and controls, and enhanced 3D environment would enhance the efficacy of the prototype.

CONCLUSION

An innovative mobile-based serious gaming intervention was developed and tested for its effectiveness as a resource to develop soft skills among surgeons in training. The results indicate that gamified interventions can serve as educational resources and supplement the self-directed learning in surgical education.

Video Commentary & Machine Learning: Tell Me What You See, I Tell You Who You Are

BACKGROUND & OBJECTIVE

Teaching and assessment of complex problem solving are a challenge for medical education. Integrating Machine Learning (ML) into medical education has the potential to revolutionize teaching and assessment of these problem-solving processes. In order to demonstrate possible applications of ML to education, we sought to apply ML in the context of a structured Video Commentary (VC) assessment, using ML to predict residents' training level.

SETTING

A secondary analysis of multi-institutional, IRB approved study. Participants had completed the VC assessment consisting of 13 short (20-40 seconds) operative video clips. They were scored in real-time using an extensive checklist by an experienced proctor in the assessment. A ML model was developed using TensorFlow and Keras. The individual scores of the 13 video clips from the VC assessment were used as the inputs for the ML model as well as for regression analysis.

PARTICIPANTS

A total of 81 surgical residents of all postgraduate years (PGY) 1-5 from 7 institutions constituted the study sample.

RESULTS

Scores from individual VC clips were strongly positively correlated with PGY level (p = 0.001). Some video clips were identified to be strongly correlated with a higher total score on the assessment; others had significant influence when used to predict trainees' PGY levels. Using a supervised machine learning model to predict trainees' PGY resulted in a 40% improvement over traditional statistical analysis.

CONCLUSIONS

Performing better in a few select video clips was key to obtaining a higher total score but not necessarily foretelling of a higher PGY level. The use of the total score as a sole measure may fail to detect deeper relationships. Our ML model is a promising tool in gauging learners' levels on an assessment as extensive as VC. The model managed to approximate residents' PGY levels with a lower MAE than using traditional statistics. Further investigations with larger datasets are needed.

"Why would you want to do that?"Surgical Interns Reflect on Discouragement From Entering Surgical Fields

INTRODUCTION

Decreasing numbers of medical students are choosing to pursue surgical careers. This study highlights individual experiences of surgical interns receiving discouragement from pursuing surgery as a career.

METHODS

We interviewed 24 incoming surgical interns from 7 institutions and 7 surgical subspecialties about their experiences with discouragement from surgery.

RESULTS

All surgical interns discussed experiencing discouragement from pursuing surgery as a career. Family, friends, the general public, and medical professionals, including surgeons, served as sources of discouragement. Reasons for discouragement fell into 3 main themes: The Surgeon and Surgical Culture, The Sacrifices, and The Sexism. Despite its pervasiveness, participants reconciled the discouragement received.

DISCUSSION

Discouragement from surgery is pervasive and centered around surgeon stereotypes and perceptions of surgical culture, significant personal sacrifices required, and traditional gender-related expectations. These results highlight the importance of individual surgeons' comments on student experience and can be used to improve the perception of surgery amongst prospective interns.

How crises work: A model of error cause and effect in surgical practice

INTRODUCTION

Surgical crises have major consequences for patients, staff and healthcare institutions. Nevertheless, their aetiology and evolution are poorly understood outside the remit of root-cause analyses.

AIMS

To develop a crisis model in surgery in order to aid the reporting and management of safety critical events.

METHODS

A narrative review surveyed the safety literature on failure causes, mechanisms and effects in the context of surgical crises. Sources were identified using non-probability sampling, with selection and inclusion being determined by author panel consensus. The data underwent thematic analysis and reporting followed the recommendation of the SALSA framework.

RESULTS

Data from 133 sources derived five principal themes. Analysis suggested that surgical care processes become destabilized in a step-wise manner. This crisis chain is initiated by four categories of threat or risk: (i) the systems in which surgeons operate; (ii) surgeons' technical, cognitive and behavioural skills; (iii) surgeons' physiological and psychological state (operational condition); and (iv) professional culture. Once triggered, the crisis chain is driven by only three types of errors: Type I. Performance errors consist of failures to diagnose, plan or execute tasks; Type II. Awareness errors are failures to recognise, comprehend or extrapolate the impact of performance failures; Type III. Rescue errors represent failures to correct faulty performance. The co-occurrence of all three error types gives rise to harm, which can lead to a crisis in the absence of mitigating actions.

CONCLUSION

Surgical crises may be triggered by four categories of threat and driven by only three types of error. These may represent universal targets for safety interventions that create new opportunities for crisis management.

Evaluating Nontechnical Skills in US Emergency Departments Using Simulation: Validating and Contextualizing a UK Assessment Tool

INTRODUCTION

Nontechnical skills (NTS) in medicine are the "cognitive, social, and personal resource skills that complement technical skills contributing to safe and efficient care." We aimed to (1) evaluate the validity and reliability of a 12-element United Kingdom emergency medicine (EM) NTS assessment tool in the context of United States (US) EM practice and (2) identify behaviors unique to US clinical practice.

METHODS

This was a mixed methods study conducted in 2 phases, following Kane's validity framework. The intended use of the NTS tool is to provide formative assessment of US EM physicians (EPs) from a video of simulated clinical encounters. In phase I, a focus group assessed the appropriateness of each aspect of the tool in the context of US EM practice by reviewing and identifying the NTS of an EP in a simulated clinical scenario. In phase II, EPs (N = 208) attending a national EM conference evaluated an EP's behaviors in 1 of 2 video simulations. Reliability in the form of internal consistency was calculated using Cronbach α. All participants suggested exemplar behaviors for the 12 elements in the context of their own clinical practice and generated new assessment elements.

RESULTS

Internal consistency was acceptable (α > 0.7) for all categories, except teamwork and cooperation. Participants proposed 4 novel behavioral elements and suggested US exemplar behaviors for all 12 original elements.

CONCLUSIONS

This tool can be used to assess US EP's NTS for the purpose of formative assessment. Refinement of exemplar behaviors and inclusion of novel US-specific elements may optimize usability.

The role of teamwork and non-technical skills for improving emergency surgical outcomes: an international perspective

The assurance of patient safety in emergency general surgery remains challenging due to the patients’ high-risk underlying conditions and the wide variability in emergency surgical care provided around the globe. The authors of this article convened as an expert panel on patient safety in surgery at the 8^th International Conference of the World Society of Emergency Surgery (WSES) in Edinburgh, Scotland, on September 7–10, 2021. This review article represents the proceedings from the expert panel discussions at the WSES congress and was designed to provide an international perspective on optimizing teamwork and non-technical skills in emergency general surgery.

Objective Measures of Surgeon Non-Technical Skills in Surgery: A Scoping Review

OBJECTIVE

The purpose of this study was to identify, synthesize, and discuss objective behavioral or physiological metrics of surgeons' nontechnical skills (NTS) in the literature.

BACKGROUND

NTS, or interpersonal or cognitive skills, have been identified to contribute to safe and efficient surgical performance; however, current assessments are subjective, checklist-based tools. Intraoperative skill evaluation, such as technical skills, has been previously utilized as an objective measure to address such limitations.

METHODS

Five databases in engineering, behavioral science, and medicine were searched following PRISMA reporting guidelines. Eligibility criteria included studies with NTS objective measurements, surgeons, and took place within simulated or live operations.

RESULTS

Twenty-three articles were included in this review. Objective metrics included communication metrics and measures from physiological responses such as changes in brain activation and motion of the eye. Frequencies of content-coded communication in surgery were utilized in 16 studies and were associated with not only the communication construct but also cognitive constructs of situation awareness and decision making. This indicates the underlying importance of communication in evaluating the NTS constructs. To synthesize the scoped literature, a framework based on the one-way communication model was used to map the objective measures to NTS constructs.

CONCLUSION

Objective NTS measurement of surgeons is still preliminary, and future work on leveraging objective metrics in parallel with current assessment tools is needed.

APPLICATION

Findings from this work identify objective NTS metrics for measurement applications in a surgical environment.

State of science: models and methods for understanding and enhancing teams and teamwork in complex sociotechnical systems

This state of the science review brings together the disparate literature of effective strategies for enhancing and accelerating team performance. The review evaluates and synthesises models and proposes recommended avenues for future research. The two major models of the Input-Mediator-Output-Input (IMOI) framework and the Big Five dimensions of teamwork were reviewed and both will need significant development for application to future teams comprising non-human agents. Research suggests that a multi-method approach is appropriate for team measurements, such as the integration of methods from self-report, observer ratings, event-based measurement and automated recordings. Simulations are recommended as the most effective team-based training interventions. The impact of new technology and autonomous agents is discussed with respect to the changing nature of teamwork. In particular, whether existing teamwork models and measures are suitable to support the design, operation and evaluation of human-nonhuman teams of the future. Practitioner summary: This review recommends a multi-method approach to the measurement and evaluation of teamwork. Team models will need to be adapted to describe interaction with non-human agents, which is what the future is most likely to hold. The most effective team training interventions use simulation-based approaches.

Virtual non-technical skills assessment training is an effective, scalable approach for novice raters

OBJECTIVES

The COVID-19 pandemic has forced a creative transition to virtual platforms due to physical distancing and travel restrictions. We designed and tested a highly scalable virtual training curriculum for novice raters using the Oxford NOTECHS non-technical skills rating system.

DESIGN

A three-day training course comprising virtual didactics, virtually facilitated simulations, and independent live observations was implemented. NOTECHS scores were submitted for eleven standardized video simulations and four live operations. Intraclass correlation coefficients (ICCs) were calculated for total NOTECHS scores and subcomponent scores. Raters previously trained in-person with the same standardized videos served as a comparator group for equivalence testing.

SETTING

All study activities were conducted in a large academic tertiary referral center in Singapore as part of an ongoing surgical safety initiative.

PARTICIPANTS

Seven staff members underwent training (three virtually and four in-person).  None had prior surgical experience or non-technical skills assessment training.

RESULTS

ICCs for total NOTECHS scores were 0.85 (95% CI, 0.73-0.98) for virtually trained raters and 0.83 for those trained in-person (95% CI, 0.68-0.99).  Scores were equivalent between groups within a 10% margin.

CONCLUSIONS

Non-technical skills assessment can be reliably taught in a highly scalable virtual format. Virtual NOTECHS training is a valuable tool for educational and quality improvement initiatives during the COVID-19 pandemic and for centers that lack ready access to onsite non-technical skills training expertise.

82 Surgeons Don’t Just Operate: Improving Junior Surgical Practitioners’ Non-Technical Skills Outside of the Operating Theatre Using High-Fidelity Simulation

The non-technical skills of surgeons play a significant role in patient confidence, experience and safety

The aim of the study was to improve non-technical skills and confidence of junior surgical practitioners regarding DNACPR conversations and obtaining operative consent.

We have developed a speciality-specific course to address the educational deficit surrounding non-technical skills outside of the operating theatre, focussing on DNACPR and consent, for junior surgical practitioners. It was piloted at Aintree University Hospital with their Orthopaedic Department. We combined lectures and workshops delivered by subject matter specialists with simulated scenarios. The simulated scenarios were debriefed by human factors specialists using the gold standard validated NOTSS tool

The pilot course in June was a resounding success; candidate confidence scores increased significantly across both skills and knowledge surrounding capacity, consent and DNACPR. All attending candidates have recommended the course to their peers and suggested integration into the Foundation training curriculum. However, identifying and implementing courses appropriate for all foundation doctors with such a specialized course would be challenging. Most foundation doctors have at least one placement in a surgical specialty. Therefore, it is more realistic to adjust the course to become less specialized, encompassing all the surgical specialities, to target all foundation doctors initially instead of juniors in each speciality. Following the capture of foundation doctors, the second phase of implementation will return to speciality-specific courses to address the learning needs of non-medical practitioners, such as Physician Associates and Advanced Nurse Practitioners, working in surgical departments.

A review of robotic surgical training: establishing a curriculum and credentialing process in ophthalmology

Ophthalmic surgery requires a highly dexterous and precise surgical approach to work within the small confines of the eye, and the use of robotics offers numerous potential advantages to current surgical techniques. However, there is a lag in the development of a comprehensive training and credentialing system for robotic eye surgery, and certification of robotic skills proficiency relies heavily on industry leadership. We conducted a literature review on the curricular elements of established robotics training programs as well as privileging guidelines from various institutions to outline key components in training and credentialing robotic surgeons for ophthalmic surgeries. Based on our literature review and informal discussions between the authors and other robotic ophthalmic experts, we recommend that the overall training framework for robotic ophthalmic trainees proceeds in a stepwise, competency-based manner from didactic learning, to simulation exercises, to finally operative experiences. Nontechnical skills such as device troubleshooting and interprofessional teamwork should also be formally taught and evaluated. In addition, we have developed an assessment tool based on validated global rating scales for surgical skills that may be used to monitor the progress of trainees. Finally, we propose a graduating model for granting privileges to robotic surgeons. Further work will need to be undertaken to assess the feasibility, efficacy and integrity of the training curriculum and credentialing practices for robotic ophthalmic surgery.

Teaching and Evaluating Nontechnical Skills for General Surgery

Surgical training programs have long used quantitative measures of knowledge, as well as subjective evaluation of technical skills, to define the competence of trainees. However, a growing body of literature has shown the importance of nontechnical surgical skills as vital components of quality surgical care. Institutions must train nontechnical surgical skills, including leadership, communication, teamwork, situational awareness, and decision making, and incorporate these attributes into their evaluative processes to maximally enhance surgical performance at every career stage.

Validity of scoring systems for the assessment of technical and non-technical skills in ophthalmic surgery-a systematic review

Evaluation and recommendation of the scoring systems for technical skills (TS) and non-technical skills (NTS) assessments in ophthalmic surgery. A literature search was performed between December 2019 and May 2020. Studies describing the development or validation of TS or NTS scoring systems in ophthalmic surgery were included. Only scoring systems for completion by hand were included. The primary outcome was the validity and reliability status for each scoring system. The secondary outcome was recommendation based on modified Oxford Centre for Evidence-Based Medicine guidelines. Nineteen and five scoring systems were identified for TS and NTS respectively. TS scoring systems exist for cataract surgery (including the steps of phacoemulsification and paediatric cataract surgery) ptosis, strabismus, lateral tarsal strip, vitrectomy, and intraocular surgery in general. NTS scoring systems apply to cataract surgery or ophthalmic surgery in general. No single scoring system satisfied all validity and reliability measures. The recommended TS scoring systems are 'International Council of Ophthalmology's Ophthalmology Surgical Competency Assessment Rubrics' (ICO-OSCAR) for phacoemulsification, strabismus and paediatric cataract surgery, and 'Objective Structured Assessment of Cataract Surgical Skill' (OSACSS). Non-Technical Skills for Surgeons (NOTSS), Observational Teamwork Assessment for Surgery (OTAS) and Anaesthetists Non-Technical Skills (ANTS) are recommended for NTS. There is a paucity of NTS scoring systems. Further research is required to validate all scoring systems to consistent standards. Limitations of the assessment tools included infrequent quantification of face and content validity, and inconsistency in terminology and statistical methods between studies.

Gathering Validity Evidence to Adapt the Non-technical Skills for Surgeons (NOTSS) Assessment Tool to the United States Context

BACKGROUND

Nontechnical skills are of increasing focus for safe and effective performance in the operating room. Assessment tools have been developed in Europe, Africa, and Asia but not adapted to the unique aspects of surgical delivery in the United States. Our objective was to use the Non-Technical Skills for Surgeons (NOTSS) assessment tool as a basis to establish consensus on essential nontechnical skills for surgical trainees and practicing surgeons in the U.S surgical context.

STUDY DESIGN

A mixed-methods research design was used in the form of a modified Delphi process to build consensus on essential NOTSS. A panel of surgical experts from hospitals across the U.S used this iterative process in 4 rounds to generate, rate, and classify behaviors. The primary outcome was consensus on behaviors as being essential for surgeons to achieve the best patient outcomes in the operating room, with a median rating of ≥6 on a 7-point scale for inclusion.

RESULTS

A total of 10 surgical experts participated. One hundred and thirty eight behaviors were generated in Round 1, and reduced to 100 behaviors in Rounds 2 and 3 based on application of inclusion criteria. The final skill list consisted of behaviors in Situation Awareness (n = 26), Decision Making (n = 18), Teamwork (n = 25), and Leadership (n = 31). No additional NOTSS categories or elements emerged from the analysis. In Round 4, all 100 behaviors were successfully grouped into 12 nontechnical skills elements. Labels and definitions were reworded to reflect the U.S. context, and an appropriate assessment scale was selected.

CONCLUSIONS

A panel of surgical experts from across the U.S. reached consensus on the essential NOTSS to achieve the best patient outcomes in the operating room. These behaviors form an empirical basis for the first context-specific nontechnical skills assessment and training tool for practicing surgeons in the U.S.

Non-technical skill assessments across levels of US surgical training

BACKGROUND

To ensure safe patient care, regulatory bodies worldwide have incorporated non-technical skills proficiency in core competencies for graduation from surgical residency. We describe normative data on non-technical skill ratings of surgical residents across training levels using the US-adapted Non-Technical Skills for Surgeons (NOTSS-US) assessment tool.

METHODS

We undertook an exploratory, prospective cohort study of 32 residents-interns (postgraduate year 1), junior residents (postgraduate years 2-3), and senior residents (postgraduate years 4-5)-across 3 US academic surgery residency programs. Faculty went through online training to rate residents, directly observed residents while operating together, then submitted NOTSS-US ratings on specific resident's intraoperative performance. Mean NOTSS-US ratings (total range 4-20, sum of category scores; situation awareness, decision-making, communication/teamwork, leadership each ranged 1-5, with 1=poor, 3=average, 5=excellent) were stratified by residents' training level and adjusted for resident-, rater-, and case-level variables, using mixed-effects linear regression.

RESULTS

For 80 operations, the overall mean total NOTSS-US rating was 12.9 (standard deviation, 3.5). The adjusted mean total NOTSS-US rating was 16.0 for senior residents, 11.6 for junior residents, and 9.5 for interns. Adjusted differences for total NOTSS-US ratings were statistically significant across the following training levels: senior residents to interns (6.5; 95% confidence interval, 4.3-8.7; P < .001), senior to junior residents (4.4; 95% confidence interval, 2.5-6.2; P < .001), and junior residents to interns (2.1; 95% confidence interval, 0.3-3.9; P = .017). Differences in adjusted NOTSS-US ratings across residents' training levels persisted for individual NOTSS-US behavior categories.

CONCLUSION

These data and online training materials can support US residency programs in determining competency-based performance milestones to develop surgical trainees' non-technical skills.

Innovations in Urologic Surgical Training

PURPOSE OF REVIEW

This review aims to summarize innovations in urologic surgical training in the past 5 years.

RECENT FINDINGS

Many assessment tools have been developed to objectively evaluate surgical skills and provide structured feedback to urologic trainees. A variety of simulation modalities (i.e., virtual/augmented reality, dry-lab, animal, and cadaver) have been utilized to facilitate the acquisition of surgical skills outside the high-stakes operating room environment. Three-dimensional printing has been used to create high-fidelity, immersive dry-lab models at a reasonable cost. Non-technical skills such as teamwork and decision-making have gained more attention. Structured surgical video review has been shown to improve surgical skills not only for trainees but also for qualified surgeons. Research and development in urologic surgical training has been active in the past 5 years. Despite these advances, there is still an unfulfilled need for a standardized surgical training program covering both technical and non-technical skills.

Nontechnical skills training in cardiothoracic surgery: A pilot study

OBJECTIVE

The importance of nontechnical skills in surgery is widely recognized. We demonstrate the feasibility of administering and assessing the results of a formal Non-Technical Skills in Surgery (NOTSS) curriculum to cardiothoracic surgery residents.

METHODS

Eight cardiothoracic surgery residents participated in the NOTSS curriculum. They were assessed on their cognitive (situation awareness, decision-making) and social (communication and teamwork, leadership) skills based on simulated vignettes. The residents underwent pretraining NOTSS assessments followed by self-administered confidence ratings regarding the 4 skills. Subsequently, a formal NOTSS lecture was delivered and additional readings from the NOTSS textbook was assigned. A month later, the residents returned for post-training NOTSS assessments and self-administered confidence ratings. Changes across days (or within-day before vs after curriculum) were assessed using Wilcoxon signed rank test.

RESULTS

There was a significant improvement in the overall NOTSS assessment score (P = .01) as well as in the individual categories (situation awareness, P = .02; decision-making, P = .02; communication and teamwork, P = .01; leadership, P = .02). There was also an increase in resident self-perception of improvement on the post-training day (P = .01).

CONCLUSIONS

We have developed a simulation-based NOTSS curriculum in cardiothoracic surgery that can be formally integrated into the current residency education. This pilot study indicates the feasibility of reproducible assessments by course educators and self-assessments by participating residents in nontechnical skills competencies.

Video Commentary: Rapid-Fire Assessment of Surgical Cognitive Skills

OBJECTIVE

We describe the use of "Video Commentary", an assessment using a set of operative videos, to assess trainees' surgical cognitive skills (operative knowledge, spatial awareness, and surgical insight).

DESIGN & SETTING

The Video Commentary assessment has been routinely administered to Postgraduate Year (PGY) 1-5 general surgery residents since 2014 as part of a biannual multistation, OSCE-type exam at Mayo Clinic, Rochester, MN. Video Commentary is a rapid-fire, 6-minute assessment, where trainees watch a series of 20 to 30 second operative video clips and comment on them as they play. Each clip varies in procedure, approach, difficulty, and complexity. The combination of video clips differs according to trainees' PGY level except for a few videos that overlap among PGY groups. The name of the procedure is provided at the beginning of each clip with a countdown timer showing in the corner of the screen. A comprehensive checklist is used to score trainees' performance in real-time.

DISCUSSION

Assessment of trainees of different levels and staff surgeons show a positive correlation with the experience level (p = 0.0001). The assessment provides a safe alternative to assess trainees in the operating room and encourages them to become more effective communicators. With the use of technology, large video databases can be created to provide just-in-time tailored feedback to the trainees.

CONCLUSIONS

Video Commentary can serve as a time and resource-efficient assessment of trainees' surgical cognitive skills and insight. The use and demand of real-time commentary on operative videos may provide a viable approach to help surgeon educators determine trainees' baseline, progression, and readiness to advance.

Simulation Based Training and Assessment in Open Vascular Surgery: A Systematic Review

OBJECTIVE

The aim of this study was to systematically review the literature and give evidence based recommendations for future initiatives for simulation based training (SBT) and assessment in open vascular surgery.

DATA SOURCES

PubMed, Embase, and the Cochrane Library.

REVIEW METHODS

A systematic review of PubMed, Embase, and the Cochrane Library was performed, with the last search on 31 March 2020, to identify studies describing SBT and assessment in open vascular surgery. Kirkpatrick's levels for efficacy of training were evaluated. Validity evidence for assessment tools was evaluated according to the recommended contemporary framework by Messick.

RESULTS

Of 2 844 studies, 51 were included for data extraction. A high degree of heterogeneity in reporting standards and varying types of simulation was found. Vascular anastomosis was the most frequently simulated technical skill (43%). Assessment was mostly carried out using the Objective Structured Assessment of Technical Skills (55%). Validity evidence for assessment tools was found using outdated frameworks, and only one study used Messick's framework. Self directed training is valuable, the low trainer to trainee ratio is important to maximise efficiency, and experienced vascular surgeons are the most effective trainers.

CONCLUSION

Carefully designed and structured SBT is effective and can improve technical skills, especially in less experienced trainees. However, the supporting evidence lacks homogeneity in the reporting standards and types of simulations. Pass/fail standards that support proficiency based learning and studies investigating skills transfer should be the focus in future studies. Validity evidence of assessment tools needs to be addressed using contemporary frameworks.

A systematic review of simulation-based training tools for technical and non-technical skills in ophthalmology

To evaluate all simulation models for ophthalmology technical and non-technical skills training and the strength of evidence to support their validity and effectiveness. A systematic search was performed using PubMed and Embase for studies published from inception to 01/07/2019. Studies were analysed according to the training modality: virtual reality; wet-lab; dry-lab models; e-learning. The educational impact of studies was evaluated using Messick's validity framework and McGaghie's model of translational outcomes for evaluating effectiveness. One hundred and thirty-one studies were included in this review, with 93 different simulators described. Fifty-three studies were based on virtual reality tools; 47 on wet-lab models; 26 on dry-lab models; 5 on e-learning. Only two studies provided evidence for all five sources of validity assessment. Models with the strongest validity evidence were the Eyesi Surgical, Eyesi Direct Ophthalmoscope and Eye Surgical Skills Assessment Test. Effectiveness ratings for simulator models were mostly limited to level 2 (contained effects) with the exception of the Sophocle vitreoretinal surgery simulator, which was shown at level 3 (downstream effects), and the Eyesi at level 5 (target effects) for cataract surgery. A wide range of models have been described but only the Eyesi has undergone comprehensive investigation. The main weakness is in the poor quality of study design, with a predominance of descriptive reports showing limited validity evidence and few studies investigating the effects of simulation training on patient outcomes. More robust research is needed to enable effective implementation of simulation tools into current training curriculums.

Non-technical skills in robotic surgery and impact on near-miss events: a multi-center study

BACKGROUND

Robotic surgery requires a set of non-technical skills (NTS), because of the complex environment. We aim to study relationship between NTS and near-miss events in robotic surgery.

METHODS

This is an observational study in five French centers. Three robotic procedures were observed and filmed by one of expert trainers in NTS. They established and scored a non-technical skills in robotic surgery (NTSRS) score, that included eight items, each scored from 1 to 5, to assess the whole surgical teams. The surgical teams also self-assessed their work. The number of near-miss events was recorded and classified as minor, or major but no harm incidents, independently by two surgeons. Correlations were Spearman coefficients.

RESULTS

Of the 26 procedures included, 15 were prostatectomy (58%), 9 nephrectomy (35%), and 2 pyeloplasty (7.7%). Half of procedures (n = 13) were performed by surgeons with extensive RS experience (more than 150 procedures). Per procedure, there was a median (quartiles) of 9 (7; 11) near-miss events. There was 1 (0; 2) major near-miss events, with no harm. The median NTSRS score was 18 (14; 21), out of 40. The number of near-miss events was strongly correlated with the NTSRS score (r = - 0.92, p < 0.001) but was not correlated with the surgeon's experience. The surgeons for fifteen (58%) procedures, and the bed-side surgeons for 11 (42%) procedures, felt that there was no need for an improvement in the quality of their NTS. None of the surgeons gave a negative self-evaluation for any procedure; in three procedures (12%), the bed-side surgeons self-assessed negatively, on ergonomics.

CONCLUSION

Occurrence of near-miss events was reduced in teams managing NTS. Specific NTS surgical team training is essential for robotic surgery as it may have a significant impact on risk management.

Effect of simulation training on nurse leadership in a shared leadership model for cardiopulmonary resuscitation in the emergency department

OBJECTIVE

Empowering a senior nurse in a shared leadership role has been proposed as a more efficient set up for the cardiac arrest team in ED. In this model, a senior nurse leads the cardiac arrest algorithm which allows cognitive off-loading of the lead emergency physician. The emergency physician is then more available to perform tasks such as echocardiography and exclude reversible causes. Simulation provides an opportunity for training and practice of this shared leadership model. We hypothesised that a structured simulation training programme that focused on implementing a nurse and doctor shared leadership model for cardiopulmonary resuscitation (CPR), would improve leadership and teamwork quality in the setting of cardiac arrest as measured by a Trauma Non-technical Skills (T-NOTECHS) teamwork scale.

METHODS

Fifteen senior ED nurses participated in this pre-interventional post-observational study. Training consisted of a didactic course on team leadership and crisis resource management (CRM) followed by 4 × 10-min resuscitation scenarios with a structured debrief focusing on team leadership skills and CRM. The primary outcome was measured on scenarios 1 and 4 using a modified T-NOTECHS teamwork scale.

RESULTS

A statistically significant increase in the T-NOTECHS scale was detected for the measures of leadership (P = 0.0028), CRM (P = 0.0001), adherence to New Zealand Resuscitation Council ALS algorithm (P = 0.0088) and situational awareness (P = 0.0002).

CONCLUSION

The present study shows that a short simulation training programme improved nurse leadership and teamwork performance in the setting of a shared leadership model for CPR in the ED which could easily be replicated in other departments.

Disparity of perspectives between teachers and learners on perioperative teaching and learning

BACKGROUND

To build a consensus about learning objectives in the operating room, the aim of the study was to evaluate both surgical teacher and learner perspectives on perioperative teaching and learning in Taiwan.

METHODS

Twelve main technical and non-technical learning objectives in the operating room were evaluated by learners and surgical teachers in Kaohsiung Medical University Hospital. The learners included postgraduate year (PGY) 1-3 residents (junior learner, JL) and PGY 4-7 residents (senior learner, SL). The definition of learning preferences were recommended learning objectives, and learning load was defined as demands of learning preferences. During the survey, surgical teachers evaluated the learning preferences for the learner, and learners evaluated their learning preferences. The learners also evaluated the learning preferences that the surgical teachers should teach.

RESULTS

Response rate of the questionnaire was 65.4%. A total of 31 learners and 39 surgical teachers completed the survey. The consensus was that the need to increase the learning loads and ethical issues were the learning preferences for SL, and indications, details of procedure, and teamwork were important to both JL and SL. The teachers intended to set specific learning objectives for different learner levels, including (i) indications, details of procedure, teamwork, and postoperative care for both JL and SL; (ii) preoperative preparation, surgical anatomy, and instrument handling for JL (P = 0.022, 0.021 and 0.006); and (iii) surgical technique, independent practice, clinical reasoning, complications, and ethical issues for SL (P = 0.010, < 0.001, < 0.001, 0.001, 0.011). Resident perspective on learning objectives differed between JL and SL, and there was discrepancy between resident's learning as perceived by teachers, particularly in the JL.

CONCLUSIONS

Our study revealed significant disparity of perspectives between teachers and learners on perioperative teaching and learning. Surgical teachers should set specific learning objectives for different learner levels, since junior and senior residents have different learning preferences even though both scrub in the same case. Effective communication between teachers and learners has the potential to improve learning experience and create a positive environment in the operating room.

Peri-Operative Patient Safety - An Interactive Workshop for Section 3 CPD Credits Developed in Collaboration with the CMPA

BACKGROUND

The Royal College of Physicians and Surgeons of Canada requires physicians to collect credit in continuing professional development courses including Section 3 credits which require feedback and self-assessment. This study aims to examine the effectiveness of offering Section 3 credits in a conference setting using an interactive workshop on peri-operative patient safety developed in collaboration with the Canadian Medical Protective Association (CMPA). Both the knowledge gained and the attitudes towards the conference were analysed.

METHODS

This was a pre/post-test study design. An interactive case studies workshop was implemented on medicolegal issues for patient care, before, during, and after surgery at the Canadian Society of Otolaryngology Head and Neck Surgery annual meeting. The workshop used small group and large interactive group educational strategies to gauge knowledge of both pre and post cases. Participants completed a questionnaire at the end of the workshop comparing their attitudes before and after the workshop.

RESULTS

There were 22 participants in the workshop. A little over half knew the requirements for Section 3 CPD credits (58%) but only 36% knew how to obtain them. The data demonstrated with 95% confidence intervals, statistically significant improvement in how participants felt about their ability to identify at-risk behaviours in surgical practice (2.10 to 2.90, 3-point Likert, p<0.001), to analyze the impact of at-risk behaviour on patient care (1.95 to 2.65, p<0.001), and to develop strategies to address at-risk behaviours in surgical practice and improve patient care (1.95 to 2.80, p<0.001). One hundred percent of participants felt similar workshops should be included in future annual meetings, and 94% felt that future meetings should include more opportunities to obtain Section 3 credits.

CONCLUSION

This study demonstrates the effectiveness of an interactive workshop in a conference setting to fulfill the need for Section 3 continuing professional development credits.

A Lack of Communication and Awareness in Nontechnical Skills Training? A Qualitative Analysis of the Perceptions of Trainers and Trainees in Surgical Training

OBJECTIVE

To examine the perceptions of surgical trainees and trainers towards nontechnical skills (NTS) as a concept, its role in training, and the challenges of developing these skills.

DESIGN

A case series of semistructured interviews using an interpretivist grounded theory approach for qualitative analysis.

SETTING

East Midlands (North) core surgical training programme in the United Kingdom.

PARTICIPANTS

Ten out of 81 volunteer core surgical trainees and academic educational supervisors (consultant surgeon trainers).

RESULTS

Understanding of NTS was consistent amongst trainers and trainees but the conceived definition of NTS was much broader than previous definitions. Most viewed NTS as important for surgeons. Trainees believed trainers did not appreciate or were unaware of NTS, likely because of a lack of discussion in practice. Trainers had several reasons for not discussing NTS including insufficient personal relationships with trainees and a lack of robust evidence on which to base discussions. A lack of insight into NTS and surgeon arrogance were suggested as barrier to effective learning.

CONCLUSIONS

Apparent discordant perceptions may be contributing to a lack of focused NTS feedback for surgeons in training. To implement NTS training changes, more will have to be done to develop a shared understanding.

Can effective pedagogy be ensured in minimally invasive surgery e-learning?

Introduction: Effectiveness of e-learning diminishes without the support of a pedagogical model to guide its use. In minimally invasive surgery (MIS), this has been reported as a limitation when technology is used to deliver contents without a sound pedagogical background.Material and methods: We describe how a generic pedagogical model, the 3D pedagogy framework, can be used for setting learning outcomes and activities in e-learning platforms focused on MIS cognitive skills. A demonstrator course on Nissen fundoplication was developed following the model step-by-step in the MISTELA learning platform. Course design was informed by Kolb's Experiential learning model. Content validation was performed by 13 MIS experts.Results: Ten experts agreed on the suitability of content structuring done according to the pedagogical model. All experts agreed that the course provides means to assess the intended learning outcomes.Conclusions: This work showcases how a general-purpose e-learning framework can be accommodated to the needs of MIS training without limiting the course designers' pedagogical approach. Key advances for its success include: (1) proving the validity of the model in the wider scope of MIS skills and (2) raising awareness amongst stakeholders on the need of developing training plans with explicit, rather than assumed, pedagogical foundations. Abbreviations: MIS: minimally invasive surgery; TEL: technology enhanced learning.

Development of the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES) for non-technical skills in cataract surgery

BACKGROUND

Nontechnical skills (NTS) are fundamental for successfully managing intraoperative complications. We aimed to develop the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES); an NTS assessment system for posterior capsule rupture (PCR) during cataract surgery.

METHODS

A literature review and a focus group consisting of three cataract surgeons and one NTS researcher elicited the important NTS for the management of intraoperative cataract surgery complications. A novel taxonomy of NTS specific for PCR management was generated. Questionnaires were distributed to ophthalmologists in one UK training region. Delphi methodology was used to develop a final HUFOES draft. One further questionnaire was used to gain feasibility, educational impact and validity data.

RESULTS

All HUFOES components achieved a mean importance rating of >8/10 and achieved high interrater agreement ratings (α = 0.953). Interrater agreement scores for HUFOES categories were: teamwork and communication (α = 0.819), leadership (α = 0.859), decision making (α = 0.753), situational awareness (α = 0.840) and professionalism (α = 0.890). In all, 92.8% (n = 13) rated HUFOES as specific for use, 85.7% (n = 12) agreed it contains appropriate assessment measures, 92.8% (n = 13) agreed that training with HUFOES would enhance preparation for PCR management and 78.6% (n = 11) declared HUFOES as the preferable training system for NTS in intraoperative ophthalmic emergencies when compared with the current gold standard.

CONCLUSIONS

HUFOES has been developed and validated as a tool for the training and assessment of NTS in PCR. An NTS training programme integrated with HUFOES should be considered in order to enhance surgical NTS for managing intraoperative complications, and improve performance and outcomes following PCR.

Assessment of the Non-Technical Skills for Surgeons (NOTSS) framework in the USA

Background

Surgeons' non-technical skills are important for patient safety. The Non-Technical Skills for Surgeons assessment tool was developed in the UK and recently adapted to the US surgical context (NOTSS-US). The aim of this study was to evaluate the reliability and distribution of non-technical skill ratings given by attending (consultant) surgeons who underwent brief online training.

Methods

Attending surgeons across six specialties at a large US academic medical centre underwent a 10-min online training, then rated 60-s standardized videos of simulated operations. Intraclass correlation coefficient (ICC), and mean(s.d.) values for NOTSS-US ratings were determined for each non-technical skill category (score range 1–5, where 1 indicates poor, 3 average and 5 excellent) and for total NOTSS-US score (range 4–20; sum of 4 category scores). Outcomes were adjusted for rater characteristics including sex, specialty and clinical experience.

Results

A total of 8889 ratings were submitted by 81 surgeon raters on 30 simulated intraoperative videos. The mean(s.d.) total NOTSS-US score for all videos was 9·5(4·8) of 20. The within-video ICC for total NOTSS-US score was 0·64 (95 per cent c.i. 0·57 to 0·70). For individual non-technical skill categories, the ICC was highest for social skills (communication/teamwork: 0·63, 95 per cent c.i. 0·56 to 0·71; leadership: 0·64, 0·55 to 0·72) and lowest for cognitive skills (situation awareness: 0·54, 0·45 to 0·62; decision-making: 0·50, 0·41 to 0·59). Women gave higher total NOTSS-US scores than men (adjusted mean difference 0·93, 95 per cent c.i. 0·44 to 1·43; P = 0·001).

Conclusion

After brief online training, the inter-rater reliability of the NOTSS-US assessment tool achieved moderate strength among trained surgeons rating simulated intraoperative videos.

Validity of the Medi-StuNTS behavioural marker system: assessing the non-technical skills of medical students during immersive simulation

Background

The Medical Students’ Non-Technical Skills (Medi-StuNTS) behavioural marker system (BMS) is the first BMS to be developed specifically for medical students to facilitate training in non-technical skills (NTS) within immersive simulated acute care scenarios. In order to begin implementing the tool in practice, validity evidence must be sought. We aimed to assess the validity of the Medi-StuNTS system with reference to Messick’s contemporary validity framework.

Methods

Two raters marked video-recorded performances of acute care simulation scenarios using the Medi-StuNTS system. Three groups were marked: third-year and fourth-year medical students (novices), final-year medical students (intermediates) and core medical trainees (experts). The scores were used to make assessments of relationships to the variable of clinical experience through expert–novice comparisons, inter-rater reliability, observability, exploratory factor analysis, inter-rater disagreements and differential item functioning.

Results

A significant difference was found between the three groups (p<0.005), with experts scoring significantly better than intermediates (p<0.005) and intermediates scoring significantly better than novices (p=0.001). There was a strong positive correlation between the two raters’ scores (r=0.79), and an inter-rater disagreement of more than one point in less than one-fifth of cases. Across all scenarios, 99.7% of skill categories and 84% of skill elements were observable. Factor analysis demonstrated appropriate grouping of skill elements. Inconsistencies in test performance across learner groups were shown specifically in the skill categories of situation awareness and decision making and prioritisation.

Conclusion

We have demonstrated evidence for several aspects of validity of the Medi-StuNTS system when assessing medical students’ NTS during immersive simulation. We can now begin to introduce this system into simulation-based education to maximise NTS training in this group.

[Methodology to develop a virtual reality training for good practices in the preparation of anti-cancer drugs]

OBJECTIVE

Describe the process for designing and creating SimUPAC 360°, a virtual reality training in anti-cancer drug production units.

METHODS

A multi-centre (a University Hospital, a General Hospital and a Cancer Control Centre), inter-professional (pharmacists, hospital pharmacy technicians and health executives) working group has been set up. It was based on videoconferencing and online document sharing. The work was divided into six phases: choice of target audience and training objectives, definition of the business model, development of the scenario, shooting and editing, creation of the training tool and finally tests, adjustments and validation of the tool.

RESULTS

After brainstorming, 77 errors were proposed. Three areas have been defined: covering area, storage and production area, and isolator. They contained 15 errors among the 77 proposed and 20 points of interest. The shooting was carried out over 2 days, in 2 hospitals. Assembly was carried out by a service provider specialist in real virtuality. Before to go online, the tool was tested and validated by experts.

DISCUSSION

The establishment of a multi-centric and interdisciplinary working group, the choice of target audience, pedagogical objectives and business model ensure the economic viability and scientific and technical robustness of the tool. The scenario development requires to define: activity areas and then, number, difficulty and typology of errors.

CONCLUSION

Creation of a virtual reality training requires a consistent and structured methodology. This methodology will make it possible to develop other training scenarios.