Observational Teamwork Assessment for Surgery: Construct Validation With Expert Versus Novice Raters

Translation, cultural adaptation and content re-validation of the observational teamwork assessment for surgery tool

Background. Poor teamwork and nontechnical skill performance are increasingly recognized as important contributing factors to errors and adverse events in the operating room. Assessment of these safety critical skills is important to facilitate improvement, however there are no tools available to assess these safety skills in Latin America. This study aimed to translate, culturally adapt and content validate the Observational Teamwork Assessment for Surgery (OTAS) tool for use in Latin America. Methods. A multi-phase, multi-method study was conducted: Phase 1: translation and back-translation; Phase 2: content validity assessed via expert consensus; Phase 3: inter-rater reliability assessed via real-time observation in 98 general surgical procedures using OTAS-S. Results. The first change in OTAS-S, was to distinguish between the surgical nurses and scrub technicians (both OR team members are captured in the nursing sub-team in the original OTAS). OTAS-S consists of 168 exemplar behaviors: 60/114 identical to the exemplars listed in the original OTAS tool, 48/114 original exemplars underwent minor modifications, 13 were duplicated (to account for the additional sub-team distinguished in OTAS-S), 6 original exemplars were removed, and 47 new exemplar behaviors were added. Inter-observer agreement was substantial (KW = 0.602; IC: 0.581-0.620). The calculated KW by phase, behaviors and teams were between 0.534 and 0.678. Conclusions. The study provides a content validated teamwork assessment tool for use within Colombian operating rooms and potentially Latin-American. OTAS-S can be used to assess the quality of teamwork in ORs, facilitate structured debriefing and thus improve patient safety and reduce team-related errors.

Assessing neurosurgical non-technical skills: an exploratory study of a new behavioural marker system

RATIONALE, AIMS AND OBJECTIVES

The management of non-technical skills is a major factor affecting teamwork quality and patient safety. This article presents a behavioural marker system for assessing neurosurgical non-technical skills (BMS-NNTS). We tested the BMS during deep brain stimulation surgery.

METHOD

We developed the BMS in three stages. First, we drew up a provisional assessment tool based on the literature and observation tools developed for other surgical specialties. We then analysed videos made in an operating room (OR) during deep brain stimulation operations in order to ensure there were no significant omissions from the skills list. Finally, we used five videos of operations to identify the behavioural markers of non-technical skills in verbal communications.

RESULTS

Analyses of more than six hours of observations revealed 3515 behaviours from which we determined the neurosurgeon's non-technical skills behaviour pattern. The neurosurgeon frequently engaged in explicit coordination, situation awareness and leadership behaviours. In addition, the neurosurgeon's behaviours differed according to the stage of the operation and the OR staff members with whom she was communicating.

CONCLUSIONS

Our behavioural marker system provides a structured approach to assessing non-technical skills in the field of neurosurgery. It can also be transferred to other surgical specialties and used in surgeon training curricula.

Re-validating the Observational Teamwork Assessment for Surgery tool (OTAS-D): cultural adaptation, refinement, and psychometric evaluation

BACKGROUND

The nontechnical and team skills of surgical teams are critical for safety and efficiency in the operating room. Assessment of nontechnical and team skills can facilitate improvement by encouraging both self-reflection and team reflection, identifying training needs, and informing operating room (OR) team training approaches. The observational teamwork assessment for surgery (OTAS) tool is a well-validated and robust tool for capturing teamwork in the operating room. The aims of the present study were to systematically adapt and refine the OTAS for German-speaking OR staff and to test the adapted assessment tool (OTAS-D) for psychometric properties and metric equivalence.

METHODS

The study was carried out in three stages: at stage 1, OTAS was translated into German. At stage 2, experienced German OR experts (surgeons, OR nurses, anesthetists) were interviewed. At stage 3, two blinded assessors observed 11 general surgical operations (general surgical and vascular procedures) and interrater reliability was tested for refined OTAS-D behavioral exemplars and scorings.

RESULTS

The German OR experts confirmed the applicability and content validity of the vast majority of translated behavioral exemplars. After their evaluation, 32 items were changed slightly, six were changed substantially, and one item was added. During observations, perfect and substantial interobserver agreement was found for 77 behavioral exemplars (67.1 % of the items, kappa coefficient >0.60). Rating at all OTAS behaviors showed acceptable levels of reliability (intraclass correlation coefficients >0.72).

CONCLUSIONS

The OTAS-D is a tool for valid and reliable assessment of nontechnical skills that contribute to safe and effective surgical performance in ORs staffed by German-speaking professionals. Furthermore, our study serves as an example for systematically adapting and customizing well-established observational tools across different healthcare environments.

Utility and assessment of non-technical skills for rapid response systems and medical emergency teams

Efforts are ongoing to improve outcomes from cardiac arrest and medical emergencies. A promising quality improvement modality is use of non-technical skills (NTS) that aim to address human factors through improvements in performance of leadership, communication, situational awareness and decision-making. Originating in the airline industry, NTS training has been successfully introduced into anaesthesia, surgery, emergency medicine and other acute medical specialities. Some aspects of NTS have already achieved acceptance for cardiac arrest teams. Leadership skills are emphasised in advanced life support training and have shown favourable results when employed in simulated and clinical resuscitation scenarios. The application of NTS in medical emergency teams as part of a rapid response system attending medical emergencies is less certain; however, observations of simulations have also shown promise. This review highlights the potential benefits of NTS competency for cardiac arrest teams and, more importantly, medical emergency teams because of the diversity of clinical scenarios encountered. Discussion covers methods to assess and refine NTS and NTS training to optimise performance in the clinical environment. Increasing attention should be applied to yielding meaningful patient and organisational outcomes from use of NTS. Similarly, implementation of any training course should receive appropriate scrutiny to refine team and institutional performance.

Practicality of intraoperative teamwork assessments

BACKGROUND

High-quality teamwork among operating room (OR) professionals is a key to efficient and safe practice. Quantification of teamwork facilitates feedback, assessment, and improvement. Several valid and reliable instruments are available for assessing separate OR disciplines and teams. We sought to determine the most feasible approach for routine documentation of teamwork in in-situ OR simulations. We compared rater agreement, hypothetical training costs, and feasibility ratings from five clinicians and two nonclinicians with instruments for assessment of separate OR groups and teams.

MATERIALS AND METHODS

Five teams of anesthesia or surgery residents and OR nurses (RN) or surgical technicians were videotaped in simulations of an epigastric hernia repair where the patient develops malignant hyperthermia. Two anesthesiologists, one OR clinical RN specialist, one educational psychologist, one simulation specialist, and one general surgeon discussed and then independently completed Anesthesiologists' Non-Technical Skills, Non-Technical Skills for Surgeons, Scrub Practitioners' List of Intraoperative Non-Technical Skills, and Observational Teamwork Assessment for Surgery forms to rate nontechnical performance of anesthesiologists, surgeons, nurses, technicians, and the whole team.

RESULTS

Intraclass correlations of agreement ranged from 0.17-0.85. Clinicians' agreements were not different from nonclinicians'. Published rater training was 4 h for Anesthesiologists' Non-Technical Skills and Scrub Practitioners' List of Intraoperative Non-Technical Skills, 2.5 h for Non-Technical Skills for Surgeons, and 15.5 h for Observational Teamwork Assessment for Surgery. Estimated costs to train one rater to use all instruments ranged from $442 for a simulation specialist to $6006 for a general surgeon.

CONCLUSIONS

Additional training is needed to achieve higher levels of agreement; however, costs may be prohibitive. The most cost-effective model for real-time OR teamwork assessment may be to use a simulation technician combined with one clinical rater to allow complete documentation of all participants.

Lessons for surgeons in the final moments of Air France Flight 447

BACKGROUND

All surgeons make mistakes, and learning from critical incidents may help improve performance. The present study aimed to highlight lessons for surgeons from analysis of the final moments of Air France Flight 447. All of the authors work in teams and situations where safety, technical performance, and non-technical skills are critical. This review was born out of discussions regarding the events of Flight 447; specifically, whether the airline disaster was relevant to their work, and whether they could learn anything from it.

METHODS

The study is based on review of the crash reports of Flight 447, which lost flight speed indication after formation of ice prevented air from entering flight speed indicators during a storm. Following a subsequent stall, the aircraft fell at a rate of >10,000 feet/min until it crashed into the Atlantic Ocean, killing 228 passengers and crew.

RESULTS

There were errors in decision making, reasoning, communication, and teamwork. Such non-technical skills failures have been recognized previously and can be addressed by existing non-technical skills training. A reliance on autopilot meant that the pilots were unfamiliar with high-altitude flying when the autopilot is disengaged. They were unprepared for and affected by such a sudden and serious problem; an event called "surprise and startle" by the accident investigation. The absence of the senior pilot (who was on a scheduled break) in the critical final minutes slowed error recognition and recovery.

CONCLUSIONS

Unintended consequences of modern safety strategies may be under-recognized and can lead to adverse events. Both simulation-based and non-simulation-based training should include "surprise and startle" events beyond the scenarios trainees might expect. Likewise, in the face of increasing reliance on modern technology, surgeons should ensure that they would be able to perform procedures in the absence of such technologies. Specific training may improve surgeons' non-technical skills, and recognition of such skills could also be used to help select future surgeons.

Development of a knowledge, skills, and attitudes framework for training in laparoscopic cholecystectomy

BACKGROUND

The implementation of duty-hour restrictions and a heightened awareness of patient safety has changed resident education and training. A new focus has been placed on high-yield training programs and simulation training has naturally grown to fill this need.

METHODS

This article discusses the development of a training framework, knowledge, skills, and attitudes, and the design of a surgical simulation curriculum. Five residents were recruited for a pilot study of the curriculum.

RESULTS

A successful framework for curriculum development was implemented using laparoscopic cholecystectomy as the example. The curriculum consisted of classroom and virtual reality simulation training and was completed in 3.1 to 4.8 hours.

CONCLUSIONS

The current curricula that have been developed for surgical education cover the breadth of a surgical residency well. This curriculum went beyond these curricula and developed a structured framework for surgical training, a method that can be applied to any procedure.

Validation of measures from a thoracoscopic esophageal atresia/tracheoesophageal fistula repair simulator

PURPOSE

A validated high fidelity simulation model would provide a safe environment to teach thoracoscopic EA/TEF repair to novices. The study purpose was to evaluate validity evidence for performance measures on an EA/TEF simulator.

METHODS

IRB-exempt data were collected from 12 self-reported "novice" and 8 "experienced" pediatric surgeons. Participants evaluated the EA/TEF repair simulator using survey ratings that were analyzed for test content validity evidence. Additionally, deidentified operative performances were videotaped and independently rated by two surgeons using the Objective Structured Assessment for Technical Skills (OSATS) instrument. Novice and experienced OSATS were compared with p<.05 significant.

RESULTS

Participants had high overall simulator ratings. Internal structure was supported by high interitem consistency (α=.95 and .96) and interrater agreement (ICC) [.52, .84] for OSATS ratings. Experienced surgeons performed at a significantly higher level than novices for all five primary and two supplemental OSATS items (p<.05).

CONCLUSION

Favorable participant ratings indicate the simulator is relevant to clinical practice and valuable as a learning tool. Further, performance ratings can discriminate experienced and novice performances of EA/TEF repair. These findings support the use of the simulator for performance assessment, representing the first validated measures from a simulator intended for pediatric surgical training.

Reliability of team-based self-monitoring in critical events: a pilot study

BACKGROUND

Teamwork is a critical component during critical events. Assessment is mandatory for remediation and to target training programmes for observed performance gaps.

METHODS

The primary purpose was to test the feasibility of team-based self-monitoring of crisis resource management with a validated teamwork assessment tool. A secondary purpose was to assess item-specific reliability and content validity in order to develop a modified context-optimised assessment tool.We conducted a prospective, single-centre study to assess team-based self-monitoring of teamwork after in-situ inter-professional simulated critical events by comparison with an assessment by observers. The Mayo High Performance Teamwork Scale (MHPTS) was used as the assessment tool with evaluation of internal consistency, item-specific consensus estimates for agreement between participating teams and observers, and content validity.

RESULTS

105 participants and 58 observers completed the MHPTS after a total of 16 simulated critical events over 8 months. Summative internal consistency of the MHPTS calculated as Cronbach's alpha was acceptable with 0.712 for observers and 0.710 for participants. Overall consensus estimates for dichotomous data (agreement/non-agreement) was 0.62 (Cohen's kappa; IQ-range 0.31-0.87). 6/16 items had excellent (kappa > 0.8) and 3/16 good reliability (kappa > 0.6). Short questions concerning easy to observe behaviours were more likely to be reliable. The MHPTS was modified using a threshold for good reliability of kappa > 0.6. The result is a 9 item self-assessment tool (TeamMonitor) with a calculated median kappa of 0.86 (IQ-range: 0.67-1.0) and good content validity.

CONCLUSIONS

Team-based self-monitoring with the MHPTS to assess team performance during simulated critical events is feasible. A context-based modification of the tool is achievable with good internal consistency and content validity. Further studies are needed to investigate if team-based self-monitoring may be used as part of a programme of assessment to target training programmes for observed performance gaps.

Training faculty in nontechnical skill assessment: national guidelines on program requirements

OBJECTIVE

To develop guidelines for a faculty training program in nontechnical skill assessment in surgery.

BACKGROUND

Nontechnical skills in the operating room are critical for patient safety. The successful integration of these skills into workplace-based assessment is dependent upon the availability of faculty who are able to teach and assess them. At present, no guidelines exist regarding the training requirements for such faculty in surgical contexts.

METHODS

The development of the guidelines was carried out in several stages: stage 1-a detailed literature review on current training for nontechnical skill assessors; stage 2-semistructured interviews with a multidisciplinary panel (consisting of clinicians and psychologists/human factors specialists) of experts in surgical nontechnical skills; and stage 3-interview findings fed into an Expert Consensus Panel (ECP) Delphi approach to establish consensus regarding training requirements for faculty assessing nontechnical skills in surgery.

RESULTS

The ECP agreed that training in nontechnical skill assessment should be delivered by a multidisciplinary team consisting of clinicians and psychologists/human factors specialists. The ECP reached consensus regarding who should be targeted to be trained as faculty (including proficiency and revalidation requirements). Consensus was reached on 7 essential training program content elements (including training in providing feedback/debriefing) and 8 essential methods of evaluating the effectiveness of a "train-the-trainers" program.

CONCLUSIONS

This study provides evidence-based guidelines that can be used to guide the development and evaluation of programs to educate faculty in the training and assessment of nontechnical skills. Uptake of these guidelines could accelerate the development of surgical expertise required for safe and high-quality patient care.

Improving decision making in multidisciplinary tumor boards: prospective longitudinal evaluation of a multicomponent intervention for 1,421 patients

BACKGROUND

Due to its complexity, cancer care is increasingly being delivered by multidisciplinary tumor boards (MTBs). Few studies have investigated how best to organize and run MTBs to optimize clinical decision making. We developed and evaluated a multicomponent intervention designed to improve the MTB's ability to reach treatment decisions.

STUDY DESIGN

We conducted a prospective longitudinal study during 16 months that evaluated MTB decision making for urological cancer patients at a university hospital in London, UK. After a baseline period, MTB improvement interventions (eg, MTBs checklist, MTB team training, and written guidance) were delivered sequentially. Outcomes measures were the MTB's ability to reach a decision, the quality of information presentation, and the quality of teamwork (as assessed by trained assessors using a previously validated observational assessment tool). The efficacy of the intervention was evaluated using multivariate analyses.

RESULTS

There were 1,421 patients studied between December 2009 and April 2, 2011. All outcomes improved considerably between baseline and intervention implementation: the MTB's ability to reach a decision rose from 82.2% to 92.7%, quality of information presentation rose from 29.6% to 38.3%, and quality of teamwork rose from 37.8% to 43.0%. The MTB's ability to reach a treatment decision was related to the quality of available information (r = 0.298; p < 0.05) and quality of teamwork within the MTB (r = 0.348; p < 0.05). The most common barriers to reaching clinical decisions were inadequate radiologic information (n = 77), inadequate pathologic information (n = 51), and inappropriate patient referrals (n = 21).

CONCLUSIONS

Multidisciplinary tumor board-delivered treatment is becoming the standard for cancer care worldwide. Our intervention is efficacious and applicable to MTBs and can improve decision making and expedite cancer care.

Assessment of teamwork during structured interdisciplinary rounds on medical units

BACKGROUND

Interdisciplinary rounds (IDR) provide a means to assemble hospital team members and improve collaboration. Little is known about teamwork during IDR.

OBJECTIVE

To evaluate and characterize teamwork during IDR.

DESIGN

Cross-sectional observational study.

SETTING

Six medical units which had implemented structured interdisciplinary rounds (SIDR).

MEASUREMENTS

We adapted the Observational Teamwork Assessment for Surgery (OTAS) tool, a behaviorally anchored rating scale shown to be reliable and valid in surgical settings. OTAS provides scores ranging from 0 to 6 (0 = problematic behavior; 6 = exemplary behavior) across 5 domains (communication, coordination, cooperation/backup behavior, leadership, and monitoring/situational awareness) and for prespecified subteams. Two researchers conducted direct observations using the adapted OTAS tool.

RESULTS

We conducted 7-8 independent observations for each unit (total = 44) and 20 joint observations. Inter-rater reliability was excellent at the unit level (Spearman's rho = 0.75), and good across domains (rho = 0.53-0.68) and subteams (rho = 0.53-0.76) with the exception of the physician subteam, for which it was poor (rho = 0.35). Though teamwork scores were generally high, we found differences across units, with a median (interquartile range [IQR]) 4.5 (3.9-4.9) for the lowest and 5.4 (5.3-5.5) for the highest performing unit (P < 0.01). Domain scores differed, with leadership receiving the lowest (median [IQR] = 5.0 [4.6-5.3]), and cooperation/backup behavior and monitoring/situational awareness receiving highest scores (median [IQR] = 5.4 [5.0-5.5] and 5.4 [5.0-5.7]). Differences across subteams were of borderline significance (P = 0.05).

CONCLUSIONS

The adapted OTAS instrument demonstrated acceptable reliability for assessing teamwork during SIDR across units, domains, and most subteams. Variation in performance suggests a need to improve consistency of teamwork and emphasizes the importance of leadership.

Protecting patients from an unsafe system: the etiology and recovery of intraoperative deviations in care

OBJECTIVE

To understand the etiology and resolution of unanticipated events in the operating room (OR).

BACKGROUND

The majority of surgical adverse events occur intraoperatively. The OR represents a complex, high-risk system. The influence of different human, team, and organizational/environmental factors on safety and performance is unknown.

METHODS

We video-recorded and transcribed 10 high-acuity operations, representing 43.7 hours of patient care. Deviations, defined as delays and/or episodes of decreased patient safety, were identified by majority consensus of a multidisciplinary team. Factors that contributed to each event and/or mitigated its impact were determined and attributed to the patient, providers, or environment/organization.

RESULTS

Thirty-three deviations (10 delays, 17 safety compromises, 6 both) occurred--with a mean of 1 every 79.4 minutes. These deviations were multifactorial (mean 3.1 factors). Problems with communication and organizational structure appeared repeatedly at the root of both types of deviations. Delays tended to be resolved with vigilance, communication, coordination, and cooperation, while mediation of safety compromises was most frequently accomplished with vigilance, leadership, communication, and/or coordination. The organization/environment was not found to play a direct role in compensation.

CONCLUSIONS

Unanticipated events are common in the OR. Deviations result from poor organizational/environmental design and suboptimal team dynamics, with caregivers compensating to avoid patient harm. Although recognized in other high-risk domains, such human resilience has not yet been described in surgery and has major implications for the design of safety interventions.

Adjusting team involvement: a grounded theory study of challenges in utilizing a surgical safety checklist as experienced by nurses in the operating room

Background

Even though the use of perioperative checklists have resulted in significant reduction in postoperative mortality and morbidity, as well as improvements of important information communication, the utilization of checklists seems to vary, and perceived barriers are likely to influence compliance. In this grounded theory study we aimed to explore the challenges and strategies of performing the WHO’s Safe Surgical Checklist as experienced by the nurses appointed as checklist coordinators.

Methods

Grounded theory was used in gathering and analyzing data from observations of the checklist used in the operating room, in conjunction with single and focus group interviews. A purposeful sample of 14 nurse-anesthetists and operating room nurses as surgical team members in a tertiary teaching hospital participated in the study.

Results

The nurses’ main concern regarding checklist utilization was identified as “how to obtain professional and social acceptance within the team”. The emergent grounded theory of “adjusting team involvement” consisted of three strategies; distancing, moderating and engaging team involvement. The use of these strategies explains how they resolved their challenges. Each strategy had corresponding conditions and consequences, determining checklist compliance, and how the checklist was used.

Conclusion

Even though nurses seem to have a loyal attitude towards the WHO’s checklist regarding their task work, they adjusted their surgical team involvement according to practical, social and professional conditions in their work environment. This might have resulted in the incomplete use of the checklist and therefore a low compliance rate. Findings also emphasized the importance of: a) management support when implementing WHO’s Safe Surgical Checklist, and b) interprofessional education approach to local adaptation of the checklists use.

Validation of a detailed scoring checklist for use during advanced cardiac life support certification

INTRODUCTION

Defining valid, reliable, defensible, and generalizable standards for the evaluation of learner performance is a key issue in assessing both baseline competence and mastery in medical education. However, before setting these standards of performance, the reliability of the scores yielding from a grading tool must be assessed. Accordingly, the purpose of this study was to assess the reliability of scores generated from a set of grading checklists used by nonexpert raters during simulations of American Heart Association (AHA) Megacodes.

METHODS

The reliability of scores generated from a detailed set of checklists, when used by 4 nonexpert raters, was tested by grading team leader performance in 8 Megacode scenarios. Videos of the scenarios were reviewed and rated by trained faculty facilitators and a group of nonexpert raters. The videos were reviewed "continuously" and "with pauses." The grading made by 2 content experts served as the reference standard, and 4 nonexpert raters were used to test the reliability of the checklists.

RESULTS

Our results demonstrate that nonexpert raters are able to produce reliable grades when using the checklists under consideration, demonstrating excellent intrarater reliability and agreement with a reference standard. The results also demonstrate that nonexpert raters can be trained in the proper use of the checklist in a short amount of time, with no discernible learning curve thereafter. Finally, our results show that a single trained rater can achieve reliable scores of team leader performance during AHA Megacodes when using our checklist in a continuous mode because measures of agreement in total scoring were very strong [Lin's (Biometrics 1989;45:255-268) concordance correlation coefficient, 0.96; intraclass correlation coefficient, 0.97].

CONCLUSIONS

We have shown that our checklists can yield reliable scores, are appropriate for use by nonexpert raters, and are able to be used during continuous assessment of team leader performance during the review of a simulated Megacode. This checklist may be more appropriate for use by advanced cardiac life support instructors during Megacode assessments than the current tools provided by the AHA.

Patient safety in surgical oncology: perspective from the operating room

Despite knowledge that most surgical adverse events occur in the operating room (OR), understanding of the intraoperative phase of care is incomplete; most studies measure surgical safety in terms of preoperative risk or postoperative morbidity and mortality. Because of the OR's complexity, human factors engineering provides an ideal methodology for studies of intraoperative safety. This article reviews models of error and resilience as delineated by human factors experts, correlating them to OR performance. Existing methodologies for studying intraoperative safety are then outlined, focusing on video-based observational research. Finally, specific human and system factors examined in the OR are detailed.

Building global capacity for patient safety: a training program for surgical safety research in developing and transitional countries

BACKGROUND

Recent studies show a significant rate of adverse events in hospitalized patients in developing/transitional countries--with approximately 18% of them related to surgical procedures. Understanding and preventing these errors requires adequate training in patient safety research methods--however, relevant training programs are currently lacking. We developed, delivered and evaluated a training program to address this gap.

METHODS

A one-day training program was developed based on the recently published WHO core competencies for patient safety research. The focus was on surgical patient safety research - including human factors, operating room (OR) teamwork, the OR environment, and safety culture. Feasibility, relevance and preliminary evaluation of the program ('proof of concept' testing) was conducted in Bogotá, Colombia in July 2011. A validated evaluation framework was utilized, assessing participants' objective knowledge, attitudes, and observational skills.

RESULTS

30 postgraduate students from a range of clinical/non-clinical disciplines signed up and 17 attended the program. Participants' knowledge of surgical patient safety significantly improved upon program completion (Mean pre-course=55% vs. Mean post-course=68%, P<0.01), as did their confidence and understanding of problems and methodologies to assess OR patient safety, and teamwork issues (P<0.05). Observational skills in recognizing safety-related behaviors using OTAS (i.e., quality of teamwork) improved on qualitative evaluation.

CONCLUSIONS

We have developed a viable, WHO-driven training program that can be delivered to clinical and non-clinical researchers to develop their competencies and thereby build capacity in developing/transitional countries to carry out surgical safety research. All program materials are available in English and Spanish for research, training and dissemination.

Observational teamwork assessment for surgery: feasibility of clinical and nonclinical assessor calibration with short-term training

OBJECTIVES

To assess the feasibility of training clinical and nonclinical novice assessors to rate teamwork behavior in the operating room with short-term structured training using the observational teamwork assessment for surgery (OTAS) tool.

BACKGROUND

Effective teamwork is fundamental to the delivery of optimal patient care in the operating room (OR). OTAS provides a comprehensive and robust measure of teamwork in surgery. To date, assessors with a background in psychology/human factors have been shown to be able to use OTAS reliably after training. However, the feasibility of observer training over a short timescale and accessibility to the wider clinical community (ie, OTAS use by clinicians) are yet to be empirically demonstrated.

METHODS

Ten general surgery cases were observed and assessed using OTAS in real-time by an expert in rating OTAS behaviors (100+ cases rated) and 4 novices: 2 psychologists and 2 surgeons. Assessors were blinded to each other's scores during observations. After each observation, scores were compared and discussed between expert and novice assessors in a debriefing session.

RESULTS

All novices were reliable with the expert to a acceptable degree at rating all OTAS behaviors by the end of training (intraclass correlation coefficients ≥0.68). For 3 of the 5 behaviors (communication, cooperation, and leadership), calibration improved most rapidly across the first 7 observed cases. For monitoring/situational awareness, calibration improved steadily across the 10 observed cases. For coordination, no significant improvement in calibration over time was observed because of high interrater reliability from the outset (ie, a ceiling effect). There was no significant difference between surgeons and psychologists in their calibration with the expert.

CONCLUSIONS

It is feasible to train both clinicians and nonclinicians to use OTAS to assess teamwork behaviors in ORs over a short structured training period. OTAS is an accessible tool that can be used robustly (ie, reliably) by assessors from both clinical and nonclinical backgrounds.

Team performance in resuscitation teams: comparison and critique of two recently developed scoring tools

Background and aim

Following high profile errors resulting in patient harm and attracting negative publicity, the healthcare sector has begun to focus on training non-technical teamworking skills as one way of reducing the rate of adverse events. Within the area of resuscitation, two tools have been developed recently aiming to assess these skills – TEAM and OSCAR. The aims of the study reported here were:1.

To determine the inter-rater reliability of the tools in assessing performance within the context of resuscitation.

2.

To correlate scores of the same resuscitation teams episodes using both tools, thereby determining their concurrent validity within the context of resuscitation.

3.

To carry out a critique of both tools and establish how best each one may be utilised.

Methods

The study consisted of two phases – reliability assessment; and content comparison, and correlation. Assessments were made by two resuscitation experts, who watched 24 pre-recorded resuscitation simulations, and independently rated team behaviours using both tools. The tools were critically appraised, and correlation between overall score surrogates was assessed.

Results

Both OSCAR and TEAM achieved high levels of inter-rater reliability (in the form of adequate intra-class coefficients) and minor significant differences between Wilcoxon tests. Comparison of the scores from both tools demonstrated a high degree of correlation (and hence concurrent validity). Finally, critique of each tool highlighted differences in length and complexity.

Conclusion

Both OSCAR and TEAM can be used to assess resuscitation teams in a simulated environment, with the tools correlating well with one another. We envisage a role for both tools – with TEAM giving a quick, global assessment of the team, but OSCAR enabling more detailed breakdown of the assessment, facilitating feedback, and identifying areas of weakness for future training.

Assessing teamwork in the trauma bay: introduction of a modified "NOTECHS" scale for trauma

BACKGROUND

A modified nontechnical skills (NOTECHS) scale for trauma (T-NOTECHS) was developed to teach and assess teamwork skills of multidisciplinary trauma resuscitation teams. In this study, T-NOTECHS was evaluated for reliability and correlation with clinical performance.

METHODS

Interrater reliability (intraclass correlation coefficient) and correlation with the speed and completeness of resuscitation tasks were assessed during simulation-based teamwork training and during actual trauma resuscitations.

RESULTS

For T-NOTECHS ratings done in real time, intraclass correlation coefficients were .44 for simulated and .48 for actual resuscitations. Reliability was higher (intraclass correlation coefficient = .71) for video review of resuscitations. Better T-NOTECHS scores were correlated with better performance during simulations, evidenced by a greater number of completed resuscitation tasks (r = .50, P < .01) and faster time to completion (r = -.38, P < .05) In actual resuscitations, T-NOTECHS ratings improved after teamwork training (P < .001). Higher T-NOTECHS scores were correlated with better clinical performance, evidenced by faster resuscitation (r = -.13, P < .05) and fewer unreported resuscitation tasks (r = -.16, P < .05).

CONCLUSIONS

Improvement in T-NOTECHS scores after teamwork training, and correlation with clinical parameters in simulated and actual trauma resuscitations, suggest its clinical relevance. Further evaluation, aiming to improve reliability, may be warranted.

Interdisciplinary teamwork in hospitals: a review and practical recommendations for improvement

Recognizing the importance of teamwork in hospitals, senior leadership from the American College of Physician Executives (ACPE), the American Hospital Association (AHA), the American Organization of Nurse Executives (AONE), and the Society of Hospital Medicine (SHM) established the High Performance Teams and the Hospital of the Future project. This collaborative learning effort aims to redesign care delivery to provide optimal value to hospitalized patients. With input from members of this initiative, we prepared this report which reviews the literature related to teamwork in hospitals. Teamwork is critically important to provide safe and effective hospital care. Hospitals with high teamwork ratings experience higher patient satisfaction, higher nurse retention, and lower hospital costs. Elements of effective teamwork have been defined and provide a framework for assessment and improvement efforts in hospitals. Measurement of teamwork is essential to understand baseline performance, and to demonstrate the utility of resources invested to enhance it and the subsequent impact on patient care. Interventions designed to improve teamwork in hospitals include localization of physicians, daily goals of care forms and checklists, teamwork training, and interdisciplinary rounds. Though additional research is needed to evaluate the impact on patient outcomes, these interventions consistently result in improved teamwork knowledge, ratings of teamwork climate, and better understanding of patients' plans of care. The optimal approach is implementation of a combination of interventions, with adaptations to fit unique clinical settings and local culture.

Surgeons' non-technical skills

The importance of non-technical skills to surgical performance is gaining wide acceptance. This article discusses the core cognitive and social skills categories thought to underpin medical knowledge and surgical expertise, and describes the rise of non-technical skill models of assessment in surgery. Behavior rating systems such as NOTSS (Non-Technical Skills for Surgeons) have been developed to support education and assessment in this regard. We now understand more about these critical skills and how they impact surgery. The challenge in the future is to incorporate them into undergraduate teaching, postgraduate training, workplace assessment, and perhaps even selection.

The impact of nontechnical skills on technical performance in surgery: a systematic review

BACKGROUND

Failures in nontechnical and teamwork skills frequently lie at the heart of harm and near-misses in the operating room (OR). The purpose of this systematic review was to assess the impact of nontechnical skills on technical performance in surgery.

STUDY DESIGN

MEDLINE, EMBASE, PsycINFO databases were searched, and 2,041 articles were identified. After limits were applied, 341 articles were retrieved for evaluation. Of these, 28 articles were accepted for this review. Data were extracted from the articles regarding sample population, study design and setting, measures of nontechnical skills and technical performance, study findings, and limitations.

RESULTS

Of the 28 articles that met inclusion criteria, 21 articles assessed the impact of surgeons' nontechnical skills on their technical performance. The evidence suggests that receiving feedback and effectively coping with stressful events in the OR has a beneficial impact on certain aspects of technical performance. Conversely, increased levels of fatigue are associated with detriments to surgical skill. One article assessed the impact of anesthesiologists' nontechnical skills on anesthetic technical performance, finding a strong positive correlation between the 2 skill sets. Finally, 6 articles assessed the impact of multiple nontechnical skills of the entire OR team on surgical performance. A strong relationship between teamwork failure and technical error was empirically demonstrated in these studies.

CONCLUSIONS

Evidence suggests that certain nontechnical aspects of performance can enhance or, if lacking, contribute to deterioration of surgeons' technical performance. The precise extent of this effect remains to be elucidated.

The role of oncologists in multidisciplinary cancer teams in the UK: an untapped resource for team leadership?

BACKGROUND AND AIMS

In the UK, cancer care is managed via multidisciplinary teams (MDT). Core members of these teams are typically surgeons, oncologists, radiologists, pathologists and clinical nurse specialists (with other health care professionals potentially present). Good teamwork, including team communication and leadership, has been shown to be a prerequisite for safe care delivery in other health care contexts, but cancer MDT team working processes are yet to be fully explored. This study aimed to assess the self-perceived contribution of oncologists to MDTs, with emphasis on their potential role as team leaders.

METHODS

Data were collected at the British Uro-oncology Group 6th Annual Meeting (Sep 11-12 2009, York, UK). Respondents completed various items related to their perceived contribution to MDTs, aspects of current and potential MDT leadership, team decision making in these meetings, and also demographic information.

RESULTS

Seventy-seven oncologists attended the meeting, of whom 61 fully completed the survey (response rate 79%). Oncologists reported that their contribution to the MDT discussion carries equal weight to those of surgeons, radiologists and pathologists. Whereas 83% of respondents reported that MDT chairmanship could rotate, only 39% reported that it does in their own MDTs. More than 90% of respondents thought that oncologists (clinical or medical) could chair these meetings, but only 25% of them had ever chaired their own MDT.

CONCLUSIONS

Despite a high level of contribution to MDTs and the respect of their colleagues, oncologists are not taking leadership roles within MDTs at the level that they expect. This study raises the question of whether a re-evaluation the leadership of MDTs is required with clinicians from a variety of specialities being given opportunities to develop skills necessary to lead cancer MDTs and improve team performance and ultimately cancer care.

Hospital quality and patient safety competencies: development, description, and recommendations for use

BACKGROUND

Hospitalists are poised to have a tremendous impact on improving the quality of care for hospitalized patients. However, many hospitalists are inadequately prepared to engage in efforts to improve quality, because medical schools and residency programs have not traditionally emphasized healthcare quality and patient safety in their curricula.

METHODS

Through a multistep process, the Society of Hospital Medicine (SHM) Quality Improvement Education (QIE) subcommittee developed the Hospital Quality and Patient Safety (HQPS) Competencies to provide a framework for developing and assessing curricula and other professional development experiences. This article describes the development, provides definitions, and makes recommendations on the use of the HQPS Competencies.

RESULTS

The 8 areas of competence include: Quality Measurement and Stakeholder Interests, Data Acquisition and Interpretation, Organizational Knowledge and Leadership Skills, Patient Safety Principles, Teamwork and Communication, Quality and Safety Improvement Methods, Health Information Systems, and Patient Centeredness. Reflecting differing levels of hospitalist involvement in healthcare quality, 3 levels of expertise within each area of competence have been established: basic, intermediate, and advanced. Standards for each competency area use carefully selected action verbs to reflect educational goals for hospitalists at each level.

CONCLUSIONS

Formal incorporation of the HQPS Competencies into professional development programs, and innovative educational initiatives and curricula, will help provide current hospitalists and the next generations of hospitalists with the needed skills to be successful.

Emotional intelligence and stress in medical students performing surgical tasks

PURPOSE

Poor stress management skills can compromise performance in the operating room, particularly in inexperienced trainees. Little is known about individual differences in managing stress. This study aimed to explore the relationship between trait emotional intelligence (EI) and objective and subjective measures of stress in medical students faced with unfamiliar surgical tasks.

METHOD

Seventeen medical undergraduates completed an unfamiliar laparoscopic task on a simulator during January to April 2008. Subjective stress before, during (retrospectively), and after the task was measured using the self-report State-Trait Anxiety Inventory. Objective stress was measured using continuous heart rate (HR) monitoring. Participants also completed the Trait Emotional Intelligence Questionnaire short form (TEIQue-SF). The authors computed scores for global trait EI and the TEIQue-SF four factors and carried out descriptive and correlational analyses.

RESULTS

The highest levels of subjective stress were reported during the task and correlated positively with trait EI as well as with the trait EI factors of well-being and emotionality. Objective stress (mean HR) during the task was positively related to the sociability factor of trait EI. Higher trait EI scores were also associated with better after-task recovery from stress experienced during the task.

CONCLUSIONS

Students with higher trait EI are more likely to experience stress during unfamiliar surgical scenarios but are also more likely to recover better compared with their lower-trait-EI peers. Trait EI has implications for the design of effective stress management training tailored to individual needs and potential applications to surgical trainee selection and development.

Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR): development and validation

Aim

The aim of the study reported here was to address the need to assess and train teamwork and non-technical skills in the context of Resuscitation. Specifically, we sought to develop a tool that is feasible to use and psychometrically sound to assess team behaviours during cardiac arrest resuscitation attempts.

Methods

To ensure validity, reliability, and feasibility, the Observational Skill based Clinical Assessment tool for Resuscitation (OSCAR) was developed in 3 phases. A review of the literature leading to initial tool development was followed by an assessment of face and content validity, and finally a thorough reliability assessment, using Cronbach's α to assess internal consistency and intraclass correlation to assess inter-rater reliability.

Results

OSCAR was developed methodically, and tested for face and content validity. Cronbach's α results ranged from 0.736 to 0.965 demonstrating high internal consistency, and intraclass correlation results ranged from 0.652 to 0.911, all of which are strongly significant and indicate good inter-rater reliability.

Conclusion

On the basis of our results, we conclude that OSCAR is psychometrically robust, scientifically sound, and clinically relevant. We have developed the Observational Skill-based Clinical Assessment tool for Resuscitation (OSCAR) for the assessment of non-technical skills in Resuscitation teams. We propose the use of this tool in simulation and real Cardiac Arrest Resuscitation attempts to assess, guide and train non-technical skills to team members, to improve patient safety and maximise the chances of successful resuscitation.

Observational teamwork assessment for surgery: content validation and tool refinement

BACKGROUND

Effective teamwork is crucial for safe surgery. Failures in nontechnical and teamwork skills are frequently implicated in adverse events. The Observational Teamwork Assessment for Surgery (OTAS) tool assesses teamwork of the entire team in the operating room. Empirical testing of OTAS has yet to explore the content validity of the tool.

STUDY DESIGN

This was a cross-sectional observational study. Data were collected in 30 procedures by 2 trained researchers. Five teamwork behaviors were scored (ie, communication, leadership, cooperation, coordination, and monitoring) and behavior exemplar completion was recorded (phase 1). Expert operating room personnel (5 surgeons, 5 anesthesiologists, and 5 scrub nurses) assessed the content validity of the OTAS exemplar behaviors. Finally, a panel of operating room patient-safety experts refined the exemplars (phase 2).

RESULTS

In total, the observability (presence/absence) of 130 exemplars was assessed by 2 blinded observers in 30 general surgical cases. Observer agreement was high (Cohen's κ ≥ 0.41) for 83.85% (109 of 130) of exemplar behaviors; 60.77% (79 of 130) of exemplar behaviors were observed frequently with high observer agreement. The majority of the exemplars were rated by expert operating room practitioners and an expert panel as substantial contributors to teamwork and patient safety. Based on expert consensus, 21 behavior exemplars were removed from OTAS and an additional 23 were modified.

CONCLUSIONS

The exemplars of OTAS demonstrated very good content validity. Taken together with recent evidence on the construct validity of the tool, these findings demonstrate that OTAS is psychometrically robust for capturing teamwork in the operating room.

Teamwork and communication in the operating room: relationship to discrete outcomes and research challenges

The literature defining and addressing teamwork and communication is abundant; however, few studies have analyzed the relationship between measures of teamwork and communication and quantifiable outcomes. The objectives of this review are: (1) to identify studies addressing teamwork and communication in the operating room in relation to discrete measures of outcome, (2) to create a classification of studies of the relationship between teamwork and communication and outcomes, (3) to assess the implications of these studies, (4) to explore the methodological challenges of teamwork and communication studies in the perioperative setting, and (5) to suggest future research directions.studies in the perioperative setting, and (5) to suggest future research directions.