Observational Teamwork Assessment for Surgery: Construct Validation With Expert Versus Novice Raters

Surgical Sabermetrics: A Scoping Review of Technology-enhanced Assessment of Nontechnical Skills in the Operating Room

OBJECTIVE

To evaluate the current evidence for surgical sabermetrics: digital methods of assessing surgical nontechnical skills and investigate the implications for enhancing surgical performance.

BACKGROUND

Surgeons need high-quality, objective, and timely feedback to optimize performance and patient safety. Digital tools to assess nontechnical skills have the potential to reduce human bias and aid scalability. However, we do not fully understand which of the myriad of digital metrics of performance assessment have efficacy for surgeons.

METHODS

A systematic review was conducted by searching PubMed, EMBASE, CINAHL, and PSYCINFO databases following PRISMA-ScR guidelines. MeSH terms and keywords included "Assessment," "Surgeons," and "Technology". Eligible studies included a digital assessment of nontechnical skills for surgeons, residents, and/or medical students within an operative context.

RESULTS

From 19,229 articles screened, 81 articles met the inclusion criteria. The studies varied in surgical specialties, settings, and outcome measurements. A total of 122 distinct objective, digital metrics were utilized. Studies digitally measured at least 1 category of surgical nontechnical skill using a single (n=54) or multiple objective measures (n=27). The majority of studies utilized simulation (n=48) over live operative settings (n=32). Surgical Sabermetrics has been demonstrated to be beneficial in measuring cognitive load (n=57), situation awareness (n=24), communication (n=3), teamwork (n=13), and leadership (n=2). No studies measured intraoperative decision-making.

CONCLUSIONS

The literature detailing the intersection between surgical data science and operative nontechnical skills is diverse and growing rapidly. Surgical Sabermetrics may provide a promising modifiable technique to achieve desirable outcomes for both the surgeon and the patient. This study identifies a diverse array of measurements possible with sensor devices and highlights research gaps, including the need for objective assessment of decision-making. Future studies may advance the integration of physiological sensors to provide a holistic assessment of surgical performance.

Beyond the self: a novel framework to enhance non-technical team skills for anesthesiologists

Human factors and non-technical skills (NTS) have been identified as essential contributors to both the propagation and prevention of medical errors in the operating room. Despite extensive study and interventions to nurture and enhance NTS in anesthesiologists, gaps to effective team practice and patient safety remain. Furthermore, the link between added NTS training and clinically significant improved outcomes has not yet been demonstrated. We performed a narrative review to summarize the literature on existing systems and initiatives used to measure and nurture NTS in the clinical operating room setting. Controlled interventions performed to nurture NTS (N.=13) were identified and compared. We comment on the body of current evidence and highlight the achievements and limitations of interventions published thus far. We then propose a novel education and training framework to further develop and enhance non-technical skills in both individual anesthesiologists and operating room teams. We use the cardiac anesthesiology environment as a starting point to illustrate its use, with clinical examples. NTS is a key component of enhancing patient safety. Effective framing of its concepts is central to apply individual characteristics and skills in team environments in the OR and achieve tangible, beneficial patient outcomes.

Successful implementation of a rater training program for medical students to evaluate simulated pediatric emergencies

Introduction

Simulation-based training is increasingly used in pediatrics to teach technical skills, teamwork, and team communication, and to improve potential deficiencies in pediatric emergency care. Team performance must be observed, analyzed, and evaluated by trained raters. The structured training of medical students for the assessment of simulated pediatric emergencies has not yet been investigated.

Methods

We developed a rater training program for medical students to assess guideline adherence, teamwork, and team communication in simulated pediatric emergencies. Interrater reliability was measured at each training stage using Kendall tau coefficients.

Results

In 10 out of 15 pairs of raters interrater reliability was moderate to high (tau>0.4), whereas it was low in the remaining 5 pairs of raters.

Discussion

The interrater reliability showed good agreement between medical students and expert raters at the end of the rater training program. Medical students can be successfully involved in the assessment of guideline adherence as well as teamwork and team communication in simulated pediatric emergencies.

Preparing and Presenting Validation Studies: A Guide for the Perplexed

SUMMARY STATEMENT

Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

Closed loop communication in the trauma bay: identifying opportunities for team performance improvement through a video review analysis

Objectives

Communication among trauma team members in the trauma bay is vulnerable to errors, which may impact patient outcomes. We used the previously validated trauma-non-technical skills (T-NOTECHS) tool to identify communication gaps during patient management in the trauma bay and to inform development strategies to improve team performance.

Methods

Two reviewers independently assessed non-technical skills of team members through video footage at Sunnybrook Health Sciences Centre. Team performance was measured using T-NOTECHS across five domains using a five-point Likert scale (lower score indicating worse performance): (1) leadership; (2) cooperation and resource management; (3) communication and interaction; (4) assessment and decision making; (5) situation awareness/coping with stress. Secondary outcomes assessed the number of callouts, closed loop communications and parallel conversations.

Results

The study included 55 trauma activations. Injury severity score (ISS) was used as a measure of trauma severity. A case with an ISS score ≥ 16 was considered severe. ISS was ≥ 16 in 37% of cases. Communication and interaction scored significantly lower compared to all other domains (p < 0.0001). There were significantly more callouts and completed closed loop communications in more severe cases compared to less severe cases (p = 0.017 for both). Incomplete closed loop communications and parallel conversations were identified, irrespective of case severity.

Conclusion

A lower communication score was identified using T-NOTECHS, attributed to incomplete closed loop communications and parallel conversations. Through video review of trauma team activations, opportunities for improvement in communication can be identified by the T-NOTECHS tool, as well as specifically identifying callouts and closed loop communication. This process may be useful for trauma programs as part of a quality improvement program on communication skills and team performance.

Supplementary Information

The online version contains supplementary material available at 10.1007/s43678-022-00295-z.

Value of Nontechnical Skills in Minimally Invasive Surgery

Nontechnical skills are of increasing importance in surgery and surgical training. The main studies on its impact on the safety and effectiveness of surgical interventions were mainly published in the first decade of the 2000s. Due to the significant technical complexity and great diversity of instruments for nontechnical skills evaluation, the interest in training and in measuring the impact on surgical safety has relatively decreased. However, the advent of minimally invasive surgery and its peculiar technical characteristics of sophisticated technique and constant innovation through the adoption of new materials and drugs has rekindled interest in this expertise area. In the present review, we have revisited the main instruments available to measure nontechnical skill of surgical teams and analyzed the role of the main competencies on which they are based, such as situational awareness, leadership and communication skills, and the consistency of the intraoperative decision-making process. We conclude that despite the great consensus that exists among all members of the health team on the importance of nontechnical skills for the surgical team in minimally invasive surgery, the reproducible evidence on the subject is scarce and laborious to obtain. To the extent that protecting and expanding nontechnical skills is fundamental to the path toward the high reliability of health institutions, it is possible to anticipate here the role of these institutions as promoters of continuity and new research models in this area of knowledge, especially in minimally invasive surgery, an access route to which more and more patients undergoing surgical treatment in these same institutions are submitted.

Single port robotic radical prostatectomy versus multi-port robotic radical prostatectomy: A human factor analysis during the initial learning curve

BACKGROUND

Studies have thus far neglected to evaluate the impact of the da Vinci single port (SP) robotic platform on surgeon experience and operating room efficiency. We sought to assess the effect of the SP platform on surgeon cognitive load measures during robotic assisted laparoscopic prostatectomy (RALP).

METHODS

We prospectively compared the first 20 SP-RALPs performed at our institution to 20 multi-port (MP)-RALPs performed by a single experienced robotic surgeon. Three multi-dimensional assessment tools were used to evaluate mental and surgical workload, teamwork and workflow disruptions.

RESULTS

No statistically significant differences were found between the MP-RALP and SP-RALP cohorts when evaluated by NASA Task Load Index, Surgery Task Load Index and Observational Teamwork Assessments.

CONCLUSIONS

The SP robotic platform did not adversely affect human factor performance of the surgeon during RALP. Multi-institutional validation will be necessary to confirm these initial findings.

Improving the Quality of Evaluation Data in Simulation-Based Healthcare Improvement Projects: A Practitioner's Guide to Choosing and Using Published Measurement Tools

STATEMENT

Simulation is increasingly being used in healthcare improvement projects. The aims of such projects can be extremely diverse. Accordingly, the outcomes or participant attributes that need to be measured can vary dramatically from project-to-project and may include a wide range of nontechnical skills, technical skills, and psychological constructs. Consequently, there is a growing need for simulation practitioners to be able to identify suitable measurement tools and incorporate them into their work. This article provides a practical introduction and guide to the key considerations for practitioners when selecting and using such tools. It also offers a substantial selection of example tools, both to illustrate the key considerations in relation to choosing a measure (including reliability and validity) and to serve as a convenient resource for those planning a study. By making well-informed choices, practitioners can improve the quality of the data they collect, and the likelihood that their projects will succeed.

Using Fully-Immersive Simulation Training with Structured Debrief to Improve Nontechnical Skills in Emergency Endovascular Surgery

OBJECTIVE

Assess whether fully-immersive simulation training with structured debriefing of a standardized emergency thoracic endovascular aortic repair (TEVAR) scenario improves team-work performance of the lead surgeon. Secondary aims: assess whether technical skills (TS) and radiation safety behaviors (RSB) improved concurrently.

DESIGN

Pre-post study.

SETTING

UK-based training days.

PARTICIPANTS

General and vascular surgical trainees (n = 16).

INTERVENTION(S)

Fully-immersive simulation training with structured debriefing of a standardized emergency TEVAR scenario. Following standardized emergency TEVAR technical training, trainees led 2 standardized fully-immersive endovascular surgery simulations, with scripted support from a passive surgical team. A non-TS (NTS) structured debrief was delivered following simulations. NTS were assessed using the validated observational teamwork assessment for surgery tool post hoc using video recordings of simulations. TS were assessed through time taken to complete each step of the procedure, as defined during technical training. RSB were assessed through checking for presence of pre-defined actions and the length of time fluoroscopy was used during each simulation.

RESULTS

Total observational teamwork assessment for surgery scores improved following structured debrief (p = 0.005, median 52.55/90 vs 73.0/90), alongside all constituent domains - communication (p < 0.001, median 11.7/20 vs 16.6/20), coordination (p < 0.001, median 8.6/15 vs 13.4/15), cooperation (p < 0.001, median 13.15/20 vs 16.35/20), leadership (p < 0.001, median 8.70/15 vs 11.30/15) and monitoring (p < 0.001, median 9.85/20 vs 14.85/20). TS improved; time to complete 12 of 13 procedural steps improved (p < 0.027). Fluoroscopy time (seconds) decreased (p = 0.339, 543.6 vs 495.5), frequency lead surgeons checked the team were wearing leads increased (p = 0.125, 3 vs 7) and asked the team to step back before screening increased (p = 0.003, frequency team asked to step back/total angiography runs before = 2/36 vs after = 14/44).

CONCLUSIONS

fully-immersive endovascular simulation with structured debrief is a robust tool to improve NTS and TS. Incorporation into surgical training may reduce operating theatres errors, increase efficiency, and improve RSB. However effective translation into the clinical workplace must be demonstrated to see these benefits.

Team Communication on Tasks Performed in Blood Pressure Clinics in Community Pharmacies

Background: To successfully deliver pharmacy services, pharmacists and technicians need to work as a team and have effective communication. Objectives: To measure the amount of communication between community pharmacists and their technicians during monthly blood pressure (BP) clinics. Methods: Cross-sectional survey method was used to assess the amount of communication pharmacists and technicians had on each task. Study participants were pharmacist-technician pairs (teams) from 10 intervention community pharmacies in the Team Education and Adherence Monitoring (TEAM) trial. Each team provided services designed to improve BP among hypertensive African American patients. Thirteen specific tasks were identified as essential in providing monthly BP clinics, and they were being performed by either the pharmacist or technician. At the end of the trial, each pharmacist and technician were asked to report his/her perception of discussion levels that occurred for each task. The level of team communication was summarized for each task (task-specific) and for each team (team-specific). Results: For task-specific communication, 3 teams had communication regarding pharmacist tasks and 5 teams engaged in communication for technician tasks. More communication was reported for newly developed technician tasks in the BP clinic. For team-specific communication, 2 teams reported no communication on any task, and another 2 teams reported having communication on all the tasks. Overall, pharmacy teams showed different levels of communication in this study. Conclusion: The amounts of communication between pharmacists and technicians were found to vary for different tasks and teams. This suggests that the nature of tasks and the unique dynamics existing in each pharmacy team could influence pharmacist-technician communication.

Evaluation of a Measurement System to Assess ICU Team Performance

OBJECTIVE

Measuring teamwork is essential in critical care, but limited observational measurement systems exist for this environment. The objective of this study was to evaluate the reliability and validity of a behavioral marker system for measuring teamwork in ICUs.

DESIGN

Instances of teamwork were observed by two raters for three tasks: multidisciplinary rounds, nurse-to-nurse handoffs, and retrospective videos of medical students and instructors performing simulated codes. Intraclass correlation coefficients were calculated to assess interrater reliability. Generalizability theory was applied to estimate systematic sources of variance for the three observed team tasks that were associated with instances of teamwork, rater effects, competency effects, and task effects.

SETTING

A 15-bed surgical ICU at a large academic hospital.

SUBJECTS

One hundred thirty-eight instances of teamwork were observed. Specifically, we observed 88 multidisciplinary rounds, 25 nurse-to-nurse handoffs, and 25 simulated code exercises.

INTERVENTIONS

No intervention was conducted for this study.

MEASUREMENTS AND MAIN RESULTS

Rater reliability for each overall task ranged from good to excellent correlation (intraclass correlation coefficient, 0.64-0.81), although there were seven cases where reliability was fair and one case where it was poor for specific competencies. Findings from generalizability studies provided evidence that the marker system dependably distinguished among teamwork competencies, providing evidence of construct validity.

CONCLUSIONS

Teamwork in critical care is complex, thereby complicating the judgment of behaviors. The marker system exhibited great potential for differentiating competencies, but findings also revealed that more context specific guidance may be needed to improve rater reliability.

A qualitative positive deviance study to explore exceptionally safe care on medical wards for older people

BACKGROUND

The positive deviance approach seeks to identify and learn from those who demonstrate exceptional performance. This study sought to explore how multidisciplinary teams deliver exceptionally safe care on medical wards for older people.

METHODS

A qualitative positive deviance study was conducted on four positively deviant and four slightly-above-average matched comparator wards, which had been identified using routinely collected NHS Safety Thermometer data. In total, 70 multidisciplinary staff participated in eight focus groups to explore staff perceptions about how their teams deliver safe patient care. A thematic analysis was conducted in two stages: first to identify the tools, processes, strategies, and cultural and social contexts that facilitated safety across all wards; and second to generate hypotheses about the characteristics that facilitated 'positively deviant' patient care.

RESULTS

Based on identifiable qualitative differences between the positively deviant and comparison wards, 14 characteristics were hypothesised to facilitate exceptionally safe care on medical wards for older people. This paper explores five positively deviant characteristics that healthcare professionals considered to be most salient. These included the relational aspects of teamworking, specifically regarding staff knowing one another and working together in truly integrated multidisciplinary teams. The cultural and social context of positively deviant wards was perceived to influence the way in which practical tools (eg, safety briefings and bedside boards) were implemented.

CONCLUSION

This study exemplifies that there are no 'silver bullets' to achieving exceptionally safe patient care on medical wards for older people. Healthcare leaders should encourage truly integrated multidisciplinary ward teams where staff know each other well and work as a team. Focusing on these underpinning characteristics may facilitate exceptional performances across a broad range of safety outcomes.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Psychological aptitude for surgery: The importance of non-technical skills

INTRODUCTION

Psychological aptitude for surgery includes all the non-technical abilities that are necessary for the surgeons. However, differently from the other careers, these skills are not considered in the selection process and their role is definitively underestimated in the field of surgery. We perform a literature review of non-technical skills for surgery to identify their role and to understand how to train and evaluate these abilities among the surgeons.

RESULTS

Different methods have been presented for the evaluation and training of non-technical skills for surgeons; based on the model of aviation and anesthesia a wide range of simulated scenarios have been proposed to practice these aptitudes and abilities. Different behavioral markers systems have been developed for correct identification and definition of these skills, these can be used in the real surgical room and even learned and trained in the simulated operating theatre.

CONCLUSION

This article shows the importance of non-practical abilities in the surgical performance and in defining the aptitude for surgery. Learning these skills and introducing them in surgical education can be useful to improve the surgical performance.

Rating the quality of teamwork-a comparison of novice and expert ratings using the Team Emergency Assessment Measure (TEAM) in simulated emergencies

Background

Training in teamwork behaviour improves technical resuscitation performance. However, its effect on patient outcome is less clear, partly because teamwork behaviour is difficult to measure. Furthermore, it is unknown who should evaluate it. In clinical practice, experts are obliged to participate in resuscitation efforts and are thus unavailable to assess teamwork quality. Consequently, we sought to determine if raters with little clinical experience and experts provide comparable evaluations of teamwork behaviour.

Methods

Novice and expert raters judged teamwork behaviour during 6 emergency medicine simulations using the Teamwork Emergency Assessment Measure (TEAM). Ratings of both groups were analysed descriptively and compared with U and t tests. We used a mixed effects model to identify the proportion of variance in TEAM scores attributable to rater status and other sources.

Results

Twelve raters evaluated 7 teams rotating through 6 cases, for a total of 84 observations. We found no significant difference between expert and novice ratings for 7 of the 11 items of the TEAM or in the sums of all item scores. Novices rated teamwork behaviour higher on 4 items and overall. Rater status accounted for 11.1% of the total variance in scores.

Conclusions

Experts’ and novices’ ratings were similarly distributed, implying that raters with limited experience can provide reliable data on teamwork behaviour. Novices show a consistent, but slightly more lenient rating behaviour. Clinical studies and real-life teams may thus employ novices using a structured observational tool such as TEAM to inform their performance review and improvement.

Electronic supplementary material

The online version of this article (10.1186/s13049-019-0591-9) contains supplementary material, which is available to authorized users.

Implementation of the surgical safety checklist in hospitals of Iran; operating room personnel's attitude, awareness and acceptance

Purpose The WHO Surgical Safety Checklist (SSC) has improved patient safety effectively. Despite the known benefits of applying the checklist before surgery, its implementation is less than universal in practice. The purpose of this paper is to determine the operating room personnel's attitude, their awareness and knowledge of the SSC, and to evaluate staff acceptance of the SSC (including personal beliefs). Design/methodology/approach This cross-sectional study was conducted in eight tertiary general hospitals in Tehran, Iran. Some 145 operating room personnel (surgeons, anaesthetists and nurses) were selected for the study. Data collection was carried out via a validated questionnaire in three parts which included socio-demographic, attitude, awareness and acceptance. Data were then analysed using the Kruskal-Wallis and χ² statistical test. Findings Out of the 145 participants in the study, 92 per cent were aware of the existence of the SSC and 73.9 per cent of them were aware of the objectives of SSC. Overall, the attitude to SSC was positive. The attitude of surgeons was positive towards the impact of the SSC on safety and teamwork. Surgeons were significantly more sensitive to the barriers of SSC application compared to nurses and anaesthetists ( p=0.046). Among the three groups, nurses had the highest level of support for SSC ( p=0.001). Practical implications Despite high acceptance of the checklist among staff, there is still a gap in knowledge about when exactly the checklist should be used. Therefore, involvement of all surgical team members to complete the checklist process, support of senior managers, on-going education and training and consideration of the barriers to its implementation are all key areas that need to be taken into account. Originality/value This is the first research to examine the operating room personnel's attitude, awareness and acceptance about SSC in Iranian hospitals. The outcomes of this study provide documentation and possible justification for effective establishment of SSC in Iran and other countries.

Non-technical skills assessment scale in nursing: construction, development and validation1

OBJECTIVES

The introduction of non-technical skills during nursing education is crucial to prepare nurses for the clinical context and increase patient safety. We found no instrument developed for this purpose. to construct, develop and validate a non-technical skills assessment scale in nursing.

METHOD

methodological research. Based on the literature review and experience of researchers on non-technical skills in healthcare and the knowledge of the principles of crisis resource management, a list of 63 items with a five-point Likert scale was constructed. The scale was applied to 177 nursing undergraduate students. Descriptive statistics, correlations, internal consistency analysis and exploratory factor analysis were performed to evaluate the psychometric properties of the scale.

RESULTS

scale items presented similar values for mean and median. The maximum and the minimum values presented a good distribution amongst all response options. Most items presented a significant and positive relationship. Cronbach alpha presented a good value (0.94), and most correlations were significant and positive. Exploratory factor analysis using the Kaiser-Meyer-Olkin test showed a value of 0.849, and the Bartlett's test showed adequate sphericity values (χ2=6483.998; p=0.000). One-factor model explained 26% of the total variance.

CONCLUSION

non-technical skills training and its measurement could be included in undergraduate or postgraduate courses in healthcare professions, or even be used to ascertain needs and improvements in healthcare contexts.

Reality check of using the surgical safety checklist: A qualitative study to observe application errors during snapshot audits

Background

The WHO Surgical Safety Checklist (SSC) was established to address important safety issues and to reduce the number of surgical deaths. So far, numerous reports have demonstrated sub-optimal implementation of the SSC in practice and limited improvements in patient outcomes. Therefore, the aim of this study was to audit the SSC-practice in a real-world setting in a university hospital setting.

Methods

From 2015 to 2016, independent observers performed snapshot audits in operating theatres and shadowed the three phases of the SSC. Using a 4-point Likert-scale to rate the compliance on each audit day, we generated a report highlighting possible improvements and provided feedback to the operating team members.

Results

Audits were performed on 36 operating days (2015: n = 19; 2016: n = 17), in which a total of 136 surgical interventions were observed. Overall, the percentage of “very good compliance” improved from 2015 to 2016: for the sign-in from 52.9% to 81.2% (p = 0.141), for the team-time-out from 33.3% to 58.8% (p = 0.181), and for the sign-out from 21.4% to 41.7% (p = 0.401). The qualitative review revealed inconsistencies when applying the SSC, of which the missing documentation of an actually performed item or the wrong timing for an item was most common.

Conclusion

Snapshot audits revealed that SSC compliance has improved over the observed period, while its application revealed inconsistencies during the three phases of the SSC. Snapshot audits proved to be a valuable tool in the qualitative analysis of SSC compliance and gave more insight than a mere completeness check of ticks in SSC documents.

The measurement of collaboration within healthcare settings: a systematic review of measurement properties of instruments

BACKGROUND

There is a growing call by consumers and governments for healthcare to adopt systems and approaches to care to improve patient safety. Collaboration within healthcare settings is an important factor for improving systems of care. By using validated measurement instruments a standardized approach to assessing collaboration is possible, otherwise it is only an assumption that collaboration is occurring in any healthcare setting.

OBJECTIVES

The objective of this review was to evaluate and compare measurement properties of instruments that measure collaboration within healthcare settings, specifically those which have been psychometrically tested and validated.

INCLUSION CRITERIA, TYPES OF PARTICIPANTS

Participants could be healthcare professionals, the patient or any non-professional who contributes to a patient's care, for example, family members, chaplains or orderlies. The term participant type means the designation of any one participant; for example 'nurse', 'social worker' or 'administrator'. More than two participant types was mandatory.

TYPES OF INTERVENTION(S)/PHENOMENA OF INTEREST

The focus of this review was the validity of tools used to measure collaboration within healthcare settings.

TYPES OF STUDIES

The types of studies considered for inclusion were validation studies, but quantitative study designs such as randomized controlled trials, controlled trials and case studies were also eligible for inclusion. Studies that focused on Interprofessional Education, were published as an abstract only, contained patient self-reporting only or were not about care delivery were excluded.

OUTCOMES

The outcome of interest was validation and interpretability of the instrument being assessed and included content validity, construct validity and reliability. Interpretability is characterized by statistics such as mean and standard deviation which can be translated to a qualitative meaning.

SEARCH STRATEGY

The search strategy aimed to find both published and unpublished studies. A three-step search strategy was utilized in this review. The databases searched included PubMed, CINAHL, Embase, Cochrane Central Register of Controlled Trials, Emerald Fulltext, MD Consult Australia, PsycARTICLES, Psychology and Behavioural Sciences Collection, PsycINFO, Informit Health Databases, Scopus, UpToDate and Web of Science. The search for unpublished studies included EThOS (Electronic Thesis Online Service), Index to Theses and ProQuest- Dissertations and Theses.

METHODOLOGICAL QUALITY

The assessment of methodological quality of the included studies was undertaken using the COSMIN checklist which is a validated tool that assesses the process of design and validation of healthcare measurement instruments.

DATA COLLECTION

An Excel spreadsheet version of COSMIN was developed for data collection which included a worksheet for extracting participant characteristics and interpretability data.

DATA SYNTHESIS

Statistical pooling of data was not possible for this review. Therefore, the findings are presented in a narrative form including tables and figures to aid in data presentation. To make a synthesis of the assessments of methodological quality of the different studies, each instrument was rated by accounting for the number of studies performed with an instrument, the appraisal of methodological quality and the consistency of results between studies.

RESULTS

Twenty-one studies of 12 instruments were included in the review. The studies were diverse in their theoretical underpinnings, target population/setting and measurement objectives. Measurement objectives included: investigating beliefs, behaviors, attitudes, perceptions and relationships associated with collaboration; measuring collaboration between different levels of care or within a multi-rater/target group; assessing collaboration across teams; or assessing internal participation of both teams and patients.Studies produced validity or interpretability data but none of the studies assessed all validity and reliability properties. However, most of the included studies produced a factor structure or referred to prior factor analysis. A narrative synthesis of the individual study factor structures was generated consisting of nine headings: organizational settings, support structures, purpose and goals; communication; reflection on process; cooperation; coordination; role interdependence and partnership; relationships; newly created professional activities; and professional flexibility.

CONCLUSIONS

Among the many instruments that measure collaboration within healthcare settings, the quality of each instrument varies; instruments are designed for specific populations and purposes, and are validated in various settings. Selecting an instrument requires careful consideration of the qualities of each. Therefore, referring to systematic reviews of measurement properties of instruments may be helpful to clinicians or researchers in instrument selection.

IMPLICATIONS FOR PRACTICE

Systematic reviews of measurement properties of instruments are valuable in aiding in instrument selection. This systematic review may be useful in instrument selection for the measurement of collaboration within healthcare settings with a complex mix of participant types. Evaluating collaboration provides important information on the strengths and limitations of different healthcare settings and the opportunities for continuous improvement via any remedial actions initiated.

IMPLICATIONS FOR RESEARCH

Development of a tool that can be used to measure collaboration within teams of healthcare professionals and non-professionals is important for practice. The use of different statistical modelling techniques, such as Item Response Theory modelling and the translation of models into Computer Adaptive Tests, may prove useful. Measurement equivalence is an important consideration for future instrument development and validation. Further development of the COSMIN tool should include appraisal for measurement equivalence. Researchers developing and validating measurement tools should consider multi-method research designs.

Self-assessment of team performance using T-NOTECHS in simulated pediatric trauma resuscitation is not consistent with expert assessment

BACKGROUND

The Trauma NOn-TECHnical Skills (T-NOTECHS) tool has been used to assess teamwork in trauma resuscitation, but its reliability and validity for self-assessment is unknown. Our purpose was to determine the reliability and validity of self-administered T-NOTECHS in pediatric trauma resuscitation.

METHODS

Simulated in situ resuscitations were evaluated using T-NOTECHS in real time by experts and immediately afterwards by team members. Reliability was analyzed with linear-weighted kappa and intra-class correlation. T-NOTECHS scores were compared between expert (gold-standard) and self-assessment.

RESULTS

Fifteen simulations were examined. T-NOTECHS scores were similar between self- and expert assessment for leadership. Self-assessment scores were higher than expert for the other domains and total composite score. Inter-rater reliability for total score was similar between the two groups, but differences were observed in the domains.

CONCLUSIONS

Self-assessment is not interchangeable with expert rating when using T-NOTECHS. Future studies need to determine how self-assessment can be best utilized.

LEVEL OF EVIDENCE

Studies of diagnostic accuracy - Level 2.

Process quality of decision-making in multidisciplinary cancer team meetings: a structured observational study

BACKGROUND

The quality of decision-making in multidisciplinary team meetings (MDTMs) depends on the quality of information presented and the quality of team processes. Few studies have examined these factors using a standardized approach. The aim of this study was to objectively document the processes involved in decision-making in MDTMs, document the outcomes in terms of whether a treatment recommendation was given (none vs. singular vs. multiple), and to identify factors related to type of treatment recommendation.

METHODS

An adaptation of the observer rating scale Multidisciplinary Tumor Board Metric for the Observation of Decision-Making (MDT-MODe) was used to assess the quality of the presented information and team processes in MDTMs. Data was analyzed using descriptive statistics and mixed logistic regression analysis.

RESULTS

N = 249 cases were observed in N = 29 MDTMs. While cancer-specific medical information was judged to be of high quality, psychosocial information and information regarding patient views were considered to be of low quality. In 25% of the cases no, in 64% one, and in 10% more than one treatment recommendations were given (1% missing data). Giving no treatment recommendation was associated with duration of case discussion, duration of the MDTM session, quality of case history, quality of radiological information, and specialization of the MDTM. Higher levels of medical and treatment uncertainty during discussions were found to be associated with a higher probability for more than one treatment recommendation.

CONCLUSIONS

The quality of different aspects of information was observed to differ greatly. In general, we did not find MDTMs to be in line with the principles of patient-centered care. Recommendation outcome varied substantially between different specializations of MDTMs. The quality of certain information was associated with the recommendation outcome. Uncertainty during discussions was related to more than one recommendation being considered. Time constraints were found to play an important role. Some of those aspects seem modifiable, which offers possibilities for the reorganization of MDTMs.

Non-technical skills assessment for prelicensure nursing students: An integrative review

BACKGROUND

In nursing, non-technical skills are recognized as playing an important role to increase patient safety and successful clinical outcomes (Pearson and McLafferty, 2011). Non-technical skills are cognitive and social resource skills that complement technical skills and contribute to safe and efficient task performance (Flin et al., 2008). In order to effectively provide non-technical skills training, it is essential to have an instrument to measure these skills.

METHODOLOGY

An online search was conducted. Articles were selected if they referred to and/or described instruments assessing non-technical skills for nurses and/or prelicensure nursing students in educational, clinical and/or simulated settings with validation evidence (inclusion criteria).

RESULTS

Of the 53 articles located, 26 met the inclusion criteria. Those referred to and/or described 16 instruments with validation evidence developed to assess non-technical skills in multidisciplinary teams including nurses.

CONCLUSION

Although articles have shown 16 valid and reliable instruments, to our knowledge, no instrument has been published or developed and validated for the assessment of non-technical skills of only nurses in general, relevant for use in high-fidelity simulation-based training for prelicensure nursing students. Therefore, there is a need for the development of such an instrument.

A realist process evaluation of robot-assisted surgery: integration into routine practice and impacts on communication, collaboration and decision-making

Background

The implementation of robot-assisted surgery (RAS) can be challenging, with reports of surgical robots being underused. This raises questions about differences compared with open and laparoscopic surgery and how best to integrate RAS into practice.

Objectives

To (1) contribute to reporting of the ROLARR (RObotic versus LAparoscopic Resection for Rectal cancer) trial, by investigating how variations in the implementation of RAS and the context impact outcomes; (2) produce guidance on factors likely to facilitate successful implementation; (3) produce guidance on how to ensure effective teamwork; and (4) provide data to inform the development of tools for RAS.

Design

Realist process evaluation alongside ROLARR. Phase 1 – a literature review identified theories concerning how RAS becomes embedded into practice and impacts on teamwork and decision-making. These were refined through interviews across nine NHS trusts with theatre teams. Phase 2 – a multisite case study was conducted across four trusts to test the theories. Data were collected using observation, video recording, interviews and questionnaires. Phase 3 – interviews were conducted in other surgical disciplines to assess the generalisability of the findings.

Findings

The introduction of RAS is surgeon led but dependent on support at multiple levels. There is significant variation in the training provided to theatre teams. Contextual factors supporting the integration of RAS include the provision of whole-team training, the presence of handpicked dedicated teams and the availability of suitably sized operating theatres. RAS introduces challenges for teamwork that can impact operation duration, but, over time, teams develop strategies to overcome these challenges. Working with an experienced assistant supports teamwork, but experience of the procedure is insufficient for competence in RAS and experienced scrub practitioners are important in supporting inexperienced assistants. RAS can result in reduced distraction and increased concentration for the surgeon when he or she is supported by an experienced assistant or scrub practitioner.

Conclusions

Our research suggests a need to pay greater attention to the training and skill mix of the team. To support effective teamwork, our research suggests that it is beneficial for surgeons to (1) encourage the team to communicate actions and concerns; (2) alert the attention of the assistant before issuing a request; and (3) acknowledge the scrub practitioner’s role in supporting inexperienced assistants. It is beneficial for the team to provide oral responses to the surgeon’s requests.

Limitations

This study started after the trial, limiting impact on analysis of the trial. The small number of operations observed may mean that less frequent impacts of RAS were missed.

Future work

Future research should include (1) exploring the transferability of guidance for effective teamwork to other surgical domains in which technology leads to the physical or perceptual separation of surgeon and team; (2) exploring the benefits and challenges of including realist methods in feasibility and pilot studies; (3) assessing the feasibility of using routine data to understand the impact of RAS on rare end points associated with patient safety; (4) developing and evaluating methods for whole-team training; and (5) evaluating the impact of different physical configurations of the robotic console and team members on teamwork.

Funding

The National Institute for Health Research Health Services and Delivery Research programme.

Tools for Assessment of Communication Skills of Hospital Action Teams: A Systematic Review

BACKGROUND

Hospital action teams comprise interdisciplinary health care providers working simultaneously to treat critically ill patients. Assessments designed to evaluate communication effectiveness or "nontechnical" performance of these teams are essential to minimize medical errors and improve team productivity. Although multiple communication tools are available, the characteristics and psychometric validity of these instruments have yet to be systematically compared.

OBJECTIVE

To identify assessments used to evaluate the communication or "nontechnical" performance of hospital action teams and summarize evidence to develop and validate these instruments.

METHOD

A literature search was conducted using MEDLINE/PubMed database to identify original articles related to assessment of communication skills in teams working in acute care medicine not exclusive to emergency room, operating room, prehospital air and ground transport, or code blue/rapid response resuscitations.

RESULTS

Ten communication assessment tools were identified. Six tools (60%) were designed to measure communication performance of the whole team, whereas 4 tools (40%) were created to assess individual team member's communication skills. Regardless of the type of analysis, the most commonly assessed behavior domains were Leadership, Teamwork, Communication, and Situation awareness. Only 1 of 16 articles describing a particular communication assessment tool reported all the validation criteria, other authors underreported efforts to validate their instruments.

CONCLUSION

A number of tools designed to measure the communication or "nontechnical" performance of hospital action teams are available. Unfortunately, limited reported validity evidence may hamper the utility of these tools in actual clinical practice until further validation studies are performed.

Strategies for increasing the feasibility of performance assessments during competency-based education: Subjective and objective evaluations correlate in the operating room

BACKGROUND

Competency-based education necessitates assessments that determine whether trainees have acquired specific competencies. The evidence on the ability of internal raters (staff surgeons) to provide accurate assessments is mixed; however, this has not yet been directly explored in the operating room. This study's objective is to compare the ratings given by internal raters vs an expert external rater (independent to the training process) in the operating room.

METHODS

Raters assessed general surgery residents during a laparoscopic cholecystectomy for their technical and nontechnical performance.

RESULTS

Fifteen cases were observed. There was a moderately positive correlation (r_s = .618, P = .014) for technical performance and a strong positive correlation (r_s = .731, P = .002) for nontechnical performance. The internal raters were less stringent for technical (mean rank 3.33 vs 8.64, P = .007) and nontechnical (mean rank 3.83 vs 8.50, P = .01) performances.

CONCLUSIONS

This study provides evidence to help operationalize competency-based assessments.

Improving Escalation of Care: Development and Validation of the Quality of Information Transfer Tool

OBJECTIVE

To develop and provide validity and feasibility evidence for the QUality of Information Transfer (QUIT) tool.

BACKGROUND

Prompt escalation of care in the setting of patient deterioration can prevent further harm. Escalation and information transfer skills are not currently measured in surgery.

METHODS

This study comprised 3 phases: the development (phase 1), validation (phase 2), and feasibility analysis (phase 3) of the QUIT tool. Phase 1 involved identification of core skills needed for successful escalation of care through literature review and 33 semistructured interviews with stakeholders. Phase 2 involved the generation of validity evidence for the tool using a simulated setting. Thirty surgeons assessed a deteriorating postoperative patient in a simulated ward and escalated their care to a senior colleague. The face and content validity were assessed using a survey. Construct and concurrent validity of the tool were determined by comparing performance scores using the QUIT tool with those measured using the Situation-Background-Assessment-Recommendation (SBAR) tool. Phase 3 was conducted using direct observation of escalation scenarios on surgical wards in 2 hospitals.

RESULTS

A 7-category assessment tool was developed from phase 1 consisting of 24 items. Twenty-one of 24 items had excellent content validity (content validity index >0.8). All 7 categories and 18 of 24 (P < 0.05) items demonstrated construct validity. The correlation between the QUIT and SBAR tools used was strong indicating concurrent validity (r = 0.694, P < 0.001). Real-time scoring of escalation referrals was feasible and indicated that doctors currently have better information transfer skills than nurses when faced with a deteriorating patient.

CONCLUSIONS

A validated tool to assess information transfer for deteriorating surgical patients was developed and tested using simulation and real-time clinical scenarios. It may improve the quality and safety of patient care on the surgical ward.

Addressing the Interprofessional Collaboration Competencies of the Association of American Medical Colleges: A Systematic Review of Assessment Instruments in Undergraduate Medical Education

PURPOSE

To summarize characteristics and validity evidence of tools that assess teamwork in undergraduate medical education (UME), and provide recommendations for addressing the interprofessional collaboration competencies of the Association of American Medical Colleges (AAMC).

METHOD

The authors conducted a systematic review, searching MEDLINE, MEDLINE In-process, CINAHL, and PsycINFO from January 1, 1979, through April 1, 2014; they searched reference lists and national meeting abstracts. They included original research reports that described a quantitative tool used to assess teamwork in UME. They abstracted characteristics and validity evidence for the tools, plus study quality, according to established frameworks. Two authors independently abstracted 25% of articles and calculated agreement. Authors then applied predefined criteria to identify tools best suited to address the AAMC's teamwork competencies.

RESULTS

Of 13,549 citations, 70 articles describing 64 teamwork assessment tools were included. Of these 64 tools, 27 (42%) assessed teamwork in classroom, 31 (48%) in simulation, and only 7 (11%) in actual clinical settings. The majority (47; 73%) of tools assessed medical students' teamwork in interprofessional teams. On the basis of content concordance, strength of validity evidence, generalizability of scores, and level of outcomes, four published tools were recommended to assess the AAMC's teamwork competencies: the Collaborative Healthcare Interdisciplinary Relationship Planning Scale, Readiness for Interprofessional Learning Scale, Communication and Teamwork Skills assessment, and Teamwork Mini-Clinical Evaluation Exercise.

CONCLUSIONS

Substantial validity evidence supports the use of several UME teamwork assessments. Four tools have been appropriately designed and sufficiently studied to constitute appropriate assessments of the AAMC's teamwork competencies.

Lessons from surgery and anaesthesia: evaluation of non-technical skills in interventional radiology

In the medical profession, surgery and anaesthesia are leading the way in identifying human errors that negatively affect patient safety. Evidence suggests that the implementation of non-technical skills assessments reduces such errors. Interventional Radiology is a procedural based speciality and therefore may also benefit from formal assessment of non-technical skills. This literature review supports the use of standardised assessment tools used in surgery and anaesthesia. Using the Downing framework of internal validity, the tools demonstrated good internal consistency but a spectrum of inter-rater variability, which can be partially improved with training. At present, a formal Interventional Radiology non-technical skills assessment tool is probably not suitable to be a stand-alone ‘high stakes’ assessment, but may be a useful adjunct to the existing array of workplace-based assessments.

Role confusion and self-assessment in interprofessional trauma teams

BACKGROUND

Trauma care requires coordinating an interprofessional team, with formative feedback on teamwork skills. We hypothesized nurses and surgeons have different perceptions regarding roles during resuscitation; that nurses' teamwork self-assessment differs from experts', and that video debriefing might improve accuracy of self-assessment.

METHODS

Trauma nurses and surgeons were surveyed regarding resuscitation responsibilities. Subsequently, nurses joined interprofessional teams in simulated trauma resuscitations. After each resuscitation, nurses and teamwork experts independently scored teamwork (T-NOTECHS). After video debriefing, nurses repeated T-NOTECHS self-assessment.

RESULTS

Nurses and surgeons assumed significantly more responsibility by their own profession for 71% of resuscitation tasks. Nurses' overall T-NOTECHS ratings were slightly higher than experts'. This was evident in all T-NOTECHS subdomains except "leadership," but despite statistical significance the difference was small and clinically irrelevant. Video debriefing did not improve the accuracy of self-assessment.

CONCLUSIONS

Nurses and physicians demonstrated discordant perceptions of responsibilities. Nurses' self-assessment of teamwork was statistically, but not clinically significantly, higher than experts' in all domains except physician leadership.

Graduating Students' and Surgery Program Directors' Views of the Association of American Medical Colleges Core Entrustable Professional Activities for Entering Residency: Where are the Gaps?

OBJECTIVE

Residency program directors have increasingly expressed concern about the preparedness of some medical school graduates for residency training. The Association of American Medical Colleges recently defined 13 core entrustable professional activities (EPAs) for entering residency that residents should be able to perform without direct supervision on the first day of training. It is not known how students' perception of their competency with these activities compares with that of surgery program directors'.

DESIGN

Cross-sectional survey.

SETTING

All surgery training programs in the United States.

PARTICIPANTS

All program directors (PDs) in the Association of Program Directors in Surgery (APDS) database (n = 222) were invited to participate in an electronic survey, and 119 complete responses were received (53.6%). Among the respondents, 83% were men and 35.2% represented community hospital programs. PDs' responses were compared with questions asking students to rate their confidence in performance of each EPA from the Association of American Medical Colleges Graduation Questionnaire (95% response).

RESULTS

PDs rated their confidence in residents' performance without direct supervision for every EPA significantly lower when compared with the rating by graduating students. Although PDs' ratings continued to be lower than students' ratings, PDs from academic programs (those associated with a medical school) gave higher ratings than those from community programs. PDs generally ranked all 13 EPAs as important to being a trustworthy physician. PDs from programs without preliminary residents gave higher ratings for confidence with EPA performance as compared with PDs with preliminary residents. Among PDs with preliminary residents, there were equal numbers of those who agreed and those who disagreed that there are no identifiable differences between categorical and preliminary residents (42.7% and 41.8%, respectively).

CONCLUSIONS

A large gap exists between confidence in performance of the 13 core EPAs for entering residency without direct supervision for graduating medical students and surgery program directors. Both the groups identified several key areas for improvement that may be addressed by medical school curricular interventions or expanding surgical boot camps in hopes to improve resident performance and patient safety.

Making health care safer: What is the contribution of health psychology?

While health care brings great benefits, all treatments, and many investigations, carry some risk. As patients, we should be told of the risks of specific treatments but we are also at risk from failings in the health care system itself. We suggest that, while there are many examples of individual health psychologists who have made important contributions, this has not yet translated into a broader disciplinary engagement. Health psychologists have devoted much more attention to patients and devoted much less attention to the potentially huge impact of studying and intervening with staff, clinical practice, and organizations. We believe that there are considerable opportunities for health psychology to engage more closely with patient safety and, more importantly, that this would be of great benefit to both patients and staff. Statement of contribution What is already known on this subject? While health care brings great benefits, all treatments, and many investigations, carry some risk. Patients are also at risk from failings in the health care system itself. Studies using review of medical records in many countries have found that between 8% and 12% of patients in hospital suffer an unintended harm due to health care. What does this study add? There are many examples of individual psychologists who have made important contributions, but this has not yet translated into a broader disciplinary engagement. There are considerable opportunities for health psychology to engage more closely with patient safety. These include health behaviour change, teamwork, communication after medical error, diagnosis and decision making, organisational culture, and improving compliance with rules and standards. Psychologists providing a clinical service to specialist services in any area could expand their remit from supporting patients to a more general support and engagement with safety and quality initiatives. Health psychologists have models to understand the behaviour of people, and recent developments in changing behaviour should be applicable to health professionals in addition to their patients.

Interprofessional collaboration milestones: advocating for common assessment criteria in graduate medical education

BACKGROUND

Milestone-based assessments of resident physicians inform critical decisions regarding resident competence and advancement. Thus, it is essential that milestone evaluations are based upon strong validity evidence and that consistent evaluation criteria are used across residency programs. A common approach to assessment of interprofessional collaboration milestones is particularly important since standardized measures of individual resident competence in interprofessional collaboration have not been established.

DISCUSSION

We propose that assessments of interprofessional collaboration in graduate medical education meet common criteria, namely, these assessments should: 1) measure competency of an individual resident, 2) occur in the context of an interprofessional team, 3) be ascertained via direct observation of the resident, 4) be performed in a real-world clinical practice setting (such as a hospital ward, outpatient clinic, or operating room). We present the evidence-based rationale for these criteria and cite examples of published assessment instruments that fulfill one or more of the criteria, however further research is needed to ensure fidelity of assessments. The proposed criteria may assist residency educators as they endeavor to provide robust and consistent assessments of interprofessional collaboration milestones.

Teamwork Assessment Tools in Modern Surgical Practice: A Systematic Review

Introduction. Deficiencies in teamwork skills have been shown to contribute to the occurrence of adverse events during surgery. Consequently, several teamwork assessment tools have been developed to evaluate trainee nontechnical performance. This paper aims to provide an overview of these instruments and review the validity of each tool. Furthermore, the present paper aims to review the deficiencies surrounding training and propose several recommendations to address these issues. Methods. A systematic literature search was conducted to identify teamwork assessment tools using MEDLINE (1946 to August 2015), EMBASE (1974 to August 2015), and PsycINFO (1806 to August 2015) databases. Results. Eight assessment tools which encompass aspects of teamwork were identified. The Nontechnical Skills for Surgeons (NOTSS) assessment was found to possess the highest level of validity from a variety of sources; reliability and acceptability have also been established for this tool. Conclusions. Deficits in current surgical training pathways have prompted several recommendations to meet the evolving requirements of surgeons. Recommendations from the current paper include integration of teamwork training and assessment into medical school curricula, standardised formal training of assessors to ensure accurate evaluation of nontechnical skill acquisition, and integration of concurrent technical and nontechnical skills training throughout training.

Validity and reliability assessment of detailed scoring checklists for use during perioperative emergency simulation training

INTRODUCTION

Few valid and reliable grading checklists have been published for the evaluation of performance during simulated high-stakes perioperative event management. As such, the purposes of this study were to construct valid scoring checklists for a variety of perioperative emergencies and to determine the reliability of scores produced by these checklists during continuous video review.

METHODS

A group of anesthesiologists, intensivists, and educators created a set of simulation grading checklists for the assessment of the following scenarios: severe anaphylaxis, cerebrovascular accident, hyperkalemic arrest, malignant hyperthermia, and acute coronary syndrome. Checklist items were coded as critical or noncritical. Nonexpert raters evaluated 10 simulation videos in a random order, with each video being graded 4 times. A group of faculty experts also graded the videos to create a reference standard to which nonexpert ratings were compared. P < 0.05 was considered significant.

RESULTS

Team leaders in the simulation videos were scored by the expert panel as having performed 56.5% of all items on the checklist (range, 43.8%-84.0%), and 67.2% of the critical items (range, 30.0%-100%). Nonexpert raters agreed with the expert assessment 89.6% of the time (95% confidence interval, 87.2%-91.6%). No learning curve development was found with repetitive video assessment or checklist use. The κ values comparing nonexpert rater assessments to the reference standard averaged 0.76 (95% confidence interval, 0.71-0.81).

CONCLUSIONS

The findings indicate that the grading checklists described are valid, are reliable, and could be used in perioperative crisis management assessment.

The Teamwork Assessment Scale: A Novel Instrument to Assess Quality of Undergraduate Medical Students' Teamwork Using the Example of Simulation-based Ward-Rounds

Background: Simulation-based teamwork trainings are considered a powerful training method to advance teamwork, which becomes more relevant in medical education. The measurement of teamwork is of high importance and several instruments have been developed for various medical domains to meet this need. To our knowledge, no theoretically-based and easy-to-use measurement instrument has been published nor developed specifically for simulation-based teamwork trainings of medical students. Internist ward-rounds function as an important example of teamwork in medicine.

Purposes: The purpose of this study was to provide a validated, theoretically-based instrument that is easy-to-use. Furthermore, this study aimed to identify if and when rater scores relate to performance.

Methods: Based on a theoretical framework for teamwork behaviour, items regarding four teamwork components (Team Coordination, Team Cooperation, Information Exchange, Team Adjustment Behaviours) were developed. In study one, three ward-round scenarios, simulated by 69 students, were videotaped and rated independently by four trained raters. The instrument was tested for the embedded psychometric properties and factorial structure. In study two, the instrument was tested for construct validity with an external criterion with a second set of 100 students and four raters.

Results: In study one, the factorial structure matched the theoretical components but was unable to separate Information Exchange and Team Cooperation. The preliminary version showed adequate psychometric properties (Cronbach’s α=.75). In study two, the instrument showed physician rater scores were more reliable in measurement than those of student raters. Furthermore, a close correlation between the scale and clinical performance as an external criteria was shown (r=.64) and the sufficient psychometric properties were replicated (Cronbach’s α=.78).

Conclusions: The validation allows for use of the simulated teamwork assessment scale in undergraduate medical ward-round trainings to reliably measure teamwork by physicians. Further studies are needed to verify the applicability of the instrument.

Development of a standardised training curriculum for robotic surgery: a consensus statement from an international multidisciplinary group of experts

OBJECTIVES

To explore the views of experts about the development and validation of a robotic surgery training curriculum, and how this should be implemented.

MATERIALS AND METHODS

An international expert panel was invited to a structured session for discussion. The study was of a mixed design, including qualitative and quantitative components based on focus group interviews during the European Association of Urology (EAU) Robotic Urology Section (ERUS) (2012), EAU (2013) and ERUS (2013) meetings. After introduction to the aims, principles and current status of the curriculum development, group responses were elicited. After content analysis of recorded interviews generated themes were discussed at the second meeting, where consensus was achieved on each theme. This discussion also underwent content analysis, and was used to draft a curriculum proposal. At the third meeting, a quantitative questionnaire about this curriculum was disseminated to attendees to assess the level of agreement with the key points.

RESULTS

In all, 150 min (19 pages) of the focus group discussion was transcribed (21 316 words). Themes were agreed by two raters (median agreement κ 0.89) and they included: need for a training curriculum (inter-rater agreement κ 0.85); identification of learning needs (κ 0.83); development of the curriculum contents (κ 0.81); an overview of available curricula (κ 0.79); settings for robotic surgery training ((κ 0.89); assessment and training of trainers (κ 0.92); requirements for certification and patient safety (κ 0.83); and need for a universally standardised curriculum (κ 0.78). A training curriculum was proposed based on the above discussions.

CONCLUSION

This group proposes a multi-step curriculum for robotic training. Studies are in process to validate the effectiveness of the curriculum and to assess transfer of skills to the operating room.

Towards the next frontier for simulation-based training: full-hospital simulation across the entire patient pathway

OBJECTIVE

To evaluate the efficacy of an entire hospital simulation in imparting skills to expert healthcare providers, encompassing both retention and transfer to clinical practice.

BACKGROUND

Studies demonstrating the effectiveness of simulation do not concentrate upon expert multidisciplinary teams. Moreover, their focus is confined to a single clinical setting, thereby not considering the complex interactions across multiple hospital departments.

METHODS

A total of 288 participants (Attending surgeons, anesthesiologists, physicians, and nurses) completed this largest simulation study to date, set in the UK Defence Medical Services' Hospital Simulator and the conflict zone in Afghanistan. The simulator termed "Hospital Exercise" (HOSPEX) is a fully immersive live-in simulation experience that covers the entire environment of a military hospital with all departments. Participants undertook a 3-day training program within HOSPEX before deployment to war zones. Primary outcome measures were assessed with IMPAcT (the Imperial Military Personnel Assessment Tool). IMPAcT measures crisis management, trauma care, hospital environment, operational readiness, and transfer of skills to civilian practice. Reliability, skills learning, and retention in the conflict zone were assessed statistically.

RESULTS

Reliability in skills assessment was excellent (Cronbach α: nontechnical skills = 0.87-0.94; environment/patient skills = 0.83-0.95). Pre/post-HOSPEX comparisons revealed significant improvements in decision making (M = 4.98, SD = 1.20 to M = 5.39, SD = 0.91; P = 0.03), situational awareness (M = 5.44, SD = 1.04 to M = 5.74, SD = 0.92; P = 0.01), trauma care (M = 5.53, SD = 1.23 to M = 5.85, SD = 1.09; P = 0.05), and knowledge of hospital environment (M = 5.19, SD = 1.17 to M = 5.42, SD = 0.97; P = 0.04). No skills decayed over time when assessed several months later in the real conflict zone. All skills transferred to civilian clinical practice.

CONCLUSIONS

This is the first study to describe the value of a full-hospital simulation across the entire patient pathway. Such macrosimulations may be the way forward for integrating the complex training needs of expert clinicians and testing organizational "fitness for purpose" of entire hospitals.