Training faculty in nontechnical skill assessment: national guidelines on program requirements. Ann Surg. 2013;258:370-375. [PMID: 23222032 DOI: 10.1097/sla.0b013e318279560b]

Development and applications of the Anaesthetists' Non-Technical Skills behavioural marker system: a systematic review

OBJECTIVES

To comprehensively synthesise evidence regarding the validity and reliability of the Anaesthetists' Non-Technical Skills (ANTS) behavioural marker system and its application as a tool for the training and assessment of non-technical skills to improve patient safety.

DESIGN

Systematic review.

DATA SOURCES

We employed a citation search strategy. The Scopus and Web of Science databases were searched for articles published from 2002 to May 2022.

ELIGIBILITY CRITERIA

English-language publications that applied the ANTS system in a meaningful way, including its use to guide data collection, analysis and reporting.

DATA EXTRACTION AND SYNTHESIS

Study screening, data extraction and quality assessment were performed by two independent reviewers. We appraised the quality of included studies using the Joanna Briggs Institute Critical Appraisal Checklists. A framework analysis approach was used to summarise and synthesise the included articles.

RESULTS

54 studies were identified. The ANTS system was applied across a wide variety of study objectives, settings and units of analysis. The methods used in these studies varied and included quantitative (n=42), mixed (n=8) and qualitative (n=4) approaches. Most studies (n=47) used the ANTS system to guide data collection. The most commonly reported reliability statistic was inter-rater reliability (n=35). Validity evidence was reported in 51 (94%) studies. The qualitative application outcomes of the ANTS system provided a reference for the analysis and generation of new theories across disciplines.

CONCLUSION

Our results suggest that the ANTS system has been used in a wide range of studies. It is an effective tool for assessing non-technical skills. Investigating the methods by which the ANTS system can be evaluated and implemented for training within clinical environments is anticipated to significantly enhance ongoing enhancements in staff performance and patient safety.

PROSPERO REGISTRATION NUMBER

CRD42022297773.

Observation of behavioural skills by medical simulation facilitators: a cross-sectional analysis of self-reported importance, difficulties, observation strategies and expertise development

BACKGROUND

The association between team performance and patient care was an immense boost for team-based education in health care. Behavioural skills are an important focus in these sessions, often provided via a mannikin-based immersive simulation experience in a (near) authentic setting. Observation of these skills by the facilitator(s) is paramount for facilitated feedback with the team. Despite the acknowledgement that trained facilitators are important for optimal learning, insight into this observation process by facilitators is limited.

OBJECTIVES

What are the self-reported current practices and difficulties regarding the observation of behavioural skills amongst facilitators during team training and how have they been trained to observe behavioural skills?

METHODS

This cross-sectional study used a pilot-tested, content-validated, multi-linguistic online survey within Europe, distributed through a non-discriminative snowball sampling method. Inclusion was limited to facilitators observing behavioural skills within a medical team setting.

RESULTS

A total of 175 persons filled in the questionnaire. All aspects of behavioural skill were perceived as very important to observe. The self-perceived difficulty of the behavioural skill aspects ranged from slightly to moderately difficult. Qualitative analysis revealed three major themes elaborating on this perceived difficulty: (1) not everything can be observed, (2) not everything is observed and (3) interpretation of observed behavioural skills is difficult. Additionally, the number of team members health care facilitators have to observe, outnumbers their self-reported maximum. Strategies and tools used to facilitate their observation were a blank notepad, co-observers and predefined learning goals. The majority of facilitators acquired observational skills through self-study and personal experience and/or observing peers. Co-observation with either peers or experts was regarded as most learn some for their expertise development. Overall, participants perceived themselves as moderately competent in the observation of behavioural skills during team training.

CONCLUSIONS

Observation of behavioural skills by facilitators in health care remains a complex and challenging task. Facilitators' limitations with respect to attention, focus and (in)ability to perform concomitant tasks, need to be acknowledged. Although strategies and tools can help to facilitate the observation process, they all have their limitations and are used in different ways.

Collecting evidence of validity for an assessment tool for Norwegian medical students' non-technical skills (NorMS-NTS): usability and reliability when used by novice raters

BACKGROUND

The NorMS-NTS tool is an assessment tool for assessing Norwegian medical students' non-technical skills (NTS). The NorMS-NTS was designed to provide student feedback, training evaluations, and skill-level comparisons among students at different study sites. Rather than requiring extensive rater training, the tool should capably suit the needs of busy doctors as near-peer educators. The aim of this study was to examine the usability and preliminary assess validity of the NorMS-NTS tool when used by novice raters.

METHODS

This study focused on the usability of the assessment tool and its internal structure. Three raters used the NorMS-NTS tool to individually rate the team leader, a medical student, in 20 video-recorded multi-professional simulation-based team trainings. Based on these ratings, we examined the tools' internal structure by calculating the intraclass correlation coefficient (ICC) (version 3.1) interrater reliability, internal consistency, and observability. After the rating process was completed, the raters answered a questionnaire about the tool's usability.

RESULTS

The ICC agreement and the sum of the overall global scores for all raters were fair: ICC (3,1) = 0.53. The correlation coefficients for the pooled raters were in the range of 0.77-0.91. Cronbach's alpha for elements, categories and global score were mostly above 0.90. The observability was high (95%-100%). All the raters found the tool easy to use, none of the elements were redundant, and the written instructions were helpful. The raters also found the tool easier to use once they had acclimated to it. All the raters stated that they could use the tool for both training and teaching.

CONCLUSIONS

The observed ICC agreement was 0.08 below the suggested ICC level for formative assessment (above 0.60). However, we know that the suggestion is based on the average ICC, which is always higher than a single-measure ICC. There are currently no suggested levels for single-measure ICC, but other validated NTS tools have single-measure ICC in the same range. We consider NorMS-NTS as a usable tool for formative assessment of Norwegian medical students' non-technical skills during multi-professional team training by raters who are new to the tool. It is necessary to further examine validity and the consequences of the tool to fully validate it for formative assessments.

A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

BACKGROUND

Non-technical skills (NTS) assessment tools are widely used to provide formative and summative assessment for healthcare professionals and there are now many of them. This study has examined three different tools designed for similar settings and gathered evidence to test their validity and usability.

METHODS

Three NTS assessment tools designed for use in the UK were used by three experienced faculty to review standardized videos of simulated cardiac arrest scenarios: ANTS (Anesthetists' Non-Technical Skills), Oxford NOTECHS (Oxford NOn-TECHnical Skills) and OSCAR (Observational Skill based Clinical Assessment tool for Resuscitation). Internal consistency, interrater reliability and quantitative and qualitative analysis of usability were analyzed for each tool.

RESULTS

Internal consistency and interrater reliability (IRR) varied considerably for the three tools across NTS categories and elements. Intraclass correlation scores of three expert raters ranged from poor (task management in ANTS [0.26] and situation awareness (SA) in Oxford NOTECHS [0.34]) to very good (problem solving in Oxford NOTECHS [0.81] and cooperation [0.84] and SA [0.87] in OSCAR). Furthermore, different statistical tests of IRR produced different results for each tool. Quantitative and qualitative examination of usability also revealed challenges in using each tool.

CONCLUSIONS

The lack of standardization of NTS assessment tools and training in their use is unhelpful for healthcare educators and students. Educators require ongoing support in the use of NTS assessment tools for the evaluation of individual healthcare professionals or healthcare teams. Summative or high-stakes examinations using NTS assessment tools should be undertaken with at least two assessors to provide consensus scoring. In light of the renewed focus on simulation as an educational tool to support and enhance training recovery in the aftermath of COVID-19, it is even more important that assessment of these vital skills is standardized, simplified and supported with adequate training.

Online non-technical skills faculty training

BACKGROUND

The Medical Students Non-Technical Skills (Medi-StuNTS) system is a behavioural marker system (BMS) designed to identify and debrief non-technical skills (NTS) for medical students during immersive simulation. Educators must be adequately trained in using the BMS. This study aimed to design and implement an online platform to deliver a faculty development course on using Medi-StuNTS and evaluate the feasibility of this platform in training faculty to identify and debrief NTS.

APPROACH

The online platform was developed by faculty with expertise in NTS, based on guidance for faculty training programme requirements and the multimodal model for online education. Content was arranged in modules, using presentations, videos of simulation scenarios and interactive discussion boards.

EVALUATION

Fifteen participants completed the course and feedback over a two-month period. A feedback form was completed to assess feasibility, based on a feasibility framework. The areas of focus were acceptability, demand, implementation, practicality, adaptation, integration, expansion and limited efficacy. Feedback indicated that the course shows promise in improving the ability of faculty to identify and debrief NTS.

IMPLICATIONS

The platform was successfully developed and implemented and was able to reach a national audience due to its online nature. Specific strengths include increased flexibility and accessibility compared to in-person training. Feasibility assessment suggests that this newly developed online platform can work as an effective method for faculty development in order to increase skills in identifying and debriefing NTS using Medi-StuNTS. Future work will focus on expansion of the online platform and dissemination to an international audience.

Cancer multidisciplinary team meetings: impact of logistical challenges on communication and decision-making

Background

Multidisciplinary teams (MDTs) are widely used in cancer care. Recent research points to logistical challenges impeding MDT decision-making and dissatisfaction among members. This study sought to identify different types of logistical issues and how they impacted team processes.

Methods

This was a secondary analysis of a cross-sectional observational study. Three cancer MDTs (breast, colorectal, and gynaecological) were recruited from UK hospitals. Validated observational instruments were used to measure decision-making (Metrics of Observational Decision-making, MDT-MODe), communication (Bales' Interaction Process Analysis, Bales' IPA), and case complexity (Measure of Case Discussion Complexity, MeDiC), including logistical challenges (Measure of Case Discussion Complexity, MeDiC), across 822 case discussions from 30 videoed meetings. Descriptive analysis and paired samples t tests were used to identify and compare frequency of different types of logistical challenges, along with partial correlations, controlling for clinical complexity of cases, to understand how such issues related to the MDT decision-making and communication.

Results

A significantly higher frequency of administrative and process issues (affecting 30 per cent of cases) was seen compared with the frequency of equipment issues (affecting 5 per cent of cases; P < 0.001) and the frequency of the attendance issues (affecting 16 per cent of cases; P < 0.001). The frequency of the attendance issues was significantly higher than the frequency of equipment issues (P < 0.001). Partial correlation analysis revealed that administrative and process issues, including attendance, were negatively correlated with quality of information (r = −0.15, P < 0.001; r = −0.11, P < 0.001), and equipment issues with the quality of contribution to meeting discussion (r = −0.14, P < 0.001). More questioning and answering by MDT members was evident with the administrative and process issues (r = 0.21, P < 0.001; r = 0.19, P < 0.001). Some differences were observed in teams' socioemotional reactions to the administrative and process issues with the gynaecological MDT showing positive correlation with positive socioemotional reactions (r = 0.20, P < 0.001), and the breast cancer MDT with negative socioemotional reactions (r = 0.17, P < 0.001).

Conclusion

Administrative and process issues were the most frequent logistical challenges for the studied teams. Where diagnostic results were unavailable, and inadequate patient details provided, the quality of decision-making was reduced.

Evaluating Nontechnical Skills in US Emergency Departments Using Simulation: Validating and Contextualizing a UK Assessment Tool

INTRODUCTION

Nontechnical skills (NTS) in medicine are the "cognitive, social, and personal resource skills that complement technical skills contributing to safe and efficient care." We aimed to (1) evaluate the validity and reliability of a 12-element United Kingdom emergency medicine (EM) NTS assessment tool in the context of United States (US) EM practice and (2) identify behaviors unique to US clinical practice.

METHODS

This was a mixed methods study conducted in 2 phases, following Kane's validity framework. The intended use of the NTS tool is to provide formative assessment of US EM physicians (EPs) from a video of simulated clinical encounters. In phase I, a focus group assessed the appropriateness of each aspect of the tool in the context of US EM practice by reviewing and identifying the NTS of an EP in a simulated clinical scenario. In phase II, EPs (N = 208) attending a national EM conference evaluated an EP's behaviors in 1 of 2 video simulations. Reliability in the form of internal consistency was calculated using Cronbach α. All participants suggested exemplar behaviors for the 12 elements in the context of their own clinical practice and generated new assessment elements.

RESULTS

Internal consistency was acceptable (α > 0.7) for all categories, except teamwork and cooperation. Participants proposed 4 novel behavioral elements and suggested US exemplar behaviors for all 12 original elements.

CONCLUSIONS

This tool can be used to assess US EP's NTS for the purpose of formative assessment. Refinement of exemplar behaviors and inclusion of novel US-specific elements may optimize usability.

Injuries during Laparoscopic Cholecystectomy: A Scoping Review of the Claims and Civil Action Judgements

Background. To define what type of injuries are more frequently related to medicolegal claims and civil action judgments. Methods. We performed a scoping review on 14 studies and 2406 patients, analyzing medicolegal claims related to laparoscopic cholecystectomy injuries. We have focalized on three phases associated with claims: phase of care, location of injuries, type of injuries. Results. The most common phase of care associated with litigation was the improper intraoperative surgical performance (47.6% ± 28.3%), related to a “poor” visualization, and the improper post-operative management (29.3% ± 31.6%). The highest rate of defense verdicts was reported for the improper post-operative management of the injury (69.3% ± 23%). A lower rate was reported in the incorrect presurgical assessment (39.7% ± 24.4%) and in the improper intraoperative surgical performance (21.39% ± 21.09%). A defense verdict was more common in cystic duct injuries (100%), lower in hepatic bile duct (42.9%) and common bile duct (10%) injuries. Conclusions. During laparoscopic cholecystectomy, the most common cause of claims, associated with lower rate of defense verdict, was the improper intraoperative surgical performance. The decision to take legal action was determined often for poor communication after the original incident.

Defining the need for faculty development in assessment

BACKGROUND

High-quality workplace-based assessments are essential for competency-based surgical education. We explored education leaders' perceptions regarding faculty competence in assessment.

METHODS

Surgical education leaders were surveyed regarding which areas faculty needed improvement, and knowledge of assessment tools. Respondents were queried on specific skills regarding (a)importance in resident/medical student education (b)competence of faculty in assessment and feedback.

RESULTS

Surveys (n = 636) were emailed, 103 responded most faculty needed improvement in: verbal (86%) and written (83%) feedback, assessing operative skill (49%) and preparation for procedures (50%). Cholecystectomy, trauma laparotomy, inguinal herniorrhaphy were "very-extremely important" in resident education (99%), but 21-24% thought faculty "moderately to not-at-all" competent in assessment. This gap was larger for non-technical skills. Regarding assessment tools, 56% used OSATS, 49% Zwisch; most were unfamiliar with all non-technical tools.

SUMMARY

These data demonstrate a significant perceived gap in competence of faculty in assessment and feedback, and unfamiliarity with assessment tools. This can inform faculty development to support competency-based surgical education.

Development and Validation of a Short Version of the Metric for the Observation of Decision-Making in Multidisciplinary Tumor Boards: MODe-Lite

Background

Evidence-based tools are necessary for scientifically improving the way MTBs work. Such tools are available but can be difficult to use. This study aimed to develop a robust observational assessment tool for use on cancer multidisciplinary tumor boards (MTBs) by health care professionals in everyday practice.

Methods

A retrospective cross-sectional observational study was conducted in the United Kingdom from September 2015 to July 2016. Three tumor boards from three teaching hospitals were recruited, with 44 members overall. Six weekly meetings involving 146 consecutive cases were video-recorded and scored using the validated MODe tool. Data were subjected to reliability and validity analysis in the current study to develop a shorter version of the MODe.

Results

Phase 1, a reduction of the original items in the MODe, was achieved through two focus group meetings with expert assessors based on previous research. The 12 original items were reduced to 6 domains, receiving full agreement by the assessors. In phase 2, the six domains were subjected to item reliability, convergent validation, and internal consistency testing against the MODe-Lite global score, the MODe global score, and the items of the MODe. Significant positive correlations were evident across all domains (p < 0.01), indicating good reliability and validity. In phase 3, feasibility and high inter-assessor reliability were achieved by two clinical assessors. Six domains measuring clinical input, holistic input, clinical collaboration, pathology, radiology, and management plan were integrated into MODe-Lite.

Conclusions

As an evidence-based tool for health care professionals in everyday practice, MODe-Lite gives cancer MTBs insight into the way they work and facilitates improvements in practice.

Supplementary Information

The online version contains supplementary material available at 10.1245/s10434-021-09989-7.

Gathering Validity Evidence to Adapt the Non-technical Skills for Surgeons (NOTSS) Assessment Tool to the United States Context

BACKGROUND

Nontechnical skills are of increasing focus for safe and effective performance in the operating room. Assessment tools have been developed in Europe, Africa, and Asia but not adapted to the unique aspects of surgical delivery in the United States. Our objective was to use the Non-Technical Skills for Surgeons (NOTSS) assessment tool as a basis to establish consensus on essential nontechnical skills for surgical trainees and practicing surgeons in the U.S surgical context.

STUDY DESIGN

A mixed-methods research design was used in the form of a modified Delphi process to build consensus on essential NOTSS. A panel of surgical experts from hospitals across the U.S used this iterative process in 4 rounds to generate, rate, and classify behaviors. The primary outcome was consensus on behaviors as being essential for surgeons to achieve the best patient outcomes in the operating room, with a median rating of ≥6 on a 7-point scale for inclusion.

RESULTS

A total of 10 surgical experts participated. One hundred and thirty eight behaviors were generated in Round 1, and reduced to 100 behaviors in Rounds 2 and 3 based on application of inclusion criteria. The final skill list consisted of behaviors in Situation Awareness (n = 26), Decision Making (n = 18), Teamwork (n = 25), and Leadership (n = 31). No additional NOTSS categories or elements emerged from the analysis. In Round 4, all 100 behaviors were successfully grouped into 12 nontechnical skills elements. Labels and definitions were reworded to reflect the U.S. context, and an appropriate assessment scale was selected.

CONCLUSIONS

A panel of surgical experts from across the U.S. reached consensus on the essential NOTSS to achieve the best patient outcomes in the operating room. These behaviors form an empirical basis for the first context-specific nontechnical skills assessment and training tool for practicing surgeons in the U.S.

Novel Assessments of Technical and Nontechnical Cardiac Surgery Quality: Protocol for a Mixed Methods Study

BACKGROUND

Of the 150,000 patients annually undergoing coronary artery bypass grafting, 35% develop complications that increase mortality 5 fold and expenditure by 50%. Differences in patient risk and operative approach explain only 2% of hospital variations in some complications. The intraoperative phase remains understudied as a source of variation, despite its complexity and amenability to improvement.

OBJECTIVE

The objectives of this study are to (1) investigate the relationship between peer assessments of intraoperative technical skills and nontechnical practices with risk-adjusted complication rates and (2) evaluate the feasibility of using computer-based metrics to automate the assessment of important intraoperative technical skills and nontechnical practices.

METHODS

This multicenter study will use video recording, established peer assessment tools, electronic health record data, registry data, and a high-dimensional computer vision approach to (1) investigate the relationship between peer assessments of surgeon technical skills and variability in risk-adjusted patient adverse events; (2) investigate the relationship between peer assessments of intraoperative team-based nontechnical practices and variability in risk-adjusted patient adverse events; and (3) use quantitative and qualitative methods to explore the feasibility of using objective, data-driven, computer-based assessments to automate the measurement of important intraoperative determinants of risk-adjusted patient adverse events.

RESULTS

The project has been funded by the National Heart, Lung and Blood Institute in 2019 (R01HL146619). Preliminary Institutional Review Board review has been completed at the University of Michigan by the Institutional Review Boards of the University of Michigan Medical School.

CONCLUSIONS

We anticipate that this project will substantially increase our ability to assess determinants of variation in complication rates by specifically studying a surgeon's technical skills and operating room team member nontechnical practices. These findings may provide effective targets for future trials or quality improvement initiatives to enhance the quality and safety of cardiac surgical patient care.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/22536.

Psychometric Testing of a Structured Assessment Instrument for Non-technical Skills (NANTS-no) for Use in Clinical Supervision of Student Nurse Anesthetists

BACKGROUND

This study evaluated psychometric properties of a structured behavioral assessment instrument, Nurse Anaesthetists' Non-Technical Skills-Norway (NANTS-no). It estimated whether reliable assessments of nontechnical skills (NTS) could be made after taking part in a workshop. An additional objective was to evaluate the instrument's acceptability and usability.

METHODS

An explorative design was used. Nurse anesthetists (n = 46) involved in clinical supervision attended a 6-hour workshop on NTS, then rated NTS in video-recorded simulated scenarios and completed a questionnaire.

RESULTS

High reliability and dependability were estimated in this setting. Participants regarded the instrument as useful for clinical supervision of student nurse anesthetists (SNAs).

CONCLUSIONS

Findings suggest that NANTS-no may be reliable for performing clinical assessments of SNAs and encouraging critical reflection. However, further research is needed to explore its use in clinical settings.

Key components of a hepatobiliary surgery curriculum for general surgery residents: results of the FULCRUM International Delphi consensus

BACKGROUND

In general surgery residency, hepatobiliary training varies significantly across the world. The aim of this study was to establish an international consensus among hepatobiliary surgeons on components of a hepatobiliary curriculum for general surgery residents.

METHODS

A three-round modified Delphi technique was employed. Fifty-two hepatobiliary surgeons involved in general surgery training programs were invited. An initial questionnaire was developed by a group of experts in hepatobiliary and educational research after a systematic literature review. It comprised 90 statements about knowledge, technical skills, attitudes, and postoperative care. Panelists could add or alter items. The survey was delivered electronically and the panel was instructed to score the items based on 5-point Likert scale. Consensus was reached when at least 80% of panelists agreed on a statement with Cronbach's alpha value >0.8.

RESULTS

Forty-one (79%) experts have participated. Sixteen panelists are based in Asia, 14 in Europe, and 11 in the Americas. Eighty percent of all proposed skills (81/101) were considered fundamental including knowledge (39/43), technical skills (16/32), attitude (15/15), and postoperative care (11/11).

CONCLUSION

An international consensus was achieved on components of a hepatobiliary curriculum. Acquiring broad knowledge is fundamental during residency. Advanced liver resection techniques require specialized hepatobiliary training.

A multicentre cross-sectional observational study of cancer multidisciplinary teams: Analysis of team decision making

BACKGROUND

Multidisciplinary teams (MDT) formulate expert informed treatment recommendations for people with cancer. We set out to examine how the factors proposed by the functional perspective of group decision making (DM), that is, interaction process, internal factors (factors emanating from within the group such as group size), external circumstances (factors coming from the outside of the team), and case-complexity affect the quality of MDT decision making.

METHODS

This was a cross-sectional observational study. Three cancer MDTs were recruited with 44 members overall and 30 of their weekly meetings filmed. Validated observational instruments were used to measure quality of DM, interactions, and complexity of 822 case discussions.

RESULTS

The full regression model with the variables proposed by the functional perspective was significant, R2  = 0.52, F(20, 801) = 43.47, P < .001, adjusted R2  = 0.51. Positive predictors of DM quality were asking questions (P = .001), providing answers (P = .001), team size (P = .007), gender balance (P = .003), and clinical complexity (P = .001), while negative socioemotional reactions (P = .007), gender imbalance (P = .003), logistical issues (P = .001), time-workload pressures (P = .002), and time spent in the meeting (P = .001) were negative predictors. Second half of the meetings also saw significant decrease in the DM quality (P = .001), interactions (P = .001), group size (P = .003), and clinical complexity (P = .001), and an increase in negative socioemotional reactions (P = .001) and time-workload pressures (P = .001).

DISCUSSION

To the best of our knowledge, this is the first study to attempt to assess the factors proposed by the functional perspective in cancer MDTs. One novel finding is the effect of sociocognitive factors on team DM quality, while another is the cognitive-catch 22 effect: while the case discussions are significantly simpler in the second half of the meeting, there is significantly less time left to discuss the remaining cases, further adding to the cognitive taxation in teams who are now rapidly attempting to close their time-workload gap. Implications are discussed in relation to quality and safety.

Reliability of assessment of medical students’ non-technical skills using a behavioural marker system: does clinical experience matter?

Introduction

Non-technical skills are recognised to play an integral part in safe and effective patient care. Medi-StuNTS (Medical Students’ Non-Technical Skills) is a behavioural marker system developed to enable assessment of medical students’ non-technical skills. This study aimed to assess whether newly trained raters with high levels of clinical experience could achieve reliability coefficients of >0.7 and to compare differences in inter-rater reliability of raters with varying clinical experience.

Methods

Forty-four raters attended a workshop on Medi-StuNTS before independently rating three videos of medical students participating in immersive simulation scenarios. Data were grouped by raters’ levels of clinical experience. Inter-rater reliability was assessed by calculating intraclass correlation coefficients (ICC).

Results

Eleven raters with more than 10 years of clinical experience achieved single-measure ICC of 0.37 and average-measures ICC of 0.87. Fourteen raters with more than or equal to 5 years and less than 10 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC of 0.59. Nineteen raters with less than 5 years of clinical experience achieved single-measure ICC of 0.09 and average-measures ICC 0.65.

Conclusions

Using 11 newly trained raters with high levels of clinical experience produced highly reliable ratings that surpassed the prespecified inter-rater reliability standard; however, a single rater from this group would not achieve sufficiently reliable ratings. This is consistent with previous studies using other medical behavioural marker systems. This study demonstrated a decrease in inter-rater reliability of raters with lower levels of clinical experience, suggesting caution when using this population as raters for assessment of non-technical skills.

A measure of case complexity for streamlining workflow in multidisciplinary tumor boards: Mixed methods development and early validation of the MeDiC tool

BACKGROUND AND OBJECTIVE

There is increasing emphasis in cancer care globally for care to be reviewed and managed by multidisciplinary teams (ie, in tumor boards). Evidence and recommendations suggest that the complexity of each patient case needs to be considered as care is planned; however, no tool currently exists for cancer teams to do so. We report the development and early validation of such a tool.

METHODS

We used a mixed-methods approach involving psychometric evaluation and expert review to develop the Measure of case-Discussion Complexity (MeDiC) between May 2014 and November 2016. The study ran in six phases and included ethnographic interviews, observations, surveys, feasibility and reliability testing, expert consensus, and multiple expert-team reviews.

RESULTS

Phase-1: case complexity factors identified through literature review and expert interviews; Phase-2: 51 factors subjected to iterative review and content validation by nine cancer teams across four England Trusts with nine further items identified; Phase 3: 60 items subjected to expert review distilled to the most relevant; Phase 4: item weighing and further content validation through a national UK survey; Phases 5 and 6: excellent interassessor reliability between clinical and nonclinical observers, and adequate validity on 903 video case discussions achieved. A final set of 27 factors, measuring clinical and logistical complexities were integrated into MeDiC.

CONCLUSIONS

MeDiC is an evidence-based and expert-driven tool that gauges the complexity of cancer cases. MeDiC may be used as a clinical quality assurance and screening tool for tumor board consideration through case selection and prioritization.

Modern Plastic Surgical Practice: Technical Competence Alone Is Not Enough

Annually, an estimated 234 million major surgical operations occur worldwide, with concomitant seven million complications and one million deaths. It is now well established that technical competence is necessary, but not sufficient for modern surgical practice and outcomes. Breakdown in non-technical skills has been attributed as a key root cause for near misses and patient harm in the operating room. This article discusses the multi-faceted skills-set that is necessary for the modern surgeon to succeed and for optimal patient outcomes. This includes technical skills, non-technical skills, with a focus on key CanMEDS framework domains, including leadership, communication, evidence-based surgery and mentorship.

Analysing Breast Cancer Multidisciplinary Patient Management: A Prospective Observational Evaluation of Team Clinical Decision-Making

Background

Multidisciplinary team (MDT)-driven cancer care is a mandatory UK national policy, widely used globally. However, few studies have examined how MDT members make decisions as a team. We report a single-centre prospective study on team working within breast cancer MDT.

Methods

This was a prospective observational study of 10 breast MDT meetings (MDM). Trained clinical observer scored quality of presented information and disciplinary contribution to case reviews in real time, using a validated tool, namely Metric for the Observation of Decision-Making. Data were analysed to evaluate quality of team working.

Results

Ten MDMs were observed (N = 346 patients). An average of 42 patients were discussed per MDM (range: 29–51) with an average 3 min 20 s (range: 31 s–9 min) dedicated to each patient. Management decision was made in 99% of cases. In terms of contribution to case reviews, breast care nurses scored significantly (p < 0.05) lower (M = 1.79, SD = 0.12) compared to other team members (e.g. surgeons, M = 4.65; oncologists, M = 3.07; pathologists, M = 4.51; radiologists, M = 3.21). Information on patient psychosocial aspects (M = 1.69, SD = 0.68), comorbidities (M = 1.36, SD = 0.39) and views on treatment options (M = 1.47, SD = 0.34) was also significantly (p < 0.05) less well represented compared to radiology (M = 3.62, SD = 0.77), pathology (M = 4.42, SD = 0.49) and patient history (M = 3.91, SD = 0.48).

Conclusion

MDT evaluation via direct observation in a meeting is feasible and reliable. We found consistent levels of quality of information coverage and contribution within the team, but certain aspects could be improved. Contribution to patient review resides predominantly with surgeons, while presented patient information is largely of biomedical nature. These findings can be fed to cancer MDTs to identify potential interventions for improvement.

Do multidisciplinary cancer care teams suffer decision-making fatigue: an observational, longitudinal team improvement study

OBJECTIVE

The objective of this study was to examine effectiveness of codesigned quality-improving interventions with a multidisciplinary team (MDT) with high workload and prolonged meetings to ascertain: (1) presence and impact of decision-making (DM) fatigue on team performance in the weekly MDT meeting and (2) impact of a short meeting break as a countermeasure of DM fatigue.

DESIGN AND INTERVENTIONS

This is a longitudinal multiphase study with a codesigned intervention bundle assessed within team audit and feedback cycles. The interventions comprised short meeting breaks, as well as change of room layout and appointing a meeting chair.

SETTING AND PARTICIPANTS

A breast cancer MDT with 15 members was recruited between 2013 and 2015 from a teaching hospital of the London (UK) metropolitan area.

MEASURES

A validated observational tool (Metric for the Observation of Decision-making) was used by trained raters to assess quality of DM during 1335 patient reviews. The tool scores quality of information and team contributions to reviews by individual disciplines (Likert-based scores), which represent our two primary outcome measures.

RESULTS

Data were analysed using multivariate analysis of variance. DM fatigue was present in the MDT meetings: quality of information (M=16.36 to M=15.10) and contribution scores (M=27.67 to M=21.52) declined from first to second half of meetings at baseline. Of the improvement bundle, we found breaks reduced the effect of fatigue: following introduction of breaks (but not other interventions) information quality remained stable between first and second half of meetings (M=16.00 to M=15.94), and contributions to team DM improved overall (M=17.66 to M=19.85).

CONCLUSION

Quality of cancer team DM is affected by fatigue due to sequential case review over often prolonged periods of time. This detrimental effect can be reversed by introducing a break in the middle of the meeting. The study offers a methodology based on 'team audit and feedback' principle for codesigning interventions to improve teamwork in cancer care.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Evaluation of a Simpler Tool to Assess Nontechnical Skills During Simulated Critical Events

INTRODUCTION

Management of critical events requires teams to employ nontechnical skills (NTS), such as teamwork, communication, decision making, and vigilance. We sought to estimate the reliability and provide evidence for the validity of the ratings gathered using a new tool for assessing the NTS of anesthesia providers, the behaviorally anchored rating scale (BARS), and compare its scores with those of an established NTS tool, the Anaesthetists' Nontechnical Skills (ANTS) scale.

METHODS

Six previously trained raters (4 novices and 2 experts) reviewed and scored 18 recorded simulated pediatric crisis management scenarios using a modified ANTS and a BARS tool. Pearson correlation coefficients were calculated separately for the novice and expert raters, by scenario, and overall.

RESULTS

The intrarater reliability of the ANTS total score was 0.73 (expert, 0.57; novice, 0.84); for the BARS tool, it was 0.80 (expert, 0.79; novice, 0.81). The average interrater reliability of BARS scores (0.58) was better than ANTS scores (0.37), and the interrater reliabilities of scores from novices (0.69 BARS and 0.52 ANTS) were better than those obtained from experts (0.47 BARS and 0.21 ANTS) for both scoring instruments. The Pearson correlation between the ANTS and BARS total scores was 0.74.

CONCLUSIONS

Overall, reliability estimates were better for the BARS scores than the ANTS scores. For both measures, the intrarater and interrater reliability was better for novices compared with domain experts, suggesting that properly trained novices can reliably assess the NTS of anesthesia providers managing a simulated critical event. There was substantial correlation between the 2 scoring instruments, suggesting that the tools measured similar constructs. The BARS tool can be an alternative to the ANTS scale for the formative assessment of NTS of anesthesia providers.

Development and evaluation of a patient-centred measurement tool for surgeons' non-technical skills

Background

Non-technical skills are essential for safe and effective surgery. Several tools to assess surgeons' non-technical skills from the clinician's perspective have been developed. However, a reliable measurement tool using a patient-centred approach does not currently exist. The aim of this study was to translate the existing Non-Technical Skills for Surgeons (NOTSS) tool into a patient-centred evaluation tool.

Methods

Data were gathered from four cohorts of patients using an iterative four-stage mixed-methods research design. Exploratory and confirmatory factor analyses were performed to establish the psychometric properties of the tool, focusing on validity, reliability, usability and parsimony.

Results

Some 534 patients were recruited to the study. A total of 24 patient-centred non-technical skill items were developed in stage 1, and reduced to nine items in stage 2 using exploratory factor analysis. In stage 3, confirmatory factor analysis demonstrated that these nine items each loaded on to one of three factors, with excellent internal consistency: decision-making, leadership, and communication and teamwork. In stage 4, validity testing established that the new tool was independent of physician empathy and predictive of surgical quality. Surgical leadership emerged as the most dominant skill that patients could recognize and evaluate.

Conclusion

A novel nine-item assessment tool has been developed. The Patients' Evaluation of Non-Technical Skills (PENTS) tool allows valid and reliable measurement of surgeons' non-technical skills from the patient perspective.

Construct and criterion validity testing of the Non-Technical Skills for Surgeons (NOTSS) behaviour assessment tool using videos of simulated operations

Background

Surgeons' non-technical skills are an important part of surgical performance and surgical education. The most widely adopted assessment tool is the Non-Technical Skills for Surgeons (NOTSS) behaviour rating system. Psychometric analysis of this tool to date has focused on inter-rater reliability and feasibility rather than validation.

Methods

NOTSS assessments were collected from two groups of consultant/attending surgeons in the UK and USA, who rated behaviours of the lead surgeon during a video-based simulated crisis scenario after either online or classroom instruction. The process of validation consisted of assessing construct validity, scale reliability and concurrent criterion validity, and undertaking a sensitivity analysis. Central to this was confirmatory factor analysis to evaluate the structure of the NOTSS taxonomy.

Results

Some 255 consultant surgeons participated in the study. The four-category NOTSS model was found to have robust construct validity evidence, and a superior fit compared with alternative models. Logistic regression and sensitivity analysis revealed that, after adjusting for technical skills, for every 1-point increase in NOTSS score of the lead surgeon, the odds of having a higher versus lower patient safety score was 2·29 times. The same pattern of results was obtained for a broad mix of surgical specialties (UK) as well as a single discipline (cardiothoracic, USA).

Conclusion

The NOTSS tool can be applied in research and education settings to measure non-technical skills in a valid and efficient manner.

Establishing the aims, format and function for multidisciplinary team-driven care within an inflammatory bowel disease service: a multicentre qualitative specialist-based consensus study

OBJECTIVE

To obtain a specialist-based consensus on the aims, format and function for MDT-driven care within an inflammatory bowel disease (IBD) service.

DESIGN

This was a prospective, multicentre study using a Delphi formal consensus-building methodology.

SETTING

Participants were recruited nationally across 13 centres from July to August 2014.

PARTICIPANTS

24 participants were included into the Delphi Specialist Consensus Panel. They included six consultant colorectal surgeons, six gastroenterologists, five consultant radiologists, three consultant histopathologists and 4 IBD nurse specialists.

INTERVENTIONS

Panellists ranked items on a Likert scale (1=not important to 5=very important). Items with a median score >3 were considered eligible for inclusion.

MAIN OUTCOME MEASURES

Consensus was defined with an IQR ≤1. Consensus on categorical responses was defined by an agreement of >60%.

RESULTS

A consensus on items (median; IQR) that described the aims of the MDT-driven care that were considered very important included: advance patient care (5;5-5), provide multidisciplinary input for the patient's care plan (5;5-5), provide shared experience and expertise (5;5-5), improve patient outcome (5;5-5), deliver the best possible care for the patient (5;5-5) and to obtain consensus on management for a patient with IBD (5;4-5). A consensus for being a core MDT member was demonstrated for colorectal surgeons (24/24), radiologists (24/24), gastroenterologists (24/24), nurse specialists (24/24), dieticians (14/23), histopathologists (21/23) and coordinators (21/24).

CONCLUSIONS

This study has provided a consensus for proposed aims, overall design, format and function MDT-driven care within an IBD service. This can provide a focus for core members, and aid a contractual recognition to ensure attendance and proactive contribution.

Predictors of Treatment Decisions in Multidisciplinary Oncology Meetings: A Quantitative Observational Study

BACKGROUND

In many healthcare systems, treatment recommendations for cancer patients are formulated by multidisciplinary tumor boards (MTBs). Evidence suggests that interdisciplinary contributions to case reviews in the meetings are unequal and information-sharing suboptimal, with biomedical information dominating over information on patient comorbidities and psychosocial factors. This study aimed to evaluate how different elements of the decision process affect the teams' ability to reach a decision on first case review.

METHODS

This was an observational quantitative assessment of 1045 case reviews from 2010 to 2014 in cancer MTBs using a validated tool, the Metric for the Observation of Decision-making. This tool allows evaluation of the quality of information presentation (case history, radiological, pathological, and psychosocial information, comorbidities, and patient views), and contribution to discussion by individual core specialties (surgeons, oncologists, radiologists, pathologists, and specialist cancer nurses). The teams' ability to reach a decision was a dichotomous outcome variable (yes/no).

RESULTS

Using multiple logistic regression analysis, the significant positive predictors of the teams' ability to reach a decision were patient psychosocial information (odds ratio [OR] 1.35) and the inputs of surgeons (OR 1.62), radiologists (OR 1.48), pathologists (OR 1.23), and oncologists (OR 1.13). The significant negative predictors were patient comorbidity information (OR 0.83) and nursing inputs (OR 0.87).

CONCLUSIONS

Multidisciplinary inputs into case reviews and patient psychosocial information stimulate decision making, thereby reinforcing the role of MTBs in cancer care in processing such information. Information on patients' comorbidities, as well as nursing inputs, make decision making harder, possibly indicating that a case is complex and requires more detailed review. Research should further define case complexity and determine ways to better integrate patient psychosocial information into decision making.

Evaluating structured assessment of anaesthesiologists' non-technical skills

BACKGROUND

Non-technical skills (NTS) are essential for safe and efficient anaesthesia. Assessment instruments with appropriate validity evidence can be used to ensure that anaesthesiologists possess the NTS necessary to deliver high-standard patient care. The aims were to collect validity evidence using a contemporary validity framework for the assessment instrument Anaesthesiologists' Non-Technical Skills in Denmark (ANTSdk) regarding response process and internal structure (including reliability), and to investigate the effect of rater training on these properties.

METHODS

An explorative study was undertaken at the Danish Institute for Medical Simulation, Copenhagen, Denmark. In a 1-day session, using ANTSdk, a convenience sample of 19 anaesthesiologists rated trainee anaesthesiologists' NTS in nine video-recorded simulation scenarios before and after a 3-h training session.

RESULTS

Response process evidence: participants considered ANTSdk useful and feasible for NTS assessment. Internal structure evidence: inter-rater reliability (single measures) largely expressed substantial agreement (ICC ≥ 0.55 and ICC ≥ 0.60 for pre- and post-training ratings respectively). Strong internal consistency of ratings was found (Spearman's correlation coefficient ≥ 0.82). Accuracy of participants' ratings compared with reference ratings (± 1 scale point) was notable (76% and 78% for pre- and post-training ratings, respectively). The results indicate that the elements 'Demonstrating self-awareness', 'Reassessing decisions', 'Assessing competencies', and 'Supporting others' need more attention in future rater training.

CONCLUSION

The validity evidence collected on content, response process, and internal structure, suggests that ANTSdk is easy to use on video-recorded simulation scenarios, indicating that ANTSdk is a feasible instrument for NTS assessment during anaesthesia training.

The anatomy of clinical decision-making in multidisciplinary cancer meetings: A cross-sectional observational study of teams in a natural context

In the UK, treatment recommendations for patients with cancer are routinely made by multidisciplinary teams in weekly meetings. However, their performance is variable.The aim of this study was to explore the underlying structure of multidisciplinary decision-making process, and examine how it relates to team ability to reach a decision.This is a cross-sectional observational study consisting of 1045 patient reviews across 4 multidisciplinary cancer teams from teaching and community hospitals in London, UK, from 2010 to 2014. Meetings were chaired by surgeons.We used a validated observational instrument (Metric for the Observation of Decision-making in Cancer Multidisciplinary Meetings) consisting of 13 items to assess the decision-making process of each patient discussion. Rated on a 5-point scale, the items measured quality of presented patient information, and contributions to review by individual disciplines. A dichotomous outcome (yes/no) measured team ability to reach a decision. Ratings were submitted to Exploratory Factor Analysis and regression analysis.The exploratory factor analysis produced 4 factors, labeled "Holistic and Clinical inputs" (patient views, psychosocial aspects, patient history, comorbidities, oncologists', nurses', and surgeons' inputs), "Radiology" (radiology results, radiologists' inputs), "Pathology" (pathology results, pathologists' inputs), and "Meeting Management" (meeting chairs' and coordinators' inputs). A negative cross-loading was observed from surgeons' input on the fourth factor with a follow-up analysis showing negative correlation (r = -0.19, P < 0.001). In logistic regression, all 4 factors predicted team ability to reach a decision (P < 0.001).Hawthorne effect is the main limitation of the study.The decision-making process in cancer meetings is driven by 4 underlying factors representing the complete patient profile and contributions to case review by all core disciplines. Evidence of dual-task interference was observed in relation to the meeting chairs' input and their corresponding surgical input into case reviews.

The Impact of Operating Room Distractions on Stress, Workload, and Teamwork

OBJECTIVE

To investigate whether distractions in the operating room (OR) are associated with higher mental workload and stress, and poorer teamwork among OR personnel.

BACKGROUND

Engaging in multiple tasks can affect performance. There is little research on the effect of distractions on surgical team members' behavior and cognitive processes.

METHODS

Ninety general surgery cases were observed in real time. Cases were assessed by a surgeon and a behavioral scientist using 4 validated tools: OR Distractions Assessment Form, the Observational Teamwork Assessment for Surgery tool, NASA-Task Load Index, and short form of the State Trait Anxiety Inventory. Analysis of variance was performed to evaluate significant differences between teamwork, workload, and stress level among team members. Correlations (Pearson r) were computed to evaluate associations between variables.

RESULTS

The most prevalent distractions were those initiated by external staff, followed by case-irrelevant conversations. Case-irrelevant conversations were associated with poorer team performance. Irrelevant conversations initiated by surgeons were associated with lower teamwork in surgeons (across team skills: r = -0.44 to -0.58, P < 0.05 to 0.01) and anesthesiologists (r = -0.38 and r = -0.40, for coordination and leadership; P < 0.05). Equipment-related distractions correlated with higher stress (r = 0.48, P < 0.05) and lower teamwork (across team skills: r = -0.42 to -0.50, P < 0.05) in nurses. Acoustic distractions correlated with higher stress in surgeons (r = 0.32, P < 0.05) and higher workload in anesthesiologists (r = 0.30, P < 0.05).

CONCLUSIONS

Although some distractions may be inevitable in the OR, they can also be detrimental to the team. A deeper understanding of the effect of distractions on teams and their outcomes can lead to targeted quality improvement.

Development and testing of the cancer multidisciplinary team meeting observational tool (MDT-MOT)

OBJECTIVE

To develop a tool for independent observational assessment of cancer multidisciplinary team meetings (MDMs), and test criterion validity, inter-rater reliability/agreement and describe performance.

DESIGN

Clinicians and experts in teamwork used a mixed-methods approach to develop and refine the tool. Study 1 observers rated pre-determined optimal/sub-optimal MDM film excerpts and Study 2 observers independently rated video-recordings of 10 MDMs.

SETTING

Study 2 included 10 cancer MDMs in England.

PARTICIPANTS

Testing was undertaken by 13 health service staff and a clinical and non-clinical observer.

INTERVENTION

None.

MAIN OUTCOME MEASURES

Tool development, validity, reliability/agreement and variability in MDT performance.

RESULTS

Study 1: Observers were able to discriminate between optimal and sub-optimal MDM performance (P ≤ 0.05). Study 2: Inter-rater reliability was good for 3/10 domains. Percentage of absolute agreement was high (≥80%) for 4/10 domains and percentage agreement within 1 point was high for 9/10 domains. Four MDTs performed well (scored 3+ in at least 8/10 domains), 5 MDTs performed well in 6-7 domains and 1 MDT performed well in only 4 domains. Leadership and chairing of the meeting, the organization and administration of the meeting, and clinical decision-making processes all varied significantly between MDMs (P ≤ 0.01).

CONCLUSIONS

MDT-MOT demonstrated good criterion validity. Agreement between clinical and non-clinical observers (within one point on the scale) was high but this was inconsistent with reliability coefficients and warrants further investigation. If further validated MDT-MOT might provide a useful mechanism for the routine assessment of MDMs by the local workforce to drive improvements in MDT performance.

Non-technical skills in minimally invasive surgery teams: a systematic review

BACKGROUND

Root cause analyses show that up to 70 % of adverse events are caused by human error. Strong non-technical skills (NTS) can prevent or reduce these errors, considerable numbers of which occur in the operating theatre. Minimally invasive surgery (MIS) requires manipulation of more complex equipment than open procedures, likely requiring a different set of NTS for each kind of team. The aims of this study were to identify the MIS teams' key NTS and investigate the effect of training and assessment of NTS on MIS teams.

METHODS

The databases of PubMed, Cochrane Library, Embase, PsycINFO, and Scopus were systematically searched according to Preferred Reporting Item for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Articles containing outcome measures related to MIS teams' key NTS, training, or assessment of NTS were included.

RESULTS

The search yielded 1984 articles, 11 of which were included. All were observational studies without blinding, and they differed in aims, types of evaluation, and outcomes. Only two studies evaluated patient outcomes other than operative time, and overall, the studies' quality of evidence was low. Different communication types were encountered in MIS compared to open surgery, mainly due to equipment- and patient-related challenges. Fixed teams improved teamwork and safety levels, while deficient planning and poor teamwork were found to obstruct workflow and increase errors. Training NTS mitigates these issues and improves staff attitudes towards NTS.

CONCLUSIONS

MIS teams' NTS are important for workflow and prevention of errors and can be enhanced by working in fixed teams. In the technological complex sphere of MIS, communication revolves around equipment- and patient-related topics, much more so than in open surgery. In all, only a few heterogeneous-design studies have examined this. In the future, the focus should shift to systematically identifying key NTS and developing effective, evidence-based team training programmes in MIS.

Assessing the Nontechnical Skills of Surgical Trainees: Views of the Theater Team

OBJECTIVE

This study aims to explore the views of members of theater teams regarding the proposed introduction of a workplace-based assessment of nontechnical skills of surgeons (NOTSS) into the Intercollegiate Surgical Curriculum Programme in the United Kingdom. In addition, the previous training and familiarity of the members of the surgical theater team with the concept and assessment of NOTSS would be evaluated.

DESIGN

A regional survey of members of theater teams (consultant surgeons, anesthetists, scrub nurses, and trainees) was performed at 1 teaching and 2 district general hospitals in South Yorkshire.

RESULTS

There were 160 respondents corresponding to a response rate of 81%. The majority (77%) were not aware of the NOTSS assessment tool with only 9% of respondents reporting to have previously used the NOTSS tool and just 3% having received training in NOTSS assessment. Overall, 81% stated that assessing NOTSS was as important as assessing technical skills. Trainees attributed less importance to nontechnical skills than the other groups (p ≤ 0.016). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee's leadership skills and decision-making capabilities, overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater. However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p = 0.001).

CONCLUSIONS

Our survey demonstrates acceptability among the theater team for the introduction of the NOTSS tool into the surgical curriculum. However, lack of familiarity highlights the importance of faculty training for assessors before such an introduction.

International recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process

BACKGROUND

Patient safety incident reporting systems (PSRS) have been established for over a decade, but uncertainty remains regarding the role that they can and ought to play in quantifying healthcare-related harm and improving care.

OBJECTIVE

To establish international, expert consensus on the purpose of PSRS regarding monitoring and learning from incidents and developing recommendations for their future role.

METHODS

After a scoping review of the literature, semi-structured interviews with experts in PSRS were conducted. Based on these findings, a survey-based questionnaire was developed and subsequently completed by a larger expert panel. Using a Delphi approach, consensus was reached regarding the ideal role of PSRSs. Recommendations for best practice were devised.

RESULTS

Forty recommendations emerged from the Delphi procedure on the role and use of PSRS. Experts agreed reporting system should not be used as an epidemiological tool to monitor the rate of harm over time or to appraise the relative safety of hospitals. They agreed reporting is a valuable mechanism for identifying organisational safety needs. The benefit of a national system was clear with respect to medication error, device failures, hospital-acquired infections and never events as these problems often require solutions at a national level. Experts recommended training for senior healthcare professionals in incident investigation. Consensus recommendation was for hospitals to take responsibility for creating safety solutions locally that could be shared nationally.

CONCLUSIONS

We obtained reasonable consensus among experts on aims and specifications of PSRS. This information can be used to reflect on existing and future PSRS, and their role within the wider patient safety landscape. The role of PSRS as instruments for learning needs to be elaborated and developed further internationally.

Exploring the relationship between anaesthesiologists' non-technical and technical skills

BACKGROUND

A combination of non-technical skills (NTS) and technical skills (TS) is crucial for anaesthetic patient management. However, a deeper understanding of the relationship between these two skills remains to be explored. We investigated the characteristics of trainee anaesthesiologists' NTS and TS in a simulated unexpected difficult airway management scenario.

METHODS

A mixed-method approach was used to explore the relationship between NTS and TS in 25 videos of 2nd year trainee anaesthesiologists managing a simulated difficult airway scenario. The videos were assessed using the customised version of the Anaesthetists' Non-Technical Skills System, ANTSdk, and an adapted TS checklist for calculating the correlation between NTS and TS. Written descriptions of the observed NTS were analysed using directed content analysis.

RESULTS

The correlation between the NTS and the TS ratings was 0.106 (two-tailed significance of 0.613). Inter-rater reliability was substantial. Themes characterising good NTS included a systematic approach, planning and communicating decisions as well as responding to the evolving situation. A list of desirable, concrete NTS for the specific airway management situation was generated.

CONCLUSION

This study illustrates that anaesthesiologist trainees' NTS and TS were not correlated in this setting, but rather intertwined and how the interplay of NTS and TS can impact patient management. Themes describing the characteristics of NTS and a list of desirable, concrete NTS were developed to aid the understanding, training and use of NTS.

Coaching Non-technical Skills Improves Surgical Residents' Performance in a Simulated Operating Room

OBJECTIVE

To investigate the effect of coaching on non-technical skills and performance during laparoscopic cholecystectomy in a simulated operating room (OR).

BACKGROUND

Non-technical skills (situation awareness, decision making, teamwork, and leadership) underpin technical ability and are critical to the success of operations and the safety of patients in the OR. The rate of developing assessment tools in this area has outpaced development of workable interventions to improve non-technical skills in surgical training and beyond.

METHOD

A randomized trial was conducted with senior surgical residents (n = 16). Participants were randomized to receive either non-technical skills coaching (intervention) or to self-reflect (control) after each of 5 simulated operations. Coaching was based on the Non-Technical Skills For Surgeons (NOTSS) behavior observation system. Surgeon-coaches trained in this method coached participants in the intervention group for 10 minutes after each simulation. Primary outcome measure was non-technical skills, assessed from video by a surgeon using the NOTSS system. Secondary outcomes were time to call for help during bleeding, operative time, and path length of laparoscopic instruments.

RESULTS

Non-technical skills improved in the intervention group from scenario 1 to scenario 5 compared with those in the control group (p = 0.04). The intervention group was faster to call for help when faced with unstoppable bleeding in the final scenario (no. 5; p = 0.03).

CONCLUSIONS

Coaching improved residents' non-technical skills in the simulated OR compared with those in the control group. Important next steps are to implement non-technical skills coaching in the real OR and assess effect on clinically important process measures and patient outcomes.

Teamwork Assessment Tools in Modern Surgical Practice: A Systematic Review

Introduction. Deficiencies in teamwork skills have been shown to contribute to the occurrence of adverse events during surgery. Consequently, several teamwork assessment tools have been developed to evaluate trainee nontechnical performance. This paper aims to provide an overview of these instruments and review the validity of each tool. Furthermore, the present paper aims to review the deficiencies surrounding training and propose several recommendations to address these issues. Methods. A systematic literature search was conducted to identify teamwork assessment tools using MEDLINE (1946 to August 2015), EMBASE (1974 to August 2015), and PsycINFO (1806 to August 2015) databases. Results. Eight assessment tools which encompass aspects of teamwork were identified. The Nontechnical Skills for Surgeons (NOTSS) assessment was found to possess the highest level of validity from a variety of sources; reliability and acceptability have also been established for this tool. Conclusions. Deficits in current surgical training pathways have prompted several recommendations to meet the evolving requirements of surgeons. Recommendations from the current paper include integration of teamwork training and assessment into medical school curricula, standardised formal training of assessors to ensure accurate evaluation of nontechnical skill acquisition, and integration of concurrent technical and nontechnical skills training throughout training.

The Development of Assessment Tools for Plastic Surgery Competencies

BACKGROUND

Objective tools to assess procedural skills in plastic surgery residency training are currently lacking. There is an increasing need to address this deficit in order to meet today's training standards in North America.

OBJECTIVES

The purpose of this pilot study was to establish a methodology for determining the essential procedural steps for two plastic surgery procedures to assist resident training and assessment.

METHODS

Following a literature review and needs assessment of resident training, the authors purposefully selected two procedures lacking robust assessment metrics (breast augmentation and facelift) and used a consensus process to complete a list of procedural steps for each. Using an online survey, plastic surgery Program Directors, Division Chiefs, and the Royal College Specialty Training Committee members in Canada were asked to indicate whether each step was considered essential or non-essential when assessing competence among graduating plastic surgery trainees. The Delphi methodology was used to obtain consensus among the panel. Panelist reliability was measured using Cronbach's alpha.

RESULTS

A total of 17 steps for breast augmentation and 24 steps for facelift were deemed essential by consensus (Cronbach's alpha 0.87 and 0.85, respectively).

CONCLUSION

Using the aforementioned technique, the essential procedural steps for two plastic surgery procedures were determined. Further work is required to develop assessment instruments based on these steps and to gather validity evidence in support of their use in surgical education.