Content Validation and Evaluation of an Endovascular Teamwork Assessment Tool

Teamwork and Surgical Team-Based Training

Effective teamwork, both in and out of the operating room, is an essential component of safe and efficient surgical performance. There are multiple available assessment tools for evaluating teamwork and important contributors to teamwork such as safety culture and nontechnical skills. Multiple types of interventions exist to improve and train providers on teamwork, and many have been demonstrated to improve not only teamwork but also patient outcomes. Teamwork strategies can be adapted to different contexts, based on provider needs and resources.

Radiation Safety Performance is More than Simply Measuring Doses! Development of a Radiation Safety Rating Scale

PURPOSE

Radiation safety performance is often evaluated using dose parameters measured by personal dosimeters and/or the C-arm, which provide limited information about teams' actual radiation safety behaviors. This study aimed to develop a rating scale to evaluate team radiation safety behaviors more accurately and investigate its reliability.

MATERIALS AND METHODS

A modified Delphi consensus was organized involving European vascular surgeons (VS), interventional radiologists, and interventional cardiologists. Initial items and anchors were drafted a priori and rated using five-point Likert scales. Participants could suggest additional items or adjustments. Consensus was defined as ≥ 80% agreement (rating ≥ 4) with Cronbach's alpha ≥ .80. Two VS with expertise in radiation safety evaluated 15 video-recorded endovascular repairs of infrarenal aortic aneurysms (EVAR) to assess usability, inter and intrarater reliability.

RESULTS

Thirty-one of 46 invited specialists completed three rating rounds to generate the final rating scale. Five items underwent major adjustments. In the final round, consensus was achieved for all items (alpha = .804; agreement > 87%): 'Pre-procedural planning', 'Preparation in angiosuite/operating room', 'Shielding equipment', 'Personal protective equipment', 'Position of operator/team', 'Radiation usage awareness', 'C-arm handling', 'Adjusting image quality', 'Additional dose reducing functions', 'Communication/leadership', and 'Overall radiation performance and ALARA principle'. All EVARs were rated, yielding excellent Cronbach's alpha (.877) with acceptable interrater and excellent intrarater reliability (ICC = .782; ICC = .963, respectively).

CONCLUSION

A reliable framework was developed to assess radiation safety behaviors in endovascular practice and provide teams with formative feedback. The final scale is provided in this publication.

Development of the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES) for non-technical skills in cataract surgery

BACKGROUND

Nontechnical skills (NTS) are fundamental for successfully managing intraoperative complications. We aimed to develop the HUman Factors in intraoperative Ophthalmic Emergencies Scoring System (HUFOES); an NTS assessment system for posterior capsule rupture (PCR) during cataract surgery.

METHODS

A literature review and a focus group consisting of three cataract surgeons and one NTS researcher elicited the important NTS for the management of intraoperative cataract surgery complications. A novel taxonomy of NTS specific for PCR management was generated. Questionnaires were distributed to ophthalmologists in one UK training region. Delphi methodology was used to develop a final HUFOES draft. One further questionnaire was used to gain feasibility, educational impact and validity data.

RESULTS

All HUFOES components achieved a mean importance rating of >8/10 and achieved high interrater agreement ratings (α = 0.953). Interrater agreement scores for HUFOES categories were: teamwork and communication (α = 0.819), leadership (α = 0.859), decision making (α = 0.753), situational awareness (α = 0.840) and professionalism (α = 0.890). In all, 92.8% (n = 13) rated HUFOES as specific for use, 85.7% (n = 12) agreed it contains appropriate assessment measures, 92.8% (n = 13) agreed that training with HUFOES would enhance preparation for PCR management and 78.6% (n = 11) declared HUFOES as the preferable training system for NTS in intraoperative ophthalmic emergencies when compared with the current gold standard.

CONCLUSIONS

HUFOES has been developed and validated as a tool for the training and assessment of NTS in PCR. An NTS training programme integrated with HUFOES should be considered in order to enhance surgical NTS for managing intraoperative complications, and improve performance and outcomes following PCR.

Psychometrics properties of the Team Interaction Scale and influencing factors of team interaction of tertiary hospital physicians in China: a cross-sectional study

OBJECTIVES

To administer a cross-cultural adaptation of the Team Interaction Scale (TIS), test its psychometric properties and investigate influencing factors of team interactions in a physician population in Chinese tertiary hospitals.

DESIGN

Cross-sectional survey.

SETTINGS

Two rounds of surveys, a pilot and a large sampling survey, were conducted in two and nine tertiary hospitals, respectively, in Liaoning Province, China.

PARTICIPANTS

In the pilot survey, 363 of 390 physicians sampled were included in the analysis, resulting in an effective response rate of 93.08%. In the large sampling survey, the effective response rate was 89.10% (3653 of 4100 physicians).

OUTCOME MEASURES

The TIS and a short version of a burn-out scale were administrated to assess the physician's team interaction and burn-out. Psychometric properties of TIS were tested by confirmatory factor analysis (CFA), exploratory factor analysis (EFA) and internal consistency analysis. Gender, age, discipline, education level, professional title, hospital scale and burn-out were explored as influencing factors with independent sample t-tests, one-way analyses of variance and a correlation analysis.

RESULTS

Based on CFA, a 17-item modified scale was developed following the pilot survey. In the large sampling survey, EFA was conducted with half of the samples, producing six dimensions: 'Communication', 'Coordination', 'Mutual help', 'Team goals', 'Work norms' and 'Cohesion and conflict resolution'. Fit of the modified model was confirmed by CFA with the other half of the samples (root mean square error of approximation=0.067, Comparative Fit Index=0.98, Normed Fit Index=0.97, Goodness of Fit Index=0.94, Adjusted Goodness of Fit Index=0.92). A high Cronbach's α coefficient of 0.98 demonstrated reliability of the modified scale. The Team Interaction Score was significantly lower in younger physicians, in men, in paediatricians and in physicians from larger-scale tertiary hospitals. Team Interaction Scores were negatively associated with burn-out.

CONCLUSIONS

The adapted TIS, containing 17 items and six dimensions, was reliable and valid for Chinese tertiary hospital physicians. To address physician burn-out, team interaction should be highlighted.

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Physician-patient interactions and communication with conscious patients during simulated cath lab procedures: an exploratory study

Background

This exploratory study investigates the feasibility for observing and evaluating intraoperative communication practices using simulation techniques. Complex procedures are increasingly performed on patients under local anaesthesia, where patients are fully conscious. Interventional cardiac procedures are one such example where patients have reported high levels of anxiety undergoing procedures. Although communication styles can serve to alleviate patient anxiety during interventions, leading to a better patient experience, there has been little observational research on communication, while patient perspectives in intraoperative contexts have been underexplored.

Methods

In this mixed-methods study, observational analysis was conducted on 20 video-recorded simulated scenarios, featuring physician operators (of varied experience levels), communication and interactions with a simulated patient (trained actor), in a controlled and highly realistic catheter laboratory setting. Two independent raters and the simulated patient embedded in scenarios retrospectively rated physician communication styles and interactions with the patient via four key parameters. Patient perspectives of communication were further explored via a quantitative measure of anxiety and semistructured qualitative interviews.

Results

While independent ratings of physician-patient communications demonstrated few discernible differences according to physicians' experience level, patient ratings were consistently higher for experienced physicians and lower for novice physicians for the four interaction styles. Furthermore, the patient's anxiety scores were differentiable according to operators' experience level. Thematic analysis provided further insights into how patient perspectives, including affective dimensions are characterised, and how physician interactions can amplify or attenuate feelings of anxiety through tone of voice, continuity in communication during the procedure, communicating while multitasking and connecting with the patient.

Conclusions

Our findings indicate underlying patient assumptions about physicians' experience levels, intraoperative communication styles and impact on anxiety. While observational methods can be applied to simulated intraoperative clinical contexts, evaluation techniques such as observational rating tools need to incorporate patient perspectives about undergoing conscious surgery.

The Impact of System Factors on Quality and Safety in Arterial Surgery: A Systematic Review

OBJECTIVE

A systems approach to patient safety proposes that a wide range of factors contribute to surgical outcome, yet the impact of team, work environment, and organisational factors, is not fully understood in arterial surgery. The aim of this systematic review is to summarize and discuss what is already known about the impact of system factors on quality and safety in arterial surgery.

DATA SOURCES

A systematic review of original research papers in English using MEDLINE, Embase, PsycINFO, and Cochrane databases, was performed according to PRISMA guidelines.

REVIEW METHODS

Independent reviewers selected papers according to strict inclusion and exclusion criteria, and using predefined data fields, extracted relevant data on team, work environment, and organisational factors, and measures of quality and/or safety, in arterial procedures.

RESULTS

Twelve papers met the selection criteria. Study endpoints were not consistent between papers, and most failed to report their clinical significance. A variety of tools were used to measure team skills in five papers; only one paper measured the relationship between team factors and patient outcomes. Two papers reported that equipment failures were common and had a significant impact on operating room efficiency. The influence of hospital characteristics on failure-to-rescue rates was tested in one large study, although their conclusions were limited to the American Medicare population. Five papers implemented changes in the patient pathway, but most studies failed to account for potential confounding variables.

CONCLUSIONS

A small number of heterogenous studies have evaluated the relationship between system factors and quality or safety in arterial surgery. There is some evidence of an association between system factors and patient outcomes, but there is more work to be done to fully understand this relationship. Future research would benefit from consistency in definitions, the use of validated assessment tools, measurement of clinically relevant endpoints, and adherence to national reporting guidelines.