A behaviourally anchored rating scale for evaluating the use of the WHO surgical safety checklist: development and initial evaluation of the WHOBARS

Adapting the World Health Organization's Surgical Safety Checklist to High-Income Settings: A Hybrid Effectiveness-Implementation Trial Protocol

Objectives

The proposed study aims to assess users' perceptions of a surgical safety checklist (SSC) reimplementation toolkit and its impact on SSC attitudes and operating room (OR) culture, meaningful checklist use, measures of surgical safety, and OR efficiency at 3 different hospital sites.

Background

The High-Performance Checklist toolkit (toolkit) assists surgical teams in modifying and implementing or reimplementing the World Health Organization's SSC. Through the explore, prepare, implement, and sustain implementation framework, the toolkit provides a process and set of tools to facilitate surgical teams' modification, implementation, training on, and evaluation of the SSC.

Methods

A pre-post intervention design will be used to assess the impact of the modified SSC on surgical processes, team culture, patient experience, and safety. This mixed-methods study includes quantitative and qualitative data derived from surveys, semi-structured interviews, patient focus groups, and SSC performance observations. Additionally, patient outcome and OR efficiency data will be collected from the study sites' health surveillance systems.

Data analysis

Statistical data will be analyzed using Statistical Product and Service Solutions, while qualitative data will be analyzed thematically using NVivo. Furthermore, interview data will be analyzed using the Consolidated Framework for Implementation Research and reach, effectiveness, adoption, implementation, maintenance implementation frameworks.

Setting

The toolkit will be introduced at 3 diverse surgical sites in Alberta, Canada: an urban hospital, university hospital, and small regional hospital.

Anticipated impact

We anticipate the results of this study will optimize SSC usage at the participating surgical sites, help shape and refine the toolkit, and improve its usability and application at future sites.

Teamwork matters: team situation awareness to build high-performing healthcare teams, a narrative review

Healthcare today is the prerogative of teams rather than of individuals. In acute care domains such as anaesthesia, intensive care, and emergency medicine, the work is complex and fast-paced, and the team members are diverse and interdependent. Three decades of research into the behaviours of high-performing teams provides us with clear guidance on team training, demonstrating positive effects on patient safety and staff wellbeing. Here we consider team performance through the lens of situation awareness. Maintaining situation awareness is an absolute requirement for safe and effective patient management. Situation awareness is a dynamic process of perceiving cues in the environment, understanding what they mean, and predicting how the situation may evolve. In the context of acute clinical care, situation awareness can be improved if the whole team actively contributes to monitoring the environment, processing information, and planning next steps. In this narrative review, we explore the concept of situation awareness at the level of the team, the conditions required to maintain team situation awareness, and the relationship between team situation awareness, shared mental models, and team performance. Our ultimate goal is to help clinicians create the conditions required for high-functioning teams, and ultimately improve the safety of clinical care.

CheckPOINT: a simple tool to measure Surgical Safety Checklist implementation fidelity

INTRODUCTION

The WHO Surgical Safety Checklist (SSC) is a communication tool that improves teamwork and patient outcomes. SSC effectiveness is dependent on implementation fidelity. Administrative audits fail to capture most aspects of SSC implementation fidelity (ie, team communication and engagement). Existing research tools assess behaviours during checklist performance, but were not designed for routine quality assurance and improvement. We aimed to create a simple tool to assess SSC implementation fidelity, and to test its reliability using video simulations, and usability in clinical practice.

METHODS

The Checklist Performance Observation for Improvement (CheckPOINT) tool underwent two rounds of face validity testing with surgical safety experts, clinicians and quality improvement specialists. Four categories were developed: checklist adherence, communication effectiveness, attitude and engagement. We created a 90 min training programme, and four trained raters independently scored 37 video simulations using the tool. We calculated intraclass correlation coefficients (ICC) to assess inter-rater reliability (ICC>0.75 indicating excellent reliability). We then trained two observers, who tested the tool in the operating room. We interviewed the observers to determine tool usability.

RESULTS

The CheckPOINT tool had excellent inter-rater reliability across SSC phases. The ICC was 0.83 (95% CI 0.67 to 0.98) for the sign-in, 0.77 (95% CI 0.63 to 0.92) for the time-out and 0.79 (95% CI 0.59 to 0.99) for the sign-out. During field testing, observers reported CheckPOINT was easy to use. In 98 operating room observations, the total median (IQR) score was 25 (23-28), checklist adherence was 7 (6-7), communication effectiveness was 6 (6-7), attitude was 6 (6-7) and engagement was 6 (5-7).

CONCLUSIONS

CheckPOINT is a simple and reliable tool to assess SSC implementation fidelity and identify areas of focus for improvement efforts. Although CheckPOINT would benefit from further testing, it offers a low-resource alternative to existing research tools and captures elements of adherence and team behaviours.

Implementation of World Health Organization behaviorally anchored rating scale and checklist utilization: promising results for LMICs

BACKGROUND

Operating teams can decrease the likelihood of patient risk by using the WHO Surgical Safety Checklist. To ascertain the impact of demographic factors on behaviorally anchored ratings and investigate operating room (OR) staff attitudes toward checklist administration, we set out to better understand how OR personnel use the checklist in a tertiary care hospital in Pakistan.

MATERIALS AND METHODS

A monocentric sequential mixed-methods study employing a quantitative approach of using World Health Organization Behaviorally Anchored Rating Scale (WHOBARS) assessments of surgical cases by OR personnel and two independent observers, who were certified surgeons having extensive experience in the rating of the WHOBARS scale for more than 1 year, followed by a qualitative approach of staff interviews were carried out in a tertiary care setting. In June and July 2022, over the period of 8 weeks, an intervention (training delivery) was implemented and evaluated. The information, skills, and behavior adjustments required to apply the checklist were taught in the course using lectures, videos, small group breakouts, participant feedback, and simulations.

RESULTS

After the introduction of WHOBARS, 50.81% of respondents reported always using the checklist, with another 30.81% using it in part. Participants' years in practice, hospital size, or surgical volume did not predict checklist use. Checklist use was associated with always counting instruments (51.08%), patient identity (67.83%), difficult intubation risk (39.72%), the risk of blood loss (51.08%), prophylactic administration of an antibiotic (52.43%), and the use of pulse oximeter (46.75%). Interviewees felt that the checklist could promote teamwork and a safe culture, particularly enabling speaking up. Senior staff were of key importance in setting the appropriate tone.

CONCLUSION

The use of a multi-disciplinary course for checklist implementation resulted in 50.81% of participants always using the checklist and an increase in counting surgical instruments. Successful checklist implementation was not predicted by the participant's length of medical service, hospital size, or surgical volume. If reproducible in other countries, widespread implementation in LMICs becomes a realistic possibility.

A Quality Improvement Initiative Using Peer Audit and Feedback to Improve Compliance with the Surgical Safety Checklist

Background

The Surgical Safety Checklist (SSC) published by the WHO in 2009 is used as standard in surgery worldwide to reduce perioperative patient mortality. However, compliance with the SSC and quality of its application are often not satisfactory. Internal audits and feedbacks seem promising for improving SSC application.

Objective

The purpose of this study is to investigate whether an intervention consisting of peer observation and immediate peer feedback can be implemented with high fidelity and acceptance.

Method

Data were obtained from a national pilot programme that was initiated in Switzerland in 2018 to measure and improve compliance with the SSC using peer audit and feedback. A total of 11 hospitals with 14 sites implemented the full intervention. Each hospital formed an interprofessional project team that should perform at least 30 observations with feedback on SSC application documented in an observation tool developed specifically for this programme. Since the SSCs of the study hospitals differ greatly regarding checklist items, for each of the three SSC sections standard items were defined: four at Sign In, five at Team Time Out and two at Sign Out. Frequency analyses were performed for initiation characteristics, SSC application at item level, feedback characteristics and programme evaluation.

Results

The 11 hospitals documented 715 valid observations, and feedback on SSC application was provided for 79% of the observations. In 61%, all team members stopped their work for the SSC application, and in 71%, the items were read off from the checklist (instead of recalled from memory). On average, 86% of the standard items were read out by the checklist coordinator, whilst the two items at Sign Out were read out only in 60% and 74%. Additional visual checks with another source (e.g. patient wristband) took place in only 41%, and verbal confirmation of the items (by someone else other than the checklist coordinator) was obtained on an average of 76% across all three checklist sections. The surgical teams reacted positively in 64% to the peer feedback.

Conclusion

Both implementation fidelity and acceptability of the intervention were high—the present intervention seems suitable for regular monitoring of the quality of SSC application with internal resources. Peer observation facilitated identifying weaknesses regarding the SSC process and application at item level. Across all hospitals, the Sign Out section in general, visual control for item checks and lack of work interruption of all team members during SSC application showed up as the main areas of improvement.

The Impact of Behavioral Anchors in the Assessment of Fellowship Applicants: Reducing Rater Biases

INTRODUCTION

No standardized evaluation tool for fellowship applicant assessment exists. Assessment tools are subject to biases and scoring tendencies which can skew scores and impact rankings. We aimed to develop and evaluate an objective assessment tool for fellowship applicants.

METHODS

We detected rater effects in our numerically scaled assessment tool (NST), which consisted of 10 domains rated from 0 to 9. We evaluated each domain, consolidated redundant categories, and removed subjective categories. For 7 remaining domains, we described each quality and developed a question with a behaviorally-anchored rating scale (BARS). Applicants were rated by 6 attendings. Ratings from the NST in 2018 were compared with the BARS from 2020 for distribution of data, skewness, and inter-rater reliability.

RESULTS

Thirty-four applicants were evaluated with the NST and 38 with the BARS. Demographics were similar between groups. The median score on the NST was 8 out of 9; scores <5 were used in less than 1% of all evaluations. Distribution of data was improved in the BARS tool. In the NST, scores from 6 of 10 domains demonstrated moderate skewness and 3 high skewness. Three of the 7 domains in the BARS showed moderate skewness and none had high skewness. Two of 10 domains in the NST vs 5 of 7 domains in the BARS achieved good inter-rater reliability.

CONCLUSION

Replacing a standard numeric scale with a BARS normalized the distribution of data, reduced skewness, and enhanced inter-rater reliability in our evaluation tool. This provides some validity evidence for improved applicant assessment and ranking.

Checklists for image-guided interventions: a systematic review

OBJECTIVES

Safety checklists have improved safety in patients undergoing surgery. Checklists have been designed specifically for use in image-guided interventions. This systematic review aimed to identify checklists designed for use in radiological interventions and to evaluate their efficacy for improving patient safety. Secondary aims were to evaluate attitudes toward checklists and barriers to their use.

METHODS

OVID, MEDLINE, CENTRAL and CINAHL were searched using terms for "interventional radiology" and "checklist". Studies were included if they described pre-procedural checklist use in vascular/body interventional radiology (IR), paediatric IR or interventional neuro-radiology (INR). Data on checklist design, implementation and outcomes were extracted.

RESULTS

Sixteen studies were included. Most studies (n = 14, 87.5%) focused on body IR. Two studies (12.5%) measured perioperative outcome after checklist implementation, but both had important limitations. Checklist use varied between 54 and 100% and completion of items on the checklists varied between 28 and 100%. Several barriers to checklist use were identified, including a lack of leadership and education and cultural challenges unique to radiology.

CONCLUSIONS

We found few reports of the use of checklists in image-guided interventions. Approaches to checklist implementation varied, and several barriers to their use were identified. Evaluation has been limited. There seems to be considerable potential to improve the effective use of checklists in radiological procedures.

ADVANCES IN KNOWLEDGE

There are few reports of the use of checklists in radiological interventions, those identified reported significant barriers to the effective use of checklists.

Using the Knowledge to Action Framework to Describe a Nationwide Implementation of the WHO Surgical Safety Checklist in Cameroon

BACKGROUND

Surgical safety has advanced rapidly with evidence of improved patient outcomes through structural and process interventions. However, knowledge of how to apply these interventions successfully and sustainably at scale is often lacking. The 2019 Global Ministerial Patient Safety Summit called for a focus on implementation strategies to maintain momentum in patient safety improvements, especially in low- and middle-income settings. This study uses an implementation framework, knowledge to action, to examine a model of nationwide World Health Organization (WHO) Surgical Safety Checklist implementation in Cameroon. Cameroon is a lower-middle-income country, and based on data from high- and low-income countries, we hypothesized that more than 50% of participants would be using the checklist (penetration) in the correct manner (fidelity) 4 months postintervention.

METHODS

A collaboration of 3 stakeholders (Ministry of Health, academic institution, and nongovernmental organization) used a prospective observational design. Based on knowledge to action, there were 3 phases to the study implementation: problem identification (lack of routine checklist use in Cameroonian hospitals), multifaceted implementation strategy (3-day multidisciplinary training course, coaching, facilitated leadership engagement, and support networks), and outcome evaluation 4 months postintervention. Validated implementation outcomes were assessed. Primary outcomes were checklist use (penetration) and fidelity; secondary outcomes were perioperative teams' reactions, learning and behavior change; and tertiary outcomes were perioperative teams' acceptability of the checklist.

RESULTS

Three hundred and fifty-one operating room staff members from 25 hospitals received training. Median time to evaluation was 4.5 months (interquartile range [IQR]: 4.5-5.5, range 3-7); checklist use (penetration) increased from 20% (95% confidence interval [CI], 16-25) to 56% (95% CI, 49-63); fidelity for adherence to 6 basic safety processes was high: verification of patient identification was 91% (95% CI, 87-95); risk assessment for difficult intubation was 79% (95% CI, 73-85): risk assessment for blood loss was 88% (95% CI, 83-93) use of pulse oximetry was 93% (95% CI, 90-97); antibiotic administration was 95% (95% CI, 91-98); surgical counting was 89% (95% CI, 84-93); and fidelity for nontechnical skills measured by the WHO Behaviorally Anchored Rating Scale was 4.5 of 7 (95% CI, 3.5-5.4). Median scores for all secondary outcomes were 10/10, and 7 acceptability measures were consistently more than 70%.

CONCLUSIONS

This study shows that a multifaceted implementation strategy is associated with successful checklist implementation in a lower-middle-income country such as Cameroon, and suggests that a theoretical framework can be used to practically drive nationwide scale-up of checklist use.

Evaluation of the effect of multidisciplinary simulation-based team training on patients, staff and organisations: protocol for a stepped-wedge cluster-mixed methods study of a national, insurer-funded initiative for surgical teams in New Zealand public hospitals

INTRODUCTION

NetworkZ is a national, insurer-funded multidisciplinary simulation-based team-training programme for all New Zealand surgical teams. NetworkZ is delivered in situ, using full-body commercial simulators integrated with bespoke surgical models. Rolled out nationally over 4 years, the programme builds local capacity through instructor training and provision of simulation resources. We aim to improve surgical patient outcomes by improving teamwork through regular simulation-based multidisciplinary training in all New Zealand hospitals.

METHODS AND ANALYSIS

Our primary hypothesis is that surgical patient outcomes will improve following NetworkZ. Our secondary hypotheses are that teamwork processes will improve, and treatment injury claims will decline. In addition, we will explore factors that influence implementation and sustainability of NetworkZ and identify organisational changes following its introduction. The study uses a stepped-wedge cluster design. The intervention will roll out at yearly intervals to four cohorts of five District Health Boards. Allocation to cohort was purposive for year 1, and subsequently randomised. The primary outcome measure is Days Alive and Out of Hospital at 90 days using patient data from an existing national administrative database. Secondary outcomes measures will include analysis of postoperative complications and treatment injury claims, surveys of teamwork and safety culture, in-theatre observations and stakeholder interviews.

ETHICS AND DISSEMINATION

We believe this is the first surgical team training intervention to be implemented on a national scale, and a unique opportunity to evaluate a nation-wide team-training intervention for healthcare teams. By using a pre-existing large administrative data set, we have the potential to demonstrate a difference to surgical patient outcomes. This will be of interest to those working in the field of healthcare teamwork, quality improvement and patient safety. New Zealand Health and Disability Ethic Committee approval (#16/NTB/143).

TRIAL REGISTRATION NUMBER

Australian and New Zealand Clinical Trials Registry ID ACTRN12617000017325 and the Universal Trial Number is U1111-1189-3992.

A video anchored rating scale leads to high inter-rater reliability of inexperienced and expert raters in the absence of rater training

BACKGROUND

Our objective was to assess the impact of incorporating videos in a behaviorally anchored performance rating scale on the inter-rater reliability (IRR) of expert, intermediate and novice raters.

METHODS

The Intra-corporeal Suturing Assessment Tool (ISAT) was modified to include short video clips demonstrating poor, average, and expert performances. Blinded raters used this tool to assess videos of trainees performing suturing on a porcine model. Three attending surgeons, 4 residents, and 4 novice raters participated; no rater training was provided. The IRR was then compared among rater groups.

RESULTS

The IRR using the modified ISAT was high at 0.80 (p < 0.001). Ratings were significantly correlated with trainee objective suturing scores for all rater groups (experts: R = 0.84, residents: R = 0.81, and novices: R = 0.69; p < 0.001).

CONCLUSIONS

Incorporating video anchors (to define performance) in the ISAT led to high IRR and enabled novices to achieve similar consistency in their ratings as experts.

Testing of the WHO Infection Prevention and Control Assessment Framework at acute healthcare facility level

BACKGROUND

Monitoring and evaluation are an essential part of infection prevention and control (IPC) implementation. The authors developed an IPC assessment framework (IPCAF) to support implementation of the World Health Organization (WHO) guidelines on core components of IPC programmes in acute healthcare facilities.

AIM

To evaluate the usability and reliability of the IPCAF tool for global use.

METHODS

The IPCAF is a questionnaire with a scoring system to measure the level of IPC implementation according to the eight WHO core components. The tool was pre-tested qualitatively, revised and translated selectively. A convenience sample of hospitals was invited to participate in the final testing. At least two IPC professionals from each hospital independently completed the IPCAF and a usability questionnaire online. The tool's internal consistency and interobserver reliability or intraclass correlation coefficient (ICC) were assessed, and usability questions were summarized descriptively.

FINDINGS

In total, 46 countries, 181 hospitals and 324 individuals participated; 52 (16%) and 55 (17%) individual respondents came from low- and lower-middle income countries, respectively. Fifty-two percent of respondents took less than 1 h to complete the IPCAF. Overall, there was adequate internal consistency and a high ICC (0.92, 95% confidence interval 0.89-0.94). Ten individual questions had poor reliability (ICC <0.4); these were considered for revision according to usability feedback and expert opinion.

CONCLUSIONS

The WHO IPCAF was tested using a robust global study and revised as necessary. It is now an effective tool for IPC improvement in healthcare facilities.

A site visit protocol for assessing recovery promotion at the program level: An example from the Veterans Health Administration

OBJECTIVE

A site visit protocol was developed to assess recovery promotion in the organizational climate and culture of programs for veterans with serious mental illnesses.

METHOD

The protocol was pilot-tested in 4 programs: 2 that had scored high on the pilot version of a staff survey measure of program-level recovery promotion and 2 that had scored low. Two-person teams conducted onsite visits and assigned global and organizational domain ratings. Interrater agreement was assessed by examining adjacent agreement and computing weighted kappa.

RESULTS

The on-site protocol had good interrater agreement and discriminated between sites that scored high and low on the staff survey.

CONCLUSIONS AND IMPLICATIONS FOR PRACTICE

This site visit protocol and procedure shows promise for evaluating recovery promotion in milieu-based programs. After further refinement of this tool, adaptations could be developed for accreditation protocols or for program self-assessment and quality improvement efforts. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Observer-based tools for non-technical skills assessment in simulated and real clinical environments in healthcare: a systematic review

Background

Over the past three decades multiple tools have been developed for the assessment of non-technical skills (NTS) in healthcare. This study was designed primarily to analyse how they have been designed and tested but also to consider guidance on how to select them.

Objectives

To analyse the context of use, method of development, evidence of validity (including reliability) and usability of tools for the observer-based assessment of NTS in healthcare.

Design

Systematic review.

Data sources

Search of electronic resources, including PubMed, Embase, CINAHL, ERIC, PsycNet, Scopus, Google Scholar and Web of Science. Additional records identified through searching grey literature (OpenGrey, ProQuest, AHRQ, King’s Fund, Health Foundation).

Study selection

Studies of observer-based tools for NTS assessment in healthcare professionals (or undergraduates) were included if they: were available in English; published between January 1990 and March 2018; assessed two or more NTS; were designed for simulated or real clinical settings and had provided evidence of validity plus or minus usability. 11,101 articles were identified. After limits were applied, 576 were retrieved for evaluation and 118 articles included in this review.

Results

One hundred and eighteen studies describing 76 tools for assessment of NTS in healthcare met the eligibility criteria. There was substantial variation in the method of design of the tools and the extent of validity, and usability testing. There was considerable overlap in the skills assessed, and the contexts of use of the tools.

Conclusion

This study suggests a need for rationalisation and standardisation of the way we assess NTS in healthcare and greater consistency in how tools are developed and deployed.

Implementation and evaluation of nationwide scale-up of the Surgical Safety Checklist

Background

The WHO Surgical Safety Checklist improves surgical outcomes, but evidence and theoretical frameworks for successful implementation in low‐income countries remain lacking. Based on previous research in Madagascar, a nationwide checklist implementation in Benin was designed and evaluated longitudinally.

Methods

This study had a longitudinal embedded mixed‐methods design. The well validated Consolidated Framework for Implementation Research (CFIR) was used to structure the approach and evaluate the implementation. Thirty‐six hospitals received 3‐day multidisciplinary training and 4‐month follow‐up. Seventeen hospitals were sampled purposively for evaluation at 12–18 months. The primary outcome was sustainability of checklist use at 12–18 months measured by questionnaire. Secondary outcomes were CFIR‐derived implementation outcomes, measured using the WHO Behaviourally Anchored Rating Scale (WHOBARS), safety questionnaires and focus groups.

Results

At 12–18 months, 86·0 per cent of participants (86 of 100) reported checklist use compared with 31·1 per cent (169 of 543) before training and 88·8 per cent (158 of 178) at 4 months. There was high‐fidelity use (median WHOBARS score 5·0 of 7; use of basic safety processes ranged from 85·0 to 99·0 per cent), and high penetration shown by a significant improvement in hospital safety culture (adapted Human Factors Attitude Questionnaire scores of 76·7, 81·1 and 82·2 per cent before, and at 4 and 12–18 months after training respectively; P < 0·001). Acceptability, adoption, appropriateness and feasibility scored 9·6–9·8 of 10. This approach incorporated 31 of 36 CFIR implementation constructs successfully.

Conclusion

This study shows successfully sustained nationwide checklist implementation using a validated implementation framework.

Implementation works

Multiple Patient Casualty Scenarios: A Measurement Tool for Teamwork

INTRODUCTION

Teamwork is a critical aspect of patient care and is especially salient in response to multiple patient casualties. Effective training and measurement improve team performance. However, the literature currently lacks a scientifically developed measure of team performance within multiple causality scenarios, making training and feedback efforts difficult. The present effort addresses this gap by integrating the input of subject matter experts and the science of multicasualty teams and training to (1) identify overarching teamwork processes and corresponding behaviors necessary for team performance and (2) develop a behavioral observation tool to optimize teamwork in multicasualty training efforts.

METHOD

A search of articles including team performance frameworks associated with team training was conducted, leading to the identification of a total of 14 articles. Trained coders extracted teamwork processes and the corresponding team behaviors indicative of effective performance from these articles. Five subject matter experts were interviewed using the critical incident technique to identify additional behaviors.

RESULTS

Team situation awareness, team leadership, coordination, and information exchange emerged as the four core team processes required for team performance in scenarios with multiple patient casualties. Relevant behaviors and subbehaviors within these overarching processes were identified to inform a pilot behavioral framework of team performance.

CONCLUSIONS

The processes and associated behaviors identified within this effort serve as scientifically grounded behaviors of team performance in the case of multiple patient casualties simulated training scenarios. Future work can use and further refine these results to ensure that measures of team performance are grounded in specific, observable, and scientifically delineated behaviors.

Protocol for process evaluation of evidence-based care pathways: the case of colorectal cancer surgery

BACKGROUND AND AIM

Care pathways are complex interventions, consisting of multiple 'active ingredients', to structure care processes around patient needs. Numerous studies have reported improved outcomes after implementation of care pathways. The structure-process-outcome framework and the context-mechanism-outcome framework both suggest that outcomes can only be achieved through a certain process within a context or structure. To understand how and why care pathways are effective, understanding of both this process and context is necessary. The aim of this article is to propose a study protocol to evaluate the implementation process of evidence-based care pathways, including the influence of the context. This protocol is explained by applying it to the implementation of a colorectal cancer surgery pathway in an international setting.

METHODS

The Medical Research Council (MRC) guidance on process evaluations for complex interventions is used as the basis for the protocol. The key components of process evaluation are intervention, context, implementation, mechanisms of impact and outcomes. In process evaluations, these components are studied using quantitative and qualitative methods. Among them are patient record analysis, questionnaires, on-site visits and interviews.

DISCUSSION

To guide our methodological choices, the MRC guidance for process evaluations of complex interventions, and published protocols for process evaluations of complex interventions were used. Our protocol is now tailored for the process evaluation of evidence-based care pathways and provides researchers and clinicians methods and tools, as well as a worked example, that can be used to study the process of care pathway implementation. As a result, healthcare professionals will be informed on context factors and implementation processes that can facilitate the implementation of care pathways, improving quality and effectiveness of care processes.

Examining reliability of WHOBARS: a tool to measure the quality of administration of WHO surgical safety checklist using generalisability theory with surgical teams from three New Zealand hospitals

OBJECTIVES

To extend reliability of WHO Behaviourally Anchored Rating Scale (WHOBARS) to measure the quality of WHO Surgical Safety Checklist administration using generalisability theory. In this context, extending reliability refers to establishing generalisability of the tool scores across populations of teams and raters by accounting for the relevant sources of measurement errors.

DESIGN

Cross-sectional random effect measurement design assessing surgical teams by the five items on the three Checklist phases, and at three sites by two trained raters simultaneously.

SETTING

The data were collected in three tertiary hospitals in Auckland, New Zealand in 2016 and included 60 teams observed in 60 different cases with an equal number of teams (n=20) per site. All elective and acute cases (adults and children) involving surgery under general anaesthesia during normal working hours were eligible.

PARTICIPANTS

The study included 243 surgical staff members, 138 (50.12%) women.

MAIN OUTCOME MEASURE

Absolute generalisability coefficient that accounts for variance due to items, phases, sites and raters for the WHOBARS measure of the quality of WHO Surgical Safety Checklist administration.

RESULTS

The WHOBARS in its present form has demonstrated good generalisability of scores across teams and raters (G absolute=0.83). The largest source of measurement error was the interaction between the surgical team and the rater, accounting for 16.7% (95% CI 16.4 to 16.9) of the total variance in the data. Removing any items from the WHOBARS led to a decrease in the overall reliability of the instrument.

CONCLUSIONS

Assessing checklist administration quality is important for promoting improvement in its use, and WHOBARS offers a reliable approach for doing this.

Sustainability of using the WHO surgical safety checklist: a mixed-methods longitudinal evaluation following a nationwide blended educational implementation strategy in Madagascar

Background

The WHO Surgical Safety Checklist reduces postoperative complications by up to 50% with the biggest gains in low-income and middle-income countries (LMICs). However in LMICs, checklist use is sporadic and widespread implementation has hitherto been unsuccessful. In 2015/2016, we partnered with the Madagascar Ministry of Health to undertake nationwide implementation of the checklist. We report a longitudinal evaluation of checklist use at 12-18 months postimplementation.

Methods

Hospitals were identified from the original cohort using purposive sampling. Using a concurrent triangulation mixed-methods design, the primary outcome was self-reported checklist use. Secondary outcomes included use of basic safety processes, assessment of team behaviour, predictors of checklist use, impact on individuals and organisational culture and identification of barriers. Data were collected during 1-day hospital visits using validated questionnaires, WHO Behaviourally Adjusted Rating Scale (WHOBARS) assessment tool and focus groups and analysed using descriptive statistics, multivariate linear regression and thematic analysis.

Results

175 individuals from 14 hospitals participated. 74% reported sustained checklist use after 15 months. Mean WHOBARS scores were high, indicating good team engagement. Sustained checklist use was associated with an improved overall understanding of patient safety but not with WHOBARS, hospital size or surgical volume. 87% reported improved understanding of patient safety and 83% increased job satisfaction. Thematic analysis identified improvements in hospital culture (teamwork and communication, preparation and organisation, trust and confidence) and hospital practice (pulse oximetry, timing of antibiotic prophylaxis, introduction of a surgical count). Lack of time in an emergency and obstructive leadership were the greatest implementation barriers.

Conclusion

74% of participants reported sustained checklist use 12-18 months following nationwide implementation in Madagascar, with associated improvements in job satisfaction, culture and compliance with safety procedures. Further work is required to examine this implementation model in other countries.

Improving the quality of administration of the Surgical Safety Checklist: a mixed methods study in New Zealand hospitals

While the WHO Surgical Safety Checklist (the Checklist) can improve patient outcomes, variable administration can erode benefits. We sought to understand and improve how operating room (OR) staff use the Checklist. Our specific aims were to: determine if OR staff can discriminate between good and poor quality of Checklist administration using a validated audit tool (WHOBARS); to determine reliability and accuracy of WHOBARS self-ratings; determine the influence of demographic variables on ratings and explore OR staff attitudes to Checklist administration.

DESIGN

Mixed methods study using WHOBARS ratings of surgical cases by OR staff and two independent observers, thematic analysis of staff interviews.

PARTICIPANTS

OR staff in three New Zealand hospitals.

OUTCOME MEASURES

Reliability of WHOBARS for self-audit; staff attitudes to Checklist administration.

RESULTS

Analysis of scores (243 participants, 2 observers, 59 cases) supported tool reliability, with 87% of WHOBARS score variance attributable to differences in Checklist administration between cases. Self-ratings were significantly higher than observer ratings, with some differences between professional groups but error variance from all raters was less than 10%. Key interview themes (33 interviewees) were: Team culture and embedding the Checklist, Information transfer and obstacles, Raising concerns and 'A tick-box exercise'. Interviewees felt the Checklist could promote teamwork and a safety culture, particularly enabling speaking up. Senior staff were of key importance in setting the appropriate tone.

CONCLUSIONS

The WHOBARS tool could be useful for self-audit and quality improvement as OR staff can reliably discriminate between good and poor Checklist administration. OR staff self-ratings were lenient compared with external observers suggesting the value of external audit for benchmarking. Small differences between ratings from professional groups underpin the value of including all members of the team in scoring. We identified factors explaining staff perceptions of the Checklist that should inform quality improvement interventions.

Crisis Resource Management in the Delivery Room: Development of Behavioral Markers for Team Performance in Emergency Simulation

Human factors are the most relevant issues contributing to adverse events in obstetrics. Specific training of Crisis Resource Management (CRM) skills (i.e., problem solving and team management, resource allocation, awareness of environment, and dynamic decision-making) is now widespread and is often based on High Fidelity Simulation. In order to be used as a guideline in simulated scenarios, CRM skills need to be mapped to specific and observable behavioral markers. For this purpose, we developed a set of observable behaviors related to the main elements of CRM in the delivery room. The observational tool was then adopted in a two-days seminar on obstetric hemorrhage where teams working in obstetric wards of six Italian hospitals took part in simulations. The tool was used as a guide for the debriefing and as a peer-to-peer feedback. It was then rated for its usefulness in facilitating the reflection upon one’s own behavior, its ease of use, and its usefulness for the peer-to-peer feedback. The ratings were positive, with a median of 4 on a 5-point scale. The CRM observational tool has therefore been well-received and presents a promising level of inter-rater agreement. We believe the tool could have value in facilitating debriefing and in the peer-to-peer feedback.