Examining reliability of WHOBARS: a tool to measure the quality of administration of WHO surgical safety checklist using generalisability theory with surgical teams from three New Zealand hospitals

CheckPOINT: a simple tool to measure Surgical Safety Checklist implementation fidelity

INTRODUCTION

The WHO Surgical Safety Checklist (SSC) is a communication tool that improves teamwork and patient outcomes. SSC effectiveness is dependent on implementation fidelity. Administrative audits fail to capture most aspects of SSC implementation fidelity (ie, team communication and engagement). Existing research tools assess behaviours during checklist performance, but were not designed for routine quality assurance and improvement. We aimed to create a simple tool to assess SSC implementation fidelity, and to test its reliability using video simulations, and usability in clinical practice.

METHODS

The Checklist Performance Observation for Improvement (CheckPOINT) tool underwent two rounds of face validity testing with surgical safety experts, clinicians and quality improvement specialists. Four categories were developed: checklist adherence, communication effectiveness, attitude and engagement. We created a 90 min training programme, and four trained raters independently scored 37 video simulations using the tool. We calculated intraclass correlation coefficients (ICC) to assess inter-rater reliability (ICC>0.75 indicating excellent reliability). We then trained two observers, who tested the tool in the operating room. We interviewed the observers to determine tool usability.

RESULTS

The CheckPOINT tool had excellent inter-rater reliability across SSC phases. The ICC was 0.83 (95% CI 0.67 to 0.98) for the sign-in, 0.77 (95% CI 0.63 to 0.92) for the time-out and 0.79 (95% CI 0.59 to 0.99) for the sign-out. During field testing, observers reported CheckPOINT was easy to use. In 98 operating room observations, the total median (IQR) score was 25 (23-28), checklist adherence was 7 (6-7), communication effectiveness was 6 (6-7), attitude was 6 (6-7) and engagement was 6 (5-7).

CONCLUSIONS

CheckPOINT is a simple and reliable tool to assess SSC implementation fidelity and identify areas of focus for improvement efforts. Although CheckPOINT would benefit from further testing, it offers a low-resource alternative to existing research tools and captures elements of adherence and team behaviours.

High-flow oxygen via oxygenating mouthguard in short upper gastrointestinal endoscopy: A randomised controlled trial

BACKGROUND

Anaesthetic care during upper gastrointestinal (GI) endoscopy has the unique challenge of maintaining ventilation and oxygenation via a shared upper airway. Supplemental oxygen is recommended by international society guidelines, however, the optimal route or rate of oxygen delivery is not known. Various oxygen delivery devices have been investigated to improve oxygenation during upper GI endoscopy, however, these are limited by commercial availability, costs and in some cases, the expertise required for insertion. Anecdotally at our centre, higher flows of supplemental oxygen can safely be delivered via an oxygenating mouthguard routinely used during upper GI endoscopic procedures.

AIM

To assess the incidence of hypoxaemia (SpO₂ < 90%) in patients undergoing upper GI endoscopy receiving supplemental oxygen using an oxygenating mouthguard at 20 L/min flow compared to standard nasal cannula (SNC) at 2 L/min flow.

METHODS

A single centre, prospective, randomised clinical trial at two sites of an Australian tertiary hospital between October 2020 and September 2021 was conducted. Patients undergoing elective upper gastrointestinal endoscopy under deep sedation were randomised to receive supplemental oxygen via high-flow via oxygenating mouthguard (HFMG) at 20 L/min flow or SNC at 2 L/min flow. The primary outcome was the incidence of hypoxaemia of any duration measured by pulse oximetry. Intraprocedural-related, procedural-related, and sedation-related adverse events and patient-reported outcomes were also recorded.

RESULTS

Three hundred patients were randomised. Eight patients were excluded after randomisation. 292 patients were included in the intention-to-treat analysis. The incidence of hypoxaemia was significantly reduced in those allocated HFMG. Six patients (4.4%) allocated to HFMG experienced an episode of hypoxaemia, compared to thirty-four (22.1%) patients allocated to SNC (P value < 0.001). No significant difference was observed in the rates of adverse events or patient-reported outcome measures.

CONCLUSION

The use of HFMG offers a novel approach to reducing the incidence of hypoxaemia during short upper gastrointestinal endoscopic procedures in low-risk patients undergoing deep sedation.

Cross-cultural validation of the stroke riskometer using generalizability theory

The Stroke Riskometer mobile application is a novel, validated way to provide personalized stroke risk assessment for individuals and motivate them to reduce their risks. Although this app is being used worldwide, its reliability across different countries has not yet been rigorously investigated using appropriate methodology. The Generalizability Theory (G-Theory) is an advanced statistical method suitable for examining reliability and generalizability of assessment scores across different samples, cultural and other contexts and for evaluating sources of measurement errors. G-Theory was applied to the Stroke Riskometer data sampled from 1300 participants in 13 countries using two-facet nested observational design (person by item nested in the country). The Stroke Riskometer demonstrated strong reliability in measuring stroke risks across the countries with coefficients G relative and absolute of 0.84, 95%CI [0.79; 0.89] and 0.82, 95%CI [0.76; 0.88] respectively. D-study analyses revealed that the Stroke Riskometer has optimal reliability in its current form in measuring stroke risk for each country and no modifications are required. These results suggest that the Stroke Riskometer's scores are generalizable across sample population and countries permitting cross-cultural comparisons. Further studies investigating reliability of the Stroke Riskometer over time in longitudinal study design are warranted.

Evaluation of the effect of multidisciplinary simulation-based team training on patients, staff and organisations: protocol for a stepped-wedge cluster-mixed methods study of a national, insurer-funded initiative for surgical teams in New Zealand public hospitals

INTRODUCTION

NetworkZ is a national, insurer-funded multidisciplinary simulation-based team-training programme for all New Zealand surgical teams. NetworkZ is delivered in situ, using full-body commercial simulators integrated with bespoke surgical models. Rolled out nationally over 4 years, the programme builds local capacity through instructor training and provision of simulation resources. We aim to improve surgical patient outcomes by improving teamwork through regular simulation-based multidisciplinary training in all New Zealand hospitals.

METHODS AND ANALYSIS

Our primary hypothesis is that surgical patient outcomes will improve following NetworkZ. Our secondary hypotheses are that teamwork processes will improve, and treatment injury claims will decline. In addition, we will explore factors that influence implementation and sustainability of NetworkZ and identify organisational changes following its introduction. The study uses a stepped-wedge cluster design. The intervention will roll out at yearly intervals to four cohorts of five District Health Boards. Allocation to cohort was purposive for year 1, and subsequently randomised. The primary outcome measure is Days Alive and Out of Hospital at 90 days using patient data from an existing national administrative database. Secondary outcomes measures will include analysis of postoperative complications and treatment injury claims, surveys of teamwork and safety culture, in-theatre observations and stakeholder interviews.

ETHICS AND DISSEMINATION

We believe this is the first surgical team training intervention to be implemented on a national scale, and a unique opportunity to evaluate a nation-wide team-training intervention for healthcare teams. By using a pre-existing large administrative data set, we have the potential to demonstrate a difference to surgical patient outcomes. This will be of interest to those working in the field of healthcare teamwork, quality improvement and patient safety. New Zealand Health and Disability Ethic Committee approval (#16/NTB/143).

TRIAL REGISTRATION NUMBER

Australian and New Zealand Clinical Trials Registry ID ACTRN12617000017325 and the Universal Trial Number is U1111-1189-3992.