A realist synthesis of multicentre comparative audit implementation: exploring what works and in which healthcare contexts

BACKGROUND

Multicentre comparative clinical audits have the potential to improve patient care, allow benchmarking and inform resource allocation. However, implementing effective and sustainable large-scale audit can be difficult within busy and resource-constrained contemporary healthcare settings. There are little data on what facilitates the successful implementation of multicentre audits. As healthcare environments are complex sociocultural organisational environments, implementing multicentre audits within them is likely to be highly context dependent.

OBJECTIVE

We aimed to examine factors that were influential in the implementation process of multicentre comparative audits within healthcare contexts-what worked, why, how and for whom?

METHODS

A realist review was conducted in accordance with the Realist and Meta-narrative Evidence Syntheses: Evolving Standards reporting standards. A preliminary programme theory informed two systematic literature searches of peer-reviewed and grey literature. The main context-mechanism-outcome (CMO) configurations underlying the implementation processes of multicentre audits were identified and formed a final programme theory.

RESULTS

69 original articles were included in the realist synthesis. Four discrete CMO configurations were deduced from this synthesis, which together made up the final programme theory. These were: (1) generating trustworthy data; (2) encouraging audit participation; (3) ensuring audit sustainability; and (4) facilitating audit cycle completion.

CONCLUSIONS

This study elucidated contexts, mechanisms and outcomes influential to the implementation processes of multicentre or national comparative audits in healthcare. The relevance of these contextual factors and generative mechanisms were supported by established theories of behaviour and findings from previous empirical research. These findings highlight the importance of balancing reliability with pragmatism within complex adaptive systems, generating and protecting human capital, ensuring fair and credible leadership and prioritising change facilitation.

Concordance between independent first-line assessors in the Western Australian Audit of Surgical Mortality

BACKGROUND

The Western Australian Audit of Surgical Mortality independently reviews deaths under a surgeon. A surgical case form completed by the surgeon is reviewed by a first-line assessor, who in 15% of cases recommends a detailed case note review. The aim of this study was to assess concordance between the original first-line assessment (FLA1) and an additional first-line assessment (FLA2) in relation to the necessity for a second-line assessment (SLA) and recognition of clinical management issues.

METHODS

Experienced assessors (n = 39) representing different specialties were invited to be FLA2 assessors for 170 random cases. They completed identical FLA forms. FLA2 assessors were consultant surgeons with experience of the peer review process. Inter-rater reliability was determined using Gwet's Agreement Coefficient.

RESULTS

A total of 170 FLAs, 158 (92.9%) were returned. Concordance between assessors for recommending an SLA was 70.5% (110 of 156), with an SLA recommended for 16 (10.3%) FLAs but not for 94 (60.3%) FLAs. When there was non-concordance, FLA1 assessors requested more SLAs than did FLA2 assessors (29 [18.6%] vs. 17 [10.9%], respectively). Three adverse events were recorded. In two the assessors differed, with one recording an adverse event and the other recording an area of concern. Additionally, in one instance the assessors differed on the preventability of the event but agreed that it may have contributed to death.

CONCLUSION

Concordance between assessors ranged between 'moderate' (requests for an SLA) through to 'almost perfect' (presence of an adverse event). This suggests the FLA screening process is reliable.

Assessment of accuracy of Australian health service death data: implications for the audits of surgical mortality

BACKGROUND

The Victorian Audit of Surgical Mortality (VASM) investigates all surgically related deaths in Victoria, Australia, as a surgical educational activity aimed to make surgery safer. Whilst data collected within the audit are regularly reviewed for accuracy, there has never been a review of the data provided from health services.

METHODS

Two-year death data provided by one Victorian health service were reviewed. Hospital notes for 4 months of each year were analysed to assess patients dying under surgical care. These data were compared to referrals to the VASM over the same period.

RESULTS

Of the 3907 patient deaths recorded, 35.1% were reviewed. During their final admission, 178 (13%) patients underwent a procedure (93 medical and 85 surgical). Only 29.2% of these were recorded in the health service data set. Eighteen patients died under the care of a surgeon without a procedure, meaning that 103 deaths should have been reported to the VASM of which only 55.3% (57/103) were reported.

CONCLUSION

There were major errors in the health service database resulting in under-reporting of deaths to the VASM which could have education and policy repercussions. For improvements to the safety and quality of health services, it is critical that all deaths are accurately recorded by health services and reported to the relevant bodies with internal verification processes.

Homogeneity score test of AC1 statistics and estimation of common AC1 in multiple or stratified inter-rater agreement studies

BACKGROUND

Cohen's κ coefficient is often used as an index to measure the agreement of inter-rater determinations. However, κ varies greatly depending on the marginal distribution of the target population and overestimates the probability of agreement occurring by chance. To overcome these limitations, an alternative and more stable agreement coefficient was proposed, referred to as Gwet's AC1. When it is desired to combine results from multiple agreement studies, such as in a meta-analysis, or to perform stratified analysis with subject covariates that affect agreement, it is of interest to compare several agreement coefficients and present a common agreement index. A homogeneity test of κ was developed; however, there are no reports on homogeneity tests for AC1 or on an estimator of common AC1. In this article, a homogeneity score test for AC1 is therefore derived, in the case of two raters with binary outcomes from K independent strata and its performance is investigated. An estimation of the common AC1 between strata and its confidence intervals is also discussed.

METHODS

Two homogeneity tests are provided: a score test and a goodness-of-fit test. In this study, the confidence intervals are derived by asymptotic, Fisher's Z transformation and profile variance methods. Monte Carlo simulation studies were conducted to examine the validity of the proposed methods. An example using clinical data is also provided.

RESULTS

Type I error rates of the proposed score test were close to the nominal level when conducting simulations with small and moderate sample sizes. The confidence intervals based on Fisher's Z transformation and the profile variance method provided coverage levels close to nominal over a wide range of parameter combination.

CONCLUSIONS

The method proposed in this study is considered to be useful for summarizing evaluations of consistency performed in multiple or stratified inter-rater agreement studies, for meta-analysis of reports from multiple groups and for stratified analysis.