Audit and feedback to reduce unwarranted clinical variation at scale: a realist study of implementation strategy mechanisms

BACKGROUND

Unwarranted clinical variation in hospital care includes the underuse, overuse, or misuse of services. Audit and feedback is a common strategy to reduce unwarranted variation, but its effectiveness varies widely across contexts. We aimed to identify implementation strategies, mechanisms, and contextual circumstances contributing to the impact of audit and feedback on unwarranted clinical variation.

METHODS

Realist study examining a state-wide value-based healthcare program implemented between 2017 and 2021 in New South Wales, Australia. Three initiatives within the program included audit and feedback to reduce unwarranted variation in inpatient care for different conditions. Multiple data sources were used to formulate the initial audit and feedback program theory: a systematic review, realist review, program document review, and informal discussions with key program stakeholders. Semi-structured interviews were then conducted with 56 participants to refute, refine, or confirm the initial program theories. Data were analysed retroductively using a context-mechanism-outcome framework for 11 transcripts which were coded into the audit and feedback program theory. The program theory was validated with three expert panels: senior health leaders (n = 19), Agency for Clinical Innovation (n = 11), and Ministry of Health (n = 21) staff.

RESULTS

The program's audit and feedback implementation strategy operated through eight mechanistic processes. The strategy worked well when clinicians (1) felt ownership and buy-in, (2) could make sense of the information provided, (3) were motivated by social influence, and (4) accepted responsibility and accountability for proposed changes. The success of the strategy was constrained when the audit process led to (5) rationalising current practice instead of creating a learning opportunity, (6) perceptions of unfairness and concerns about data integrity, 7) development of improvement plans that were not followed, and (8) perceived intrusions on professional autonomy.

CONCLUSIONS

Audit and feedback strategies may help reduce unwarranted clinical variation in care where there is engagement between auditors and local clinicians, meaningful audit indicators, clear improvement plans, and respect for clinical expertise. We contribute theoretical development for audit and feedback by proposing a Model for Audit and Feedback Implementation at Scale. Recommendations include limiting the number of audit indicators, involving clinical staff and local leaders in feedback, and providing opportunities for reflection.

Improving Emergency Department Throughput Using Audit-and-Feedback With Peer Comparison Among Emergency Department Physicians

INTRODUCTION

We sought to determine if audit-and-feedback with peer comparison among emergency physicians is associated with improved emergency department (ED) throughput and decreased variation in physician performance.

METHODS

We implemented an audit-and-feedback with peer comparison tool at a single urban academic ED from March 1, 2013, to July 1, 2018. In the first study period, physicians received no reports. In the second period, they received daily reports. In the third period, they received daily, quarterly, and annual reports. Outcomes included patients per hour, admission rate, time to admission, and time to discharge.

RESULTS

A total of 272,032 patient visits and 36 ED physicians were included. The mean admission rate decreased 6.8%; the mean time to admission decreased 43.8 minutes; and the mean time to discharge decreased 40.6 minutes. Variation among physicians decreased for admission rate, time to admission, and time to discharge. Low-performing outliers showed disproportionately larger improvements in patients per hour, admission rate, time to admission, and time to discharge.

CONCLUSIONS

Automated peer comparison reports for academic emergency physicians was associated with lower admission rates, shorter times to admission, and shorter times to discharge at the departmental level, as well as decreased practice variation at the individual level.

Addressing unwarranted clinical variation: A rapid review of current evidence

INTRODUCTION

Unwarranted clinical variation (UCV) can be described as variation that can only be explained by differences in health system performance. There is a lack of clarity regarding how to define and identify UCV and, once identified, to determine whether it is sufficiently problematic to warrant action. As such, the implementation of systemic approaches to reducing UCV is challenging. A review of approaches to understand, identify, and address UCV was undertaken to determine how conceptual and theoretical frameworks currently attempt to define UCV, the approaches used to identify UCV, and the evidence of their effectiveness.

DESIGN

Rapid evidence assessment (REA) methodology was used.

DATA SOURCES

A range of text words, synonyms, and subject headings were developed for the major concepts of unwarranted clinical variation, standards (and deviation from these standards), and health care environment. Two electronic databases (Medline and Pubmed) were searched from January 2006 to April 2017, in addition to hand searching of relevant journals, reference lists, and grey literature.

DATA SYNTHESIS

Results were merged using reference-management software (Endnote) and duplicates removed. Inclusion criteria were independently applied to potentially relevant articles by 3 reviewers. Findings were presented in a narrative synthesis to highlight key concepts addressed in the published literature.

RESULTS

A total of 48 relevant publications were included in the review; 21 articles were identified as eligible from the database search, 4 from hand searching published work and 23 from the grey literature. The search process highlighted the voluminous literature reporting clinical variation internationally; yet, there is a dearth of evidence regarding systematic approaches to identifying or addressing UCV.

CONCLUSION

Wennberg's classification framework is commonly cited in relation to classifying variation, but no single approach is agreed upon to systematically explore and address UCV. The instances of UCV that warrant investigation and action are largely determined at a systems level currently, and stakeholder engagement in this process is limited. Lack of consensus on an evidence-based definition for UCV remains a substantial barrier to progress in this field.

Practice Variations in the Diagnosis, Monitoring, and Treatment of Systemic Lupus Erythematosus in Canada

OBJECTIVE

To evaluate the diagnosis, monitoring, and treatment of systemic lupus erythematosus (SLE) in Canada.

METHODS

A 63-question electronic survey was developed with the Canadian Rheumatology Association and others. Descriptive analyses of responses were performed.

RESULTS

Survey respondents (n = 175) reported varying practices in the diagnosis, monitoring, and treatment of SLE. Performance of laboratory investigations for diagnosis and monitoring varied, with 78% of responders performing them at least every 6 months. Validated measures of SLE disease activity and damage were not commonly used. Most common first-line agents besides steroids for induction therapy for class III or IV lupus nephritis included intravenous cyclophosphamide and mycophenolate mofetil. Antimalarial use was common, with 96% of respondents using these in active skin disease. Over 60% of respondents indicated that 80-100% of their patients were taking antimalarials, while another 25% indicated they used these drugs in up to 80% of their patients. There were 71% of responders who reported completing frequent (6-12 mos) ophthalmology screening in patients taking antimalarials. Biologics were infrequently used. Responders were more likely to stop azathioprine and chloroquine than hydroxychloroquine in pregnant patients with SLE. Other aspects of routine care including vaccination and cardiovascular risk management varied considerably. The majority (80%) agreed that a dedicated multidisciplinary care team would improve SLE care.

CONCLUSION

Considerable practice variation in SLE management was noted. This may help inform future recommendations for the diagnosis, monitoring, and treatment of SLE in Canada.

Spine Surgeon Treatment Variability: The Impact on Costs

STUDY DESIGN

Cross-sectional analysis.

OBJECTIVES

Given the lack of strong evidence/guidelines on appropriate treatment for lumbar spine disease, substantial variability exists among surgical treatments utilized, which is associated with differences in costs to treat a given pathology. Our goal was to investigate the variability in costs among spine surgeons nationally for the same pathology in similar patients.

METHODS

Four hundred forty-five spine surgeons completed a survey of clinical and radiographic case scenarios on patients with recurrent lumbar disc herniation, low back pain, and spondylolisthesis. Those surveyed were asked to provide various details including their geographical location, specialty, and fellowship training. Treatment options included no surgery, anterior lumbar interbody fusion, posterolateral fusion, and transforaminal/posterior lumbar interbody fusion. Costs were estimated via Medicare national payment amounts.

RESULTS

For recurrent lumbar disc herniation, no difference in costs existed for patients undergoing their first revision microdiscectomy. However, for patients undergoing another microdiscectomy, surgeons who operated <100 times/year had significantly lower costs than those who operated >200 times/year (P < .001) and those with 5-15 years of experience had significantly higher costs than those with >15 years (P < .001). For the treatment of low back pain, academic surgeons kept costs about 55% lower than private practice surgeons (P < .001). In the treatment of spondylolisthesis, there was significant treatment variability without significant differences in costs.

CONCLUSIONS

Significant variability in surgical treatment paradigms exists for different pathologies. Understanding why variability in treatment selection exists in similar clinical contexts across practices is important to ensure the most cost-effective delivery of care among spine surgeons.

Variation in Antiosteoporotic Drug Prescribing and Spending Across Spain. A Population-Based Ecological Cross-Sectional Study

Introduction: Evidence has shown that utilization of antiosteoporotic medications does not correspond with risk, and studies on other therapies have shown that adequacy of pharmaceutical prescribing might vary between regions. Nevertheless, very few studies have addressed the variability in osteoporotic drug consumption. We aimed to describe variations in pharmaceutical utilization and spending on osteoporotic drugs between Health Areas (HA) in Spain.

Methods: Population-based cross-sectional ecological study of expenditure and utilization of the five therapeutic groups marketed for osteoporosis treatment in Spain in 2009. Small area variation analysis (SAVA) methods were used. The units of analysis were the 168 HA of 13 Spanish regions, including 7.2 million women aged 50 years and older. The main outcomes were the defined daily dose (DDD) per 1000 inhabitants and day (DDD/1000/Day) dispensed according to the pharmaceutical claims reimbursed, and the expenditure on antiosteoporotics at retail price per woman ≥50 years old and per year.

Results: The average osteoporosis drug consumption was 116.8 DDD/1000W/Day, ranging from 78.5 to 158.7 DDD/1000W/Day between the HAs in the 5th and 95th percentiles. Seventy-five percent of the antiosteoporotics consumed was bisphosphonates, followed by raloxifene, strontium ranelate, calcitonins, and parathyroid hormones including teriparatide. Regarding variability by therapeutic groups, biphosphonates showed the lowest variation, while calcitonins and parathyroid hormones showed the highest variation. The annual expenditure on antiosteoporotics was €426.5 million, translating into an expenditure of €59.2 for each woman ≥50 years old and varying between €38.1 and €83.3 between HAs in the 5th and 95th percentiles. Biphosphonates, despite accounting for 79% of utilization, only represented 63% of total expenditure, while parathyroid hormones with only 1.6% of utilization accounted for 15% of the pharmaceutical spending.

Conclusion: This study highlights a marked geographical variation in the prescription of antiosteoporotics, being more pronounced in the case of costly drugs such as parathyroid hormones. The differences in rates of prescribing explained almost all of the variance in drug spending, suggesting that the difference in prescription volume between territories, and not the price of the drugs, is the main source of variation in this setting. Data on geographical variation of prescription can help guide policy proposals for targeting areas with inadequate antiosteoporotic drug use.

Practice patterns in longitudinal lupus care provision: patient and physician perspectives

Background/purpose

To plan a quality improvement project, we need to understand the practice patterns of physicians. We undertook an online survey of systemic lupus erythematosus (SLE) patients and physicians providing care to SLE patients to determine the patterns of medical care provided to SLE patients.

Materials and methods

Two self-report surveys were developed. A 12-item survey for the patients and a 13-item survey for physicians enquired about longitudinal care for SLE. Surveys were administered online to physicians providing care to SLE patients, and to patients who self-identified as having SLE, through the Lupus Society of Illinois. Patient and physician data were analyzed for physician practice patterns for SLE care, using chi square tests and t tests. A P value of 0.05 or less was considered significant on two-tailed tests.

Results

A total of 283 patients completed the survey. Mean (SD) age and disease duration of patients were 45.9 (13.2) and 12.7 (9.7) years. Half of the participants were being seen at 3–4-month intervals. More than 70% of patients reported being tested for antinuclear antibody (ANA), and 20–30% anti-ENA antibody and Sjögren’s (SSA/SSB) antibodies, respectively, at each follow-up visit. Eighty-six rheumatologists completed the surveys. Mean (SD) age was 55 (12) years and 56% were men. More than half (54%) provided care only in a private practice setting. More than 80% of physicians reported seeing their SLE patients at 3–4-month interval. Only 2% reported performing ANA tests at each visit, while 4–5% performed anti-ENA and anti-SSA/SSB antibody tests at each visit for their SLE patients. More than 75% of physicians in private practice also ordered sedimentation rate at each visit for their SLE patients.

Conclusions

Unnecessary laboratory investigations may be being ordered routinely for patients at every visit. These results indicate a need for physician education on indications and utility of some of the laboratory tests such as ANA.

Understanding the Extent and Drivers of Interphysician Cost Variation for Spine Procedures

STUDY DESIGN

Retrospective analysis of billing data, medical records, and hospital cost data.

OBJECTIVE

To quantify intersurgeon variation for hospital costs of four spine procedures while adjusting for patient comorbidities and demographic factors.

SUMMARY OF BACKGROUND DATA

Spine care accounts for $90 billion in health care expenditures in the United States. Past findings demonstrate regional variation in surgery rates and high intersurgeon variation for anterior cervical discectomies/fusions. However, less has been done to examine intersurgeon variation in resource use across multiple procedures while adjusting for patient characteristics outside of a surgeon's control.

METHODS

We examined intersurgeon variation for 1241 elective spine procedures at one facility for 3 years. The procedures included 1 to 2 level cases of anterior cervical discectomies/fusions, posterior lumbar decompressions/fusions, posterior laminectomies, and lumbar discectomies. We isolated mean and median costs by surgeon and adjusted for patient demographics, comorbidities, and procedure types. Finally, we examined variation in subcategories such as instrumentation and inpatient stay costs to determine which contribute to total cost variation.

RESULTS

Unadjusted costs per surgeon varied by a factor of 1.32 to 1.81 between lowest and highest cost surgeon depending on procedure. After adjusting for patient features and procedure, variation was reduced to 1.31x. Of the seven surgeons who had sufficient patient volume, one was significantly less costly (-$1,462 per procedure) whereas three were significantly more costly than mean (+$685, +$839, +$702 per procedure). Intersurgeon differences in supply and operating room costs largely accounted for total variation, though actual drivers of variation were surgeon-specific.

CONCLUSION

Surgeons vary in average cost for spine procedures, though variation is more modest once adjusted for patient characteristics. Data on procedure-level variation should be discussed with individual surgeons to shift practice patterns. Finally, the comparison methodology can be applied to other procedures and specialties.

LEVEL OF EVIDENCE

4.

Quantification of biological variation in blood-based therapy--a summary of a meta-analysis to inform manufacturing in the clinic

Background and Objectives

Biological raw materials, the basis for cellular therapies such as stem cells, have a significantly greater degree of complexity than their traditional pharmaceutical counterparts. This can be attributed to the inherent variation of its source – human beings. Currently, cell therapies are made in small, ad hoc batches, but larger scale production is a prerequisite to meeting future demand and will require a quality‐by‐design approach to manufacturing that will be designed around, or be robust to this variation. Quantification of variation will require understanding of the current baseline and stratification of its sources.

Materials and Methods

Haematopoietic stem cell therapy was chosen as a case study to explore this variation, and a PRISMA‐guided (Preferred Reporting Items for Systematic Reviews and Meta‐Analyses) systematic meta‐analysis was carried out for a number of predetermined cell measurements.

Results

From this data set, it appears that the extent of variation in therapeutic dose (in terms of transplanted total nucleated cells and CD34⁺ cells per kilogram) for HSCT is between one and four orders of magnitude of the median.

Conclusions

This is tolerated under the practice of medicine but would be unmanageable from a biomanufacturing perspective and raises concerns about comparable levels of efficacy and treatment. A number of sources that will contribute towards this variation are also reported, as is the direction of travel for 4 greater clarity of the scale of this challenge.

Regional variation in the use of surgery in Ontario

BACKGROUND

Regional variation in the use of surgery implies that there is uncertainty regarding appropriate use. The objectives of this study were to identify which surgical procedures are most commonly performed in the province of Ontario and measure the extent of variation in the use of surgical procedures across Ontario counties.

METHODS

We used the Canadian Institute for Health Information Discharge Abstract Database, Same Day Surgery Database and National Ambulatory Care Reporting System to retrieve information on all inpatient and day surgery visits in Ontario between Apr. 1, 2002, and Mar. 31, 2011. We identified the 84 most common procedures according to Canadian Classification of Interventions codes. We calculated rates of use for each procedure throughout the 49 Ontario counties and then calculated measures of variation (quartile ratio and systematic component of variation) in use between the counties.

RESULTS

Colonoscopy was the most commonly performed procedure during the study period, with an average adjusted rate of 2012 per 100 000 population. The procedure with the highest measure of variation was iridectomy, with a quartile ratio of 6.7, followed by colposcopy (5.2), cervical biopsy (4.2) and femoral arteriography (4.1). These procedures were less commonly performed. Common procedures such as colonoscopy, cataract extraction and vaginal delivery had lower quartile ratios. Analysis using the systematic component of variation as the measure of variation gave similar results.

INTERPRETATION

Colonoscopy was the most commonly performed procedure in Ontario, and cataract extraction was the most common surgical procedure. Procedures with the highest measures of variation between counties tended to be those that occurred less commonly in Ontario, and common procedures were associated with less regional variation.

Variation in use of reverse total shoulder arthroplasty across hospitals

BACKGROUND

The opportunity for variation exists in the choice between anatomic and reverse total shoulder arthroplasty. Quality improvement methods seek to reduce variation. We used supply-chain data to characterize variation in the selection of anatomic versus reverse total shoulder arthroplasty across hospitals and to analyze the effect of hospital volume on this variation.

METHODS

Mendenhall Associates, Inc (Ann Arbor, MI, USA) provided us with a database of hospital supply-chain data from orthopaedic surgical cases. This study included hospitals in which at least one total shoulder arthroplasty was performed. We calculated, for each hospital, the percentages of each type of prosthesis implanted and examined the distribution of these percentages across all hospitals. We also divided the sample of hospitals into tertiles, by volume of total shoulder arthroplasties performed, and examined the distributions of percentage reverse shoulder arthroplasty performed in each tertile.

RESULTS

Across all hospitals, we saw wide variation in the volume of total shoulder arthroplasties and the percentage of reverse shoulder arthroplasties performed. Hospitals with lower total shoulder arthroplasty volumes exhibited greater variation in the percentages of each type of total shoulder arthroplasty performed. Higher volume hospitals exhibited smaller variation.

CONCLUSIONS

This study revealed wide variation in the selection of anatomic and reverse total shoulder arthroplasty across all hospitals and an inverse relationship between hospital volume and variation. This variation signals uncertainty about the best application of each device and that there is need for improvement in the consistency of treatment of patients with shoulder disease.

LEVEL OF EVIDENCE

Level II, cost-effectiveness study, economic and decision analysis.

From data to decisions? Exploring how healthcare payers respond to the NHS Atlas of Variation in Healthcare in England

PURPOSE

Although information on variations in health service performance is now more widely available, relatively little is known about how healthcare payers use this information to improve resource allocation. We explore to what extent and how Primary Care Trusts (PCTs) in England have used the NHS Atlas of Variation in Healthcare, which has highlighted small area variation in rates of expenditure, activity and outcome.

METHODS

Data collection involved an email survey among PCT Chief Executives and a telephone follow-up to reach non-respondents (total response: 53 of 151 of PCTs, 35%). 45 senior to mid-level staff were interviewed to probe themes emerging from the survey. The data were analysed using a matrix-based Framework approach.

FINDINGS

Just under half of the respondents (25 of 53 PCTs) reported not using the Atlas, either because they had not been aware of it, lacked staff capacity to analyse it, or did not perceive it as applicable to local decision-making. Among the 28 users, the Atlas served as a prompt to understand variations and as a visual tool to facilitate communication with clinicians. Achieving clarity on which variations are unwarranted and agreeing on responsibilities for action appeared to be important factors in moving beyond initial information gathering towards decisions about resource allocation and behaviour change.

CONCLUSIONS

Many payers were unable to use information on small area variations in expenditure, activity and outcome. To change this what is additionally required are appropriate tools to understand causes of unexplained variation, in particular unwarranted variation, and enable remedial actions to be prioritised in terms of their contribution to population health.