Economic evaluations of audit and feedback interventions: a systematic review

A systematic review of experimentally tested implementation strategies across health and human service settings: evidence from 2010-2022

BACKGROUND

Studies of implementation strategies range in rigor, design, and evaluated outcomes, presenting interpretation challenges for practitioners and researchers. This systematic review aimed to describe the body of research evidence testing implementation strategies across diverse settings and domains, using the Expert Recommendations for Implementing Change (ERIC) taxonomy to classify strategies and the Reach Effectiveness Adoption Implementation and Maintenance (RE-AIM) framework to classify outcomes.

METHODS

We conducted a systematic review of studies examining implementation strategies from 2010-2022 and registered with PROSPERO (CRD42021235592). We searched databases using terms "implementation strategy", "intervention", "bundle", "support", and their variants. We also solicited study recommendations from implementation science experts and mined existing systematic reviews. We included studies that quantitatively assessed the impact of at least one implementation strategy to improve health or health care using an outcome that could be mapped to the five evaluation dimensions of RE-AIM. Only studies meeting prespecified methodologic standards were included. We described the characteristics of studies and frequency of implementation strategy use across study arms. We also examined common strategy pairings and cooccurrence with significant outcomes.

FINDINGS

Our search resulted in 16,605 studies; 129 met inclusion criteria. Studies tested an average of 6.73 strategies (0-20 range). The most assessed outcomes were Effectiveness (n=82; 64%) and Implementation (n=73; 56%). The implementation strategies most frequently occurring in the experimental arm were Distribute Educational Materials (n=99), Conduct Educational Meetings (n=96), Audit and Provide Feedback (n=76), and External Facilitation (n=59). These strategies were often used in combination. Nineteen implementation strategies were frequently tested and associated with significantly improved outcomes. However, many strategies were not tested sufficiently to draw conclusions.

CONCLUSION

This review of 129 methodologically rigorous studies built upon prior implementation science data syntheses to identify implementation strategies that had been experimentally tested and summarized their impact on outcomes across diverse outcomes and clinical settings. We present recommendations for improving future similar efforts.

Implementation Strategies to Promote Short-Course Radiation for Bone Metastases

Importance

For patients with nonspine bone metastases, short-course radiotherapy (RT) can reduce patient burden without sacrificing clinical benefit. However, there is great variation in uptake of short-course RT across practice settings.

Objective

To evaluate whether a set of 3 implementation strategies facilitates increased adoption of a consensus recommendation to treat nonspine bone metastases with short-course RT (ie, ≤5 fractions).

Design, Setting, and Participants

This prospective, stepped-wedge, cluster randomized quality improvement study was conducted at 3 community-based cancer centers within an existing academic-community partnership. Rollout was initiated in 3-month increments between October 2021 and May 2022. Participants included treating physicians and patients receiving RT for nonspine bone metastases. Data analysis was performed from October 2022 to May 2023.

Exposures

Three implementation strategies-(1) dissemination of published consensus guidelines, (2) personalized audit-and-feedback reports, and (3) an email-based electronic consultation platform (eConsult)-were rolled out to physicians.

Main Outcomes and Measures

The primary outcome was adherence to the consensus recommendation of short-course RT for nonspine bone metastases. Mixed-effects logistic regression at the bone metastasis level was used to model associations between the exposure of physicians to the set of strategies (preimplementation vs postimplementation) and short-course RT, while accounting for patient and physician characteristics and calendar time, with a random effect for physician. Physician surveys were administered before implementation and after implementation to assess feasibility, acceptability, and appropriateness of each strategy.

Results

Forty-five physicians treated 714 patients (median [IQR] age at treatment start, 67 [59-75] years; 343 women [48%]) with 838 unique nonspine bone metastases during the study period. Implementing the set of strategies was not associated with use of short-course RT (odds ratio, 0.78; 95% CI, 0.45-1.34; P = .40), with unadjusted adherence rates of 53% (444 lesions) preimplementation vs 56% (469 lesions) postimplementation; however, the adjusted odds of adherence increased with calendar time (odds ratio, 1.68; 95% CI, 1.20-2.36; P = .003). All 3 implementation strategies were perceived as being feasible, acceptable, and appropriate; only the perception of audit-and-feedback appropriateness changed before vs after implementation (19 of 29 physicians [66%] vs 27 of 30 physicians [90%]; P = .03, Fisher exact test), with 20 physicians (67%) preferring reports quarterly.

Conclusions and Relevance

In this quality improvement study, a multicomponent set of implementation strategies was not associated with increased use of short-course RT within an academic-community partnership. However, practice improved with time, perhaps owing to secular trends or physician awareness of the study. Audit-and-feedback was more appropriate than anticipated. Findings support the need to investigate optimal approaches for promoting evidence-based radiation practice across settings.

Implementation of an audit and feedback module targeting low-value clinical practices in a provincial trauma quality assurance program: a cost-effectiveness study

BACKGROUND

Audit and Feedback (A&F) interventions based on quality indicators have been shown to lead to significant improvements in compliance with evidence-based care including de-adoption of low-value practices (LVPs). Our primary aim was to evaluate the cost-effectiveness of adding a hypothetical A&F module targeting LVPs for trauma admissions to an existing quality assurance intervention targeting high-value care and risk-adjusted outcomes. A secondary aim was to assess how certain A&F characteristics might influence its cost-effectiveness.

METHODS

We conducted a cost-effectiveness analysis using a probabilistic static decision analytic model in the Québec trauma care continuum. We considered the Québec Ministry of Health perspective. Our economic evaluation compared a hypothetical scenario in which the A&F module targeting LVPs is implemented in a Canadian provincial trauma quality assurance program to a status quo scenario in which the A&F module is not implemented. In scenarios analyses we assessed the impact of A&F characteristics on its cost-effectiveness. Results are presented in terms of incremental costs per LVP avoided.

RESULTS

Results suggest that the implementation of A&F module (Cost = $1,480,850; Number of LVPs = 6,005) is associated with higher costs and higher effectiveness compared to status quo (Cost = $1,124,661; Number of LVPs = 8,228). The A&F module would cost $160 per LVP avoided compared to status quo. The A&F module becomes more cost-effective with the addition of facilitation visits; more frequent evaluation; and when only high-volume trauma centers are considered.

CONCLUSION

A&F module targeting LVPs is associated with higher costs and higher effectiveness than status quo and has the potential to be cost-effective if the decision-makers' willingness-to-pay is at least $160 per LVP avoided. This likely represents an underestimate of true ICER due to underestimated costs or missed opportunity costs. Results suggest that virtual facilitation visits, frequent evaluation, and implementing the module in high-volume centers can improve cost-effectiveness.

EASY-NET Program: Effectiveness of an Audit and Feedback Intervention in the Emergency Care for Acute Conditions in the Lazio Region

The EASY-NET network program (NET-2016-02364191)-effectiveness of audit and feedback (A&F) strategies to improve health practice and equity in various clinical and organizational settings), piloted a novel and more structured A&F strategy. This study compared the effectiveness of the novel strategy against the sole periodic dissemination of indicators in enhancing the appropriateness and timeliness of emergency health interventions for patients diagnosed with acute myocardial infarction (AMI) and ischemic stroke in the Lazio Region. The efficacy of the intervention was assessed through a prospective quasi-experimental design employing a pre- and post-intervention (2021-2022) comparison with a control group. Participating hospitals in the Lazio Region, where professional teams voluntarily engaged in the intervention, constituted the exposed group, while the control group exclusively engaged in routine reporting activities. Effectiveness analysis was conducted at the patient level, utilizing regional health information systems to compute process and outcome indicators. The effectiveness of the intervention was evaluated using difference-in-difference models, comparing pre- and post-intervention periods between exposed and control groups. Estimates were calculated in terms of the difference in percentage points (PP) between absolute risks. Sixteen facilities for the AMI pathway and thirteen for the stroke pathway participated in the intervention. The intervention yielded a reduction in the proportion of 30-day readmissions following hospitalization for ischemic stroke by 0.54 pp in the exposed patients demonstrating a significant difference of -3.80 pp (95% CI: -6.57; -1.03; 5453 patients, 63.7% cases) in the exposed group compared to controls. However, no statistically significant differences attributable to the implemented A&F intervention were observed in other indicators considered. These results represent the first evidence in Italy of the impact of A&F interventions in an emergency setting, utilizing aggregated data from hospitals involved in the Lazio Region's emergency network.

Evaluation of SINERGIAPS, an intervention to improve patient safety in primary healthcare centers in Spain based on patients' perceptions and experiences: a protocol for a hybrid type I randomized clinical trial

Background

Adverse events in the primary care setting result in a direct cost equivalent to at least 2.5% of total healthcare spending. Across OECD countries, they lead to more than seven million avoidable hospital admissions annually. In this manuscript, we describe the protocol of a trial aimed at evaluating the effectiveness of SinergiAPS (a patient-centered audit and feedback intervention) in reducing avoidable hospital admission and explore the factors that may affect its implementation.

Methods

We will conduct a 24-month, parallel, open-label, multicenter, pragmatic, hybrid type 1 randomized clinical trial. 118 primary healthcare centers with wide geographical distribution in Spain will be randomly assigned (ratio 1:1) to two groups. The intervention group will receive two audits (baseline and intermediate at 12 months) based on information collected through the administration of the PREOS-PC questionnaire (a measure of patient-reported patient safety) to a convenience sample of 100 patients per center. The intervention group will receive reports on the results of both audits, along with educational resources aimed at facilitating the design and implementation of safety improvement plans. The control group will receive care as usual. The primary outcome will be the rate of avoidable hospitalizations (administrative data). Secondary outcomes: patient-reported patient safety experiences and outcomes (PREOS-PC questionnaire); patient safety culture as perceived by professionals (MOSPSC questionnaire); adverse events reported by healthcare professionals (ad hoc questionnaire); the number of safety improvement actions which the re has implemented (ad hoc questionnaire). Outcome data will be collected at baseline and 24 months follow-up. For the evaluation of the implementation of the SinergiAPS intervention, we will draw on the Consolidated Framework for Implementation Research (CFIR). We will collect and analyze qualitative and quantitative data (30 individual interviews, implementation logbooks; questionnaires for professionals from intervention centers, and level of use of the SinergiAPS web tool).

Discussion

This study will expand the scarce body of evidence existing regarding the effects and implementation of interventions aimed at promoting patient and family engagement in primary healthcare, specifically for enhancing patient safety. The study has the potential to produce an impact on clinical practice, healthcare systems, and population health.Clinical Trial Registration: https://clinicaltrials.gov/study/NCT05958108?term=sinergiAPS&rank=1 (NCT05958108).

Postpandemic Evaluation of the Eco-Efficiency of Personal Protective Equipment Against COVID-19 in Emergency Departments: Proposal for a Mixed Methods Study

BACKGROUND

The COVID-19 pandemic has had a profound impact on emergency department (ED) care in Canada and around the world. To prevent transmission of COVID-19, personal protective equipment (PPE) was required for all ED care providers in contact with suspected cases. With mass vaccination and improvements in several infection prevention components, our hypothesis is that the risks of transmission of COVID-19 will be significantly reduced and that current PPE use will have economic and ecological consequences that exceed its anticipated benefits. Evidence is needed to evaluate PPE use so that recommendations can ensure the clinical, economic, and environmental efficiency (ie, eco-efficiency) of its use.

OBJECTIVE

To support the development of recommendations for the eco-efficient use of PPE, our research objectives are to (1) estimate the clinical effectiveness (reduced transmission, hospitalizations, mortality, and work absenteeism) of PPE against COVID-19 for health care workers; (2) estimate the financial cost of using PPE in the ED for the management of suspected or confirmed COVID-19 patients; and (3) estimate the ecological footprint of PPE use against COVID-19 in the ED.

METHODS

We will conduct a mixed method study to evaluate the eco-efficiency of PPE use in the 5 EDs of the CHU de Québec-Université Laval (Québec, Canada). To achieve our goals, the project will include four phases: systematic review of the literature to assess the clinical effectiveness of PPE (objective 1; phase 1); cost estimation of PPE use in the ED using a time-driven activity-based costing method (objective 2; phase 2); ecological footprint estimation of PPE use using a life cycle assessment approach (objective 3; phase 3); and cost-consequence analysis and focus groups (integration of objectives 1 to 3; phase 4).

RESULTS

The first 3 phases have started. The results of these phases will be available in 2023. Phase 4 will begin in 2023 and results will be available in 2024.

CONCLUSIONS

While the benefits of PPE use are likely to diminish as health care workers' immunity increases, it is important to assess its economic and ecological impacts to develop recommendations to guide its eco-efficient use.

TRIAL REGISTRATION

PROSPERO CRD42022302598; https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=302598.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/50682.

Improving Knowledge of Audit and Feedback among Health Care Professionals in Sicily

Audit and Feedback (A&F) is considered one of the most significant tools for implementing continuous Quality Improvement (QI) in the healthcare field. The audit process is a structured inspection of professional practice against known standards or targets. The results of this inspection are subsequently feedback from professionals in order to implement an improvement process. The Italian Ministry of Health has recently funded the network project EASY-NET, with the main objective of evaluating the effectiveness of A&F strategies to improve healthcare practice and equity in various clinical and organizational settings in seven Italian regions. The Sicily region is represented within the EASY-NET project by the IRCCS Centro Neurolesi Bonino-Pulejo of Messina as the Work Package 7 (WP7). One of the objectives of the WP7 is to assess mechanisms and tools to enhance the effectiveness of A&F strategies. The purpose of this study is to investigate the influence that training interventions can have on improving knowledge of A&F strategies among healthcare professionals. The study employed a quasi-experimental design with a pretest-posttest evaluation strategy. The participants' initial knowledge of A&F strategies was evaluated through a baseline survey. Subsequently, the participants attended an online training workshop led by A&F experts, and a follow-up survey consisting of the same set of questions was conducted at the end of the process. Results showed statistically significant positive changes in the level of knowledge of A&F among participants following the training intervention. Furthermore, dividing the participants into two subgroups based on their professional background revealed significant differences in the level of knowledge of A&F methodologies between the observed categories of healthcare professionals. In conclusion, the study revealed that training interventions can be facilitators to implementing effective A&F programs.

Evaluating the effectiveness of a multifaceted intervention to reduce low-value care in adults hospitalized following trauma: a protocol for a pragmatic cluster randomized controlled trial

BACKGROUND

While simple Audit & Feedback (A&F) has shown modest effectiveness in reducing low-value care, there is a knowledge gap on the effectiveness of multifaceted interventions to support de-implementation efforts. Given the need to make rapid decisions in a context of multiple diagnostic and therapeutic options, trauma is a high-risk setting for low-value care. Furthermore, trauma systems are a favorable setting for de-implementation interventions as they have quality improvement teams with medical leadership, routinely collected clinical data, and performance-linked to accreditation. We aim to evaluate the effectiveness of a multifaceted intervention for reducing low-value clinical practices in acute adult trauma care.

METHODS

We will conduct a pragmatic cluster randomized controlled trial (cRCT) embedded in a Canadian provincial quality assurance program. Level I-III trauma centers (n = 30) will be randomized (1:1) to receive simple A&F (control) or a multifaceted intervention (intervention). The intervention, developed using extensive background work and UK Medical Research Council guidelines, includes an A&F report, educational meetings, and facilitation visits. The primary outcome will be the use of low-value initial diagnostic imaging, assessed at the patient level using routinely collected trauma registry data. Secondary outcomes will be low-value specialist consultation, low-value repeat imaging after a patient transfer, unintended consequences, determinants for successful implementation, and incremental cost-effectiveness ratios.

DISCUSSION

On completion of the cRCT, if the intervention is effective and cost-effective, the multifaceted intervention will be integrated into trauma systems across Canada. Medium and long-term benefits may include a reduction in adverse events for patients and an increase in resource availability. The proposed intervention targets a problem identified by stakeholders, is based on extensive background work, was developed using a partnership approach, is low-cost, and is linked to accreditation. There will be no attrition, identification, or recruitment bias as the intervention is mandatory in line with trauma center designation requirements, and all outcomes will be assessed with routinely collected data. However, investigators cannot be blinded to group allocation and there is a possibility of contamination bias that will be minimized by conducting intervention refinement only with participants in the intervention arm.

TRIAL REGISTRATION

This protocol has been registered on ClinicalTrials.gov (February 24, 2023, # NCT05744154 ).