Testing Behavior Change Techniques to Encourage Primary Care Physicians to Access Cancer Screening Audit and Feedback Reports: Protocol for a Factorial Randomized Experiment of Email Content

Developing an Audit and Feedback Dashboard for Family Physicians: User-Centered Design Process

BACKGROUND

Audit and feedback (A&F), the summary and provision of clinical performance data, is a common quality improvement strategy. Successful design and implementation of A&F-or any quality improvement strategy-should incorporate evidence-informed best practices as well as context-specific end user input.

OBJECTIVE

We used A&F theory and user-centered design to inform the development of a web-based primary care A&F dashboard. We describe the design process and how it influenced the design of the dashboard.

METHODS

Our design process included 3 phases: prototype development based on A&F theory and input from clinical improvement leaders; workshop with family physician quality improvement leaders to develop personas (ie, fictional users that represent an archetype character representative of our key users) and application of those personas to design decisions; and user-centered interviews with family physicians to learn about the physician's reactions to the revised dashboard.

RESULTS

The team applied A&F best practices to the dashboard prototype. Personas were used to identify target groups with challenges and behaviors as a tool for informed design decision-making. Our workshop produced 3 user personas, Dr Skeptic, Frazzled Physician, and Eager Implementer, representing common users based on the team's experience of A&F. Interviews were conducted to further validate findings from the persona workshop and found that (1) physicians were interested in how they compare with peers; however, if performance was above average, they were not motivated to improve even if gaps compared to other standards in their care remained; (2) burnout levels were high as physicians are trying to catch up on missed care during the pandemic and are therefore less motivated to act on the data; and (3) additional desired features included integration within the electronic medical record, and more up-to-date and accurate data.

CONCLUSIONS

We found that carefully incorporating data from user interviews helped operationalize generic best practices for A&F to achieve an acceptable dashboard that could meet the needs and goals of physicians. We demonstrate such a design process in this paper. A&F dashboards should address physicians' data skepticism, present data in a way that spurs action, and support physicians to have the time and capacity to engage in quality improvement work; the steps we followed may help those responsible for quality improvement strategy implementation achieve these aims.

The preparation phase in the multiphase optimization strategy (MOST): a systematic review and introduction of a reporting checklist

Multicomponent behavioral interventions developed using the multiphase optimization strategy (MOST) framework offer important advantages over alternative intervention development models by focusing on outcomes within constraints relevant for effective dissemination. MOST consists of three phases: preparation, optimization, and evaluation. The preparation phase is critical to establishing the foundation for the optimization and evaluation phases; thus, detailed reporting is critical to enhancing rigor and reproducibility. A systematic review of published research using the MOST framework was conducted. A structured framework was used to describe and summarize the use of MOST terminology (i.e., preparation phase and optimization objective) and the presentation of preparation work, the conceptual model, and the optimization. Fifty-eight articles were reviewed and the majority focused on either describing the methodology or presenting results of an optimization trial (n = 38, 66%). Although almost all articles identified intervention components (96%), there was considerable variability in the degree to which authors fully described other elements of MOST. In particular, there was less consistency in use of MOST terminology. Reporting on the MOST preparation phase is varied, and there is a need for increased focus on explicit articulation of key design elements and rationale of the preparation phase. The proposed checklist for reporting MOST studies would significantly advance the use of this emerging methodology and improve implementation and dissemination of MOST. Accurate reporting is essential to reproducibility and rigor of scientific trials as it ensures future research fully understands not only the methodology, but the rationale for intervention and optimization decisions.

Audit feedback interventions to address high-risk prescriptions in long-term care homes: a costing study and return on investment analysis

BACKGROUND

Audit and feedback is a common implementation strategy, but few studies describe its costs. 'MyPractice' is a province-wide audit and feedback initiative to improve prescribing in nursing homes. This study sought to estimate the costs of 'MyPractice' and assess whether the financial benefit of 'MyPractice' offsets those costs.

METHODS

We conducted a costing study from the perspective of the Ontario government. Total cost of 'MyPractice' was calculated as the sum of the costs of producing and disseminating the reports (covering three report releases) which were obtained from Ontario Health staff interviews and document reviews. Return on investment (ROI) was calculated as the ratio of net cost-savings and the intervention cost. Cost savings were based on the effectiveness of 'MyPractice' derived from a published cohort study. Cost-savings attributable to 'MyPractice' were estimated from the changes in the rates of antipsychotics over time between physicians who signed up and viewed the reports and those who did not sign up to the reports.

RESULTS

Total intervention costs were C$223,691 (C$838 per physician and C$74,564 per release). Costs incurred during the development phase accounted for 74% of the total cost (C$166,117), while implementation costs for three report releases were responsible for 26% of the total costs (C$57,575). The ROI for every C$1 spent on the 'MyPractice' intervention was 1.02 (95% CI 0.51, 1.93) for three report releases.

CONCLUSION

'MyPractice' report offers a good return on investment and the value for money could improve with greater number of report releases.

Engagement of General Practice in an Australian Organised Bowel Cancer Screening Program: A Cross-Sectional Survey of Knowledge and Practice

Background:

Understanding factors causing variation in family physicians/general practitioners (GPs) screening knowledge, understanding and support of organised population-based colorectal cancer (CRC) programs can direct interventions that maximise the influence of a CRC screening recommendation from a GP. This study aims to assess contextual factors that influence knowledge and quality improvement (QI) practice directed to CRC screening in Australian general practice.

Methods:

A convenience sample of anonymous general practice staff from all Australian states and territories completed a web-based survey. Multivariate analyses assessed the association between CRC screening knowledge and QI-CRC practice scores and patient, organisational and environmental-level contextual factors.

Results:

Of 1,013 survey starts, 918 respondents (90.6%) completed the survey. Respondents less likely to recommend FOBT screening had lower knowledge and QI practice scores directed to CRC screening. Controlling for individual and practice characteristics, respondents’ rating of the Australian National Bowel Cancer Screening Program (NBCSP) support for preventive care, attending external education, and sufficient practice resources to implement QI practice (generally) were the strongest factors associated with QI practice directed towards CRC screening. Knowledge scores were less amenable to the influence of contextual factors explored.

Conclusion:

More active engagement of family medicine/general practice to improve screening promotion could be achieved through better QI resourcing without changing the fundamental design of population-based CRC screening programs.

Effectiveness of confidential reports to physicians on their prescribing of antipsychotic medications in nursing homes

BACKGROUND

Antipsychotic medication use in nursing homes is associated with potential for harms. In Ontario, Canada, an agency of the provincial government offers nursing home physicians quarterly audit and feedback on their antipsychotic prescribing. We compared the characteristics of physicians who did and did not engage with the intervention, and assessed early changes in prescribing.

METHODS

This population-level, retrospective cohort study used linked administrative databases to track prescribing practices in nursing homes pre-intervention (baseline), immediately post-initiative (3 months), and at follow-up (6 months). Exposure variables identified whether a physician signed up to participate (or not) or viewed the feedback following sign up (or not). Differences in the proportion of days that residents received antipsychotic medications at 6 months compared to baseline by exposure(s) were assessed using a linear mixed effects regression analysis to adjust for a range of resident, physician, and nursing home factors. Benzodiazepine and statin prescribing were assessed as a balance and tracer measures, respectively.

RESULTS

Of 944 eligible physicians, 210 (22.3%) signed up to recieve the feedback report and 132 (13.9%) viewed their feedback. Physicians who signed up for feedback were more likely to have graduated from a Canadian medical school, work in urban nursing homes, and care for a larger number of residents. The clinical and functional characteristics of residents were similar across physician exposure groups. At 6 months, antipsychotic prescribing had decreased in all exposure groups. Those who viewed their feedback report had a signicantly greater reduction in antipsychotic prescribing than those who did not sign up (0.94% patient-days exposed; 95% CI 0.35 to 1.54%, p = 0.002). Trends in prescribing patterns across exposure groups for benzodiazepines and statins were not statistically significant.

INTERPRETATION

Almost a quarter of eligible physicians engaged early in a voluntary audit and feedback intervention related to antipsychotic prescribing in nursing homes. Those who viewed their feedback achieved a small but statistically significant change in prescribing, equivalent to approximately 14,000 fewer days that nursing home residents received antipsychotic medications over 6 months. This study adds to the literature regarding the role of audit and feedback interventions to improve quality of care.

Testing e-mail content to encourage physicians to access an audit and feedback tool: a factorial randomized experiment

Background

In Ontario, an online audit and feedback tool that provides primary care physicians with detailed information about patients who are overdue for cancer screening is underused. In the present study, we aimed to examine the effect of messages operationalizing 3 behaviour change techniques on access to the audit and feedback tool and on cancer screening rates.

Methods

During May-September 2017, a pragmatic 2×2×2 factorial experiment tested 3 behaviour change techniques: anticipated regret, material incentive, and problem-solving. Outcomes were assessed using routinely collected administrative data. A qualitative process evaluation explored how and why the e-mail messages did or did not support Screening Activity Report access.

Results

Of 5449 primary care physicians randomly allocated to 1 of 8 e-mail messages, fewer than half opened the messages and fewer than 1 in 10 clicked through the messages. Messages with problem-solving content were associated with a 12.9% relative reduction in access to the tool (risk ratio: 0.871; 95% confidence interval: 0.791 to 0.958; p = 0.005), but a 0.3% increase in cervical cancer screening (rate ratio: 1.003; 95% confidence interval: 1.001 to 1.006; p = 0.003). If true, that association would represent 7568 more patients being screened. No other significant effects were observed.

Conclusions

For audit and feedback to work, recipients must engage with the data; for e-mail messages to prompt activity, recipients must open and review the message content. This large factorial experiment demonstrated that small changes in the content of such e-mail messages might influence clinical behaviour. Future research should focus on strategies to make cancer screening more user-centred.

Designing Emails Aimed at Increasing Family Physicians' Use of a Web-Based Audit and Feedback Tool to Improve Cancer Screening Rates: Cocreation Process

BACKGROUND

Providing clinical performance data to health professionals, a process known as audit and feedback, can play an important role in health system improvement. However, audit and feedback tools can only be effective if the targeted health professionals access and actively review their data. Email is used by Cancer Care Ontario, a provincial cancer agency, to promote access to a Web-based audit and feedback tool called the Screening Activity Report (SAR); however, current emails that lack behavior change content have been ineffective at encouraging log-in to the SAR.

OBJECTIVE

The objective of our study was to describe the process and experience of developing email content that incorporates user input and behavior change techniques (BCTs) to promote the use of the SAR among Ontario primary care providers.

METHODS

Our interdisciplinary research team first identified BCTs shown to be effective in other settings that could be adapted to promote use of the SAR. We then developed draft BCT-informed email content. Next, we conducted cocreation workshops with physicians who had logged in to the SAR more than once over the past year. Participants provided reactions to researcher-developed BCT-informed content and helped to develop an email that they believed would prompt their colleagues to use the SAR. Content from cocreation workshops was brought to focus groups with physicians who had not used the SAR in the past year. We analyzed notes from the cocreation workshops and focus groups to inform decisions about content. Finally, 8 emails were created to test BCT-informed content in a 2×2×2 factorial randomized experiment.

RESULTS

We identified 3 key tensions during the development of the email that required us to balance user input with scientific evidence, organizational policies, and our scientific objectives, which are as follows: conflict between user preference and scientific evidence, privacy constraints around personalizing unencrypted emails with performance data, and using cocreation methods in a study with the objective of developing an email that featured BCT-informed content.

CONCLUSIONS

Teams tasked with developing content to promote health professional engagement with audit and feedback or other quality improvement tools might consider cocreation processes for developing communications that are informed by both users and BCTs. Teams should be cautious about making decisions solely based on user reactions because what users seem to prefer is not always the same as what works. Furthermore, implementing user recommendations may not always be feasible. Teams may face challenges when using cocreation methods to develop a product with the simultaneous goal of having clearly defined variables to test in later studies. The expected role of users, evidence, and the implementation context all warrant consideration to determine whether and how cocreation methods could help to achieve design and scientific objectives.