Designing clinical practice feedback reports: three steps illustrated in Veterans Health Affairs long-term care facilities and programs

Impact of Clinician Feedback Reports on Antibiotic Use in Children Hospitalized With Community-acquired Pneumonia

BACKGROUND

Feedback reports summarizing clinician performance are effective tools for improving antibiotic use in the ambulatory setting, but the effectiveness of feedback reports in the hospital setting is unknown.

METHODS

Quasi-experimental study conducted between December 2021 and November 2023 within a pediatric health system measuring the impact of clinician feedback reports delivered by email and reviewed in a monthly meeting on appropriate antibiotic use in children hospitalized with community-acquired pneumonia (CAP). We used an interrupted time series analysis (ITSA) to estimate the immediate change and change over time in the proportion of CAP encounters adherent to validated metrics of antibiotic choice and duration, then used Poisson regression to estimate intervention effect as a risk ratio (RR).

RESULTS

Preintervention, 213 of 413 (52%) encounters received the appropriate antibiotic choice and duration, which increased to 308 of 387 (80%) postintervention. The ITSA demonstrated an immediate 18% increase in the proportion of CAP encounters receiving both the appropriate antibiotic choice and duration (95% confidence interval [CI]: 3-33%), with no further change over time (-0.3% per month, 95% CI: -2%-2%). In the Poisson model adjusted for age, sex, race, season, site, and intensive care unit admission, the intervention was associated with a 32% increase in the rate of appropriate antibiotic choice and duration (RR 1.32, 95% CI: 1.12-1.56, P < .01). No difference in length of stay or revisits were detected postintervention.

CONCLUSIONS

The intervention was associated with an increase in clinician adherence to antibiotic choice and duration recommendations for children hospitalized with CAP.

Improving audit and feedback: A user-centred approach to designing feedback techniques for an online experiment

Objective: Audit and feedback (A&F) programmes aim to improve patient care by providing summary data on performance to clinicians. They generally have modest, but variable, effects on patient care and questions remain about how best to provide performance feedback. It is not feasible to test all ways of providing feedback in 'real-world' randomised trials. Online screening experiments that screen feedback techniques prior to real-world evaluations of optimised versions offer a systematic approach. User-centred design methodologies can inform the design of such online experiments. Methods: We report the use of an innovative user-centred design approach to create feedback techniques for an online screening experiment and reflect on its usefulness. This approach included the involvement of patients and stakeholders. Results and Conclusion: We highlight lessons on ways to engage with partners, considering the feasibility of online A&F feedback delivery, fidelity, and usability. We demonstrate how the approach was implemented to co-create a set of feedback techniques for an online experiment and could also be applied to the design of other digital interventions.

Precision feedback: A conceptual model

Introduction

When performance data are provided as feedback to healthcare professionals, they may use it to significantly improve care quality. However, the question of how to provide effective feedback remains unanswered, as decades of evidence have produced a consistent pattern of effects-with wide variation. From a coaching perspective, feedback is often based on a learner's objectives and goals. Furthermore, when coaches provide feedback, it is ideally informed by their understanding of the learner's needs and motivation. We anticipate that a "coaching"-informed approach to feedback may improve its effectiveness in two ways. First, by aligning feedback with healthcare professionals' chosen goals and objectives, and second, by enabling large-scale feedback systems to use new types of data to learn what kind of performance information is motivating in general. Our objective is to propose a conceptual model of precision feedback to support these anticipated enhancements to feedback interventions.

Methods

We iteratively represented models of feedback's influence from theories of motivation and behavior change, visualization, and human-computer interaction. Through cycles of discussion and reflection, application to clinical examples, and software development, we implemented and refined the models in a software application to generate precision feedback messages from performance data for anesthesia providers.

Results

We propose that precision feedback is feedback that is prioritized according to its motivational potential for a specific recipient. We identified three factors that influence motivational potential: (1) the motivating information in a recipient's performance data, (2) the surprisingness of the motivating information, and (3) a recipient's preferences for motivating information and its visual display.

Conclusions

We propose a model of precision feedback that is aligned with leading theories of feedback interventions to support learning about the success of feedback interventions. We plan to evaluate this model in a randomized controlled trial of a precision feedback system that enhances feedback emails to anesthesia providers.

Communicate to Care: Implementing Health Literacy in a Pediatric Ears, Nose, and Throat Clinic

BACKGROUND

Despite positive outcomes in controlled trials, organizations have been slow to adopt health literacy practices. The purpose of the Communicate to CARE (Clear Communication, Achieve Understanding with Teach-Back, Receptive to our patient family needs, Empathetic care delivery) study was to use theories and strategies from implementation science to scale up health literacy practices in a pediatric Ears, Nose, and Throat (ENT) clinic.

BRIEF DESCRIPTION OF ACTIVITY

Expanding on previous efforts that simply reflected on barriers, the CARE team identified barriers within the local context pre-implementation to select strategies to directly address barriers during health literacy implementation. The RE-AIM framework was used to evaluate the reach, effectiveness, adoption, implementation, and maintenance of health literacy practices.

IMPLEMENTATION

Over 18 months, the CARE team delivered multiple implementation strategies, including external facilitator, microlessons, preparing champions, audit and feedback, local consensus discussions, and small test of change. We tailored health literacy practices to clinic team roles to accommodate the clinic workflow.

RESULTS

ENT team mean ratings on acceptability, appropriateness, and feasibility remained >4 indicating a high likelihood of successful implementation. Caregiver always ratings significantly increased from baseline to 12 months for easy-to-understand medication instructions (74%-96%), test results (54%-96%), know what to do if had questions (89%-96%), and encouraged to talk about health problems (76%-90%). Caregiver ratings dropped slightly at 18 months, indicating a need for booster training. While one third of caregivers reported Teach-Back practice across all time periods, the ENT team reported increased practice from baseline (42%), 6 (61%) and 12 months (70%).

LESSONS LEARNED

Over the first 12 months, the external facilitator delivered implementation strategies with weekly contact, tapering contact over the final 6 months. The local champion became engaged in the CARE study through a quality improvement project with meaningful outcomes for the clinic and an incentivization program for scholarly endeavors. Lunch and learn sessions helped build relationships between the CARE and ENT team to discuss and problem solve issues. The 5-item CAHPS health literacy composite proved to be sensitive to changes during implementation of health literacy practices. Integrating these items into standard follow up surveys with patients and families would help realize the return on investment for health literacy implementation. [HLRP: Health Literacy Research and Practice. 2024;8(3):e166-e174.].

Evaluating feedback reports to support documentation of veterans' care preferences in home based primary care

BACKGROUND

To evaluate the effectiveness of delivering feedback reports to increase completion of LST notes among VA Home Based Primary Care (HBPC) teams. The Life Sustaining Treatment Decisions Initiative (LSTDI) was implemented throughout the Veterans Health Administration (VHA) in the United States in 2017 to ensure that seriously ill Veterans have care goals and LST decisions elicited and documented.

METHODS

We distributed monthly feedback reports summarizing LST template completion rates to 13 HBPC intervention sites between October 2018 and February 2020 as the sole implementation strategy. We used principal component analyses to match intervention to 26 comparison sites and used interrupted time series/segmented regression analyses to evaluate the differences in LST template completion rates between intervention and comparison sites. Data were extracted from national databases for VA HBPC in addition to interviews and surveys in a mixed methods process evaluation.

RESULTS

LST template completion rose from 6.3 to 41.9% across both intervention and comparison HBPC teams between March 1, 2018, and February 26, 2020. There were no statistically significant differences for intervention sites that received feedback reports.

CONCLUSIONS

Feedback reports did not increase documentation of LST preferences for Veterans at intervention compared with comparison sites. Observed increases in completion rates across intervention and comparison sites can likely be attributed to implementation strategies used nationally as part of the national roll-out of the LSTDI. Our results suggest that feedback reports alone were not an effective implementation strategy to augment national implementation strategies in HBPC teams.

Developing an Audit and Feedback Dashboard for Family Physicians: User-Centered Design Process

BACKGROUND

Audit and feedback (A&F), the summary and provision of clinical performance data, is a common quality improvement strategy. Successful design and implementation of A&F-or any quality improvement strategy-should incorporate evidence-informed best practices as well as context-specific end user input.

OBJECTIVE

We used A&F theory and user-centered design to inform the development of a web-based primary care A&F dashboard. We describe the design process and how it influenced the design of the dashboard.

METHODS

Our design process included 3 phases: prototype development based on A&F theory and input from clinical improvement leaders; workshop with family physician quality improvement leaders to develop personas (ie, fictional users that represent an archetype character representative of our key users) and application of those personas to design decisions; and user-centered interviews with family physicians to learn about the physician's reactions to the revised dashboard.

RESULTS

The team applied A&F best practices to the dashboard prototype. Personas were used to identify target groups with challenges and behaviors as a tool for informed design decision-making. Our workshop produced 3 user personas, Dr Skeptic, Frazzled Physician, and Eager Implementer, representing common users based on the team's experience of A&F. Interviews were conducted to further validate findings from the persona workshop and found that (1) physicians were interested in how they compare with peers; however, if performance was above average, they were not motivated to improve even if gaps compared to other standards in their care remained; (2) burnout levels were high as physicians are trying to catch up on missed care during the pandemic and are therefore less motivated to act on the data; and (3) additional desired features included integration within the electronic medical record, and more up-to-date and accurate data.

CONCLUSIONS

We found that carefully incorporating data from user interviews helped operationalize generic best practices for A&F to achieve an acceptable dashboard that could meet the needs and goals of physicians. We demonstrate such a design process in this paper. A&F dashboards should address physicians' data skepticism, present data in a way that spurs action, and support physicians to have the time and capacity to engage in quality improvement work; the steps we followed may help those responsible for quality improvement strategy implementation achieve these aims.

Randomized controlled trial of tailored audit with feedback in VHA long-term care settings

BACKGROUND

The Long-Term Care QUERI program supported implementation of the Life-Sustaining Treatment Decisions Initiative in US Veterans Health Administration long-term care settings. The program worked with eleven Community Living Centers (CLCs) and twelve Home-Based Primary Care (HBPC) programs to increase rates of completed templates, using audit with feedback. We distributed monthly feedback reports to site champions showing the number of Veterans with appropriate documentation. Although feedback reports are a common implementation tool, little is known about the most effective ways to design, distribute, and support them. We sought to test tailoring reports with tips using site-specific data, as well as national comparator data.

METHODS

We conducted a cluster randomized controlled trial of monthly feedback reports utilizing site-tailored tips and national comparator data compared to our original feedback reports that included only graphical and numerical data. CLC and HBPC team members were invited to participate in brief surveys each quarter to determine if they had received and used the feedback reports. The outcome for CLC residents was the percent with a completed LST template any time prior to the 14th day of their stay. The outcome for HBPC residents was the percent of Veterans with a completed LST template by their second HBPC visit.

RESULTS

The response rate to the survey ranged between 6.8 and 19.3% of staff members across the CLC and HBPC sites with 12.8-25.5% of survey respondents reporting that they had seen the feedback reports. The linear regression models showed no significant association between receiving the enhanced feedback reports and having a higher documentation completion rate.

CONCLUSIONS

Receiving feedback reports tailored to sites by including tips based on baseline context assessments and qualitative findings, and reports showing national comparator data, did not have an impact on the number of Veterans with a completed LST template. Having a higher proportion of CLC or HBPC team members view the reports was not associated with an increase in LST template completion. These findings suggest that tailored audit with feedback may not have been effective at the program level, although the proportion of respondents who reported seeing the reports was small.

Characterizing the Gaps Between Best-Practice Implementation Strategies and Real-world Implementation: Qualitative Study Among Family Physicians Who Engaged With Audit and Feedback Reports

BACKGROUND

In Ontario, Canada, a government agency known as Ontario Health is responsible for making audit and feedback reports available to all family physicians to encourage ongoing quality improvement. The confidential report provides summary data on 3 key areas of practice: safe prescription, cancer screening, and diabetes management.

OBJECTIVE

This report was redesigned to improve its usability in line with evidence. The objective of this study was to explore how the redesign was perceived, with an emphasis on recipients' understanding of the report and their engagement with it.

METHODS

We conducted qualitative semistructured interviews with family physicians who had experience with both versions of the report recruited through purposeful and snowball sampling. We analyzed the transcripts following an emergent and iterative approach.

RESULTS

Saturation was reached after 17 family physicians participated. In total, 2 key themes emerged as factors that affected the perceived usability of the report: alignment between the report and the recipients' expectations and capacity to engage in quality improvement. Family physicians expected the report and its quality indicators to reflect best practices and to be valid and accurate. They also expected the report to offer feedback on the clinical activities they perceived to be within their control to change. Furthermore, family physicians expected the goal of the report to be aligned with their perspective on feasible quality improvement activities. Most of these expectations were not met, limiting the perceived usability of the report. The capacity to engage with audit and feedback was hindered by several organizational and physician-level barriers, including the lack of fit with the existing workflow, competing priorities, time constraints, and insufficient skills for bridging the gaps between their data and the corresponding desired actions.

CONCLUSIONS

Despite recognized improvements in the design of the report to better align with best practices, it was not perceived as highly usable. Improvements in the presentation of the data could not overcome misalignment with family physicians' expectations or the limited capacity to engage with the report. Integrating iterative evaluations informed by user-centered design can complement evidence-based guidance for implementation strategies. Creating a space for bringing together audit and feedback designers and recipients may help improve usability and effectiveness.

Coping with ‘the grey area’ of antibiotic prescribing: a theory-informed qualitative study exploring family physician perspectives on antibiotic prescribing

Background

Unnecessary antibiotic use is associated with adverse side effects and rising rates of resistance at the individual and population level. This study used a theory-informed approach to identify potentially modifiable determinants of antibiotic prescribing for patients presenting to primary care with upper respiratory tract infection symptoms.

Methods

Qualitative interviews were conducted with primary care physicians in Ontario, Canada who were identified as medium- or high-volume antibiotic prescribers (high volume defined as top 20^th percentile versus “medium” defined as 40^th to 60^th percentile). The interview guide and analysis were informed by the Theoretical Domains Framework. Each interview was coded by two research team members. Sampling and analysis continued until thematic saturation was achieved.

Results

Twenty family physicians were interviewed. Physicians felt that many decisions about prescribing for upper respiratory tract infection symptoms were straightforward (i.e., black and white). However, intention to avoid prescribing in cases where an antibiotic was not indicated clinically did not always align with the provider action or expectation of the patient. Clinical decisions were influenced by the Theoretical Domain Framework domains that were both internal to the physician (Knowledge, Skills, Social/Professional Role, and Belief about Capabilities) and external to the physician (Social Influence, Belief about Consequences, Reinforcement, Emotions, and Behavioural Regulation). The Environmental Context and Resources played a key role. Physicians reported significant differences in their approach to antibiotic prescribing within episodic (walk-in) or continuity of care settings, as the presence (or not) of longitudinal physician–patient relationships seemed to moderate the role of these factors on the decision-making process in cases of uncertainty.

Conclusions

Antibiotic prescribing in primary care is a complex decision-making process in which context may outweigh biology during encounters featuring clinical uncertainty. Differential skill in handling uncertainty and tactics used to operationalize guideline recommendations in the real world seems to contribute to observed variation in prescribing patterns, as much or more than differences in knowledge of best practices.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12875-022-01806-8.

'Give me something meaningful': GPs perspectives on how to improve an audit and feedback report provided by health insurers - an exploratory qualitative study

BACKGROUND

Audit and feedback (A&F) is a valuable quality improvement strategy, which can contribute to de-implementation of low-value care. In the Netherlands, all health insurers collaboratively provide A&F to general practitioners (GPs), the 'Primary Care Practice Report' (PCPR). Unfortunately, the use of this report by GPs is limited. This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR.

METHOD

We used an interpretative qualitative design, with think-aloud tasks to uncover thoughts of GPs on the usability of the PCPR and semistructured interview questions to ask GPs' recommendations for improvement of the PCPR. Interviews were audiorecorded and transcribed ad verbatim. Data were analysed using thematic content analysis.

RESULTS

We identified two main themes: 'poor usability of the PCPR', and 'minimal motivation to change based on the PCPR'. The GPs found the usability of the PCPR poor due to the feedback not being clinically meaningful, the data not being recent, individual and reliable, the performance comparators offer insufficient guidance to assess clinical performance, the results are not discussed with peers and the definitions and visuals are unclear. The GPs recommended improving these issues. The GPs motivation to change based on the PCPR was minimal.

CONCLUSIONS

The GPs evaluated the PCPR as poorly usable and were minimally motivated to change. The PCPR seems developed from the perspective of the reports' commissioners, health insurers, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback could contribute to their motivation to improve clinical performance.Furthermore, the GPs stated that they receive a multitude of A&F reports, which they hardly use. Thus, we see a need for policy makers to invest in less, but more usable A&F reports.

Evaluating implementation strategies to support documentation of veterans' care preferences

OBJECTIVE

To evaluate the effectiveness of feedback reports and feedback reports + external facilitation on completion of life-sustaining treatment (LST) note the template and durable medical orders. This quality improvement program supported the national roll-out of the Veterans Health Administration (VA) LST Decisions Initiative (LSTDI), which aims to ensure that seriously-ill veterans have care goals and LST decisions elicited and documented.

DATA SOURCES

Primary data from national databases for VA nursing homes (called Community Living Centers [CLCs]) from 2018 to 2020.

STUDY DESIGN

In one project, we distributed monthly feedback reports summarizing LST template completion rates to 12 sites as the sole implementation strategy. In the second involving five sites, we distributed similar feedback reports and provided robust external facilitation, which included coaching, education, and learning collaboratives. For each project, principal component analyses matched intervention to comparison sites, and interrupted time series/segmented regression analyses evaluated the differences in LSTDI template completion rates between intervention and comparison sites.

DATA COLLECTION METHODS

Data were extracted from national databases in addition to interviews and surveys in a mixed-methods process evaluation.

PRINCIPAL FINDINGS

LSTDI template completion rose from 0% to about 80% throughout the study period in both projects' intervention and comparison CLCs. There were small but statistically significant differences for feedback reports alone (comparison sites performed better, coefficient estimate 3.48, standard error 0.99 for the difference between groups in change in trend) and feedback reports + external facilitation (intervention sites performed better, coefficient estimate -2.38, standard error 0.72).

CONCLUSIONS

Feedback reports + external facilitation was associated with a small but statistically significant improvement in outcomes compared with comparison sites. The large increases in completion rates are likely due to the well-planned national roll-out of the LSTDI. This finding suggests that when dissemination and support for widespread implementation are present and system-mandated, significant enhancements in the adoption of evidence-based practices may require more intensive support.

Preferences Elicited and Respected for Seriously Ill Veterans through Enhanced Decision-Making (PERSIVED): a protocol for an implementation study in the Veterans Health Administration

BACKGROUND

Empirical evidence supports the use of structured goals of care conversations and documentation of life-sustaining treatment (LST) preferences in durable, accessible, and actionable orders to improve the care for people living with serious illness. As the largest integrated healthcare system in the USA, the Veterans Health Administration (VA) provides an excellent environment to test implementation strategies that promote this evidence-based practice. The Preferences Elicited and Respected for Seriously Ill Veterans through Enhanced Decision-Making (PERSIVED) program seeks to improve care outcomes for seriously ill Veterans by supporting efforts to conduct goals of care conversations, systematically document LST preferences, and ensure timely and accurate communication about preferences across VA and non-VA settings.

METHODS

PERSIVED encompasses two separate but related implementation projects that support the same evidence-based practice. Project 1 will enroll 12 VA Home Based Primary Care (HBPC) programs and Project 2 will enroll six VA Community Nursing Home (CNH) programs. Both projects begin with a pre-implementation phase during which data from diverse stakeholders are gathered to identify barriers and facilitators to adoption of the LST evidence-based practice. This baseline assessment is used to tailor quality improvement activities using audit with feedback and implementation facilitation during the implementation phase. Site champions serve as the lynchpin between the PERSIVED project team and site personnel. PERSIVED teams support site champions through monthly coaching sessions. At the end of implementation, baseline site process maps are updated to reflect new steps and procedures to ensure timely conversations and documentation of treatment preferences. During the sustainability phase, intense engagement with champions ends, at which point champions work independently to maintain and improve processes and outcomes. Ongoing process evaluation, guided by the RE-AIM framework, is used to monitor Reach, Adoption, Implementation, and Maintenance outcomes. Effectiveness will be assessed using several endorsed clinical metrics for seriously ill populations.

DISCUSSION

The PERSIVED program aims to prevent potentially burdensome LSTs by consistently eliciting and documenting values, goals, and treatment preferences of seriously ill Veterans. Working with clinical operational partners, we will apply our findings to HBPC and CNH programs throughout the national VA healthcare system during a future scale-out period.

Usability-Focused Development and Usage of NeoTree-Beta, an App for Newborn Care in a Low-Resource Neonatal Unit, Malawi

Background

Neonatal mortality is high in low-resource settings. NeoTree is a digital intervention for neonatal healthcare professionals (HCPs) aiming to achieve data-driven quality improvement and improved neonatal survival in low-resource hospitals. Optimising usability with end-users could help digital health interventions succeed beyond pilot stages in low-resource settings. Usability is the quality of a user's experience when interacting with an intervention, encompassing their effectiveness, efficiency, and overall satisfaction.

Objective

To evaluate the usability and usage of NeoTree beta-app and conduct Agile usability-focused intervention development.

Method

A real-world pilot of NeoTree beta-app was conducted over 6 months at Kamuzu Central Hospital neonatal unit, Malawi. Prior to deployment, think-aloud interviews were conducted to guide nurses through the app whilst voicing their thoughts aloud (n = 6). System Usability Scale (SUS) scores were collected before the implementation of NeoTree into usual clinical care and 6 months after implementation (n = 8 and 8). During the pilot, real-world user-feedback and user-data were gathered. Feedback notes were subjected to thematic analysis within an Agile “product backlog.” For usage, number of users, user-cadre, proportion of admissions/outcomes recorded digitally, and median app-completion times were calculated.

Results

Twelve overarching usability themes generated 57 app adjustments, 39 (68%) from think aloud analysis and 18 (32%) from the real-world testing. A total of 21 usability themes/issues with corresponding app features were produced and added to the app. Six themes relating to data collection included exhaustiveness of data schema, prevention of errors, ease of progression, efficiency of data entry using shortcuts, navigation of user interface (UI), and relevancy of content. Six themes relating to the clinical care included cohesion with ward process, embedded education, locally coherent language, adaptability of user-interface to available resources, and printout design to facilitate handover. SUS scores were above average (88.1 and 89.4 at 1 and 6 months, respectively). Ninety-three different HCPs of 5 cadres, recorded 1,323 admissions and 1,197 outcomes over 6 months. NeoTree achieved 100% digital coverage of sick neonates admitted. Median completion times were 16 and 8 min for admissions and outcomes, respectively.

Conclusions

This study demonstrates optimisation of a digital health app in a low-resource setting and could inform other similar usability studies apps in similar settings.

A Scalable Service to Improve Health Care Quality Through Precision Audit and Feedback: Proposal for a Randomized Controlled Trial

BACKGROUND

Health care delivery organizations lack evidence-based strategies for using quality measurement data to improve performance. Audit and feedback (A&F), the delivery of clinical performance summaries to providers, demonstrates the potential for large effects on clinical practice but is currently implemented as a blunt one size fits most intervention. Each provider in a care setting typically receives a performance summary of identical metrics in a common format despite the growing recognition that precisionizing interventions hold significant promise in improving their impact. A precision approach to A&F prioritizes the display of information in a single metric that, for each recipient, carries the highest value for performance improvement, such as when the metric's level drops below a peer benchmark or minimum standard for the first time, thereby revealing an actionable performance gap. Furthermore, precision A&F uses an optimal message format (including framing and visual displays) based on what is known about the recipient and the intended gist meaning being communicated to improve message interpretation while reducing the cognitive processing burden. Well-established psychological principles, frameworks, and theories form a feedback intervention knowledge base to achieve precision A&F. From an informatics perspective, precision A&F requires a knowledge-based system that enables mass customization by representing knowledge configurable at the group and individual levels.

OBJECTIVE

This study aims to implement and evaluate a demonstration system for precision A&F in anesthesia care and to assess the effect of precision feedback emails on care quality and outcomes in a national quality improvement consortium.

METHODS

We propose to achieve our aims by conducting 3 studies: a requirements analysis and preferences elicitation study using human-centered design and conjoint analysis methods, a software service development and implementation study, and a cluster randomized controlled trial of a precision A&F service with a concurrent process evaluation. This study will be conducted with the Multicenter Perioperative Outcomes Group, a national anesthesia quality improvement consortium with >60 member hospitals in >20 US states. This study will extend the Multicenter Perioperative Outcomes Group quality improvement infrastructure by using existing data and performance measurement processes.

RESULTS

The proposal was funded in September 2021 with a 4-year timeline. Data collection for Aim 1 began in March 2022. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024.

CONCLUSIONS

The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. By implementing and evaluating a demonstration system for precision feedback, we create the potential to observe the conditions under which feedback interventions are effective.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/34990.

Opportunities to improve the impact of two national clinical audit programmes: a theory-guided analysis

Background

Audit and feedback is widely used in healthcare improvement, with evidence of modest yet potentially important effects upon professional practice. There are approximately 60 national clinical audit programmes in the UK. These programmes often develop and adapt new ways of delivering feedback to optimise impacts on clinical practice. Two such programmes, the National Diabetes Audit (NDA) and the Trauma Audit Research Network (TARN), recently introduced changes to their delivery of feedback. We assessed the extent to which the design of these audit programmes and their recent changes were consistent with best practice according to the Clinical Performance Feedback Intervention Theory (CP-FIT). This comprehensive framework specifies how variables related to the feedback itself, the recipient, and the context operate via explanatory mechanisms to influence feedback success.

Methods

We interviewed 19 individuals with interests in audit and feedback, including researchers, audit managers, healthcare staff, and patient and public representatives. This range of expert perspectives enabled a detailed exploration of feedback from the audit programmes. We structured interviews around the CP-FIT feedback cycle and its component processes (e.g. Data collection and analysis, Interaction). Our rapid analytic approach explored the extent to which both audits applied features consistent with CP-FIT.

Results

Changes introduced by the audit programmes were consistent with CP-FIT. Specifically, the NDA’s increased frequency of feedback augmented existing strengths, such as automated processes (CP-FIT component: Data collection and analysis) and being a credible source of feedback (Acceptance). TARN’s new analytic tool allowed greater interactivity, enabling recipients to interrogate their data (Verification; Acceptance). We also identified scope for improvement in feedback cycles, such as targeting of feedback recipients (Interaction) and feedback complexity (Perception) for the NDA and specifying recommendations (Intention) and demonstrating impact (Clinical performance improvement) for TARN.

Conclusions

The changes made by the two audit programmes appear consistent with suggested best practice, making clinical improvement more likely. However, observed weaknesses in the feedback cycle may limit the benefits of these changes. Applying CP-FIT via a rapid analysis approach helps identify strengths and remediable weaknesses in the design of audit programmes that can be shared with them in a timely manner.

Performance of a Supervisor Observational Coding System and an Audit and Feedback Intervention

Workplace-based clinical supervision is common in community based mental health care for youth and families and could be a leveraged to scale and improve the implementation of evidence-based treatment (EBTs). Accurate methods are needed to measure, monitor, and support supervisor performance with limited disruption to workflow. Audit and Feedback (A&F) interventions may offer some promise in this regard. The study-a randomized controlled trial with 60 clinical supervisors measured longitudinally for 7 months-had two parts: (1) psychometric evaluation of an observational coding system for measuring adherence and competence of EBT supervision and (2) evaluation of an experimental Supervisor Audit and Feedback (SAF) intervention on outcomes of supervisor adherence and competence. All supervisors recorded and uploaded weekly supervision sessions for 7 months, and those in the experimental condition were provided a single, monthly web-based feedback report. Psychometric performance was evaluated using measurement models based in Item Response Theory, and the effect of the SAF intervention was evaluated using mixed-effects regression models. The observational instrument performed well across psychometric indicators of dimensionality, rating scale functionality, and item fit; however, coder reliability was lower for competence than for adherence. Statistically significant A&F effects were largely in the expected directions and consistent with hypotheses. The observational coding system performed well, and a monthly electronic feedback report showed promise in maintaining or improving community-based clinical supervisors' adherence and, to a lesser extent, competence. Limitations discussed include unknown generalizability to the supervision of other EBTs.

Applying the Tailored Implementation in Chronic Diseases framework to inform implementation of the Preferences Elicited and Respected for Seriously Ill Veterans through enhanced decision-making program in the United States Veterans Health Administration

In 2017, the National Center for Ethics in Health Care for the United States Department of Veterans Affairs (VA) commenced national roll-out of the Life-Sustaining Treatment Decisions Initiative. This national VA initiative aimed to promote personalized, proactive, patient-driven care for seriously ill Veterans by documenting Veterans' goals and preferences for life-sustaining treatments in a durable electronic health record note template known as the life-sustaining treatment template. The Preferences Elicited and Respected for Seriously Ill Veterans through Enhanced Decision-Making (PERSIVED) quality improvement program was created to address the high variation in life-sustaining treatment template completion in VA Home Based Primary Care (HBPC) and Community Nursing Home programs. This manuscript describes the program that focuses on improving life sustaining treatment template completion rates amongst HBPC programs. To increase life-sustaining treatment template completion for Veterans receiving care from HBPC programs, the PERSIVED team applies two implementation strategies: audit with feedback and implementation facilitation. The PERSIVED team conducts semi-structured interviews, needs assessments, and process mapping with HBPC programs in order to identify barriers and facilitators to life-sustaining treatment template completion to inform tailored facilitation. Our interview data is analyzed using the Tailored Implementation in Chronic Diseases (TICD) framework, which identifies 57 determinants that might influence practice or implementation of interventions. To quickly synthesize and use baseline data to inform the tailored implementation plan, we adapted a rapid analysis process for our purposes. This paper describes a six-step process for conducting and analyzing baseline interviews through applying the TICD that can be applied and adapted by implementation scientists to rapidly inform tailoring of implementation facilitation.

Partnered innovation to implement timely and personalized care: A case study

BACKGROUND

Understanding how to translate research discoveries into solutions for healthcare improvement is a priority of NIH-funded Clinical and Translational Science Awards (CTSA). This study, supported by one CTSA, aims to capture one process of shaping and implementing innovations to advance the timeliness and patient-centeredness of cardiovascular care. Specifically, we sought to understand a partnership between a private digital health startup company, a university innovation lab, and an academic health system's cardiology program pursuing this goal.

FINDINGS

The collaboration proceeded through clear phases to address the questions and challenges: problem definition, exploration and formalization of the partnership, innovation co-creation and pilot test, and scale-up planning. Phases were punctuated by key decisions, such as forming the partnership, negotiating terms of the partnership, iterating form and features of the innovation, and exploring sufficiency of its value-add for scale-up and sustainment. Key implementation concepts were apparent, including implementation strategies (e.g., champions and iterative trialing) and the implementation outcomes of acceptability, sustainment, and scale-up. Participants identified potential risks of collaboration, reflected on their co-creation process, and the value of engaging stakeholders in innovation design. Findings may inform subsequent collaborations between innovators and translational researchers.

METHODS

We conducted a case study to understand the partnership; characterize the questions they pursued, their decision points, information and data sources; and identify the challenges and risks. Data were collected through a series of four focus groups with members of each partnering organization. A transdisciplinary research team iteratively worked to condense and synthesize data from audio recorded transcripts into a case narrative.

Small and Large Animal Veterinarian Perceptions of Antimicrobial Use Metrics for Hospital-Based Stewardship in the United States

Background: Robust measurement and tracking of antimicrobial use (AMU) is a fundamental component of stewardship interventions. Feeding back AMU metrics to individual clinicians is a common approach to changing prescribing behavior. Metrics must be meaningful and comprehensible to clinicians. Little is known about how veterinary clinicians working in the United States (US) hospital setting think about AMU metrics for antimicrobial stewardship. Objective: To identify hospital-based veterinary clinicians' attitudes toward audit and feedback of AMU metrics, their perceptions of different AMU metrics, and their response to receiving an individualized prescribing report. Methods: Semi-structured interviews were conducted with veterinarians working at two hospitals in the Eastern US. Interviews elicited perceptions of antimicrobial stewardship in veterinary medicine. Respondents were shown a personalized AMU Report characterizing their prescribing patterns relative to their peers and were asked to respond. Interviews were recorded, transcribed, and analyzed using the framework method with matrices. Results: Semi-structured interviews were conducted with 34 veterinary clinicians (22 small animal and 12 large animal). Respondents generally felt positive about the reports and were interested in seeing how their prescribing compared to that of their peers. Many respondents expressed doubt that the reports accurately captured the complexities of their prescribing decisions and found metrics associated with animal daily doses (ADDs) confusing. Only 13 (38.2%) respondents felt the reports would change how they used antimicrobials. When asked how the impact of the reports could be optimized, respondents recommended providing a more detailed explanation of how the AMU metrics were derived, education prior to report roll-out, guidance on how to interpret the metrics, and development of meaningful benchmarks for goal-setting. Conclusions: These findings provide important insight that can be used to design veterinary-specific AMU metrics as part of a stewardship intervention that are meaningful to clinicians and more likely to promote judicious prescribing.