The Calgary Audit and Feedback Framework: a practical, evidence-informed approach for the design and implementation of socially constructed learning interventions using audit and group feedback

Navigating discourses of feedback: developing a pattern system of feedback

Although feedback is often presented as if it were a well-understood concept in health professions education, in practice it can mean many things. For some, feedback is a conversation about defining and improving performance, while for others it is the information generated by assessments and tools. Indeed, feedback has variously been defined as a process, as data, as a conversation, and as a reflective exercise. As a result, for a concept so central to what educators do, 'feedback' is ambiguous and has multiple meanings. Pattern theory affords opportunities to examine what scholars and practitioners mean when they use the term 'feedback'. Elaborating feedback as a pattern system can connect otherwise disjointed discourses of feedback. In this paper, the authors describe the development of a pattern system of feedback in medical education. Arksey & O'Malley's 5-stages of scoping reviews were adapted to enact a 6-step pattern system development methodology that included (1) Identifying the research question and scope of inquiry; (2) elaborating a strategy for pattern identification; (3) study selection; (4) abductive pattern representation development; (5) pattern system testing; and (6) summarizing and reporting the results. A pattern system of feedback was developed based on review of 218 full text articles and testing against an additional 2833 citations. This pattern system is made up of 36 pattern representations organized under 6 domains: feedback referent, feedback intentions, feedback information, feedback processing, feedback response, and feedback meta. The pattern system was applied to two models of feedback to demonstrate its utility as a lens through which to analyze various instances of feedback and to foreshadow its potential broader applicability as a tool to facilitate knowledge synthesis in the feedback problem space.

Re-Purposing the Ordering of Routine Laboratory Tests in Hospitalized Medical Patients (RePORT): protocol for a multicenter stepped-wedge cluster randomised trial to evaluate the impact of a multicomponent intervention bundle to reduce laboratory test over-utilization

BACKGROUND

Laboratory test overuse in hospitals is a form of healthcare waste that also harms patients. Developing and evaluating interventions to reduce this form of healthcare waste is critical. We detail the protocol for our study which aims to implement and evaluate the impact of an evidence-based, multicomponent intervention bundle on repetitive use of routine laboratory testing in hospitalized medical patients across adult hospitals in the province of British Columbia, Canada.

METHODS

We have designed a stepped-wedge cluster randomized trial to assess the impact of a multicomponent intervention bundle across 16 hospitals in the province of British Columbia in Canada. We will use the Knowledge to Action cycle to guide implementation and the RE-AIM framework to guide evaluation of the intervention bundle. The primary outcome will be the number of routine laboratory tests ordered per patient-day in the intervention versus control periods. Secondary outcome measures will assess implementation fidelity, number of all common laboratory tests used, impact on healthcare costs, and safety outcomes. The study will include patients admitted to adult medical wards (internal medicine or family medicine) and healthcare providers working in these wards within the participating hospitals. After a baseline period of 24 weeks, we will conduct a 16-week pilot at one hospital site. A new cluster (containing approximately 2-3 hospitals) will receive the intervention every 12 weeks. We will evaluate the sustainability of implementation at 24 weeks post implementation of the final cluster. Using intention to treat, we will use generalized linear mixed models for analysis to evaluate the impact of the intervention on outcomes.

DISCUSSION

The study builds upon a multicomponent intervention bundle that has previously demonstrated effectiveness. The elements of the intervention bundle are easily adaptable to other settings, facilitating future adoption in wider contexts. The study outputs are expected to have a positive impact as they will reduce usage of repetitive laboratory tests and provide empirically supported measures and tools for accomplishing this work.

TRIAL REGISTRATION

This study was prospectively registered on April 8, 2024, via ClinicalTrials.gov Protocols Registration and Results System (NCT06359587). https://classic.

CLINICALTRIALS

gov/ct2/show/NCT06359587?term=NCT06359587&recrs=ab&draw=2&rank=1.

Exploring the content and delivery of feedback facilitation co-interventions: a systematic review

BACKGROUND

Policymakers and researchers recommend supporting the capabilities of feedback recipients to increase the quality of care. There are different ways to support capabilities. We aimed to describe the content and delivery of feedback facilitation interventions delivered alongside audit and feedback within randomised controlled trials.

METHODS

We included papers describing feedback facilitation identified by the latest Cochrane review of audit and feedback. The piloted extraction proforma was based upon a framework to describe intervention content, with additional prompts relating to the identification of influences, selection of improvement actions and consideration of priorities and implications. We describe the content and delivery graphically, statistically and narratively.

RESULTS

We reviewed 146 papers describing 104 feedback facilitation interventions. Across included studies, feedback facilitation contained 26 different implementation strategies. There was a median of three implementation strategies per intervention and evidence that the number of strategies per intervention is increasing. Theory was used in 35 trials, although the precise role of theory was poorly described. Ten studies provided a logic model and six of these described their mechanisms of action. Both the exploration of influences and the selection of improvement actions were described in 46 of the feedback facilitation interventions; we describe who undertook this tailoring work. Exploring dose, there was large variation in duration (15-1800 min), frequency (1 to 42 times) and number of recipients per site (1 to 135). There were important gaps in reporting, but some evidence that reporting is improving over time.

CONCLUSIONS

Heterogeneity in the design of feedback facilitation needs to be considered when assessing the intervention's effectiveness. We describe explicit feedback facilitation choices for future intervention developers based upon choices made to date. We found the Expert Recommendations for Implementing Change to be valuable when describing intervention components, with the potential for some minor clarifications in terms and for greater specificity by intervention providers. Reporting demonstrated extensive gaps which hinder both replication and learning. Feedback facilitation providers are recommended to close reporting gaps that hinder replication. Future work should seek to address the 'opportunity' for improvement activity, defined as factors that lie outside the individual that make care or improvement behaviour possible.

REVIEW REGISTRATION

The study protocol was published at: https://www.protocols.io/private/4DA5DE33B68E11ED9EF70A58A9FEAC02 .

A realist synthesis of multicentre comparative audit implementation: exploring what works and in which healthcare contexts

BACKGROUND

Multicentre comparative clinical audits have the potential to improve patient care, allow benchmarking and inform resource allocation. However, implementing effective and sustainable large-scale audit can be difficult within busy and resource-constrained contemporary healthcare settings. There are little data on what facilitates the successful implementation of multicentre audits. As healthcare environments are complex sociocultural organisational environments, implementing multicentre audits within them is likely to be highly context dependent.

OBJECTIVE

We aimed to examine factors that were influential in the implementation process of multicentre comparative audits within healthcare contexts-what worked, why, how and for whom?

METHODS

A realist review was conducted in accordance with the Realist and Meta-narrative Evidence Syntheses: Evolving Standards reporting standards. A preliminary programme theory informed two systematic literature searches of peer-reviewed and grey literature. The main context-mechanism-outcome (CMO) configurations underlying the implementation processes of multicentre audits were identified and formed a final programme theory.

RESULTS

69 original articles were included in the realist synthesis. Four discrete CMO configurations were deduced from this synthesis, which together made up the final programme theory. These were: (1) generating trustworthy data; (2) encouraging audit participation; (3) ensuring audit sustainability; and (4) facilitating audit cycle completion.

CONCLUSIONS

This study elucidated contexts, mechanisms and outcomes influential to the implementation processes of multicentre or national comparative audits in healthcare. The relevance of these contextual factors and generative mechanisms were supported by established theories of behaviour and findings from previous empirical research. These findings highlight the importance of balancing reliability with pragmatism within complex adaptive systems, generating and protecting human capital, ensuring fair and credible leadership and prioritising change facilitation.

Randomized controlled trial of tailored audit with feedback in VHA long-term care settings

BACKGROUND

The Long-Term Care QUERI program supported implementation of the Life-Sustaining Treatment Decisions Initiative in US Veterans Health Administration long-term care settings. The program worked with eleven Community Living Centers (CLCs) and twelve Home-Based Primary Care (HBPC) programs to increase rates of completed templates, using audit with feedback. We distributed monthly feedback reports to site champions showing the number of Veterans with appropriate documentation. Although feedback reports are a common implementation tool, little is known about the most effective ways to design, distribute, and support them. We sought to test tailoring reports with tips using site-specific data, as well as national comparator data.

METHODS

We conducted a cluster randomized controlled trial of monthly feedback reports utilizing site-tailored tips and national comparator data compared to our original feedback reports that included only graphical and numerical data. CLC and HBPC team members were invited to participate in brief surveys each quarter to determine if they had received and used the feedback reports. The outcome for CLC residents was the percent with a completed LST template any time prior to the 14th day of their stay. The outcome for HBPC residents was the percent of Veterans with a completed LST template by their second HBPC visit.

RESULTS

The response rate to the survey ranged between 6.8 and 19.3% of staff members across the CLC and HBPC sites with 12.8-25.5% of survey respondents reporting that they had seen the feedback reports. The linear regression models showed no significant association between receiving the enhanced feedback reports and having a higher documentation completion rate.

CONCLUSIONS

Receiving feedback reports tailored to sites by including tips based on baseline context assessments and qualitative findings, and reports showing national comparator data, did not have an impact on the number of Veterans with a completed LST template. Having a higher proportion of CLC or HBPC team members view the reports was not associated with an increase in LST template completion. These findings suggest that tailored audit with feedback may not have been effective at the program level, although the proportion of respondents who reported seeing the reports was small.

Repurposing the Ordering of Routine Laboratory Tests in Hospitalised Medical Patients (RePORT): results of a cluster randomised stepped-wedge quality improvement study

BACKGROUND

Low-value use of laboratory tests is a global challenge. Our objective was to evaluate an intervention bundle to reduce repetitive use of routine laboratory testing in hospitalised patients.

METHODS

We used a stepped-wedge design to implement an intervention bundle across eight medical units. Our intervention included educational tools and social comparison reports followed by peer-facilitated report discussion sessions. The study spanned October 2020-June 2021, divided into control, feasibility testing, intervention and a follow-up period. The primary outcomes were the number and costs of routine laboratory tests ordered per patient-day. We used generalised linear mixed models, and analyses were by intention to treat.

RESULTS

We included a total of 125 854 patient-days. Patient groups were similar in age, sex, Charlson Comorbidity Index and length of stay during the control, intervention and follow-up periods. From the control to the follow-up period, there was a 14% (incidence rate ratio (IRR)=0.86, 95% CI 0.79 to 0.92) overall reduction in ordering of routine tests with the intervention, along with a 14% (β coefficient=-0.14, 95% CI -0.07 to -0.21) reduction in costs of routine testing. This amounted to a total cost savings of $C1.15 per patient-day. There was also a 15% (IRR=0.85, 95% CI 0.79, 0.92) reduction in ordering of all common tests with the intervention and a 20% (IRR=1.20, 95% CI 1.10 to 1.30) increase in routine test-free patient-days. No worsening was noted in patient safety endpoints with the intervention.

CONCLUSIONS

A multifaceted intervention bundle using education and facilitated multilevel social comparison was associated with a safe and effective reduction in use of routine daily laboratory testing in hospitals. Further research is needed to understand how system-level interventions may increase this effect and which intervention elements are necessary to sustain results.

How do healthcare providers use national audit data for improvement?

BACKGROUND

Substantial resources are invested by Health Departments worldwide in introducing National Clinical Audits (NCAs). Yet, there is variable evidence on the NCAs' effectiveness and little is known on factors underlying the successful use of NCAs to improve local practice. This study will focus on a single NCA (the National Audit of Inpatient Falls -NAIF 2017) to explore: (i) participants' perspectives on the NCA reports, local feedback characteristics and actions undertaken following the feedback underpinning the effective use of the NCA feedback to improve local practice; (ii) reported changes in local practice following the NCA feedback in England and Wales.

METHODS

Front-line staff perspectives were gathered through interviews. An inductive qualitative approach was used. Eighteen participants were purposefully sampled from 7 of the 85 participating hospitals in England and Wales. Analysis was guided by constant comparative techniques.

RESULTS

Regarding the NAIF annual report, interviewees valued performance benchmarking with other hospitals, the use of visual representations and the inclusion of case studies and recommendations. Participants stated that feedback should target front-line healthcare professionals, be straightforward and focused, and be delivered through an encouraging and honest discussion. Interviewees highlighted the value of using other relevant data sources alongside NAIF feedback and the importance of continuous data monitoring. Participants reported that engagement of front-line staff in the NAIF and following improvement activities was critical. Leadership, ownership, management support and communication at different organisational levels were perceived as enablers, while staffing level and turnover, and poor quality improvement (QI) skills, were perceived as barriers to improvement. Reported changes in practice included increased awareness and attention to patient safety issues and greater involvement of patients and staff in falls prevention activities.

CONCLUSIONS

There is scope to improve the use of NCAs by front-line staff. NCAs should not be seen as isolated interventions but should be fully embedded and integrated into the QI strategic and operational plans of NHS trusts. The use of NCAs could be optimised, but knowledge of them is poor and distributed unevenly across different disciplines. More research is needed to provide guidance on key elements to consider throughout the whole improvement process at different organisational levels.

Snakes and ladders: A qualitative study understanding the active ingredients of social interaction around the use of audit and feedback

Explore characteristics of the facilitator, group, and interaction that influence whether a group discussion about data leads to the identification of a clearly specified action plan. Peer-facilitated group discussions among primary care physicians were carried out and recorded. A follow-up focus group was conducted with peer facilitators to explore which aspects of the discussion promoted action planning. Qualitative data was analyzed using an inductive-deductive thematic analysis approach using the conceptual model developed by Cooke et al. Group discussions were coded case-specifically and then analyzed to identify which themes influenced action planning as it relates to performance improvement. Physicians were more likely to interact with practice-level data and explore actions for performance improvement when the group facilitator focused the discussion on action planning. Only one of the three sites (Site C) converged on an action plan following the peer-facilitated group discussion. At Site A, physicians shared skepticism of the data, were defensive about performance, and explained performance as a product of factors beyond their control. Site B identified several potential actions but had trouble focusing on a single indicator or deciding between physician- and group-level actions. None of the groups discussed variation in physician-level performance indicators, or how physician actions might contribute to the reported outcomes. Peer facilitators can support data interpretation and practice change; however their success depends on their personal beliefs about the data and their ability to identify and leverage change cues that arise in conversation. Further research is needed to understand how to create a psychologically safe environment that welcomes open discussion of physician variation.

Characterizing the Gaps Between Best-Practice Implementation Strategies and Real-world Implementation: Qualitative Study Among Family Physicians Who Engaged With Audit and Feedback Reports

BACKGROUND

In Ontario, Canada, a government agency known as Ontario Health is responsible for making audit and feedback reports available to all family physicians to encourage ongoing quality improvement. The confidential report provides summary data on 3 key areas of practice: safe prescription, cancer screening, and diabetes management.

OBJECTIVE

This report was redesigned to improve its usability in line with evidence. The objective of this study was to explore how the redesign was perceived, with an emphasis on recipients' understanding of the report and their engagement with it.

METHODS

We conducted qualitative semistructured interviews with family physicians who had experience with both versions of the report recruited through purposeful and snowball sampling. We analyzed the transcripts following an emergent and iterative approach.

RESULTS

Saturation was reached after 17 family physicians participated. In total, 2 key themes emerged as factors that affected the perceived usability of the report: alignment between the report and the recipients' expectations and capacity to engage in quality improvement. Family physicians expected the report and its quality indicators to reflect best practices and to be valid and accurate. They also expected the report to offer feedback on the clinical activities they perceived to be within their control to change. Furthermore, family physicians expected the goal of the report to be aligned with their perspective on feasible quality improvement activities. Most of these expectations were not met, limiting the perceived usability of the report. The capacity to engage with audit and feedback was hindered by several organizational and physician-level barriers, including the lack of fit with the existing workflow, competing priorities, time constraints, and insufficient skills for bridging the gaps between their data and the corresponding desired actions.

CONCLUSIONS

Despite recognized improvements in the design of the report to better align with best practices, it was not perceived as highly usable. Improvements in the presentation of the data could not overcome misalignment with family physicians' expectations or the limited capacity to engage with the report. Integrating iterative evaluations informed by user-centered design can complement evidence-based guidance for implementation strategies. Creating a space for bringing together audit and feedback designers and recipients may help improve usability and effectiveness.

Benefits of Providing Feedback and Utilisation Metrics to Specialists on Their Participation in eConsult

Our study evaluates the impact of feedback sent to specialists participating in eConsult services. eConsult Specialists from two eConsult services in Ontario, Canada, received feedback on their use of eConsult via bi-annual specialist reports. An 11-item survey was developed to evaluate the impact, content, and distribution process of these specialist reports. We distributed 742 specialist reports in March 2021 and surveyed the specialists in July 2021. Our findings show that specialists largely felt that the feedback received validated their efforts (83%) and that receiving the report made them more likely to continue to participate in the eConsult service (59%). Most did not feel judged (74%) or distressed (79%) by the reports, and 72% said that reporting the median self-reported billing time did not impact their own billing times. Overall, eConsult services can capture, report and aggregate data valuable to specialists and is useful for Continuing Professional Development. Benefits and lack of risk implementing this type of feedback should encourage other services to consider similar processes.

Implementation science in nursing education research: An exemplar

INTRODUCTION

Using evidence-based teaching approaches can improve nursing students' learning. However, variation in how - or if - these approaches are implemented by faculty and nursing educators is prevalent. A thorough, applied understanding of how evidence-based teaching approaches can best be implemented in the educational setting is lacking.

OBJECTIVES

The purpose of this project was to use an implementation science framework to implement and evaluate the quality of a doctor of nursing practice (DNP) course before and after implementing evidence-based revisions to course delivery and composition.

DESIGN

A pre/post design was used to evaluate course outcomes following implementation of evidence-based teaching approaches.

SETTING

A small, private university in the southeastern United States.

PARTICIPANTS

Students who enrolled in a DNP healthcare quality improvement course.

METHODS

An implementation science framework for integrating evidence-based teaching approaches was used to guide this project. Revisions were made to a DNP course, with evidence-based teaching approaches implemented using strategies including a dedicated course facilitator, faculty education, interactive assistance to course faculty, and detailed rubrics to ensure consistency in grading between sections. Outcomes included course evaluation scores, qualitative student comments, and student engagement measured via the average number of discussion board posts authored and read.

RESULTS

After using the implementation science framework to translate evidence-based teaching approaches, there was a statistically significant improvement in three course evaluation questions and the overall course mean. Qualitative comments showed that students found the revisions beneficial to their learning. There was no change in student engagement.

CONCLUSIONS

Using a structured implementation science framework and plan to translate and evaluate evidence-based teaching approaches resulted in significant improvements in course outcomes. Nurse educators should consider using an implementation framework to guide course revisions.

The effectiveness of a multifaceted, group-facilitated audit and feedback intervention to increase tranexamic acid use during total joint arthroplasty

PURPOSE

Intraoperative tranexamic acid (TXA) is used to reduce blood loss and the need for transfusions following total hip arthroplasty (THA) and total knee arthroplasty (TKA). Despite evidence in literature and local practice protocols supporting TXA as a part of standard of care for joint arthroplasty, TXA administration is underutilized. We aimed to use group-facilitated audit and feedback as the foundation of a knowledge translation strategy to increase TXA use for THA and TKA procedures.

METHODS

Anesthesiologists consented to receive two data reports summarizing their individual rates of TXA use and postoperative blood transfusions compared with site peers. Variables collected included patient demographics, TXA usage, and the frequency and volume of red blood cell transfusions administered in the 72-hr postoperative period. The facilitated feedback session discussed report findings and focused on factors contributing to local practice patterns and opportunities for change.

RESULTS

Tranexamic acid use increased for THA procedures at the intervention site from 66.6 to 74.4% (absolute change, 7.9%; 95% confidence interval [CI], 2.4 to 13.3). Likewise, TXA use for TKA procedures increased from 62.4 to 82.3% (absolute change, 19.9%; 95% CI 15.0 to 25.0).

CONCLUSIONS

Physicians and their teams were able to review their practice data on TXA utilization, reflect on differences compared with evidence-based guidelines, discuss findings with peers, and identify opportunities for improvement. The intervention increased the use of TXA for both TKA and THA and shifted the dosage to better align with evidence-based practice guidelines.

How does nursing-sensitive indicator feedback with nursing or interprofessional teams work and shape nursing performance improvement systems? A rapid realist review

BACKGROUND

Care quality varies between organizations and even units within an organization. Inadequate care can have harmful financial and social consequences, e.g. nosocomial infection, lengthened hospital stays or death. Experts recommend the implementation of nursing performance improvement systems to assess team performance and monitor patient outcomes as well as service efficiency. In practice, these systems provide nursing or interprofessional teams with nursing-sensitive indicator feedback. Feedback is essential since it commits teams to improve their practice, although it appears somewhat haphazard and, at times, overlooked. Research findings suggest that contextual dynamics, initial system performance and feedback modes interact in unknown ways. This rapid review aims to produce a theorization to explain what works in which contexts, and how feedback to nursing or interprofessional teams shape nursing performance improvement systems.

METHODS

Based on theory-driven realist methodology, with reference to an innovative combination of Actor-Network Theory and Critical Realist philosophy principles, this realist rapid review entailed an iterative procedure: 8766 documents in French and English, published between 2010 and 2018, were identified via 5 databases, and 23 were selected and analysed. Two expert panels (scientific and clinical) were consulted to improve the synthesis and systemic modelling of an original feedback theorization.

RESULTS

We identified three hypotheses, subdivided into twelve generative configurations to explain how feedback mobilizes nursing or interprofessional teams. Empirically founded and actionable, these propositions are supported by expert panels. Each configuration specifies contextualized mechanisms that explain feedback and team outcomes. Socially mediated mechanisms are particularly generative of action and agency.

CONCLUSIONS

This rapid realist review provides an informative theoretical proposition to embrace the complexity of nursing-sensitive indicator feedback with nursing or interdisciplinary teams. Building on general explanations previously observed, this review provides insight into a deep explanation of feedback mechanisms.

SYSTEMATIC REVIEW REGISTRATION

Prospero CRD42018110128 .

Integrating training, practice, and reflection within a new model for Canadian medical licensure: a concept paper prepared for the Medical Council of Canada

In 2020 the Medical Council of Canada created a task force to make recommendations on the modernization of its practices for granting licensure to medical trainees. This task force solicited papers on this topic from subject matter experts. As outlined within this Concept Paper, our proposal would shift licensure away from the traditional focus on high-stakes summative exams in a way that integrates training, clinical practice, and reflection. Specifically, we propose a model of graduated licensure that would have three stages including: a trainee license for trainees that have demonstrated adequate medical knowledge to begin training as a closely supervised resident, a transition to practice license for trainees that have compiled a reflective educational portfolio demonstrating the clinical competence required to begin independent practice with limitations and support, and a fully independent license for unsupervised practice for attendings that have demonstrated competence through a reflective portfolio of clinical analytics. This proposal was reviewed by a diverse group of 30 trainees, practitioners, and administrators in medical education. Their feedback was analyzed and summarized to provide an overview of the likely reception that this proposal would receive from the medical education community.

‘More than just numbers on a page?’ A qualitative exploration of the use of data collection and feedback in youth mental health services

Objectives

This study aimed to explore current data collection and feedback practice, in the form of monitoring and evaluation, among youth mental health (YMH) services and healthcare commissioners; and to identify barriers and enablers to this practice.

Design

Qualitative semi-structured interviews were conducted via Zoom videoconferencing software. Data collection and analysis were informed by the Theoretical Domains Framework (TDF). Data were deductively coded to the 14 domains of the TDF and inductively coded to generate belief statements.

Setting

Healthcare commissioning organisations and YMH services in Australia.

Participants

Twenty staff from healthcare commissioning organisations and twenty staff from YMH services.

Results

The umbrella behaviour ‘monitoring and evaluation’ (ME) can be sub-divided into 10 specific sub-behaviours (e.g. planning and preparing, providing technical assistance, reviewing and interpreting data) performed by healthcare commissioners and YMH services. One hundred belief statements relating to individual, social, or environmental barriers and enablers were generated. Both participant groups articulated a desire to improve the use of ME for quality improvement and had particular interest in understanding the experiences of young people and families. Identified enablers included services and commissioners working in partnership, data literacy (including the ability to set appropriate performance indicators), relational skills, and provision of meaningful feedback. Barriers included data that did not adequately depict service performance, problems with data processes and tools, and the significant burden that data collection places on YMH services with the limited resources they have to do it.

Conclusions

Importantly, this study illustrated that the use of ME could be improved. YMH services, healthcare commissioners should collaborate on ME plans and meaningfully involve young people and families where possible. Targets, performance indicators, and outcome measures should explicitly link to YMH service quality improvement; and ME plans should include qualitative data. Streamlined data collection processes will reduce unnecessary burden, and YMH services should have the capability to interrogate their own data and generate reports. Healthcare commissioners should also ensure that they provide meaningful feedback to their commissioned services, and local and national organisations collecting youth mental health data should facilitate the sharing of this data. The results of the study should be used to design theory-informed strategies to improve ME use.

Family Physician Quality Improvement Plans: A Realist Inquiry Into What Works, for Whom, Under What Circumstances

INTRODUCTION

Evaluation of quality improvement programs shows variable impact on physician performance often neglecting to examine how implementation varies across contexts and mechanisms that affect uptake. Realist evaluation enables the generation, refinement, and testing theories of change by unpacking what works for whom under what circumstances and why. This study used realist methods to explore relationships between outcomes, mechanisms (resources and reasoning), and context factors of a national multisource feedback (MSF) program.

METHODS

Linked data for 50 physicians were examined to determine relationships between action plan completion status (outcomes), MSF ratings, MSF comments and prescribing data (resource mechanisms), a report summarizing the conversation between a facilitator and physician (reasoning mechanism), and practice risk factors (context). Working backward from outcomes enabled exploration of similarities and differences in mechanisms and context.

RESULTS

The derived model showed that the completion status of plans was influenced by interaction of resource and reasoning mechanisms with context mediating the relationships. Two patterns were emerged. Physicians who implemented all their plans within six months received feedback with consistent messaging, reviewed data ahead of facilitation, coconstructed plan(s) with the facilitator, and had fewer risks to competence (dyscompetence). Physicians who were unable to implement any plans had data with fewer repeated messages and did not incorporate these into plans, had difficult plans, or needed to involve others and were physician-led, and were at higher risk for dyscompetence.

DISCUSSION

Evaluation of quality improvement initiatives should examine program outcomes taking into consideration the interplay of resources, reasoning, and risk factors for dyscompetence.

Program Evaluation of Implementation Science Outcomes From an Intervention to Improve Compliance With Chlorhexidine Gluconate Bathing: A Qualitative Study

BACKGROUND AND OBJECTIVES

Evaluation of implementation science research is warranted to better understand and determine the success of translating evidence-based infection prevention practices at the bedside. The purpose of this program evaluation was to evaluate implementation outcomes from the perspectives of nurses and nursing leaders regarding a previously conducted chlorhexidine gluconate (CHG) bathing implementation science study among 14 critical care units.

METHODS

Focus groups and interviews, using semistructured interview questions, were conducted to examine the perceptions of nurses who participated in a CHG bathing implementation science study. A deductive qualitative analysis using Proctor and colleagues' implementation outcomes framework was used. Transcripts were analyzed and categorized using the framework as a predetermined code list to structure the implementation outcomes of acceptability, appropriateness, adoption, feasibility, and sustainability.

FINDINGS

A total of 19 nurses and nurse leaders participated in a focus group or interview. Participants noted that both implementation strategies used in the initial study (educational outreach and audit and feedback) were acceptable and appropriate and expressed that the evidence-based CHG bathing practice was feasible to integrate into practice and was being adopted.

DISCUSSION

The program evaluation identified strengths and opportunities for improvement related to the implementation strategies and evidence-based CHG bathing protocol. Findings can inform future studies that seek to implement CHG bathing protocols in the critical care setting using audit and feedback and educational outreach strategies.

Confidence-competence alignment and the role of self-confidence in medical education: A conceptual review

CONTEXT

There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.

METHODS

This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.

RESULTS

A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: (a) confidence can shape how we act and is optimised when it closely corresponds to reality; (b) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualisation, the surrounding system and society; (c) confidence is shaped by many external factors and the context of the situation; (d) confidence must be considered in conjunction with competence and (e) the confidence-competence ratio (CCR) changes over time. It is important to track learners' CCRs and work with them to maintain balance.

CONCLUSION

Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR and explore the role of CCR in continuing professional development for individuals and teams.

A Realist Evaluation of a 72-Hour Readmission Audit and Feedback (A&F) Intervention in Emergency Medicine

Introduction

Audit and feedback (A&F) interventions are intended to increase accountability and improve the quality of care; however, their impact can vary significantly. As performance feedback is implemented in healthcare, there is a growing need to determine how users interact with the data and how systems can achieve more consistent performance outcomes. This study aimed to understand the contexts, mechanisms, and outcomes of an emergency department 72-hour readmission A&F intervention.

Methods

Semi-structured interviews with key stakeholders were conducted and analyzed using thematic and template analysis techniques specifically aimed at identifying context, mechanism, and outcome configurations.

Results

Seventeen (17) physician interviews were conducted. We identified five outcomes of the intervention and the contexts and mechanisms contributing to them. Importantly, we identified that this A&F strategy could potentially have positive (improved follow-up of cases, improved discharge communication) and negative impacts (increased physician anxiety, potentially increased resource use) on physicians and departmental efficiency.

Conclusion

The 72-hour readmission alert A&F intervention generates a number of distinct outcome patterns that result from a variety of mechanisms acting in different contexts. Knowledge of these context-mechanism-outcome relationships may help implementers design and tailor performance feedback strategies.

What the COVID-19 Pandemic Can Teach Health Professionals About Continuing Professional Development

The world's health care providers have realized that being agile in their thinking and growth in times of rapid change is paramount and that continuing education can be a key facet of the future of health care. As the world recovers from the COVID-19 pandemic, educators at academic health centers are faced with a crucial question: How can continuing professional development (CPD) within teams and health systems be improved so that health care providers will be ready for the next disruption? How can new information about the next disruption be collected and disseminated so that interprofessional teams will be able to effectively and efficiently manage a new disease, new information, or new procedures and keep themselves safe? Unlike undergraduate and graduate/postgraduate education, CPD does not always have an identified educational home and has had uneven and limited innovation during the pandemic. In this commentary, the authors explore the barriers to change in this sector and propose 4 principles that may serve to guide a way forward: identifying a home for interprofessional continuing education at academic health centers, improving workplace-based learning, enhancing assessment for individuals within health care teams, and creating a culture of continuous learning that promotes population health.

Enabling positive practice improvement through data-driven feedback: A model for understanding how data and self-perception lead to practice change

PURPOSE

This article aims to identify the factors that affect physicians' experiences of receiving practice data and to use these data to develop a model describing how individuals interact with the data.

METHODS

We designed an interview guide to study physicians' perspectives on audit and feedback. By intentional sampling, we recruited 15 physicians amongst gender groups, types of practice (academic vs community), and durations of practice. The interviews were conducted by a single author and transcribed without identifiers. We then began with an open coding analysis for all of the transcripts, and thereafter conducted axial coding to group the data into larger themes.

RESULTS

Several attributes were identified as either enabling or counterproductive attributes for participant improvement. The final proposed model identifies different zones of engagement on the basis of both the individual practitioner's growth mindset and the quality of the existing data system. In the highest engagement zone, the mindset of the collective leadership is one of growth. Systemic supports are in place, which potentiates learning that may come from an individual motivated to use their own data.

CONCLUSION

Our novel model depicts the relationship between data feedback systems and individuals' mindsets interact to augment or hinder clinical practice improvement. This model may provide leaders with a framework to examine their academic and administrative structures and how they might interface with performance feedback systems with clinicians.

Rationale and protocol for a cluster randomized pragmatic clinical trial testing behavioral economic implementation strategies to improve tobacco treatment rates for cancer patients who smoke

BACKGROUND

Routine evidence-based tobacco use treatment minimizes cancer-specific and all-cause mortality, reduces treatment-related toxicity, and improves quality of life among patients receiving cancer care. Few cancer centers employ mechanisms to systematically refer patients to evidence-based tobacco cessation services. Implementation strategies informed by behavioral economics can increase tobacco use treatment engagement within oncology care.

METHODS

A four-arm cluster-randomized pragmatic trial will be conducted across nine clinical sites within the Implementation Science Center in Cancer Control Implementation Lab to compare the effect of behavioral economic implementation strategies delivered through embedded messages (or "nudges") promoting patient engagement with the Tobacco Use Treatment Service (TUTS). Nudges are electronic medical record (EMR)-based messages delivered to patients, clinicians, or both, designed to counteract known patient and clinician biases that reduce treatment engagement. We used rapid cycle approaches (RCA) informed by relevant stakeholder experiences to refine and optimize our implementation strategies and methods prior to trial initiation. Data will be obtained via the EMR, clinician survey, and semi-structured interviews with a subset of clinicians and patients. The primary measure of implementation is penetration, defined as the TUTS referral rate. Secondary outcome measures of implementation include patient treatment engagement (defined as the number of patients who receive FDA-approved medication or behavioral counseling), quit attempts, and abstinence rates. The semi-structured interviews, guided by the Consolidated Framework for Implementation Research, will assess contextual factors and patient and clinician experiences with the nudges.

DISCUSSION

This study will be the first in the oncology setting to compare the effectiveness of nudges to clinicians and patients, both head-to-head and in combination, as implementation strategies to improve TUTS referral and engagement. We expect the study to (1) yield insights into the effectiveness of nudges as an implementation strategy to improve uptake of evidence-based tobacco use treatment within cancer care, and (2) advance our understanding of the multilevel contextual factors that drive response to these strategies. These results will lay the foundation for how patients with cancer who smoke are best engaged in tobacco use treatment and may lead to future research focused on scaling this approach across diverse centers.

TRIAL REGISTRATION

Clinicaltrials.gov, NCT04737031 . Registered 3 February 2021.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Interactive and Participatory Audit and Feedback (IPAF): theory-based development and multi-site implementation outcomes with specialty clinic staff

BACKGROUND

Theory-based implementation strategies, such as audit and feedback (A&F), can improve the adoption of evidence-based practices. However, few strategies have been developed and tested to meet the needs of specialty clinics. In particular, frontline staff can execute cardiovascular disease (CVD) risk reduction protocols, but A&F strategies to support them are not well examined. Our objective was to develop and evaluate a theory-based approach to A&F, Interactive and Participatory A&F (IPAF).

METHODS

We developed IPAF informed by two complementary theories, self-regulation theory (SRT) and self-determination theory (SDT). IPAF applies concepts from these theories to inform (1) what to address with staff to improve rates of best practices (SRT) and (2) how to interact with staff to improve behaviors aligned with best practices (SDT). We promoted IPAF fidelity by developing a semi-structured guide to facilitate staff discussion of target behaviors, perceived barriers, goals, and action plans. We evaluated IPAF in the context of eight quasi-experimental implementations in specialty clinics across two health systems. Following a hybrid type 2 effectiveness-implementation design, we reported intervention outcomes for CVD risk reduction elsewhere. This paper reports implementation outcomes associated with IPAF, focusing on feasibility, appropriateness, acceptability, fidelity, and adoption. We evaluated implementation outcomes using mixed-methods data including electronic health record (EHR) data, team records, and staff questionnaire responses.

RESULTS

Eighteen staff participated in 99 monthly, individual, synchronous (face-to-face or phone) IPAF sessions during the first 6 months of implementation. Subsequently, we provided over 375 monthly feedback emails. Feasibility data revealed high staff attendance (90-93%) and engagement in IPAF sessions. Staff highly rated questionnaire items about IPAF acceptability. Team records and staff responses demonstrated fidelity of IPAF delivery and receipt. Adoption of target behaviors increased significantly (all P values < 0.05), and adoption or behaviors were maintained for over 24 months.

CONCLUSIONS

We developed and evaluated a theory-based approach to A&F with frontline staff in specialty clinics to improve the implementation of evidence-based interventions. The findings support feasibility, appropriateness, acceptability, and fidelity of IPAF, and staff adoption and maintenance of target behaviors. By evaluating multi-site implementation outcomes, we extended prior research on clinic protocols and A&F beyond primary care settings and providers.

"I made a mistake!": A narrative analysis of experienced physicians' stories of preventable error

RATIONALE, AIMS, AND OBJECTIVES

The complexity of healthcare systems makes errors unavoidable. To strengthen the dialogue around how physicians experience and share medical errors, the objective of this study was to understand how generalist physicians make meaning of and grow from their medical errors.

METHODS

This study used a narrative inquiry approach to conduct and analyse in-depth interviews from 26 physicians from the generalist specialties of emergency, internal, and family medicine. We gathered stories via individual interview, analysed them for key components, and rewrote a "meta-story" in a chronological sequence. We conceptualized the findings into a metaphor to draw similarities, learn from, and apply new principles from other fields of practice.

RESULTS

Through analysis we interpreted the story of a physician who is required to make numerous decisions in a short period of time in different clinical environments among the patient's family and whilst abiding by existing rules and regulations. Through sharing stories of success and failure, the clinical supervisor can help optimize the physician's emotional growth and professional development. Similarly, through sharing and learning from stories, colleagues and trainees can also contribute to the growth of the protagonist's character and the development of clinic, hospital, and healthcare system.

CONCLUSION

We draw parallels between the clinical setting and a generalist physician's experiences of a medical error with the environment and practices within professional sports. Using this comparison, we discuss the potential for meaningful coaching in medical education.

Implementation and Evaluation of Audit and Feedback for Monitoring Treat-to-Target (T2T) Strategies in Rheumatoid Arthritis Using Performance Measures

INTRODUCTION

In collaboration with the Alberta Medical Association's Physician Learning Program we developed individualized physician reports and held a group feedback session on rheumatoid arthritis (RA) performance measures (PM) to facilitate treat-to-target (T2T) strategies and evaluated physician experiences with this process.

METHODS

5 PMs addressing T2T concepts from an established Canadian quality framework were operationalized for physician practice reports at 2 university-affiliated rheumatology clinics. Rheum4U, a quality improvement and research platform, was the data source. The audit results were reviewed in a facilitated group feedback session. Rheumatologists provided experiential feedback on the process through survey and/or an interview. Transcripts from interviews were analyzed using a 6-step thematic analysis.

RESULTS

11 of 12 eligible rheumatologists consented to receive practice reports and provided feedback through surveys (n = 5) and interviews (n = 6). The practice reports from Rheum4U (n = 448 patients) revealed high rates of yearly follow-up (> 85%, PM1) and 100% performance on documentation of disease activity at ≥ 50% of visits (PM2). Only 34% of patients were seen within 3 months if not in remission (PM3) with 62% (2017) and 69% (2018) of those with active RA achieving a LDA state within 6 months (PM4). Approximately 70% of patients were in remission at any time point (PM5). All survey respondents agreed or strongly agreed comparison to peers was valuable and helped them reflect on their practice. Several strategies for improvement were identified, including but not limited to, leveraging of electronic records for future audit and feedback reports, providing additional granularity of results, additional stratification of results, and using high-performing peers as the comparator rather than the group mean.

CONCLUSIONS

Audit and feedback was perceived by clinicians as a useful strategy for evaluating T2T efforts in RA. Future work will focus on longitudinal evaluation of the clinical impact of this quality improvement initiative.

Group-facilitated audit and feedback to improve bronchiolitis care in the emergency department

Objective

Despite strong evidence recommending supportive care as the mainstay of management for most infants with bronchiolitis, prior studies show that patients still receive low-value care (e.g., respiratory viral testing, salbutamol, chest radiography). Our objective was to decrease low-value care by delivering individual physician reports, in addition to group-facilitated feedback sessions to pediatric emergency physicians.

Methods

Our cohort included 3,883 patients ≤ 12 months old who presented to pediatric emergency departments in Calgary, Alberta, with a diagnosis of bronchiolitis from April 1, 2013, to April 30, 2018. Using administrative data, we captured baseline characteristics and therapeutic interventions. Consenting pediatric emergency physicians received two audit and feedback reports, which included their individual data and peer comparators. A multidisciplinary group-facilitated feedback session presented data and identified barriers and enablers of reducing low-value care. The primary outcome was the proportion of patients who received any low-value intervention and was analysed using statistical process control charts.

Results

Seventy-eight percent of emergency physicians consented to receive their audit and feedback reports. Patient characteristics were similar in the baseline and intervention period. Following the baseline physician reports and the group feedback session, low-value care decreased from 42.6% to 27.1% (absolute difference: −15.5%; 95% CI: −19.8% to −11.2%) and 78.9% to 64.4% (absolute difference: −14.5%; 95% CI: −21.9% to −7.2%) in patients who were not admitted and admitted, respectively. Balancing measures, such as intensive care unit admission and emergency department revisit, were unchanged.

Conclusion

The combination of audit and feedback and a group-facilitated feedback session reduced low-value care for patients with bronchiolitis.

Regional needs assessment for emergency physician audit and feedback

Objectives

Audit and feedback is widely used to improve physician performance. Many data metrics are being provided to physicians, yet most of these are driven by the regulatory environment. We sought to conduct a needs assessment of audit and feedback metrics that were most useful to clinicians within our health care region.

Methods

We conducted a Web-based survey of five clinical practice sites in our region and asked that physicians rank 49 clinical practice metrics. In addition, we assessed their readiness for audit and feedback and their preferences for data confidentiality. We collected data on duration of training, gender, and site of practice (academic v. community) allowing for comparison between groups.

Results

A total of 104 emergency medicine physicians participated in the survey (52.3% response rate). There was a significant readiness for participation in audit and feedback activities. Top ranked metrics were emergency department return rates and colleague's assessment of collegiality and quality of care, which were common across all sites. Small yet significant differences were noted between genders and academic v. community practitioners.

Conclusion

This study represents the first regional analysis of physician preferences for audit and feedback activities and implementation. It demonstrates that physicians are interested in audit and feedback activities and provides a roadmap for the development of a regional audit and feedback structure. It will also be used as a guiding document for regional change management.

How does feedback shared with interprofessional health care teams shape nursing performance improvement systems? A rapid realist review protocol

BACKGROUND

Nursing care quality varies between hospitals, and even between departments within the same institution. Suboptimal care can have deleterious consequences for patients such as lengthened hospital stay, nosocomial infection, pressure ulcers or death. Experts recommend the implementation of nursing performance improvement systems to assess team performance and monitor patient outcomes and efficiency savings. In practice, these systems are expected to include feedback processes directed towards nursing teams and interprofessional staff in order to facilitate adjustments and improve their performance. Unfortunately, feedback appears somewhat haphazard and, at times, overlooked. This could be explained by an ongoing absence of clear recommendations. As a result, feedback effects are inconclusive: some teams improve their practice, others do not. Although feedback has been conceptualised and studied from different theoretical perspectives, ongoing empirical inconsistencies remain unexplained. The goal of this rapid realist review protocol is to develop a theory that explains how feedback shared with interprofessional health care teams shape nursing performance improvement systems.

METHOD

This study follows standard guidelines established for realist reviews. Mechanisms at work will be analysed using Actor-Network Theory. All scientific documents are selected from five databases, are published in both English and French between 2010 and 2018, and include empirical research, reviews and grey literature. First, selection of documents will proceed on the basis of titles and abstracts; followed by a second selection by reading the remaining full texts. Inclusion criteria and a data extraction form will be pilot tested with 40 articles prior to completion by two reviewers. Data will be summarised in the form of [context, mechanism, outcome] equations to theorise operational feedback.

DISCUSSION

The innovative combination of Actor-Network Theory with a realist methodology holds promise for the identification of explanatory equations in complex systems and theory development. A rapid realist review is relevant to address an enduring knowledge gap which requires theory development. This preliminary study lays the groundwork for a pioneering theory on feedback in nursing performance improvement systems that will subsequently inform a multiple case study.

SYSTEMATIC REVIEW REGISTRATION

Prospero CRD42018110128.

What motivates general practitioners to change practice behaviour? A qualitative study of audit and feedback group sessions in Dutch general practice

OBJECTIVES

Adopting an attributional perspective, the current article investigates how audit and feedback group sessions contribute to general practitioners' (GPs) motivation to change their practice behaviour to improve care. We focus on the contributions of the audit and feedback itself (content) and the group discussion (process).

METHODS

Four focus groups, comprising a total of 39 participating Dutch GPs, discussed and compared audit and feedback of their practices. The focus groups were analysed thematically.

RESULTS

Audit and feedback contributed to GPs' motivation to change in two ways: by raising awareness about aspects of their current care practice and by providing indications of the possible impact of change. For these contributions to play out, the audit and feedback should be reliable and valid, specific, recent and recurrent and concern GPs' own practices or practices within their own influence sphere. Care behaviour attributed to external, uncontrollable or unstable causes would not induce change. The added value of the group is twofold as well: group discussion contributed to GPs' motivation to change by providing a frame of reference and by affording insights that participants would not have been able to achieve on their own.

CONCLUSIONS

In audit and feedback group sessions, both audit and feedback information and group discussion can valuably contribute to GPs' motivation to change care practice behaviour. Peer interaction can positively contribute to explore alternative practices and avenues for improvement. Local or regional peer meetings would be beneficial in facilitating reflection and discussion. An important avenue for future studies is to explore the contribution of audit and feedback and small-group discussion to actual practice change.