How do physicians behave when they participate in audit and feedback activities in a group with their peers?

Lessons learned from a pay-for-performance scheme for appropriate prescribing using electronic health records from general practices in the Netherlands

INTRODUCTION

A nationwide pay-for-performance (P4P) scheme was introduced in the Netherlands between 2018 and 2023 to incentivize appropriate prescribing in general practice. Appropriate prescribing was operationalised as adherence to prescription formularies and measured based on electronic health records (EHR) data. We evaluated this P4P scheme from a learning health systems perspective.

METHODS

We conducted semi-structured interviews with 15 participants representing stakeholders of the scheme: general practitioners (GPs), health insurers, pharmacists, EHR suppliers and formulary committees. We used a thematic approach for data analysis.

RESULTS

Using EHR data showed several benefits, but lack of uniformity of EHR systems hindered consistent measurements. Specific indicators were favoured over general indicators as they allow GPs to have more control over their performance. Most participants emphasized the need for GPs to jointly reflect on their performance. Communication to GPs appeared to be challenging. Partly because of these challenges, impact of the scheme on prescribing behaviour was perceived as limited. However, several unexpected positive effects of the scheme were mentioned, such as better EHR recording habits.

CONCLUSIONS

This study identified benefits and challenges useful for future P4P schemes in promoting appropriate care with EHR data. Enhancing uniformity in EHR systems is crucial for more consistent quality measurements. Future P4P schemes should focus on high-quality feedback, peer-to-peer learning and establish a single point of communication for healthcare providers.

Re-Purposing the Ordering of Routine Laboratory Tests in Hospitalized Medical Patients (RePORT): protocol for a multicenter stepped-wedge cluster randomised trial to evaluate the impact of a multicomponent intervention bundle to reduce laboratory test over-utilization

BACKGROUND

Laboratory test overuse in hospitals is a form of healthcare waste that also harms patients. Developing and evaluating interventions to reduce this form of healthcare waste is critical. We detail the protocol for our study which aims to implement and evaluate the impact of an evidence-based, multicomponent intervention bundle on repetitive use of routine laboratory testing in hospitalized medical patients across adult hospitals in the province of British Columbia, Canada.

METHODS

We have designed a stepped-wedge cluster randomized trial to assess the impact of a multicomponent intervention bundle across 16 hospitals in the province of British Columbia in Canada. We will use the Knowledge to Action cycle to guide implementation and the RE-AIM framework to guide evaluation of the intervention bundle. The primary outcome will be the number of routine laboratory tests ordered per patient-day in the intervention versus control periods. Secondary outcome measures will assess implementation fidelity, number of all common laboratory tests used, impact on healthcare costs, and safety outcomes. The study will include patients admitted to adult medical wards (internal medicine or family medicine) and healthcare providers working in these wards within the participating hospitals. After a baseline period of 24 weeks, we will conduct a 16-week pilot at one hospital site. A new cluster (containing approximately 2-3 hospitals) will receive the intervention every 12 weeks. We will evaluate the sustainability of implementation at 24 weeks post implementation of the final cluster. Using intention to treat, we will use generalized linear mixed models for analysis to evaluate the impact of the intervention on outcomes.

DISCUSSION

The study builds upon a multicomponent intervention bundle that has previously demonstrated effectiveness. The elements of the intervention bundle are easily adaptable to other settings, facilitating future adoption in wider contexts. The study outputs are expected to have a positive impact as they will reduce usage of repetitive laboratory tests and provide empirically supported measures and tools for accomplishing this work.

TRIAL REGISTRATION

This study was prospectively registered on April 8, 2024, via ClinicalTrials.gov Protocols Registration and Results System (NCT06359587). https://classic.

CLINICALTRIALS

gov/ct2/show/NCT06359587?term=NCT06359587&recrs=ab&draw=2&rank=1.

Exploring the content and delivery of feedback facilitation co-interventions: a systematic review

BACKGROUND

Policymakers and researchers recommend supporting the capabilities of feedback recipients to increase the quality of care. There are different ways to support capabilities. We aimed to describe the content and delivery of feedback facilitation interventions delivered alongside audit and feedback within randomised controlled trials.

METHODS

We included papers describing feedback facilitation identified by the latest Cochrane review of audit and feedback. The piloted extraction proforma was based upon a framework to describe intervention content, with additional prompts relating to the identification of influences, selection of improvement actions and consideration of priorities and implications. We describe the content and delivery graphically, statistically and narratively.

RESULTS

We reviewed 146 papers describing 104 feedback facilitation interventions. Across included studies, feedback facilitation contained 26 different implementation strategies. There was a median of three implementation strategies per intervention and evidence that the number of strategies per intervention is increasing. Theory was used in 35 trials, although the precise role of theory was poorly described. Ten studies provided a logic model and six of these described their mechanisms of action. Both the exploration of influences and the selection of improvement actions were described in 46 of the feedback facilitation interventions; we describe who undertook this tailoring work. Exploring dose, there was large variation in duration (15-1800 min), frequency (1 to 42 times) and number of recipients per site (1 to 135). There were important gaps in reporting, but some evidence that reporting is improving over time.

CONCLUSIONS

Heterogeneity in the design of feedback facilitation needs to be considered when assessing the intervention's effectiveness. We describe explicit feedback facilitation choices for future intervention developers based upon choices made to date. We found the Expert Recommendations for Implementing Change to be valuable when describing intervention components, with the potential for some minor clarifications in terms and for greater specificity by intervention providers. Reporting demonstrated extensive gaps which hinder both replication and learning. Feedback facilitation providers are recommended to close reporting gaps that hinder replication. Future work should seek to address the 'opportunity' for improvement activity, defined as factors that lie outside the individual that make care or improvement behaviour possible.

REVIEW REGISTRATION

The study protocol was published at: https://www.protocols.io/private/4DA5DE33B68E11ED9EF70A58A9FEAC02 .

Sepsis screening protocol implementation: a clinician-validated rapid realist review

INTRODUCTION

The failed or partial implementation of clinical practices negatively impacts patient safety and increases systemic inefficiencies. Implementation of sepsis screening guidelines has been undertaken in many settings with mixed results. Without a theoretical understanding of what leads to successful implementation, improving implementation will continue to be ad hoc or intuitive. This study proposes a programme theory for how and why the successful implementation of sepsis screening guidelines can occur.

METHODS

A rapid realist review was conducted to develop a focused programme theory for the implementation of sepsis screening guidelines. An independent two-reviewer approach was used to iteratively extract and synthesise context and mechanism data. Theoretical context-mechanism-outcome propositions were refined and validated by clinicians using a focus group and individual realist interviews. Implementation resources and clinical reasoning were differentiated in articulating mechanisms.

RESULTS

Eighteen articles were included in the rapid review. The theoretical domains framework was identified as the salient substantive theory informing the programme theory. The theory consisted of five main middle-range propositions. Three promoting mechanisms included positive belief about the benefits of the protocol, belief in the legitimacy of using the protocol and trust within the clinical team. Two inhibiting mechanisms included pessimism about the protocol being beneficial and pessimism about the team. Successful implementation was defined as achieving fidelity and sustained use of the intervention. Two intermediate outcomes, acceptability and feasibility of the intervention, and adoption, were necessary to achieve before successful implementation.

CONCLUSION

This rapid realist review synthesised key information from the literature and clinician feedback to develop a theory-based approach to clinical implementation of sepsis screening. The programme theory presents knowledge users with an outline of how and why clinical interventions lead to successful implementation and could be applied in other clinical areas to improve quality and safety.

Reconsidering performance management to support innovative changes in health care services

PURPOSE

A large number of studies indicate that coercive forms of organizational control and performance management in health care services often backfire and initiate dysfunctional consequences. The purpose of this article is to discuss new approaches to performance management in health care services when the purpose is to support innovative changes in the delivery of services.

DESIGN/METHODOLOGY/APPROACH

The article represents cross-boundary work as the theoretical and empirical material used to discuss and reconsider performance management comes from several relevant research disciplines, including systematic reviews of audit and feedback interventions in health care and extant theories of human motivation and organizational control.

FINDINGS

An enabling approach to performance management in health care services can potentially contribute to innovative changes. Key design elements to operationalize such an approach are a formative and learning-oriented use of performance measures, an appeal to self- and social-approval mechanisms when providing feedback and support for local goals and action plans that fit specific conditions and challenges.

ORIGINALITY/VALUE

The article suggests how to operationalize an enabling approach to performance management in health care services. The framework is consistent with new governance and managerial approaches emerging in public sector organizations more generally, supporting a higher degree of professional autonomy and the use of nonfinancial incentives.

Access to automated comparative feedback reports in primary care - a study of intensity of use and relationship with clinical performance among Swedish primary care practices

BACKGROUND

Digital applications that automatically extract information from electronic medical records and provide comparative visualizations of the data in the form of quality indicators to primary care practices may facilitate local quality improvement (QI). A necessary condition for such QI to work is that practices actively access the data. The purpose of this study was to explore the use of an application that visualizes quality indicators in Swedish primary care, developed by a profession-led QI initiative ("Primärvårdskvalitet"). We also describe the characteristics of practices that used the application more or less extensively, and the relationships between the intensity of use and changes in selected performance indicators.

METHODS

We studied longitudinal data on 122 primary care practices' visits to pages (page views) in the application over a period up to 5 years. We compared high and low users, classified by the average number of monthly page views, with respect to practice and patient characteristics as well as baseline measurements of a subset of the performance indicators. We estimated linear associations between visits to pages with diabetes-related indicators and the change in measurements of selected diabetes indicators over 1.5 years.

RESULTS

Less than half of all practices accessed the data in a given month, although most practices accessed the data during at least one third of the observed months. High and low users were similar in terms of most studied characteristics. We found statistically significant positive associations between use of the diabetes indicators and changes in measurements of three diabetes indicators.

CONCLUSIONS

Although most practices in this study indicated an interest in the automated feedback reports, the intensity of use can be described as varying and on average limited. The positive associations between the use and changes in performance suggest that policymakers should increase their support of practices' QI efforts. Such support may include providing a formalized structure for peer group discussions of data, facilitating both understanding of the data and possible action points to improve performance, while maintaining a profession-led use of applications.

Better with GPs as managers? - Variation in perceptions of feedback messages, goal-clarity and performance across manager´s in Swedish primary care

BACKGROUND

Primary care in several countries is developing towards team-based and multi-professional care, requiring leadership and management capabilities at the primary care practice level. This article reports findings from a study of primary care managers in Sweden, focusing variation in performance and perceptions of feedback messages and goal-clarity, depending on managers' professional background.

METHODS

The study was designed as a cross-sectional analysis of primary care practice managers' perceptions combined with registered data on patient-reported performance. Managers perceptions was collected through a survey to all 1 327 primary care practice managers in Sweden. Data about patient-reported performance was collected from the 2021 National Patient Survey in primary care. We used bivariate (Pearson correlation) and multivariate (ordinary least square regression analysis) statistical methods to describe and analyse the possible association between managers' background, responses to survey statements and patient-reported performance.

RESULTS

Both GP and non-GP managers had positive perceptions of the quality and support of feedback messages from professional committees focusing medical quality indicators, although managers perceived that the feedback facilitated improvement work to a lower degree. Feedback from the regions as payers scored consistently lower in all dimensions, especially among GP-managers. Results from regression analysis indicate that GP-managers correlate with better patient-reported performance when controlling for selected primary care practice and managerial characteristics. A significant positive relationship with patient-reported performance was also found for female managers, a smaller size of the primary care practice and a good staffing situation of GPs.

CONCLUSIONS

Both GP and non-GP managers rated the quality and support of feedback messages from professional committees higher than feedback from regions as payers. Differences in perceptions were especially striking among GP-managers. Patient-reported performance was significantly better in primary care practices managed by GPs and female managers. Variables reflecting structural and organizational, rather than managerial, characteristics contributed with additional explanations behind the variation in patient-reported performance across primary care practices. As we cannot exclude reversed causality, the findings may reflect that GPs are more likely to accept being a manager of a primary care practice with favourable characteristics.

JBI's approach to evidence implementation: a 7-phase process model to support and guide getting evidence into practice

ABSTRACT

In this paper, we provide an overview of JBI's approach to evidence implementation and describe the supporting process model that aligns with this approach. The central tenets of JBI's approach to implementing evidence into practice include the use of evidence-based audit and feedback, identification of the context in which evidence is being implemented, facilitation of any change, and an evaluation process. A pragmatic and practical seven-phased approach is outlined to assist with the 'planning' and 'doing' of getting evidence into practice, focusing on clinicians as change agents for implementing evidence in clinical and policy settings. Further research and development is required to formally evaluate the robustness of the approach to better understand the complex nature of evidence implementation.

Walking the line between assessment, improvement and learning: a qualitative study on opportunities and risks of incorporating peer discussion of audit and feedback within quality improvement in general practice

OBJECTIVES

There is a broad call for change towards 'new era' quality systems in healthcare, in which the focus lies on learning and improving. A promising way to establish this in general practice care is to combine audit and feedback with peer group discussion. However, it is not known what different stakeholders think of this type of quality improvement. The aim of this research was to explore the opinions of different stakeholders in general practice on peer discussion of audit and feedback and on its opportunities and risks. Second, their thoughts on transparency versus accountability, regarding this system, were studied.

DESIGN

An exploratory qualitative study within a constructivist paradigm. Semistructured interviews and focus group discussions were held and coded using thematic analysis. Included stakeholders were general practitioners (GP), patients, professional organisations and insurance companies.

SETTING

General practice in the Netherlands.

PARTICIPANTS

22 participants were purposively sampled for eight interviews and two focus group discussions.

RESULTS

Three main opportunities of peer discussion of audit and feedback were identified: deeper levels of reflection on data, adding context to numbers and more ownership; and three main risks: handling of unwilling colleagues, lacking a safe group and the necessity of patient involvement. An additional theme concerned disagreement on the amount of transparency to be offered: insurance companies and patients advocated for complete transparency on data and improvement of outcomes, while GPs and professional organisations urged to restrict transparency to giving insight into the process.

CONCLUSIONS

Peer discussion of audit and feedback could be part of a change movement, towards a quality system based on learning and trust, that is initiated by the profession. Creating a safe learning environment and involving patients is key herein. Caution is needed when complete transparency is asked, since it could jeopardise practitioners' reflection and learning in safety.

Snakes and ladders: A qualitative study understanding the active ingredients of social interaction around the use of audit and feedback

Explore characteristics of the facilitator, group, and interaction that influence whether a group discussion about data leads to the identification of a clearly specified action plan. Peer-facilitated group discussions among primary care physicians were carried out and recorded. A follow-up focus group was conducted with peer facilitators to explore which aspects of the discussion promoted action planning. Qualitative data was analyzed using an inductive-deductive thematic analysis approach using the conceptual model developed by Cooke et al. Group discussions were coded case-specifically and then analyzed to identify which themes influenced action planning as it relates to performance improvement. Physicians were more likely to interact with practice-level data and explore actions for performance improvement when the group facilitator focused the discussion on action planning. Only one of the three sites (Site C) converged on an action plan following the peer-facilitated group discussion. At Site A, physicians shared skepticism of the data, were defensive about performance, and explained performance as a product of factors beyond their control. Site B identified several potential actions but had trouble focusing on a single indicator or deciding between physician- and group-level actions. None of the groups discussed variation in physician-level performance indicators, or how physician actions might contribute to the reported outcomes. Peer facilitators can support data interpretation and practice change; however their success depends on their personal beliefs about the data and their ability to identify and leverage change cues that arise in conversation. Further research is needed to understand how to create a psychologically safe environment that welcomes open discussion of physician variation.

Characterizing the Gaps Between Best-Practice Implementation Strategies and Real-world Implementation: Qualitative Study Among Family Physicians Who Engaged With Audit and Feedback Reports

BACKGROUND

In Ontario, Canada, a government agency known as Ontario Health is responsible for making audit and feedback reports available to all family physicians to encourage ongoing quality improvement. The confidential report provides summary data on 3 key areas of practice: safe prescription, cancer screening, and diabetes management.

OBJECTIVE

This report was redesigned to improve its usability in line with evidence. The objective of this study was to explore how the redesign was perceived, with an emphasis on recipients' understanding of the report and their engagement with it.

METHODS

We conducted qualitative semistructured interviews with family physicians who had experience with both versions of the report recruited through purposeful and snowball sampling. We analyzed the transcripts following an emergent and iterative approach.

RESULTS

Saturation was reached after 17 family physicians participated. In total, 2 key themes emerged as factors that affected the perceived usability of the report: alignment between the report and the recipients' expectations and capacity to engage in quality improvement. Family physicians expected the report and its quality indicators to reflect best practices and to be valid and accurate. They also expected the report to offer feedback on the clinical activities they perceived to be within their control to change. Furthermore, family physicians expected the goal of the report to be aligned with their perspective on feasible quality improvement activities. Most of these expectations were not met, limiting the perceived usability of the report. The capacity to engage with audit and feedback was hindered by several organizational and physician-level barriers, including the lack of fit with the existing workflow, competing priorities, time constraints, and insufficient skills for bridging the gaps between their data and the corresponding desired actions.

CONCLUSIONS

Despite recognized improvements in the design of the report to better align with best practices, it was not perceived as highly usable. Improvements in the presentation of the data could not overcome misalignment with family physicians' expectations or the limited capacity to engage with the report. Integrating iterative evaluations informed by user-centered design can complement evidence-based guidance for implementation strategies. Creating a space for bringing together audit and feedback designers and recipients may help improve usability and effectiveness.

A Qualitative Study to Understand the Cultural Factors That Influence Clinical Data Use for Continuing Professional Development

INTRODUCTION

The use of data to inform lifelong learning has become increasingly important in continuing professional development (CPD) practice. Despite the potential benefits of data-driven learning, our understanding of how physicians engage in data-informed learning activities, particularly for CPD, remains unclear and warrants further study. The purpose of this study was to explore how physicians perceive cultural factors (individual, organizational, and systemic) that influence the use of clinical data to inform lifelong learning and self-initiated CPD activities.

METHODS

This qualitative study is part of an explanatory sequential mixed-methods study examining data-informed learning. Participants were psychiatrists and general surgeons from Canada and the United States. Recruitment occurred between April 2019 and November 2019, and the authors conducted semistructured telephone interviews between May 2019 and November 2019. The authors performed thematic analysis using an iterative, inductive method of constant comparative analysis.

RESULTS

The authors interviewed 28 physicians: 17 psychiatrists (61%) and 11 general surgeons (39%). Three major themes emerged from the continuous, iterative analysis of interview transcripts: (1) a strong relationship between data and trust, (2) a team-based approach to data-informed learning for practice improvement, and (3) a need for organizational support and advocacy to put data into practice.

CONCLUSION

Building trust, taking a team-based approach, and engaging multiple stakeholders, such as data specialists and organizational leadership, may significantly improve the use of data-informed learning. The results are situated in the existing literature, and opportunities for future research are summarized.

'Give me something meaningful': GPs perspectives on how to improve an audit and feedback report provided by health insurers - an exploratory qualitative study

BACKGROUND

Audit and feedback (A&F) is a valuable quality improvement strategy, which can contribute to de-implementation of low-value care. In the Netherlands, all health insurers collaboratively provide A&F to general practitioners (GPs), the 'Primary Care Practice Report' (PCPR). Unfortunately, the use of this report by GPs is limited. This study examined the thoughts of GPs on the usability of the PCPR and GPs recommendations for improving the PCPR.

METHOD

We used an interpretative qualitative design, with think-aloud tasks to uncover thoughts of GPs on the usability of the PCPR and semistructured interview questions to ask GPs' recommendations for improvement of the PCPR. Interviews were audiorecorded and transcribed ad verbatim. Data were analysed using thematic content analysis.

RESULTS

We identified two main themes: 'poor usability of the PCPR', and 'minimal motivation to change based on the PCPR'. The GPs found the usability of the PCPR poor due to the feedback not being clinically meaningful, the data not being recent, individual and reliable, the performance comparators offer insufficient guidance to assess clinical performance, the results are not discussed with peers and the definitions and visuals are unclear. The GPs recommended improving these issues. The GPs motivation to change based on the PCPR was minimal.

CONCLUSIONS

The GPs evaluated the PCPR as poorly usable and were minimally motivated to change. The PCPR seems developed from the perspective of the reports' commissioners, health insurers, and does not meet known criteria for effective A&F design and user-centred design. Importantly, the GPs did state that well-designed feedback could contribute to their motivation to improve clinical performance.Furthermore, the GPs stated that they receive a multitude of A&F reports, which they hardly use. Thus, we see a need for policy makers to invest in less, but more usable A&F reports.

An exploration into physician and surgeon data sensemaking: a qualitative systematic review using thematic synthesis

Providing electronic health data to medical practitioners to reflect on their performance can lead to improved clinical performance and quality of care. Understanding the sensemaking process that is enacted when practitioners are presented with such data is vital to ensure an improvement in performance. Thus, the primary objective of this research was to explore physician and surgeon sensemaking when presented with electronic health data associated with their clinical performance. A systematic literature review was conducted to analyse qualitative research that explored physicians and surgeons experiences with electronic health data associated with their clinical performance published between January 2010 and March 2022. Included articles were assessed for quality, thematically synthesised, and discussed from the perspective of sensemaking. The initial search strategy for this review returned 8,829 articles that were screened at title and abstract level. Subsequent screening found 11 articles that met the eligibility criteria and were retained for analyses. Two articles met all of the standards within the chosen quality assessment (Standards for Reporting Qualitative Research, SRQR). Thematic synthesis generated five overarching themes: data communication, performance reflection, infrastructure, data quality, and risks. The confidence of such findings is reported using CERQual (Confidence in the Evidence from Reviews of Qualitative research). The way the data is communicated can impact sensemaking which has implications on what is learned and has impact on future performance. Many factors including data accuracy, validity, infrastructure, culture can also impact sensemaking and have ramifications on future practice. Providing data in order to support performance reflection is not without risks, both behavioural and affective. The latter of which can impact the practitioner's ability to effectively make sense of the data. An important consideration when data is presented with the intent to improve performance.Registration This systematic review was registered with Prospero, registration number: CRD42020197392.

Integrating training, practice, and reflection within a new model for Canadian medical licensure: a concept paper prepared for the Medical Council of Canada

In 2020 the Medical Council of Canada created a task force to make recommendations on the modernization of its practices for granting licensure to medical trainees. This task force solicited papers on this topic from subject matter experts. As outlined within this Concept Paper, our proposal would shift licensure away from the traditional focus on high-stakes summative exams in a way that integrates training, clinical practice, and reflection. Specifically, we propose a model of graduated licensure that would have three stages including: a trainee license for trainees that have demonstrated adequate medical knowledge to begin training as a closely supervised resident, a transition to practice license for trainees that have compiled a reflective educational portfolio demonstrating the clinical competence required to begin independent practice with limitations and support, and a fully independent license for unsupervised practice for attendings that have demonstrated competence through a reflective portfolio of clinical analytics. This proposal was reviewed by a diverse group of 30 trainees, practitioners, and administrators in medical education. Their feedback was analyzed and summarized to provide an overview of the likely reception that this proposal would receive from the medical education community.

Family Physician Quality Improvement Plans: A Realist Inquiry Into What Works, for Whom, Under What Circumstances

INTRODUCTION

Evaluation of quality improvement programs shows variable impact on physician performance often neglecting to examine how implementation varies across contexts and mechanisms that affect uptake. Realist evaluation enables the generation, refinement, and testing theories of change by unpacking what works for whom under what circumstances and why. This study used realist methods to explore relationships between outcomes, mechanisms (resources and reasoning), and context factors of a national multisource feedback (MSF) program.

METHODS

Linked data for 50 physicians were examined to determine relationships between action plan completion status (outcomes), MSF ratings, MSF comments and prescribing data (resource mechanisms), a report summarizing the conversation between a facilitator and physician (reasoning mechanism), and practice risk factors (context). Working backward from outcomes enabled exploration of similarities and differences in mechanisms and context.

RESULTS

The derived model showed that the completion status of plans was influenced by interaction of resource and reasoning mechanisms with context mediating the relationships. Two patterns were emerged. Physicians who implemented all their plans within six months received feedback with consistent messaging, reviewed data ahead of facilitation, coconstructed plan(s) with the facilitator, and had fewer risks to competence (dyscompetence). Physicians who were unable to implement any plans had data with fewer repeated messages and did not incorporate these into plans, had difficult plans, or needed to involve others and were physician-led, and were at higher risk for dyscompetence.

DISCUSSION

Evaluation of quality improvement initiatives should examine program outcomes taking into consideration the interplay of resources, reasoning, and risk factors for dyscompetence.

Confidence-competence alignment and the role of self-confidence in medical education: A conceptual review

CONTEXT

There have been significant advances in competency-based medical education (CBME) within health professions education. While most of the efforts have focused on competency, less attention has been paid to the role of confidence as a factor in preparing for practice. This paper seeks to address this deficit by exploring the role of confidence and the calibration of confidence with regard to competence.

METHODS

This paper presents a conceptual review of confidence and the calibration of confidence in different medical education contexts. Building from an initial literature review, the authors engaged in iterative discussions exploring divergent and convergent perspectives, which were then supplemented with targeted literature reviews. Finally, a stakeholder consultation was conducted to situate and validate the provisional findings.

RESULTS

A series of axioms were developed to guide perceptions and responses to different states of confidence in health professionals: (a) confidence can shape how we act and is optimised when it closely corresponds to reality; (b) self-confidence is task-specific, but also inextricably influenced by the individual self-conceptualisation, the surrounding system and society; (c) confidence is shaped by many external factors and the context of the situation; (d) confidence must be considered in conjunction with competence and (e) the confidence-competence ratio (CCR) changes over time. It is important to track learners' CCRs and work with them to maintain balance.

CONCLUSION

Confidence is expressed in different ways and is shaped by a variety of modifiers. While CBME primarily focuses on competency, proportional confidence is an integral component in ensuring safe and professional practice. As such, it is important to consider both confidence and competence, as well as their relationship in CBME. The CCR can serve as a key construct in developing mindful and capable health professionals. Future research should evaluate strategies for assessing CCR, identify best practices for teaching confidence and guiding self-calibration of CCR and explore the role of CCR in continuing professional development for individuals and teams.

A Realist Evaluation of a 72-Hour Readmission Audit and Feedback (A&F) Intervention in Emergency Medicine

Introduction

Audit and feedback (A&F) interventions are intended to increase accountability and improve the quality of care; however, their impact can vary significantly. As performance feedback is implemented in healthcare, there is a growing need to determine how users interact with the data and how systems can achieve more consistent performance outcomes. This study aimed to understand the contexts, mechanisms, and outcomes of an emergency department 72-hour readmission A&F intervention.

Methods

Semi-structured interviews with key stakeholders were conducted and analyzed using thematic and template analysis techniques specifically aimed at identifying context, mechanism, and outcome configurations.

Results

Seventeen (17) physician interviews were conducted. We identified five outcomes of the intervention and the contexts and mechanisms contributing to them. Importantly, we identified that this A&F strategy could potentially have positive (improved follow-up of cases, improved discharge communication) and negative impacts (increased physician anxiety, potentially increased resource use) on physicians and departmental efficiency.

Conclusion

The 72-hour readmission alert A&F intervention generates a number of distinct outcome patterns that result from a variety of mechanisms acting in different contexts. Knowledge of these context-mechanism-outcome relationships may help implementers design and tailor performance feedback strategies.

Quality Improvement of Primary Care Provider Communication on Hospital Discharge

BACKGROUND AND OBJECTIVES

Effective communication between inpatient and primary care providers (PCPs) is important for safe transition of care for hospitalized patients. In 2017, communication with PCPs was prioritized for the pediatric hospital medicine division. Our primary aim was to improve documented attempted communication with PCPs within 72 hours of discharge from 41% to at least 60% by January 1, 2018, and maintain this performance through 2019.

METHODS

This study included all inpatient encounters discharged by a pediatric hospital medicine provider from March 2017 to April 2020. An electronic health record phrase debuted March 2017. Successful documentation was defined as any attempt to contact the PCP, regardless of whether actual communication occurred. Group and individual audit and feedback occurred in July 2017 to April 2020. Provider communication was financially incentivized in July 2018 to June 2019. An annotated P-chart for the proportion of encounters with documented PCP communication occurring within 72 hours was established. Special-cause variation was determined by using Shewhart rules.

RESULTS

The mean proportion of encounters with documented PCP communication increased from 41% at baseline (March 2017 through July 2017) to 60% in August 2017 and 66% in December 2017. After the financial incentive was removed in July 2019, documentation decreased to 54%. Phone calls with clinic staff were the most common communication method (40% to 71%). Direct conversations with the PCP occurred rarely (0% to 3%).

CONCLUSIONS

Even when coupled with audit and feedback with EHR interventions, our work suggests that shifting to external financial motivation may hinder sustainability of behavior change to improve attempted documented PCP communication.

Enabling positive practice improvement through data-driven feedback: A model for understanding how data and self-perception lead to practice change

PURPOSE

This article aims to identify the factors that affect physicians' experiences of receiving practice data and to use these data to develop a model describing how individuals interact with the data.

METHODS

We designed an interview guide to study physicians' perspectives on audit and feedback. By intentional sampling, we recruited 15 physicians amongst gender groups, types of practice (academic vs community), and durations of practice. The interviews were conducted by a single author and transcribed without identifiers. We then began with an open coding analysis for all of the transcripts, and thereafter conducted axial coding to group the data into larger themes.

RESULTS

Several attributes were identified as either enabling or counterproductive attributes for participant improvement. The final proposed model identifies different zones of engagement on the basis of both the individual practitioner's growth mindset and the quality of the existing data system. In the highest engagement zone, the mindset of the collective leadership is one of growth. Systemic supports are in place, which potentiates learning that may come from an individual motivated to use their own data.

CONCLUSION

Our novel model depicts the relationship between data feedback systems and individuals' mindsets interact to augment or hinder clinical practice improvement. This model may provide leaders with a framework to examine their academic and administrative structures and how they might interface with performance feedback systems with clinicians.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Interactive and Participatory Audit and Feedback (IPAF): theory-based development and multi-site implementation outcomes with specialty clinic staff

BACKGROUND

Theory-based implementation strategies, such as audit and feedback (A&F), can improve the adoption of evidence-based practices. However, few strategies have been developed and tested to meet the needs of specialty clinics. In particular, frontline staff can execute cardiovascular disease (CVD) risk reduction protocols, but A&F strategies to support them are not well examined. Our objective was to develop and evaluate a theory-based approach to A&F, Interactive and Participatory A&F (IPAF).

METHODS

We developed IPAF informed by two complementary theories, self-regulation theory (SRT) and self-determination theory (SDT). IPAF applies concepts from these theories to inform (1) what to address with staff to improve rates of best practices (SRT) and (2) how to interact with staff to improve behaviors aligned with best practices (SDT). We promoted IPAF fidelity by developing a semi-structured guide to facilitate staff discussion of target behaviors, perceived barriers, goals, and action plans. We evaluated IPAF in the context of eight quasi-experimental implementations in specialty clinics across two health systems. Following a hybrid type 2 effectiveness-implementation design, we reported intervention outcomes for CVD risk reduction elsewhere. This paper reports implementation outcomes associated with IPAF, focusing on feasibility, appropriateness, acceptability, fidelity, and adoption. We evaluated implementation outcomes using mixed-methods data including electronic health record (EHR) data, team records, and staff questionnaire responses.

RESULTS

Eighteen staff participated in 99 monthly, individual, synchronous (face-to-face or phone) IPAF sessions during the first 6 months of implementation. Subsequently, we provided over 375 monthly feedback emails. Feasibility data revealed high staff attendance (90-93%) and engagement in IPAF sessions. Staff highly rated questionnaire items about IPAF acceptability. Team records and staff responses demonstrated fidelity of IPAF delivery and receipt. Adoption of target behaviors increased significantly (all P values < 0.05), and adoption or behaviors were maintained for over 24 months.

CONCLUSIONS

We developed and evaluated a theory-based approach to A&F with frontline staff in specialty clinics to improve the implementation of evidence-based interventions. The findings support feasibility, appropriateness, acceptability, and fidelity of IPAF, and staff adoption and maintenance of target behaviors. By evaluating multi-site implementation outcomes, we extended prior research on clinic protocols and A&F beyond primary care settings and providers.

"I made a mistake!": A narrative analysis of experienced physicians' stories of preventable error

RATIONALE, AIMS, AND OBJECTIVES

The complexity of healthcare systems makes errors unavoidable. To strengthen the dialogue around how physicians experience and share medical errors, the objective of this study was to understand how generalist physicians make meaning of and grow from their medical errors.

METHODS

This study used a narrative inquiry approach to conduct and analyse in-depth interviews from 26 physicians from the generalist specialties of emergency, internal, and family medicine. We gathered stories via individual interview, analysed them for key components, and rewrote a "meta-story" in a chronological sequence. We conceptualized the findings into a metaphor to draw similarities, learn from, and apply new principles from other fields of practice.

RESULTS

Through analysis we interpreted the story of a physician who is required to make numerous decisions in a short period of time in different clinical environments among the patient's family and whilst abiding by existing rules and regulations. Through sharing stories of success and failure, the clinical supervisor can help optimize the physician's emotional growth and professional development. Similarly, through sharing and learning from stories, colleagues and trainees can also contribute to the growth of the protagonist's character and the development of clinic, hospital, and healthcare system.

CONCLUSION

We draw parallels between the clinical setting and a generalist physician's experiences of a medical error with the environment and practices within professional sports. Using this comparison, we discuss the potential for meaningful coaching in medical education.

Peer Learning Through Multi-Institutional Case Conferences: Abdominal and Cardiothoracic Radiology Experience

RATIONALE AND OBJECTIVES

We describe a model of multi-institutional, multisociety, online case conferences that is a case-based group discussion of selected (nonrandom) cases which are subsequently hosted on social media and online platforms (e.g., YouTube, websites) to be available for a wider audience.

MATERIALS AND METHODS

Using online conferencing software (Zoom, GoToMeeting), participants from both abdominal and cardiothoracic radiologists engage in separate, subspecialty one-hour meetings discussing a variety of meaningful cases. Participants take turns presenting their cases to the group and discuss significant findings, interpretations, differential diagnoses, and any other teaching points. All of the case conferences for both societies are recorded and edited to be uploaded on YouTube and their respective websites.

RESULTS

Participants from these conferences log in from 14 institutions in 7 states across the United States. The YouTube videos reach thousands of people around the world. The abdominal case conference on YouTube has received almost 1,300 views with 90 videos uploaded. The thoracic (the Society of Thoracic Radiology) case conference has been running for over 7 years, with 226 videos uploaded to YouTube and 38,200 views, 1426 subscribers, and a total watch time of over 525,800 minutes. Twitter has been utilized by both groups to promote online viewership.

CONCLUSION

Our model is feasible and effective compared to traditional peer review. The cases selected are deliberate and focused on quality improvement and/or education. We harness online engagement, specifically social media presence, which has opened new opportunities to educate our peers and reach a global audience, including the nonradiologic community, to learn about radiology and unique practices.

Implementation and Evaluation of Audit and Feedback for Monitoring Treat-to-Target (T2T) Strategies in Rheumatoid Arthritis Using Performance Measures

INTRODUCTION

In collaboration with the Alberta Medical Association's Physician Learning Program we developed individualized physician reports and held a group feedback session on rheumatoid arthritis (RA) performance measures (PM) to facilitate treat-to-target (T2T) strategies and evaluated physician experiences with this process.

METHODS

5 PMs addressing T2T concepts from an established Canadian quality framework were operationalized for physician practice reports at 2 university-affiliated rheumatology clinics. Rheum4U, a quality improvement and research platform, was the data source. The audit results were reviewed in a facilitated group feedback session. Rheumatologists provided experiential feedback on the process through survey and/or an interview. Transcripts from interviews were analyzed using a 6-step thematic analysis.

RESULTS

11 of 12 eligible rheumatologists consented to receive practice reports and provided feedback through surveys (n = 5) and interviews (n = 6). The practice reports from Rheum4U (n = 448 patients) revealed high rates of yearly follow-up (> 85%, PM1) and 100% performance on documentation of disease activity at ≥ 50% of visits (PM2). Only 34% of patients were seen within 3 months if not in remission (PM3) with 62% (2017) and 69% (2018) of those with active RA achieving a LDA state within 6 months (PM4). Approximately 70% of patients were in remission at any time point (PM5). All survey respondents agreed or strongly agreed comparison to peers was valuable and helped them reflect on their practice. Several strategies for improvement were identified, including but not limited to, leveraging of electronic records for future audit and feedback reports, providing additional granularity of results, additional stratification of results, and using high-performing peers as the comparator rather than the group mean.

CONCLUSIONS

Audit and feedback was perceived by clinicians as a useful strategy for evaluating T2T efforts in RA. Future work will focus on longitudinal evaluation of the clinical impact of this quality improvement initiative.

Peer Learning Through Multi-Institutional Web-based Case Conferences: Perceived Value (and Challenges) From Abdominal, Cardiothoracic, and Musculoskeletal Radiology Case Conference Participants

RATIONALE AND OBJECTIVES

Peer learning is a case-based group-learning model intended to improve performance. In this descriptive paper, we describe multi-institutional, multi-subspecialty, web-based radiology case conferences and summarize the participants' experiences.

MATERIALS AND METHODS

A semi-structured, 27-question survey was administered to radiologists participating in abdominal, cardiothoracic, and musculoskeletal case conferences. Survey questions included demographics, perceived educational value and challenges experienced. Survey question formats were continuous, binary, five-point Likert scale or text-based. The measures of central tendencies, proportions of responses and patterns were tabulated.

RESULTS

From 57 responders, 12/57 (21.1%) were abdominal, 16/57 (28.1%) were cardiothoracic, and 29/57 (50.8%) were musculoskeletal conference participants; 50/56 (89.3%) represented academic practice. Median age was 45 years (range 35-74); 43/57 (75.4%) were male. Geographically, 16/52 (30.8%) of participants were from the East Coast, 16/52 (30.8%) Midwest, 18/52 (34.6%) West Coast, and 2/52 (3.8%) International. The median reported educational value was 5/5 (interquartile range 5-5). Benefits of the case conference included education (50/95, 52.6%) and networking (39/95, 41.1%). Participants reported presenting the following cases: "great call" 32/48 (66.7%), learning opportunity 32/48 (66.7%), new knowledge 41/49 (83.7%), "zebras" 46/49 (93.9%), and procedural-based 16/46 (34.8%). All 51/51 (100%) of responders reportedly gained new knowledge, 49/51 (96.1%) became more open to group discussion, 34/51 (66.7%) changed search patterns, and 50/51 (98%) would continue to participate. Reported challenges included time zone differences and support from departments for a protected time to participate.

CONCLUSION

Peer learning through multi-institutional case conferences provides educational and networking opportunities. Current challenges and desires include having department-supported protected time and ability to receive continuing medical education credit.

Collective Intelligence of Peer Learning: Promoting Culture of Learning and Improvement Among Radiologists

The traditional Scoring-Based Peer Review system has been the predominant radiology performance quality assurance model, which can become a condemning, ineffective process. In 2015, the Institute of Medicine called for "policies and practices that promote a non-punitive culture that values open discussion and feedback on diagnostic performance." The development of Peer Learning (PL), a process that encompasses peer feedback, learning, and improvement, has positively impacted radiology through the recognition of success, identification of mistakes as learning opportunities, and development of a professional culture of trust. Furthermore, collective intelligence advances the PL process within the learning community, optimizing the abilities of a group effort that outperform that of a single individual, especially in the setting of complex medical and diagnostic imaging decision-making. The objective of the review article is to highlight the collective intelligence aspect of PL program, which allows PL to be more effective than established peer review model.

Clinical Improvement Interventions for Residents and Practicing Physicians: A Scoping Review of Coaching and Mentoring for Practice Improvement

BACKGROUND

Graduate medical education (GME) bodies are beginning to mandate coaching as an integral part of the learning process, in addition to current requirements for mentorship. Once an emergency medicine physician transitions beyond graduate training, there is no requirement and little focus on coaching as a method of improving or maintaining clinical practice. Our objective was to understand and describe the current state of the published literature with regard to the use of coaching and mentorship for both GME and practicing physicians.

METHODS

We conducted a structured review of the literature through PubMed and Google Scholar and included all articles applying coaching or mentorship modalities to GME trainees or practicing physicians. A Google Form was used for standardized data abstraction. Data were collected pertaining to the settings of intervention, the nature of the intervention, its effect, and its resource requirements.

RESULTS

A total of 3,546 papers were isolated during the literature review. After exclusion, 186 underwent full-text review by the authors of which 126 articles were included in the final data analysis. Eighty-two articles (65%) pertained to mentorship and 14 (11%) to coaching; the remainder of the articles discussed a combination or variation of these two concepts. Fifty-three (42%) articles were descriptive studies and 35 (28%) were narrative reviews or commentaries. Forty-seven (37%) articles originated from within surgical specialties and coaching was most commonly applied to procedural or manual skills with 22 (17%) instances among all studies.

CONCLUSIONS

Most literature on coaching and mentorship is descriptive or narrative, and few papers are in the specialty of emergency medicine. Most interventions are limited to single instances of coaching or mentorship without longitudinal application of the intervention. There is an important need to study and publish further evidence on coaching interventions.

What motivates general practitioners to change practice behaviour? A qualitative study of audit and feedback group sessions in Dutch general practice

OBJECTIVES

Adopting an attributional perspective, the current article investigates how audit and feedback group sessions contribute to general practitioners' (GPs) motivation to change their practice behaviour to improve care. We focus on the contributions of the audit and feedback itself (content) and the group discussion (process).

METHODS

Four focus groups, comprising a total of 39 participating Dutch GPs, discussed and compared audit and feedback of their practices. The focus groups were analysed thematically.

RESULTS

Audit and feedback contributed to GPs' motivation to change in two ways: by raising awareness about aspects of their current care practice and by providing indications of the possible impact of change. For these contributions to play out, the audit and feedback should be reliable and valid, specific, recent and recurrent and concern GPs' own practices or practices within their own influence sphere. Care behaviour attributed to external, uncontrollable or unstable causes would not induce change. The added value of the group is twofold as well: group discussion contributed to GPs' motivation to change by providing a frame of reference and by affording insights that participants would not have been able to achieve on their own.

CONCLUSIONS

In audit and feedback group sessions, both audit and feedback information and group discussion can valuably contribute to GPs' motivation to change care practice behaviour. Peer interaction can positively contribute to explore alternative practices and avenues for improvement. Local or regional peer meetings would be beneficial in facilitating reflection and discussion. An important avenue for future studies is to explore the contribution of audit and feedback and small-group discussion to actual practice change.

Use of performance reports among trauma medical directors and programme managers in the American College of Surgeons' Trauma Quality Improvement Program: a qualitative analysis

BACKGROUND

The American College of Surgeons' Trauma Quality Improvement Program (TQIP) provides trauma centres with performance reports on their processes and outcomes of care relative to their peers. This study explored how performance reports are used by trauma centre leaders to engage in performance improvement and perceived barriers to use.

STUDY DESIGN

Qualitative focus group study with trauma medical directors (TMDs) and trauma programme managers (TPMs) in US trauma centres. Consistent with qualitative descriptive analysis, data collection and interpretation were inductively and iteratively completed. Major themes were derived using a constant comparative technique.

RESULTS

Six focus groups were conducted involving 22 TMDs and 22 TPMs. Three major themes were captured: (1) technical uses of performance reports; (2) cultural uses of performance reports; (3) opportunities to enhance the role and value of TQIP. First, technical uses included using reports to assess data collection procedures, data quality and areas of poor performance relative to peers. In this domain, barriers to report use included not trusting others' data quality and challenges with report interpretation. Second, reports were used to influence practice change by fostering inter-specialty discussions, leveraging resources for quality improvement, community engagement and regional collaboratives. Perceived lack of specialist engagement was viewed as an impediment in this domain. Lastly, identified opportunities for TQIP to support report use involved clarifying the relationship between verification and performance reports, and increasing partnerships with nursing associations.

CONCLUSION

Trauma centre improvement leaders indicated practical and social uses of performance reports that can affect intention and ability to change. Recommendations to optimise programme participation include a focus on data quality, adequate resource provision and enhanced support for regional collaboratives.

Examining Associations Between Physician Data Utilization for Practice Improvement and Lifelong Learning

INTRODUCTION

Practice data can inform the selection of educational strategies; however, it is not widely used, even when available. This study's purpose was to determine factors that influence physician engagement with practice data to advance competence and drive practice change.

METHODS

A practice-based, pan-Canadian survey was administered to three physician subspecialties: psychiatrists (Psy), radiation oncologists (RO), and general surgeons (GS). The survey was distributed through national specialty society membership lists. The survey assessed factors that influence the use of data for practice improvement and orientation to lifelong learning, using the Jefferson Scale of Physician Lifelong Learning (JeffSPLL). Linear regression was used to model the relationship between the outcome variable frequency of data use and independent predictors of continuous learning to improving practice.

RESULTS

A total of 305 practicing physicians (Psy = 203, RO = 53, GS = 49) participated in this study. Most respondents used data for practice improvement (n = 177, 61.7%; Psy = 115, 40.1%; RO = 35; 12.2%; GS = 27, 9.4%) and had high orientation to lifelong learning (JeffSPLL mean scores: Psy = 47.4; RO = 43.5; GS = 45.1; Max = 56). Linear regression analysis identified significant predictors of data use in practice being: frequency of assessing learning needs, helpfulness of data to improve practice, and frequency to develop learning plans. Together, these predictors explained 42.9% of the variance in physicians' orientation toward integrating accessible data into practice (R = 0.426, P < .001).

DISCUSSION

This study demonstrates an association between practice data use and perceived data utility, reflection on learning needs and learning plan development. Implications for this work include process development for data-informed action planning for practice improvement for physicians.

The Calgary Audit and Feedback Framework: a practical, evidence-informed approach for the design and implementation of socially constructed learning interventions using audit and group feedback

Background

Audit and feedback interventions may be strengthened using social interaction. The Calgary office of the Alberta Physician Learning Program (CPLP) developed a process for audit and group feedback for physicians. This paper extends previous work in which we developed a conceptual model of physician responses to audit and group feedback based on a qualitative analysis of six audit and group feedback sessions. The present study explored the mediating factors for successfully engaging physician groups in change planning through audit and group feedback.

Methods

To understand why some groups were more interactive than others, we completed a comparative case analysis of the six audit and group feedback projects from the prior study. We used framework analysis to build the case studies, triangulated our observations across data sources to validate findings, compared the case studies for similarities and differences that influenced social interaction (mediating factors), and thematically categorized mediating factors into an organizing framework.

Results

Mediating factors for socially interactive AGFS were a pre-existing relationship between the program team and the physician group, projects addressing important, actionable questions, easily interpretable data visualization in the reports, and facilitation of the groups that included reflective questioning. When these factors were in place (cases 1, 2A, 3), the audit and group feedback sessions were dynamic, with physicians sharing and comparing practices, and raising change cues (such as declaring commitments to de-prescribing, planning educational interventions, and improving documentation). In cases 2C–D, the mediating factors were less well established and in these cases, the sessions showed little physician reflection or change planning. We organized the mediating factors into a framework linking the factors for successful sessions to the conceptual model of physician behaviors which these mediating factors drive.

Conclusions

We propose the Calgary Audit and Feedback Framework as a practical tool to help foster socially constructed learning in audit and group feedback sessions. Ensuring that the four factors, relationship, question choice, data visualization, and facilitation, are considered for design and implementation of audit and group feedback will help physicians move from reactions to their data towards planning for change.