Audit and feedback in mental healthcare: staff experiences

Champion and audit and feedback strategy fidelity and their relationship to depression intervention fidelity: A mixed method study

Background

Globally, mental health disorders rank as the greatest cause of disability. Low and middle-income countries (LMICs) hold a disproportionate share of the mental health burden, especially as it pertains to depression. Depression is highly prevalent among those with non-communicable diseases (NCDs), creating a barrier to successful treatment. While some treatments have proven efficacy in LMIC settings, wide dissemination is challenged by multiple factors, leading researchers to call for implementation strategies to overcome barriers to care provision. However, implementation strategies are often not well defined or documented, challenging the interpretation of study results and the uptake and replication of strategies in practice settings. Assessing implementation strategy fidelity (ISF), or the extent to which a strategy was implemented as designed, overcomes these challenges. This study assessed fidelity of two implementation strategies (a 'basic' champion strategy and an 'enhanced' champion + audit and feedback strategy) to improve the integration of a depression intervention, measurement based care (MBC), at 10 NCD clinics in Malawi. The primary goal of this study was to assess the relationship between the implementation strategies and MBC fidelity using a mixed methods approach.

Methods

We developed a theory-informed mixed methods fidelity assessment that first combined an implementation strategy specification technique with a fidelity framework. We then created corresponding fidelity indicators to strategy components. Clinical process data and one-on-one in-depth interviews with 45 staff members at 6 clinics were utilized as data sources. Our final analysis used descriptive statistics, reflexive-thematic analysis (RTA), data merging, and triangulation to examine the relationship between ISF and MBC intervention fidelity.

Results

Our mixed methods analysis revealed how ISF may moderate the relationship between the strategies and MBC fidelity. Leadership engagement and implementation climate were critical for clinics to overcome implementation barriers and preserve implementation strategy and MBC fidelity. Descriptive statistics determined champion strategy fidelity to range from 61 to 93% across the 10 clinics. Fidelity to the audit and feedback strategy ranged from 82 to 91% across the 5 clinics assigned to that condition. MBC fidelity ranged from 54 to 95% across all clinics. Although correlations between ISF and MBC fidelity were not statistically significant due to the sample of 10 clinics, associations were in the expected direction and of moderate effect size. A coefficient for shared depression screening among clinicians had greater face validity compared to depression screening coverage and functioned as a proximal indicator of implementation strategy success.

Conclusion

Fidelity to the basic and enhanced strategies varied by site and were influenced by leadership engagement and implementation climate. Champion strategies may benefit from the addition of leadership strategies to help address implementation barriers outside the purview of champions. ISF may moderate the relationship between strategies and implementation outcomes.

Improving dementia care: insights from audit and feedback in interdisciplinary primary care sites

Background

Many primary care sites have implemented models to improve detection, diagnosis, and management of dementia, as per Canadian guidelines. The aim of this study is to describe the responses of clinicians, managers, and staff of sites that have implemented these models when presented with audit results, their insights on the factors that explain their results, their proposed solutions for improvement and how these align to one another.

Methods

One audit and feedback cycle was carried out in eight purposefully sampled sites in Ontario, Canada, that had previously implemented dementia care models. Audit consisted of a) chart review to assess quality of dementia care indicators, b) questionnaire to assess the physicians’ knowledge, attitudes and practice toward dementia care, and c) semi-structured interviews to understand barriers and facilitators to implementing these models. Feedback was given to clinicians, managers, and staff in the form of graphic and oral presentations, followed by eight focus groups (one per site). Discussions revolved around: what audit results elicited more discussion from the participants, 2) their insights on the factors that explain their audit results, and 3) solutions they propose to improve dementia care. Deductive content and inductive thematic analyses, grounded in causal pathways models’ theory was performed.

Findings

The audit and feedback process allowed the 63 participants to discuss many audit results and share their insights on a) organizational factors (lack of human resources, the importance of organized links with community services, clear roles and support from external memory clinics) and b) clinician factors (perceived competency practice and attitudes on dementia care), that could explain their audit results. Participants also provided solutions to improve dementia care in primary care (financial incentives, having clear pathways, adding tools to improve chart documentation, establish training on dementia care, and the possibility of benchmarking with other institutions). Proposed solutions were well aligned with their insights and further nuanced according to contextual details.

Conclusions

This study provides valuable information on solutions proposed by primary care clinicians, managers, and staff to improve dementia care in primary care. The solutions are grounded in clinical experience and will inform ongoing and future dementia strategies.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12913-022-07672-5.

Prevalence of Effective Audit-and-Feedback Practices in Primary Care Settings: A Qualitative Examination Within Veterans Health Administration

OBJECTIVE

The purpose of this study is to uncover and catalog the various practices for delivering and disseminating clinical performance in various Veterans Affairs (VA) locations and to evaluate their quality against evidence-based models of effective feedback as reported in the literature.

BACKGROUND

Feedback can enhance clinical performance in subsequent performance episodes. However, evidence is clear that the way in which feedback is delivered determines whether performance is harmed or improved.

METHOD

We purposively sampled 16 geographically dispersed VA hospitals based on high, low, consistently moderate, and moderately average highly variable performance on a set of 17 outpatient clinical performance measures. We excluded four sites due to insufficient interview data. We interviewed four key personnel from each location (n = 48) to uncover effective and ineffective audit and feedback strategies. Interviews were transcribed and analyzed qualitatively using a framework-based content analysis approach to identify emergent themes.

RESULTS

We identified 102 unique strategies used to deliver feedback. Of these strategies, 64 (62.74%) have been found to be ineffective according to the audit-and-feedback research literature. Comparing features common to effective (e.g., individually tailored, computerized feedback reports) versus ineffective (e.g., large staff meetings) strategies, most ineffective strategies delivered feedback in meetings, whereas strategies receiving the highest effectiveness scores delivered feedback via visually understood reports that did not occur in a group setting.

CONCLUSIONS

Findings show that current practices are leveraging largely ineffective feedback strategies. Future research should seek to identify the longitudinal impact of current feedback and audit practices on clinical performance.

APPLICATION

Feedback in primary care has little standardization and does not follow available evidence for effective feedback design. Future research in this area is warranted.

Audit and feedback of therapist-assisted internet-delivered cognitive behaviour therapy within routine care: A quality improvement case study

With the growing use of ICBT in routine care clinics there is a need for literature on how to monitor and improve the quality of therapist behaviours in clinical practice. In this paper, we first provide background literature on Audit and Feedback (A&F), a common quality improvement technique, and then present a case study regarding the use of A&F to improve quality of therapist behaviours in emails sent to patients provided with ICBT in routine care. The A&F measure used was derived from previous research on therapist's email behaviours in ICBT. Fifteen undesirable therapist behaviours (e.g., Did Not Message, Unresponsive to Symptom Increase, Does Not Address Patient Concern) were audited in 1840 emails sent from eight therapists to 198 randomly selected patients, representing 18% of 1114 patients who started between one and five lessons of ICBT in the previous year and did not formally withdraw from treatment (n = 31 patients). The therapists who were audited were provided feedback four times over a one-year period from October 2018 to September 2019. Overall, in all audit periods, we found a low percentage of undesirable therapist behaviours (i.e., therapists displayed the behaviour in 12% or less of the total emails sent). For most therapist behaviours, we saw a trend towards improvement across the four audit cycles. Three therapist behaviours (i.e., Failure to Ask One Question to the Patient, Poor Instructions, Not Linking Email to Course Content) did not follow this pattern and were flagged for clinical discussion to determine why behaviours were elevated and whether these behaviours represented unrealistic expectations. The process was valuable for monitoring and improving therapist behaviours and highlights the need for future research on standards for therapist behaviours (e.g., which behaviours to focus on, setting acceptable levels of undesirable behaviour).

First-line managers' experience of the use of audit and feedback cycle in specialist mental health care: A qualitative case study

BACKGROUND

Audit and feedback was the main strategy to facilitate implementation of The National Guideline for Persons with Concurrent Substance Use Disorders and Mental Disorders in specialist mental health services. Studies have shown that leadership support contributes to implementation success. The aim of the study was to explore how first-line managers in a District Psychiatric Centre experienced using audit and feedback cycle.

METHOD

The study had a qualitative case study design with individual interviews with five first-line managers from a District Psychiatric Centre in Norway. Qualitative content analysis was conducted.

RESULTS

First-line managers were positive to contribute to better practice for the patient group and apply available tools. Four themes emerged: 1) Lack of endurance, where first-line managers saw their role as being process leaders, but failed to persist, 2) Lack of support in the process, where first-line managers called for a stronger organisational focus 3) Lack of ownership, where first-line managers felt the process was imposed on them, and 4) Lack of leader autonomy, where first-line managers seemed insecure about their role between professional leadership and own management.

CONCLUSION

First-line managers were not sufficiently experienced or equipped to solve the implementation process satisfactorily. They were torn between different commitments, without the autonomy to act as process drivers or facilitators, and without taking the necessary leadership role. The potential impact of the use of audit and feedback may thus not be fully realized, in part, because of limited organisational support and capacity to respond effectively.

A Quality Improvement Assessment of the Delivery of Mental Health Services among WTC Responders Treated in the Community

The World Trade Center Health Program (WTCHP) provides mental health services through diverse service delivery mechanisms, however there are no current benchmarks to evaluate utilization or quality. This quality improvement (QI) initiative sought to examine the delivery and effectiveness of WTCHP mental health services for World Trade Center (WTC) responders who receive care through the Northwell Health Clinical Center of Excellence (CCE), and to characterize the delivery of evidence-based treatments (EBT) for mental health (MH) difficulties in this population. Methods include an analysis of QI data from the Northwell CCE, and annual WTCHP monitoring data for all responders certified for mental health treatment. Nearly 48.9% of enrolled responders with a WTC-certified diagnosis utilized treatment. The majority of treatment delivered was focused on WTC-related conditions. There was significant disagreement between provider-reported EBT use and independently-evaluated delivery of EBT (95.6% vs. 54.8%, p ≤ 0.001). EBT delivery was associated with a small decrease in Posttraumatic Stress Disorder (PTSD) symptoms over time. Providers engaged in the process of data collection, but there were challenges with adherence to outcome monitoring and goal setting. Data from this report can inform continued QI efforts in the WTCHP, as well as the implementation and evaluation of EBT.