How Has Intervention Fidelity Been Assessed in Smoking Cessation Interventions? A Systematic Review

Adapted motivational interviewing for brief healthcare consultations: A systematic review and meta-analysis of treatment fidelity in real-world evaluations of behaviour change counselling

BACKGROUND

Behaviour change counselling (BCC) is an adaptation of motivational interviewing (MI) designed to maximize the effectiveness of time-limited health behaviour change consultations. To improve intervention quality and understanding of treatment effects, it is recommended that evaluations of health behaviour change interventions incorporate existing fidelity frameworks (e.g. The National Institutes of Health [NIH] Behaviour Change Consortium) and ensure that treatment fidelity is assessed and reported.

PURPOSE

This systematic review was designed to examine (a) adherence to NIH fidelity recommendations, (b) provider fidelity to BCC and (c) impact of these variables on the real-world effectiveness of BCC for adult health behaviours and outcomes.

METHODS AND RESULTS

Searches of 10 electronic databases yielded 110 eligible publications describing 58 unique studies examining BCC delivered within real-world healthcare settings by existing providers. Mean study adherence to NIH fidelity recommendations was 63.31% (Range 26.83%-96.23%). Pooled effect size (Hedges g) for short-term and long-term outcomes was .19 (95% CI [.11, .27]) and .09 (95% CI [.04, .13]), respectively. In separate, random-effects meta-regressions, neither short-term nor long-term effect sizes were significantly modified by adherence to NIH fidelity recommendations. For the subgroup of short-term alcohol studies (n = 10), a significant inverse relationship was detected (Coefficient = -.0114, 95% CI [-.0187, -.0041], p = .0021). Inadequate and inconsistent reporting within the included studies precluded planned meta-regression between provider fidelity and BCC effect size.

CONCLUSIONS

Further evidence is needed to clarify whether adherence to fidelity recommendations modifies intervention effects. Efforts to promote transparent consideration, evaluation and reporting of fidelity are urgently needed. Research and clinical implications are discussed.

Measures of fidelity of delivery and engagement in self-management interventions: A systematic review of measures

BACKGROUND/AIMS

Self-management interventions are increasingly being developed and researched to improve long-term condition outcomes. To understand and interpret findings, it is essential that fidelity of intervention delivery and participant engagement are measured and reported. Before developing fidelity checklists to assess treatment fidelity of interventions, current recommendations suggest that a synthesis of fidelity measures reported in the literature is completed. Therefore, here we aim to identify what the current measures of fidelity of intervention delivery and engagement for self-management interventions for long-term conditions are and whether there is treatment fidelity.

METHODS

Four databases (MEDLINE, PubMed, CINAHL Plus and ScienceDirect) and the journal implementation science were systematically searched to identify published reports from inception to December 2020 for experimental studies measuring fidelity of intervention delivery and/or participant engagement in self-management interventions for long-term conditions. Data on fidelity of delivery and engagement measures and the findings were extracted and synthesised.

RESULTS

Thirty-nine articles were identified as eligible, with 25 studies measuring fidelity of delivery, 19 reporting engagement and 5 measuring both. For fidelity of delivery, measures included structured checklists, participant completed measures and researcher observations/notes. These were completed by researchers, participants and intervention leaders. Often there was little information around the development of these measures, particularly when the measure had been developed by the researchers, rather than building on others work. Eighteen of 25 studies reported there was fidelity of intervention delivery. For engagement, measures included data analytics, participant completed measures and researcher observations. Ten out of 19 studies reported participants were engaged with the intervention.

CONCLUSION

In complex self-management interventions, it is essential to assess whether treatment fidelity of each core component of interventions is delivered, as outlined in the protocol, to understand which components are having an effect. Treatment fidelity checklists comparing what was planned to be delivered, with what was delivered should be developed with pre-defined cut-offs for when fidelity has been achieved. Similarly, when measuring engagement, while data analytics continue to rise with the increase in digital interventions, clear cut-offs for participant use and content engaged with to be considered an engagement participant need to be pre-determined.

Exploring the Black Box of an mHealth Intervention (LIFE4YOUth): A Qualitative Process and Outcome Evaluation of End-User Engagement

The effectiveness of mHealth interventions rely on whether the content successfully activate mechanisms necessary for behavior change. These mechanisms may be affected by end-users' experience of the intervention content. The aim of this study was to explore how the content of a novel mHealth intervention (LIFE4YOUth) was understood, interpreted, and applied by high school students, and the consequences of engaging with the content. Qualitative content analysis was used inductively and deductively to analyze interview data (n = 16) based on think-aloud techniques with Swedish high school students aged 16-19 years. Theoretical constructs from social cognitive theory framed the deductive analysis. The analysis resulted in four categories which describe central activities of intervention engagement among end-users: defining, considering, centralizing, and personalizing. End-users engaged in these activities to different degrees as illustrated by four typologies: Literal, Vague, Rigid, and Creative engagement. Most informants knew about the risks and benefits of health behaviors, but engagement with intervention content generally increased informants' awareness. In conclusion, this study provides in-depth knowledge on the cognitive process when engaging with mHealth content and suggests that deliberate and flexible engagement most likely deepens end-users' understanding of why and how health behavior change can be managed.

The case for prioritizing implementation strategy fidelity measurement: benefits and challenges

Implementation strategies are systematic approaches to improve the uptake and sustainability of evidence-based interventions. They frequently focus on changing provider behavior through the provision of interventions such as training, coaching, and audit-and-feedback. Implementation strategies often impact intermediate behavioral outcomes like provider guideline adherence, in turn improving patient outcomes. Fidelity of implementation strategy delivery is defined as the extent to which an implementation strategy is carried out as it was designed. Implementation strategy fidelity measurement is under-developed and under-reported, with the quality of reporting decreasing over time. Benefits of fidelity measurement include the exploration of the extent to which observed effects are moderated by fidelity, and critical information about Type-III research errors, or the likelihood that null findings result from implementation strategy fidelity failure. Reviews of implementation strategy efficacy often report wide variation across studies, commonly calling for increased implementation strategy fidelity measurement to help explain variations. Despite the methodological benefits of rigorous fidelity measurement, implementation researchers face multi-level challenges and complexities. Challenges include the measurement of a complex variable, multiple data collection modalities with varying precision and costs, and the need for fidelity measurement to change in-step with adaptations. In this position paper, we weigh these costs and benefits and ultimately contend that implementation strategy fidelity measurement and reporting should be improved in trials of implementation strategies. We offer pragmatic solutions for researchers to make immediate improvements like the use of mixed methods or innovative data collection and analysis techniques, the inclusion of implementation strategy fidelity assessment in reporting guidelines, and the staged development of fidelity tools across the evolution of an implementation strategy. We also call for additional research into the barriers and facilitators of implementation strategy fidelity measurement to further clarify the best path forward.

How is the Behavior Change Technique Content of the NHS Diabetes Prevention Program Understood by Participants? A Qualitative Study of Fidelity, With a Focus on Receipt

Background

The National Health Service (NHS) Diabetes Prevention Program (DPP) is a nationally implemented behavioral intervention for adults at high risk of developing Type 2 diabetes in England, based on a program specification that stipulates inclusion of 19 specific behavior change techniques (BCTs). Previous work has identified drift in fidelity from these NHS England specifications through providers’ program manuals, training, and delivery, especially in relation to BCTs targeting self-regulatory processes.

Purpose

This qualitative study investigates intervention receipt, i.e., how the self-regulatory BCT content of the NHS-DPP is understood by participants.

Methods

Twenty participants from eight NHS-DPP locations were interviewed; topics included participants’ understanding of self-monitoring of behavior, goal setting, feedback, problem solving, and action planning. Transcripts were analyzed thematically using the framework method.

Results

There was a wide variation in understanding among participants for some BCTs, as well as between BCTs. Participants described their understanding of “self-monitoring of behaviors” with ease and valued BCTs focused on outcomes (weight loss). Some participants learned how to set appropriate behavioral goals. Participants struggled to recall “action planning” or “problem solving” or found these techniques challenging to understand, unless additional support was provided (e.g., through group discussion).

Conclusions

Participants’ lack of understanding of some self-regulatory BCTs is consistent with the drift across fidelity domains previously identified from NHS design specifications. Behavioral interventions should build-in necessary support for participants to help them understand some BCTs such as action planning and problem solving. Alternatively, these self-regulatory BCTs may be intrinsically difficult to use for this population.