Intensive Quality Assurance of Therapist Adherence to Behavioral Interventions for Adolescent Substance Use Problems

A mixed-methods study to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings

It has been well-documented that the degree to which interventions are implemented with fidelity in typical service settings has varied. Frequently, interventions are developed and tested in highly controlled or early adopter settings. Less attention has been given to what implementation looks like in usual care, and which factors promote practitioners' ability to implement with fidelity. Individuals and organizations implementing interventions in the real world receive varying levels of external supports and may apply a new intervention unaided. The purpose of this mixed-methods study was to explore factors that support implementation as intended in local community agencies. In the quantitative phase of this study, 32 case planners implementing Family Connections (FC), a child maltreatment preventive intervention, completed a survey about their perceptions of practitioner and organizational factors related to fidelity. The survey data were connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner's ability to implement an intervention in usual care. The quantitative and qualitative results suggest supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioners' ability to learn and use FC. The quantitative results suggest that the level of education was positively associated with fidelity and perceptions of the intervention's limitations may be negatively related to implementation. Additional components that influence implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

Plain Language Abstract

This mixed-methods study sought to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings. The study first asked case planners about their perceptions of practitioner and organizational factors related to fidelity through an online survey. This survey was connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner’s ability to implement a maltreatment prevention intervention. Both methods of the study suggest that various aspects of supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioner's ability to learn and use the intervention. Additional components that influence the implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

A systematic review of empirical studies examining mechanisms of implementation in health

BACKGROUND

Understanding the mechanisms of implementation strategies (i.e., the processes by which strategies produce desired effects) is important for research to understand why a strategy did or did not achieve its intended effect, and it is important for practice to ensure strategies are designed and selected to directly target determinants or barriers. This study is a systematic review to characterize how mechanisms are conceptualized and measured, how they are studied and evaluated, and how much evidence exists for specific mechanisms.

METHODS

We systematically searched PubMed and CINAHL Plus for implementation studies published between January 1990 and August 2018 that included the terms "mechanism," "mediator," or "moderator." Two authors independently reviewed title and abstracts and then full texts for fit with our inclusion criteria of empirical studies of implementation in health care contexts. Authors extracted data regarding general study information, methods, results, and study design and mechanisms-specific information. Authors used the Mixed Methods Appraisal Tool to assess study quality.

RESULTS

Search strategies produced 2277 articles, of which 183 were included for full text review. From these we included for data extraction 39 articles plus an additional seven articles were hand-entered from only other review of implementation mechanisms (total = 46 included articles). Most included studies employed quantitative methods (73.9%), while 10.9% were qualitative and 15.2% were mixed methods. Nine unique versions of models testing mechanisms emerged. Fifty-three percent of the studies met half or fewer of the quality indicators. The majority of studies (84.8%) only met three or fewer of the seven criteria stipulated for establishing mechanisms.

CONCLUSIONS

Researchers have undertaken a multitude of approaches to pursue mechanistic implementation research, but our review revealed substantive conceptual, methodological, and measurement issues that must be addressed in order to advance this critical research agenda. To move the field forward, there is need for greater precision to achieve conceptual clarity, attempts to generate testable hypotheses about how and why variables are related, and use of concrete behavioral indicators of proximal outcomes in the case of quantitative research and more directed inquiry in the case of qualitative research.

A systematic review of empirical studies examining mechanisms of implementation in health

BACKGROUND

Understanding the mechanisms of implementation strategies (i.e., the processes by which strategies produce desired effects) is important for research to understand why a strategy did or did not achieve its intended effect, and it is important for practice to ensure strategies are designed and selected to directly target determinants or barriers. This study is a systematic review to characterize how mechanisms are conceptualized and measured, how they are studied and evaluated, and how much evidence exists for specific mechanisms.

METHODS

We systematically searched PubMed and CINAHL Plus for implementation studies published between January 1990 and August 2018 that included the terms "mechanism," "mediator," or "moderator." Two authors independently reviewed title and abstracts and then full texts for fit with our inclusion criteria of empirical studies of implementation in health care contexts. Authors extracted data regarding general study information, methods, results, and study design and mechanisms-specific information. Authors used the Mixed Methods Appraisal Tool to assess study quality.

RESULTS

Search strategies produced 2277 articles, of which 183 were included for full text review. From these we included for data extraction 39 articles plus an additional seven articles were hand-entered from only other review of implementation mechanisms (total = 46 included articles). Most included studies employed quantitative methods (73.9%), while 10.9% were qualitative and 15.2% were mixed methods. Nine unique versions of models testing mechanisms emerged. Fifty-three percent of the studies met half or fewer of the quality indicators. The majority of studies (84.8%) only met three or fewer of the seven criteria stipulated for establishing mechanisms.

CONCLUSIONS

Researchers have undertaken a multitude of approaches to pursue mechanistic implementation research, but our review revealed substantive conceptual, methodological, and measurement issues that must be addressed in order to advance this critical research agenda. To move the field forward, there is need for greater precision to achieve conceptual clarity, attempts to generate testable hypotheses about how and why variables are related, and use of concrete behavioral indicators of proximal outcomes in the case of quantitative research and more directed inquiry in the case of qualitative research.

Systematic literature review and meta-analysis of the relationship between adherence, competence and outcome in psychotherapy for children and adolescents

The extent to which therapist adherence to guidelines and clinician skill or competence may play a role in the prediction of therapeutic outcomes remains inconclusive. This systematic literature review and meta-analysis considers whether adherence or competence predicts youth outcome in child and adolescent psychotherapy, and whether there are any identifiable factors which moderate the strength or direction of this relationship. A systematic literature search identified 35 studies in 52 papers. The studies contained 29 effect sizes for the relationship between adherence and outcome, while nine effect sizes were extracted for competence, and a further five effects measured a composite of adherence and competence constructs, referred to as fidelity in this report. The meta-analysis indicated a small but significant relationship between therapist adherence and outcome, although the small size of effect suggests that outcomes are likely to be more strongly associated with factors other than adherence. No significant relationship was identified between competence or composite fidelity and outcome. Although variance was observed in effect sizes, no significant moderation by client group, intervention type, or implementation measure informant was identified. Further study is needed to understand the specific circumstances under which adherence and outcome are related.

A Web-Delivered Multicomponent Intervention for Adolescents with Poorly Controlled Type 1 Diabetes: A Pilot Randomized Controlled Trial

Background

Type 1 diabetes is associated with significant mortality and economic cost. Management of type 1 diabetes involves completing multiple daily adherence behaviors, and many adolescents struggle with self-management and show poor glycemic control.

Purpose

The purpose was to conduct an unblinded pilot randomized controlled parallel-group study of a web-delivered multicomponent intervention targeting self-monitoring of blood glucose, working memory, and parent supervision of diabetes care among adolescents with type 1 diabetes. Intervention components included high magnitude incentives for adolescents and parents, motivational and cognitive behavioral therapy and working memory training for adolescents, and training in contingency contracting for parents.

Methods

Adolescents (N = 114) with poorly controlled type 1 diabetes were screened, and N = 61 were randomized using minimum likelihood allocation to usual care (usual care, N = 31) or to a 25-week/15-session web-delivered intervention (WebRx, N = 30).

Results

At the end of treatment, adolescents in WebRx had higher self-monitoring of blood glucose (d = 0.58) (primary outcome), better visual spatial working memory (d = 0.48) and inhibition (d = 0.98), and lower HbA1c (d = 0.45) than those in usual care. WebRx parents reported more frequent review of the adolescent's glucometer (d = 1.30) and reduced family conflict (d = 0.56). Between-condition differences were maintained 6 months later in self-monitoring of blood glucose (d = 0.42), visual spatial working memory (d = 0.76), family conflict (d = 0.50), and HbA1c (d = 0.44).

Conclusions

Results showing sustained effects on self-monitoring of blood glucose and HbA1c support moving forward with a larger trial to test this innovative web-delivered and multicomponent intervention. ClinicalTrials.gov Number (NCT01722643).

Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice

A step toward the development of optimally effective, efficient, and feasible implementation strategies that increase evidence-based treatment integration in mental health services involves identification of the multilevel mechanisms through which these strategies influence implementation outcomes. This article (a) provides an orientation to, and rationale for, consideration of multilevel mediating mechanisms in implementation trials, and (b) systematically reviews randomized controlled trials that examined mediators of implementation strategies in mental health. Nine trials were located. Mediation-related methodological deficiencies were prevalent and no trials supported a hypothesized mediator. The most common reason was failure to engage the mediation target. Discussion focuses on directions to accelerate implementation strategy development in mental health.

It's not just the Therapist: Therapist and Country-Wide Experience Predict Therapist Adherence and Adolescent Outcome

Objective

Therapist adherence is a quality indicator in routine clinical care when evaluating the success of the implementation of an intervention. The current study investigated whether therapist adherence mediates the association between therapist, team, and country-wide experience (i.e. number of years since implementation in the country) on the one hand, and treatment outcome on the other hand. We replicated and extended a study by Löfholm et al. (2014).

Method

Data over a 10-year period were obtained from 4290 adolescents (12–17 years) with antisocial or delinquent problem behavior, who were treated with Multisystemic Therapy (MST) by 222 therapists, working in 27 different teams in the Netherlands. Multilevel structural equation modeling was used to assess the associations between experience, therapist adherence, and post-treatment outcomes.

Results

Treatment outcomes were directly predicted by therapist experience, countrywide experience, and therapist adherence, but not by team experience. Moreover, therapist adherence mediated the association between therapist and country-wide experience, and treatment outcomes. The association between therapist experience and therapist adherence was not affected by the number of years of team experience or country-wide experience.

Conclusion

The effect of country-wide experience on outcome may reflect increasing experience of training and supporting the therapists. It suggests that nation-wide quality control may relate to better therapist adherence and treatment outcome for adolescents treated with systemic therapy.

Contingency Management Approaches for Adolescent Substance Use Disorders

Multiple interventions for treating adolescents with substance use disorders have demonstrated efficacy, but a majority of teens do not show an enduring positive response to these treatments. Contingency management (CM)-based strategies provide a promising alternative, and clinical research focused on the development and testing of innovative CM models continues to grow. This article provides an updated review on the progress made in this area. It is important to continue to search for more effective models, focus on post-treatment maintenance (reduce relapse), and strive for high levels of integrity and fidelity during dissemination efforts to optimize outcomes.

Multilevel Mechanisms of Implementation Strategies in Mental Health: Integrating Theory, Research, and Practice

A step toward the development of optimally effective, efficient, and feasible implementation strategies that increase evidence-based treatment integration in mental health services involves identification of the multilevel mechanisms through which these strategies influence implementation outcomes. This article (a) provides an orientation to, and rationale for, consideration of multilevel mediating mechanisms in implementation trials, and (b) systematically reviews randomized controlled trials that examined mediators of implementation strategies in mental health. Nine trials were located. Mediation-related methodological deficiencies were prevalent and no trials supported a hypothesized mediator. The most common reason was failure to engage the mediation target. Discussion focuses on directions to accelerate implementation strategy development in mental health.

Evidence base on outpatient behavioral treatments for adolescent substance use: updates and recommendations 2007-2013

This article updates the evidence base on outpatient behavioral treatments for adolescent substance use (ASU) since publication of the previous review completed for this journal by Waldron and Turner ( 2008 ). It first summarizes the Waldron and Turner findings as well as those from more recent literature reviews and meta-analytic studies of ASU treatment. It then presents study design and methods criteria used to select 19 comparative studies subjected to Journal of Clinical Child & Adolescent Psychology level of support evaluation. These 19 studies are grouped by study category (efficacy or effectiveness) and described for sample characteristics, methodological quality, and substance use outcomes. Cumulative level of support designations are then made for each identified treatment approach: ecological family-based treatment, group cognitive-behavioral therapy, and individual cognitive-behavioral therapy are deemed Well Established; behavioral family-based treatment and motivational interviewing are deemed Probably Efficacious; drug counseling is deemed Possibly Efficacious; and four integrated treatment models combining more than one approach are deemed Well Established or Probably Efficacious. The remainder of the article (a) articulates fidelity, mediator, and moderator effects reported for evidence-based approaches since 2008 and (b) recommends four enhancements to the prevailing business model of ASU outpatient services to accelerate penetration of evidence-based approaches into the underserved consumer base: pursue partnerships with influential governmental systems, utilize web-based technology to extend reach and control costs, adapt effective methods for linking services across sectors of care, and promote uptake and sustainability by emphasizing return on investment.

Implementation of Treatment Integrity Procedures

This systematic review evaluates the implementation of treatment integrity procedures in outcome studies of youth interventions targeting behavioral problems. The Implementation of Treatment Integrity Procedures Scale (ITIPS), developed by Perepletchikova, Treat, and Kazdin (2007) , was adapted (ITIPS-A) and used to evaluate 32 outcome studies of evidence-based interventions for youths with externalizing behavioral problems. Integrity measures were found to be still rare in these studies. Of the studies that took integrity into account, 80% approached adequacy in implementing procedures for treatment integrity. The ITIPS-A is recommended as an instrument to guide development of integrity instruments and the implementation of treatment integrity procedures in youth care.

Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review

OBJECTIVE

Although there has been a dramatic increase in the number of evidence-based practices (EBPs) to improve child and adolescent mental health, the poor uptake of these EBPs has led to investigations of factors related to their successful dissemination and implementation. The purpose of this systematic review was to identify key findings from empirical studies examining the dissemination and implementation of EBPs for child and adolescent mental health.

METHOD

Of 14,247 citations initially identified, 73 articles drawn from 44 studies met inclusion criteria. The articles were classified by implementation phase (exploration, preparation, implementation, and sustainment) and specific implementation factors examined. These factors were divided into outer (i.e., system level) and inner (i.e., organizational level) contexts.

RESULTS

Few studies used true experimental designs; most were observational. Of the many inner context factors that were examined in these studies (e.g., provider characteristics, organizational resources, leadership), fidelity monitoring and supervision had the strongest empirical evidence. Albeit the focus of fewer studies, implementation interventions focused on improving organizational climate and culture were associated with better intervention sustainment as well as child and adolescent outcomes. Outer contextual factors such as training and use of specific technologies to support intervention use were also important in facilitating the implementation process.

CONCLUSIONS

The further development and testing of dissemination and implementation strategies is needed to more efficiently move EBPs into usual care.