MONITORING FIDELITY IN THE ADOLESCENT COMMUNITY REINFORCEMENT APPROACH (A-CRA): THE TRAINING PROCESS FOR A-CRA RATERS

Expanding outcomes when considering the relative effectiveness of two evidence-based outpatient treatment programs for adolescents

The current study seeks to advance understanding about how to address substance use and co-occurring mental health problems in adolescents. Specifically, we compared the effectiveness of two evidence-based treatment programs (Motivational Enhancement Treatment/Cognitive Behavior Therapy, 5 Sessions [MET/CBT5] and Adolescent Community Reinforcement Approach [A-CRA]) for both substance use and mental health outcomes (i.e., crossover effects). We used statistical methods designed to approximate randomized controlled trials when comparing nonequivalent groups using observational study data. Our methods also included an assessment of the potential impact of omitted variables. We found that after applying balancing weighting to ensure similarity of the baseline samples (given the nonrandomized study design), both groups significantly improved on the two substance use outcomes (days abstinent and percent of youth in recovery) and on the two mental health outcomes (post-traumatic stress disorder (PTSD) symptoms and general emotional problems). Youth in A-CRA were significantly more likely to be in recovery at the 3-month follow-up compared to youth in MET/CBT5, but the size of this effect was very small. Youth receiving MET/CBT5 appeared to show significantly more improvement in the two mental health measures compared to youth in A-CRA, though these effect sizes were also very small. The findings indicate that adolescents with co-occurring substance use and mental health problems improve on both substance use and mental health outcomes with both treatments even though they are not specifically targeting mental health problems.

Toward Evidence-Based Measures of Implementation: Examining the Relationship Between Implementation Outcomes and Client Outcomes

BACKGROUND

Developing consistent, valid, and efficient implementation outcome measures is necessary to advance implementation science. However, development of such measures has been limited to date, especially for validating the extent to which such measures are associated with important improvements in client outcomes. This study seeks to address this gap by developing one or more evidence-based measures of implementation (EBMIs; i.e., implementation outcome measure that is predictive of improvements in key client outcomes) for the Adolescent Community Reinforcement Approach (A-CRA), an evidence-based practice (EBP) for adolescent substance use.

METHODS

Data for the current study were collected as part of a large-scale federally funded EBP dissemination and implementation initiative. The multilevel dataset included 65 substance use treatment organizations, 308 clinicians, and 5873 adolescent clients. Adjusted multilevel regression analyses were used to examine the extent to which client-level outcome measures assessed at 6-month follow-up (i.e., substance use, emotional problems) could be predicted by four implementation outcomes: two measures of fidelity (i.e., session exposure, procedure exposure) and two measures of penetration (i.e., absolute client penetration, absolute staff penetration).

RESULTS

Adjusting for client substance use at intake, as well as several client characteristics (e.g., age, race, criminal justice involvement), client substance use at follow-up was significantly lower for treatment organizations that had higher procedure exposure (B=-1.227, standard error [SE]=0.583, 95% confidence interval=-2.370, 0.252; p<.05). None of the other three implementation outcome measures were found to predict improvements in client outcomes.

CONCLUSIONS

The current study provides support for procedure exposure as an organizational-level EBMI for A-CRA. Thus, future efforts focused on implementing A-CRA could be improved by measuring and monitoring the extent to which A-CRA procedures are being delivered to clients. Additionally, given the dearth of studies that have examined the relationship between organizational-level measures of implementation and client outcomes, this article provides a prototype for future research to identify EBMIs for other behavioral treatments.

Associations between implementation characteristics and evidence-based practice sustainment: a study of the Adolescent Community Reinforcement Approach

Background

Few empirical studies longitudinally examine evidence-based practice (EBP) sustainment and the hypothesized factors that influence it. In an effort to address this gap, the current study examined sustainment of an EBP for adolescent substance use called the adolescent community reinforcement approach (A-CRA).

Methods

A-CRA sustainment was assessed via information collected as part of key informant interviews and surveys with clinical staff from community-based treatment organizations that had received federal funding to implement A-CRA. Administrative data from the funding period on implementation was also used. Using discrete-time survival analysis, we regressed A-CRA sustainment on several factors theorized to influence EBP sustainment. Factors examined included outer setting, inner setting, implementation quality during the funding period, and intervention-related characteristics.

Results

Overall, data from 83 % of the targeted sample of treatment organizations was collected. A-CRA sustainment was strongly related to the time since funding loss. Strong relationships were found between sustainment status and implementation quality during the funding period, agency focus, funding stability, and political support for the treatment along with staff perceptions of the treatment’s complexity and implementation difficulty.

Conclusions

Consistent with the Consolidated Framework for Implementation Research, the current study found several factors related to the outer setting (e.g., funding stability), inner setting (e.g., agency focus), implementation quality during the funding period (e.g., staff trained, clients served), and characteristics of the intervention (e.g., implementation complexity) to be associated with EBP sustainment. Future research is warranted to examine the extent to which these relationships are stable over time. Efforts to ensure that adequate implementation occurs during the initial implementation period and that adequate funding, infrastructure, and staff support following the ending of initial support are critical to a program’s survival.

Examining the sustainment of the Adolescent-Community Reinforcement Approach in community addiction treatment settings: protocol for a longitudinal mixed method study

BACKGROUND

Although evidence-based treatments are considered the gold standard for clinical practice, it is widely recognized that evidence-based treatment implementation in real world practice settings has been limited. To address this gap, the federal government provided three years of funding, training and technical assistance to 84 community-based treatment programs to deliver an evidence-based treatment called the Adolescent-Community Reinforcement Approach (A-CRA). Little is known about whether such efforts lead to long-term A-CRA sustainment after the initial funding ends.

METHODS/DESIGN

We will use a longitudinal mixed method data analytic approach to characterize sustainment over time and to examine the factors associated with the extent to which A-CRA is sustained. We will use implementation data collected during the funding period (e.g., organizational functioning, staff certification rates and penetration) and supplement it with additional data collected during the proposed project period regarding implementation quality and the hypothesized predictors of sustainment (i.e., inner and outer contextual variables) collected over three waves from 2013 to 2015 representing program sustainment up to five years post-initial funding.

DISCUSSION

Gaining a better understanding of the factors that influence the evidence-based treatment sustainment may lead to more effective dissemination strategies and ultimately improve the quality of care being delivered in community-based addiction treatment settings.

A Behaviorally-Anchored Rating System to Monitor Treatment Integrity for Community Clinicians Using the Adolescent Community Reinforcement Approach

This study evaluated a process for training raters to reliably rate clinicians delivering the Adolescent Community Reinforcement Approach (A-CRA) in a national dissemination project. The unique A-CRA coding system uses specific behavioral anchors throughout its 73 procedure components. Five randomly-selected raters each rated "passing" and "not passing" examples of the 19 A-CRA procedures. Ninety-four percent of the final ICCs were at least 'good' (≥.60) and 66.7% were 'excellent' (≥.75), and 95% of the ratings exceeded the 60% or better agreement threshold between raters and the gold standard. Raters can be trained to provide reliable A-CRA feedback for large-scale dissemination projects.

Training and retaining staff to competently deliver an evidence-based practice: the role of staff attributes and perceptions of organizational functioning

Within the context of an initiative to implement evidence-based practices (EBPs) for adolescents with substance use disorders, this study examined the extent to which staff factors measured at an initial EBP training workshop were predictive of EBP competence and turnover status of staff (N = 121) measured 6, 9, and 12 months posttraining. By the final assessment point, 52.3% of staff transitioned to the employed/EBP-competent category, 26.6% transitioned to the not employed/not EBP-competent category, 4.6% transitioned to the not employed/EBP-competent category, and 16.5% had not transitioned out of the initial category. Multilevel multinomial regression analysis identified several measures that were significant predictors of staff transitions to the not employed/not EBP-competent category (e.g., program needs, job satisfaction, burnout) and transitions to the employed/EBP-competent category (e.g., months in position, pressures for change, influence). Findings have implications for the development and testing of strategies to train and retain staff to deliver EBPs in practice settings.

A Large-scale Dissemination and Implementation Model for Evidence-based Treatment and Continuing Care

Multiple evidence-based treatments for adolescents with substance use disorders are available; however, the diffusion of these treatments in practice remains minimal. A dissemination and implementation model incorporating research-based training components for simultaneous implementation across 33 dispersed sites and over 200 clinical staff is described. Key elements for the diffusion of the Adolescent Community Reinforcement Approach and Assertive Continuing Care were: (a) three years of funding to support local implementation; (b) comprehensive training, including a 3.5 day workshop, bi-weekly coaching calls, and ongoing performance feedback facilitated by a web tool; (c) a clinician certification process; (d) a supervisor certification process to promote long-term sustainability; and (e) random fidelity reviews after certification. Process data are summarized for 167 clinicians and 64 supervisors.

The community reinforcement approach: an update of the evidence

The Community Reinforcement Approach (CRA), originally developed for individuals with alcohol use disorders, has been successfully employed to treat a variety of substance use disorders for more than 35 years. Based on operant conditioning, CRA helps people rearrange their lifestyles so that healthy, drug-free living becomes rewarding and thereby competes with alcohol and drug use. Consequently, practitioners encourage clients to become progressively involved in alternative non-substance-related pleasant social activities, and to work on enhancing the enjoyment they receive within the "community" of their family and job. Additionally, in the past 10-15 years, researchers have obtained scientific evidence for two off-shoots of CRA that are based on the same operant mechanism. The first variant is Adolescent Community Reinforcement Approach (A-CRA), which targets adolescents with substance use problems and their caregivers. The second approach, Community Reinforcement and Family Training (CRAFT), works through family members to engage treatment-refusing individuals into treatment. An overview of these treatments and their scientific backing is presented.

The Reinforcing Therapist Performance (RTP) experiment: study protocol for a cluster randomized trial

Background

Rewarding provider performance has been recommended by the Institute of Medicine as an approach to improve the quality of treatment, yet little empirical research currently exists that has examined the effectiveness and cost-effectiveness of such approaches. The aim of this study is to test the effectiveness and cost-effectiveness of providing monetary incentives directly to therapists as a method to improve substance abuse treatment service delivery and subsequent client treatment outcomes.

Design

Using a cluster randomized design, substance abuse treatment therapists from across 29 sites were assigned by site to either an implementation as usual (IAU) or pay-for-performance (P4P) condition.

Participants

Substance abuse treatment therapists participating in a large dissemination and implementation initiative funded by the Center for Substance Abuse Treatment.

Intervention

Therapists in both conditions received comprehensive training and ongoing monitoring, coaching, and feedback. However, those in the P4P condition also were given the opportunity to earn monetary incentives for achieving two sets of measurable behaviors related to quality implementation of the treatment.

Outcomes

Effectiveness outcomes will focus on the impact of the monetary incentives to increase the proportion of adolescents who receive a targeted threshold level of treatment, months that therapists demonstrate monthly competency, and adolescents who are in recovery following treatment. Similarly, cost-effectiveness outcomes will focus on cost per adolescent receiving targeted threshold level of treatment, cost per month of demonstrated competence, and cost per adolescent in recovery.

Trial Registration

Trial Registration Number: NCT01016704