Community Implementation of MI-Enhanced Behavior Therapy for Adolescent ADHD: Linking Fidelity to Effectiveness

The Interrater Reliability of a Coding System for Measuring Mental Health Professionals' Decisions and Actions

OBJECTIVE

The clinical decisions and actions of evidence-based practice in psychology (EBPP) are largely underspecified and poorly understood, in part due to the lack of measurement methods. We tested the reliability of a behavioral coding system that characterizes a flow of interrelated activities that includes problem detection and prioritization, intervention selection and implementation, and review of intervention integrity and impact.

METHOD

The context included two publicly funded youth mental health service organizations located in geographically distinct and underresourced communities in the U.S. where service inequities are common. We sampled 84 digitally recorded and transcribed supervision events that included a sample of professionals who were mostly women (93.02%) and BIPOC (86.04%) whose self-reported race/ethnicity matched the youth populations they served. We coded these events for activities (e.g., considering) and their predicate content (i.e., problems or practices) and examined reliability of these codes applied to excerpts (i.e., small contiguous units of dialogue) as well as to complete events.

RESULTS

Interrater reliability estimates showed that, overall, coders reliably rated the occurrence and extensiveness of activities and content. Excerpt coding was generally more reliable than event coding. However, mathematical aggregation of excerpt coding offered a superior method for estimating event codes reliably, reducing individual subjectivity while providing event level synthesis of activities that are grounded in excerpt level details.

CONCLUSIONS

The assessment of clinical decisions and actions has the potential to unpack the black box of EBPP, with different methods best suited to different research questions and resource considerations.

Validating a Pragmatic Measure of Evidence-Based Practice (EBP) Delivery: Therapist Reports of EBP Strategy Delivery and Associations with Child Outcome Trajectories

Pragmatic measures of evidence-based practice (EBP) implementation can support and evaluate implementation efforts. We examined the predictive validity of therapist reports of EBP strategy delivery for children's mental health outcomes. Data were obtained from 1,380 sessions with 248 children delivered by 76 therapists in two county systems. Children (Mage=11.8 years, SD = 3.7) presented with internalizing (52%), externalizing (27%), trauma (16%), and other (5%) concerns. Therapists reported their delivery of EBP strategies on a revised version of the EBP Concordant Care Assessment (ECCA; Brookman-Frazee, et al., Administration and Policy in Mental Health and Mental Health Services Research, 48, 155-170, 2021) that included 25 content (e.g., parenting, cognitive behavioral) and 12 technique strategies (e.g., modeling, practice/role-play). On average, 5.6 ECCA session reports (SD = 2.3) were obtained for each client, and caregivers reported symptoms on the Brief Problem Checklist (Chorpita, et al., Journal of Consulting and Clinical Psychology, 78(4), 526-536, 2010) at baseline, weekly over two months, and again at four months. Multilevel models examined whether the mean extensiveness of each EBP strategy predicted trajectories of child outcomes. More individual technique (6 of 12) than content strategies (1 of 25) were associated with outcome trajectories. For techniques, more extensive use of Performance Feedback and Live Coaching and less extensive use of Addressing Barriers were associated with greater declines in total symptoms, and more extensive use of Establishing/Reviewing Goals, Tracking/Reviewing Progress, and Assigning/Reviewing Homework was associated with declines in externalizing symptoms. For content, more extensive use of Cognitive Restructuring was associated with declines in total symptoms. In addition, higher average extensiveness ratings of the top content strategy across sessions was associated with greater declines in total and externalizing symptoms. Therapist-reported delivery of some EBP strategies showed evidence of predictive validity and may hold utility in indexing quality of care.

Community-Delivered Evidence-Based Practice and Usual Care for Adolescent Attention-Deficit/Hyperactivity Disorder: Examining Mechanistic Outcomes

Previous research suggests that routine psychosocial care for adolescents with attention-deficit/hyperactivity disorder (ADHD) is an eclectic and individualized mix of diluted evidence-based practices (EBPs) and low-value approaches. This study evaluated the extent to which a community-delivered EBP and usual care (UC) for adolescents with ADHD produce differential changes in theorized behavioral, psychological, and cognitive mechanisms of ADHD. A randomized community-based trial was conducted with double randomization of adolescent and community therapists to EBP delivery supports (Supporting Teens' Autonomy Daily [STAND]) versus UC delivery. Participants were 278 culturally diverse adolescents (ages 11-17) with ADHD and caregivers. Mechanistic outcomes were measured at baseline, post-treatment, and follow-up using parent-rated, observational, and task-based measures. Results using linear mixed models indicated that UC demonstrated superior effects on parent-rated and task-based executive functioning relative to STAND. However, STAND demonstrated superior effects on adolescent motivation and reducing parental intrusiveness relative to UC when it was delivered by licensed therapists. Mechanisms of community-delivered STAND and UC appear to differ. UC potency may occur through improved executive functioning, whereas STAND potency may occur through improved teen motivation and reducing low-value parenting practices. However, when delivered by unlicensed, community-based therapists, STAND did not enact proposed mechanisms. Future adaptations of community-delivered EBPs for ADHD should increase supports for unlicensed therapists, who comprise the majority of the community mental health workforce.

A Randomized Community-Based Trial of Behavior Therapy vs. Usual Care for Adolescent ADHD: Secondary Outcomes and Effects on Comorbidity

Though behavior therapy (BT) for ADHD in adolescence is evidence-based, almost no work examines its implementation and effectiveness in community settings. A recent randomized community-based trial of an evidence-based BT for adolescent ADHD (Supporting Teens' Autonomy Daily; STAND; N = 278) reported high clinician, parent, and youth acceptability but variable implementation fidelity. Primary outcome analyses suggested no significant differences between STAND and usual care (UC) unless the clinician delivering STAND was licensed. The present study reports secondary outcomes for this trial on indices of comorbidity (anxiety, depression, oppositional defiant disorder, conduct disorder) and ADHD outcomes not targeted by the active treatment (social skills, sluggish cognitive tempo). We also examine whether therapist licensure moderated treatment effects (as in primary outcome analyses). Using intent-to-treat and per protocol linear mixed models, patients randomized to STAND were compared to those randomized to UC over approximately 10 months of follow-up. Group × Time effects revealed that, overall, STAND did not outperform usual care when implemented by community clinicians. However, a Group × Time × Licensure interaction revealed a significant effect on conduct problems when STAND was delivered by licensed clinicians (d = .19-.47). When delivered in community settings, behavior therapy for adolescent ADHD can outperform UC with respect to conduct problems reduction. Community mental health clinics should consider: (1) assigning adolescent ADHD cases to licensed professionals to maximize impact and (2) choosing psychosocial approaches when ADHD presents with comorbid conduct problems. There is also a need to reduce implementation barriers for unlicensed clinicians in community settings.

Are There Long-Term Effects of Behavior Therapy for Adolescent ADHD? A Qualitative Study

We utilized qualitative methodology to characterize potential long-term effects (therapeutic and iatrogenic) of behavior therapy for adolescents with ADHD. Forty-two in-depth interviews were conducted with adolescents with ADHD and parents, 4 years post-treatment. Grounded theory methods identified and reported prevalence of themes. All reported long-term effects were classified as benefits; no iatrogenic effects were noted. Long-term impact themes reported for a majority of participants included: development of organization skills (81.0%), enhanced motivation (57.1%), improved self-awareness (57.1%), improved parental knowledge of ADHD (76.2%), increased parent autonomy granting (61.9%), enhanced parental engagement with the youth (52.4%), and improved parent-teen relationships (52.4%). Fourteen themes were present for smaller subsamples, including reduced need for medication (3 of 9 medicated participants). Experimental studies of behavior therapy for adolescent ADHD should measure themes detected herein and directly test the possibility of long-term treatment effects.

Building a Theoretical Model for Supporting Teens' Autonomy Daily (STAND): A Network Analysis of Family-Perceived Changes

Little is known about processes through which behavior therapy (BT) for adolescent ADHD improves outcomes. The purpose of this study was to build a theoretical model for the processes through which a BT for adolescent ADHD (Supporting Teens' Autonomy Daily; STAND) impacts functioning. Seventy-eight audio recordings from a standard therapeutic task in the final STAND session were analyzed as parents and adolescents (ages 11-16) reflected upon what changed during STAND and why. Qualitative coding sorted parent and teen statements into orthogonal categories of perceived changes. Network analysis examined inter-relations between categories. Results indicated twenty-one categories of perceived change areas. Parent use of behavioral strategies, adolescent motivation, and adolescent organization skills were central nodes in the network of perceived changes, with strong relations to academic and parent-teen relationship outcomes. A model is proposed in which skills training in STAND increases parent behavioral strategy use and teen organization skills, while Motivational Interviewing (MI) in STAND increase parent behavioral strategy use and initial adolescent motivation. In turn, parent behavioral strategy use is proposed to further reinforce teen motivation through contingency management, thereby increasing teen application of organization skills to daily life. As a result of improved teen motivation and organization skills, the model proposes that ADHD symptoms, academic problems, and parent-teen conflict abate. We discuss secondary mechanisms and outcomes in this model, the possibility of person-specific processes, implications for community-based adaptation of STAND, and plans to validate this conceptual model using sophisticated mediational models.

Stakeholder-Generated Implementation Strategies to Promote Evidence-Based ADHD Treatment in Community Mental Health

Community implementation of evidence-based practices (EBPs) for Attention Deficit/Hyperactivity Disorder (ADHD) is greatly lacking. A recent randomized community-based trial of an EBP for ADHD (Supporting Teens' Autonomy Daily; STAND) demonstrated suboptimal implementation and effectiveness outcomes. In the present study, we conducted an Innovation Tournament (IT) with agency staff stakeholders (N = 26) to identify barriers to successful implementation of STAND and implementation strategies for a revised service delivery model. We conducted member-checking of agency staff-generated ideas with parents (N = 226) and subsequent querying of additional parent (N = 226) and youth-generated (N = 205) strategies to improve care. Go-Zone plots were utilized to identify strategies with the highest feasibility and importance. Practical barriers (i.e., transportation, scheduling difficulties) and parent/youth engagement were the most commonly cited obstacles to successful implementation of STAND in community contexts. Eighteen "winning" implementation strategies were identified that survived member checking. These were classified as train and educate stakeholders (n = 5; e.g., train agency supervisors to deliver supervision, digitize treatment materials and trainings), engage consumers (n = 9; e.g., begin treatment with rapport building sessions, increase psychoeducation), provide interactive assistance (n = 2; e.g., add group supervision, increase roleplay in supervision), and use of evaluative/iterative strategies (n = 2; e.g., perform fidelity checks, supervisor review of session recordings). Parents and youth desired longer duration of treatment and increased focus on maintenance. Strategies will be developed and tested as part of a pilot effectiveness trial designed to refine STAND's service delivery model.Trial Registration NCT02694939 www.clinicaltrials.gov.