Therapist Self-Report of Fidelity to Core Elements of Family Therapy for Adolescent Behavior Problems: Psychometrics of a Pragmatic Quality Indicator Tool

Validating a Pragmatic Measure of Evidence-Based Practice (EBP) Delivery: Therapist Reports of EBP Strategy Delivery and Associations with Child Outcome Trajectories

Pragmatic measures of evidence-based practice (EBP) implementation can support and evaluate implementation efforts. We examined the predictive validity of therapist reports of EBP strategy delivery for children's mental health outcomes. Data were obtained from 1,380 sessions with 248 children delivered by 76 therapists in two county systems. Children (Mage=11.8 years, SD = 3.7) presented with internalizing (52%), externalizing (27%), trauma (16%), and other (5%) concerns. Therapists reported their delivery of EBP strategies on a revised version of the EBP Concordant Care Assessment (ECCA; Brookman-Frazee, et al., Administration and Policy in Mental Health and Mental Health Services Research, 48, 155-170, 2021) that included 25 content (e.g., parenting, cognitive behavioral) and 12 technique strategies (e.g., modeling, practice/role-play). On average, 5.6 ECCA session reports (SD = 2.3) were obtained for each client, and caregivers reported symptoms on the Brief Problem Checklist (Chorpita, et al., Journal of Consulting and Clinical Psychology, 78(4), 526-536, 2010) at baseline, weekly over two months, and again at four months. Multilevel models examined whether the mean extensiveness of each EBP strategy predicted trajectories of child outcomes. More individual technique (6 of 12) than content strategies (1 of 25) were associated with outcome trajectories. For techniques, more extensive use of Performance Feedback and Live Coaching and less extensive use of Addressing Barriers were associated with greater declines in total symptoms, and more extensive use of Establishing/Reviewing Goals, Tracking/Reviewing Progress, and Assigning/Reviewing Homework was associated with declines in externalizing symptoms. For content, more extensive use of Cognitive Restructuring was associated with declines in total symptoms. In addition, higher average extensiveness ratings of the top content strategy across sessions was associated with greater declines in total and externalizing symptoms. Therapist-reported delivery of some EBP strategies showed evidence of predictive validity and may hold utility in indexing quality of care.

Provider Adherence to Modular Cognitive Behavioral Therapy for Children and Adolescents

This study explored predictors of community-based providers' adherence to MATCH, a modular cognitive behavioral therapy for children and adolescents. Provider-reported adherence to MATCH was measured using three increasingly strict criteria: (1) session content (whether the session covered MATCH content consistent with the client's target problem), (2) session content and sequencing (whether the session covered MATCH content in the expected sequence for the client's target problem), and (3) session content, sequencing, and participant (whether the session covered MATCH content in the expected sequence and with the expected participant(s) for the client's target problem). Session, client, provider, and organizational predictors of adherence to MATCH were assessed using multilevel modeling. Results revealed that nearly all providers delivered MATCH content that corresponded to the target problem, but only one-third of providers delivered MATCH content in the expected sequence and with the expected participant for the client's target problem. This difference underscores the need for nuanced adherence measurement to capture important implementation information that broad operationalizations of adherence miss. Regardless of the criteria used providers were most adherent to MATCH during sessions when clients presented with interfering comorbid mental health symptoms. This suggests that the design of MATCH, which offers flexibility and structured guidance to address comorbid mental health problems, may allow providers to personalize treatment to address interfering comorbidity symptoms while remaining adherent to evidence-based practices. Additional guidance for providers on managing other types of session interference (e.g., unexpected events) may improve treatment integrity in community settings.

Standard Versus Family-Based Screening, Brief Intervention, and Referral to Treatment for Adolescent Substance Use in Primary Care: Protocol for a Multisite Randomized Effectiveness Trial

BACKGROUND

Screening, brief intervention, and referral to treatment for adolescents (SBIRT-A) is widely recommended to promote detection and early intervention for alcohol and other drug (AOD) use in pediatric primary care. Existing SBIRT-A procedures rely almost exclusively on adolescents alone, despite the recognition of caregivers as critical protective factors in adolescent development and AOD use. Moreover, controlled SBIRT-A studies conducted in primary care have yielded inconsistent findings about implementation feasibility and effects on AOD outcomes and overall developmental functioning. There is urgent need to investigate the value of systematically incorporating caregivers in SBIRT-A procedures.

OBJECTIVE

This randomized effectiveness trial will advance research and scope on SBIRT-A in primary care by conducting a head-to-head test of 2 conceptually grounded, evidence-informed approaches: a standard adolescent-only approach (SBIRT-A-Standard) versus a more expansive family-based approach (SBIRT-A-Family). The SBIRT-A-Family approach enhances the procedures of the SBIRT-A-Standard approach by screening for AOD risk with both adolescents and caregivers; leveraging multidomain, multireporter AOD risk and protection data to inform case identification and risk categorization; and directly involving caregivers in brief intervention and referral to treatment activities.

METHODS

The study will include 2300 adolescents (aged 12-17 y) and their caregivers attending 1 of 3 hospital-affiliated pediatric settings serving diverse patient populations in major urban areas. Study recruitment, screening, randomization, and all SBIRT-A activities will occur during a single pediatric visit. SBIRT-A procedures will be delivered digitally on handheld tablets using patient-facing and provider-facing programming. Primary outcomes (AOD use, co-occurring behavior problems, and parent-adolescent communication about AOD use) and secondary outcomes (adolescent quality of life, adolescent risk factors, and therapy attendance) will be assessed at screening and initial assessment and 3-, 6-, 9-, and 12-month follow-ups. The study is well powered to conduct all planned main and moderator (age, sex, race, ethnicity, and youth AOD risk status) analyses.

RESULTS

This study will be conducted over a 5-year period. Provider training was initiated in year 1 (December 2023). Participant recruitment and follow-up data collection began in year 2 (March 2024). We expect the results from this study to be published in early 2027.

CONCLUSIONS

SBIRT-A is widely endorsed but currently underused in pediatric primary care settings, and questions remain about optimal approaches and overall effectiveness. In particular, referral to treatment procedures in primary care remains virtually untested among youth. In addition, whereas research strongly supports involving families in interventions for adolescent AOD, SBIRT-A effectiveness trial testing approaches that actively engage family members in primary care are absent. This trial is designed to help fill these research gaps to inform the critical health decision of whether and how to include caregivers in SBIRT-A activities conducted in pediatric primary care.

TRIAL REGISTRATION

ClinicalTrials.gov NCT05964010; https://www.clinicaltrials.gov/study/NCT05964010.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54486.

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes

PURPOSE

Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions.

METHOD

Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events.

ANALYSIS

We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures).

RESULTS

For chart notes, practices had moderate interrater reliability (M k = 0.50, M ICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes.

CONCLUSION

Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

Pilot Trial of Online Measurement Training and Feedback in Family Therapy for Adolescent Behavior Problems

OBJECTIVE

Pragmatic procedures for sustaining high-fidelity delivery of evidence-based interventions are needed to support implementation in usual care. This study tested an online therapist training system, featuring observational coder training and self-report fidelity feedback, to promote self-report acumen and routine use of family therapy (FT) techniques for adolescent behavior problems.

METHOD

Therapists (N = 84) from nine substance use and mental health treatment sites reported on 185 adolescent clients. Therapists submitted baseline data on FT technique use with clients, completed a workshop introducing the 32-week training system, and were randomly assigned by site to Core Training versus Core Training + Consultation. Core Training included a therapist coder training course (didactic instruction and mock session coding exercises in 13 FT techniques) and fidelity feedback procedures depicting therapist-report data on FT use. Consultation convened therapists and supervisors for one-hour monthly sessions with an external FT expert. During the 32 weeks of training, therapists submitted self-report data on FT use along with companion session audiotapes subsequently coded by observational raters.

RESULTS

Therapist self-report reliability and accuracy both increased substantially during training. Observers reported no increase over time in FT use; therapists self-reported a decrease in FT use, likely an artifact of their improved self-report accuracy. Consultation did not enhance therapist self-report acumen or increase FT use.

CONCLUSIONS

Online training methods that improve therapist-report reliability and accuracy for FT use may confer important advantages for treatment planning and fidelity monitoring. More intensive and/or different training interventions appear needed to increase routine FT delivery.

Relationship-Oriented Recovery System for Youth (RORSY): Clinical Protocol for Transition-Age Youth with Opioid Use Disorders

This article introduces the Relationship-Oriented Recovery System for Youth (RORSY) protocol, which is designed to increase uptake of Medications for Opioid Use Disorder (MOUD) and related services among adolescents and young adults. Youth exhibit alarmingly poor rates of MOUD initiation and adherence, OUD services involvement and long-term recovery success. RORSY attends to three developmentally unique recovery needs of this age group: assess and bolster youth recovery capital, prioritize involvement of concerned significant others, and use digital direct-to-consumer recovery supports. RORSY contains five evidence-informed intervention modules that can be flexibly tailored to meet the individual and relationship needs of a given youth: Relational Orientation, Youth Recovery Management Planning, Relational Recovery Management Planning, Relationship Skills Building, and Digital Recovery Support Planning. The article concludes with practice and policy recommendations for making relationship-building a top clinical priority for addressing youth OUD.

Pragmatic fidelity measurement in youth service settings

Background

Fidelity measurement is critical for developing, evaluating, and implementing evidence-based treatments (EBTs). However, traditional fidelity measurement tools are often not feasible for community-based settings. We developed a short fidelity rating form for the Collaborative Problem Solving (CPS) approach from an existing manualized coding system that requires extensive training. We examined the reliability and accuracy of this short form when completed by trained observers, untrained observers, and self-reporting providers to evaluate multiple options for reducing barriers to fidelity measurement in community-based settings.

Methods

Community-based treatment providers submitted recordings of youth service sessions in which they did, or did not, use CPS. For 60 recordings, we compared short-form fidelity ratings assigned by trained observers and untrained observers to those provided by trained observers on the manualized coding system. For 141 recordings, we compared providers' self-reported fidelity on the short form to ratings provided by trained observers on the manualized coding system and examined providers' accuracy as a function of their global fidelity.

Results & Conclusions

The short form was reliable and accurate for trained observers. An assigned global integrity score and a calculated average of component scores on the short form, but not component scores themselves, were reliable and accurate for observers who had CPS expertise but no specific training on rating CPS fidelity. When providers self-reported fidelity on the short form, their global integrity score was a reliable estimate of their CPS integrity; however, providers with better CPS fidelity were most accurate in their self-reports. We discuss the costs and benefits of these more pragmatic fidelity measurement options in community-based settings.

What is the Status of Multi-Informant Treatment Fidelity Research?

OBJECTIVE

The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these informants often do not agree on ratings. This scoping review aims to ascertain the state of science around multi-informant assessment of treatment fidelity.

METHOD

A literature search of articles published through December 2021 identified 673 articles. Screening reduced the number of articles to 44, and the final study set included 35 articles.

RESULTS

There was substantial variability across studies regarding study design, how fidelity was operationalized, and how reliability was defined and assessed. Most studies evaluated the agreement between independent observers and practitioner-report, though several other informant pairs were assessed. Overall, findings suggest that concordance across fidelity informants was low to moderate, with a few key exceptions.

CONCLUSIONS

It is difficult to draw clear conclusions about the degree to which single versus multiple informant assessment is needed to produce an accurate and complete picture of treatment fidelity. The field needs to take steps to determine how to leverage multi-informant assessment to accurately assess treatment fidelity.

Self-Coding of Fidelity as a Potential Active Ingredient of Consultation to Improve Clinicians' Fidelity

A key goal for implementation science is the identification of evidence-based consultation protocols and the active ingredients within these protocols that drive clinician behavior change. The current study examined clinicians' self-coding of fidelity as a potential active ingredient of consultation for the Attachment and Biobehavioral Catch-up (ABC) intervention. It also examined two other potential predictors of clinician fidelity in response to consultation: dosage of consultation and working alliance. Twenty-nine clinicians (97% female, 62% White, M age = 34 years) participated in a year of weekly fidelity-focused ABC consultation sessions, for which clinicians self-coded fidelity and received consultant feedback on both their coding and their fidelity. Data from the ABC fidelity measure were available for 1067 sessions coded by consultants, and clinicians' self-coding accuracy was calculated from 1044 sessions coded by both clinicians and consultants. Alliance was measured with the Working Alliance Inventory-Trainee and Supervisor Versions. The study was observational, and fidelity and self-coding accuracy were modeled across time using hierarchical linear modeling. Clinicians' ABC fidelity, as well as their self-coding accuracy, increased over the course of consultation. Clinicians' self-coding accuracy predicted their initial fidelity and growth in fidelity. Working alliance was also linked to fidelity and self-coding accuracy. These results suggest that clinician self-coding should be further examined as an active ingredient of consultation. The study has important implications for the design of consultation procedures and fidelity assessments.