Exploring the black box: measuring youth treatment process and progress in usual care

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes

PURPOSE

Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions.

METHOD

Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events.

ANALYSIS

We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures).

RESULTS

For chart notes, practices had moderate interrater reliability (M k = 0.50, M ICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes.

CONCLUSION

Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

Initial Development of Pragmatic Behavioral Activation Fidelity Assessments

PURPOSE

Our goal was to develop brief pragmatic assessments of Behavioral Activation (BA) fidelity to support its dissemination in low-resource settings.

METHODS

We used qualitative and quantitative methods across three investigations to develop pragmatic assessments rated from the perspective of therapists, patients, and observers: (1) we developed an initial comprehensive pool of 119 items and adapted/refined the item pool to 32 items through stakeholder focus groups and cognitive interviews; (2) independent blind judges rated each of items in the refined item pool on an early session of BA for 64 patients to support the selection of items based on predictive validity; and (3) we conducted a preliminary evaluation of the acceptability and feasibility of the assessments of BA fidelity from the perspective of therapists and patients.

RESULTS

The internal consistency reliability for the 10-item total score was .83 rated from the perspective of independent observers. The assessment was completed by patients following 90% of sessions and by clinicians following 93% of sessions. Items were rated high on overall satisfaction by both therapists (M = 4.6, SD = 0.89) and patients (M = 4.8, SD = 0.41).

CONCLUSION

Our findings suggest that these brief assessments of BA fidelity are reliable, feasible, and acceptable to community stakeholders.

Integrating Measurement-Based Care into Treatment for Autism Spectrum Disorder: Insights from a Community Clinic

Measurement-based care (MBC), an evidence-based approach that has demonstrated efficacy for improving treatment outcomes, has yet to be investigated in clients with Autism Spectrum Disorder. The current paper investigates the use of MBC in autistic (n = 20) and non-autistic (n = 20) clients matched on age, sex, and presenting problem. Results of change score analysis indicated that utilizing routine symptom monitoring can enhance treatment evaluation. Autistic clients participated in significantly more sessions, made significantly less progress, and were less compliant with MBC than non-autistic clients. Though hierarchical linear modeling demonstrated no significant differences in treatment slope, results indicated moderate effect size. These results inform the use of MBC in community clinics, highlighting policy implications and need for targeted measurement.

Improving Mental Health Services: A 50-Year Journey from Randomized Experiments to Artificial Intelligence and Precision Mental Health

This conceptual paper describes the current state of mental health services, identifies critical problems, and suggests how to solve them. I focus on the potential contributions of artificial intelligence and precision mental health to improving mental health services. Toward that end, I draw upon my own research, which has changed over the last half century, to highlight the need to transform the way we conduct mental health services research. I identify exemplars from the emerging literature on artificial intelligence and precision approaches to treatment in which there is an attempt to personalize or fit the treatment to the client in order to produce more effective interventions.

What Gets Measured Gets Done: How Mental Health Agencies can Leverage Measurement-Based Care for Better Patient Care, Clinician Supports, and Organizational Goals

Mental health clinicians and administrators are increasingly asked to collect and report treatment outcome data despite numerous challenges to select and use instruments in routine practice. Measurement-based care (MBC) is an evidence-based practice for improving patient care. We propose that data collected from MBC processes with patients can be strategically leveraged by agencies to also support clinicians and respond to accountability requirements. MBC data elements are outlined using the Precision Mental Health Framework (Bickman et al. in Adm Policy Mental Health Mental Health Serv Res 43:271-276, 2016), practical guidance is provided for agency administrators, and conceptual examples illustrate strategic applications of one or more instruments to meet various needs throughout the organization.

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services

Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also explored. Therapists and independent observers rated the extensiveness of therapist (n = 103) EBP strategy delivery in 680 community psychotherapy sessions in which six EBPs were used. Concordance between therapist- and observer-report of the extensiveness of therapist EBP strategy use was at least fair (ICC ≥ .40) for approximately half of the items. Greater therapist-observer concordance was observed in sessions where a structured EBP was delivered and in sessions where therapists reported being able to carry out planned activities. Findings highlighted conditions that may improve or hinder therapists' ability to report on their own EBP strategy delivery in a way that is consistent with independent observers. These results can help inform the development of pragmatic therapist-report measures of EBP strategy delivery and implementation efforts more broadly.

Factors Associated with Community-Partnered School Behavioral Health Clinicians' Adoption and Implementation of Evidence-Based Practices

Community-partnered school behavioral health (CP-SBH) is a model whereby schools partner with local community agencies to deliver services. This mixed-methods study examined 80 CP-SBH clinicians' adoption and implementation of evidence-based practice (EBP) approaches following mandated training. Forty-four clinicians were randomly assigned to one of two training conditions for a modular common elements approach to EBPs; 36 clinicians were preselected for training in a non-modular EBP. EBP knowledge improved for all training conditions at 8-month follow-up and practice element familiarity improved for modular approach training conditions, but the modular condition including ongoing consultation did not yield better results. Qualitative interviews (N = 17) highlighted multi-level influences of the CP-SBH service system and individual clinician characteristics on adoption and implementation.

Factor Analysis of Therapist-Identified Treatment Targets in Community-Based Children's Mental Health

The present study used exploratory and confirmatory factor analyses to identify underlying latent factors affecting variation in community therapists' endorsement of treatment targets. As part of a statewide practice management program, therapist completed monthly reports of treatment targets (up to 10 per month) for a sample of youth (n = 790) receiving intensive in-home therapy. Nearly 75 % of youth were diagnosed with multiple co-occurring disorders. Five factors emerged: Disinhibition, Societal Rules Evasion, Social Engagement Deficits, Emotional Distress, and Management of Biodevelopmental Outcomes. Using logistic regression, primary diagnosis predicted therapist selection of Disinhibition and Emotional Distress targets. Client age predicted endorsement of Societal Rules Evasion targets. Practice-to-research implications are discussed.

Reliability of therapist self-report on treatment targets and focus in family-based intervention

Reliable therapist-report methods appear to be an essential component of quality assurance procedures to support adoption of evidence-based practices in usual care, but studies have found weak correspondence between therapist and observer ratings of treatment techniques. This study examined therapist reliability and accuracy in rating intervention target (i.e., session participants) and focus (i.e., session content) in a manual-guided, family-based preventive intervention implemented with 50 inner-city adolescents at risk for substance use. A total of 106 sessions selected from three phases of treatment were rated via post-session self-report by the participating therapist and also via videotape by nonparticipant coders. Both groups estimated the amount of session time devoted to model-prescribed treatment targets (adolescent, parent, conjoint) and foci (family, school, peer, prosocial, drugs). Therapists demonstrated excellent reliability with coders for treatment targets and moderate to high reliability for treatment foci across the sample and within each phase. Also, therapists did not consistently overestimate their degree of activity with targets or foci. Implications of study findings for fidelity assessment in routine settings are discussed.

Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: factor and construct validity

This study introduces a therapist-report measure of evidence-based practices for adolescent conduct and substance use problems. The Inventory of Therapy Techniques-Adolescent Behavior Problems (ITT-ABP) is a post-session measure of 27 techniques representing four approaches: cognitive-behavioral therapy (CBT), family therapy (FT), motivational interviewing (MI), and drug counseling (DC). A total of 822 protocols were collected from 32 therapists treating 71 adolescents in six usual care sites. Factor analyses identified three clinically coherent scales with strong internal consistency across the full sample: FT (8 items; α = .79), MI/CBT (8 items; α = .87), and DC (9 items, α = .90). The scales discriminated between therapists working in a family-oriented site versus other sites and showed moderate convergent validity with therapist reports of allegiance and skill in each approach. The ITT-ABP holds promise as a cost-efficient quality assurance tool for supporting high-fidelity delivery of evidence-based practices in usual care.

Feedback mechanisms of change: How problem alerts reported by youth clients and their caregivers impact clinician-reported session content

OBJECTIVE

This study explored how clinician-reported content addressed in treatment sessions was predicted by clinician feedback group and multi-informant cumulative problem alerts that appeared in computerized feedback reports for 299 clients aged 11-18 years receiving home-based community mental health treatment.

METHOD

Measures included a clinician report of content addressed in sessions and additional measures of treatment progress and process (e.g., therapeutic alliance) completed by clinicians, clients, and their caregivers. Item responses in the top 25th percentile in severity from these measures appeared as "problem alerts" on corresponding computerized feedback reports. Clinicians randomized to the feedback group received feedback weekly while the control group did not. Analyses were conducted using the Cox proportional hazards regression for recurrent events.

RESULTS

For all content domains, the results of the survival analyses indicated a robust effect of the feedback group on addressing specific content in sessions, with feedback associated with shorter duration to first occurrence and increased likelihood of addressing or focusing on a topic compared to the non-feedback group.

CONCLUSION

There appears to be an important relationship between feedback and cumulative problem alerts reported by multiple informants as they influence session content.

A Meta-Analysis of Personalized Treatment Goals in Psychotherapy: A Preliminary Report and Call for More Studies

The aim of this study was to explore the hypothesis that psychotherapy has larger effect sizes for personalized treatment goals than for symptom checklists. We conducted a meta-analysis of clinical trials that measured treatment success both in terms of symptom checklists and personalized treatment goals. Our search of the literature yielded 12 studies that met our inclusion criteria. Effect sizes were substantially larger for personalized treatment goals (ES = .86, p < .0001) than for symptom checklists (ES = .32, p = .003). The magnitude of this difference was significant (p < .05). Our results suggest that psychotherapy is perhaps more effective in helping patients with individual goals than reducing scores on broad measures of symptoms. Estimates of the effectiveness of psychotherapy that are based on symptom checklists perhaps underestimate the true benefit of psychotherapy. We discuss the implications for research and clinical practice.

Community-based treatment for youth with co- and multimorbid disruptive behavior disorders

Little is known about the types of psychotherapeutic practices delivered to youth with comorbid and multimorbid diagnoses in community settings. The present study, based on therapists' self-reported practices with 569 youth diagnosed with a disruptive behavior disorder (ODD or CD), examined whether specific therapeutic practice applications varied as a function of the number and type of comorbid disorders. While type of comorbid disorder (AD/HD or internalizing) did not predict therapists' practices, youth with more than two diagnoses (multimorbid) received treatment characterized by a more diverse set and a higher dosage of practices.

Predictors, Moderators, and Treatment Parameters of Community and Clinic-Based Treatment for Child Disruptive Behavior Disorders

This study examines predictors, moderators, and treatment parameters associated with two key child outcomes in a recent clinical trial comparing the effects of a modular treatment that was applied by study clinicians in the community (COMM) or a clinic (CLINIC) for children with oppositional defiant disorder (ODD) or conduct disorder (CD). Based on a literature review, moderator and predictor variables across child, parent, and family domains were examined in relation to changes in parental ratings of the severity of externalizing behavior problems or the number of ODD and CD symptoms endorsed on psychiatric interview at pretreatment, posttreatment, and 36-month posttreatment follow-up. In addition, associations between parameters of treatment (e.g., hours of child, parent, and parent-child treatment received, treatment completion, referral for additional services at discharge) and child outcomes were explored. Path models identified few moderators (e.g., level of child impairment, attention deficit hyperactivity disorder diagnosis, level of family conflict) and several predictors (child trauma history, family income, parental employment, parental depression) of treatment response. Treatment response was also related to a few treatment parameters (e.g., hours of child and parent treatment received, treatment completion, referral for additional services at discharge). We discuss the implications of these findings for maximizing the benefits of modular treatment by optimizing or personalizing intervention approaches for children with behavior disorders.

Assessing usual care in clinical trials

Researchers designing clinical trials often specify usual care received by participants as the control condition expecting that all participants receive usual care regardless of group assignment. The assumption is that the groups in the study are affected similarly. We describe the assessment of usual care within the 16 studies in Multisite Adherence Collaboration in HIV (MACH 14), a multisite collaboration on adherence to antiretroviral therapy. Only five of the studies in MACH 14 assessed usual care. Assessment protocols varied as did the timing and frequency of assessments. All usual care assessments addressed patient education focused on HIV, HIV medications, and medication adherence. Our findings support earlier work that calls for systematic assessments of usual care within the study design, inclusion of descriptions of usual care in reports of the study, and the influence of usual care on the experimental condition in clinical trials.

Family voice with informed choice: coordinating wraparound with research-based treatment for children and adolescents

The wraparound process is a type of individualized, team-based care coordination that has become central to many state and system efforts to reform children's mental health service delivery for youths with the most complex needs and their families. Although the emerging wraparound research base is generally positive regarding placements and costs, effect sizes are smaller for clinical and functional outcomes. This article presents a review of literature on care coordination and wraparound models, with a focus on theory and research that indicates the need to better connect wraparound-enrolled children and adolescents to evidence-based treatment (EBT). The article goes on to describe how recently developed applications of EBT that are based on quality improvement and flexible application of "common elements" of research-based care may provide a more individualized approach that better aligns with the philosophy and procedures of the wraparound process. Finally, this article presents preliminary studies that show the feasibility and potential effectiveness of coordinating wraparound with the Managing and Adapting Practice system, and discusses intervention development and research options that are currently under way.

Innovations in Practice: Adolescent Mentalization-Based Integrative Therapy (AMBIT) - a new integrated approach to working with the most hard to reach adolescents with severe complex mental health needs

BACKGROUND

'Hard to reach' young people are associated by virtue of their serious, multiple, and complex needs, the difficulty of delivering effective help to them, and their poor long-term outcomes. There is a lack of published evidence relating to the effectiveness of interventions directed at this group.

METHOD

We review these concerns and the options available to service commissioners and clinicians seeking, if not an evidence-based approach then at least an evidence-oriented one. A mentalization-based multimodal intervention (AMBIT) is briefly described, proposing a new kind of specialist practitioner and taking a radically different approach to treatment manualization.

RESULTS

A brief description is given of the different settings in which AMBIT is currently being developed, deployed, and evaluated, and of lessons learned.

CONCLUSIONS

AMBIT offers promise as an evolving 'open source' framework supporting development of evidence-based local practice in chaotic complex settings.

The Peabody Treatment Progress Battery: history and methods for developing a comprehensive measurement battery for youth mental health

There is increased need for comprehensive, flexible, and evidence-based approaches to measuring the process and outcomes of youth mental health treatment. This paper introduces a special issue dedicated to the Peabody Treatment Progress Battery (PTPB), a battery of measures created to meet this need. The PTPB is an integrated set of brief, reliable, and valid instruments that can be administered efficiently at low cost and can provide systematic feedback for use in treatment planning. It includes eleven measures completed by youth, caregivers, and/or clinicians that assess clinically-relevant constructs such as symptom severity, therapeutic alliance, life satisfaction, motivation for treatment, hope, treatment expectations, caregiver strain, and service satisfaction. This introductory article describes the rationale for the PTPB and its development and evaluation, detailing the specific analytic approaches utilized by the different papers in the special issue and a description of the study and samples from which the participants were taken.

The Session Report Form (SRF): are clinicians addressing concerns reported by youth and caregivers?

This study explores the relationship between clinician-reported content addressed in sessions, measured with the Session Report Form (SRF), and multi-informant problem alerts stemming from a larger battery of treatment process and progress measures. Multilevel Multinomial Logit Models were conducted with 133 clinicians and 299 youths receiving home-based treatment (N = 3,143 sessions). Results indicate a strong relationship between session content and problems related to youth symptoms and functioning as reported by clinicians in the same session. Session content was related to emotional, family, and friend/peer problems reported by youth and youth behavioral problems reported by caregivers. High-risk problems (alcohol/substance use, harm to self or others) were strongly related to session content regardless of informant. Session content was not related to problem alerts associated with the treatment process, caregiver strain, or client/caregiver strengths. The SRF appears to be a useful measure for assessing common themes addressed in routine mental health settings.

Toward the effective and efficient measurement of implementation fidelity

Implementation science in mental health is informed by other academic disciplines and industries. Conceptual and methodological territory charted in psychotherapy research is pertinent to two elements of the conceptual model of implementation posited by Aarons and colleagues (2010)--implementation fidelity and innovation feedback systems. Key characteristics of scientifically validated fidelity instruments, and of the feasibility of their use in routine care, are presented. The challenges of ensuring fidelity measurement methods are both effective (scientifically validated) and efficient (feasible and useful in routine care) are identified as are examples of implementation research attempting to balance these attributes of fidelity measurement.

Moderators and predictors of clinical outcome in a randomized trial for behavior problems in pediatric primary care

OBJECTIVES

To evaluate putative moderator, predictor, and treatment parameter variables in relation to three outcomes in a clinical trial that compared a modular protocol for on-site, nurse-administered intervention (PONI) and enhanced usual care (EUC) for pediatric behavioral problems in primary care.

METHODS

Patients were 163 clinically referred children for behavior problems in six primary care offices. PONI consisted of seven treatment modules adapted from prior treatment trials with this population, whereas EUC involved a facilitated referral to a community provider. Outcome measures were based on standardized scales reflecting one parent-rated aggregate (child dysfunction) and one child-rated aggregate (child health), and diagnostic interviews with both informants (remission in oppositional defiant disorder).

RESULTS

Moderator analyses revealed that PONI was more effective than EUC in reducing child dysfunction by 12-month follow-up among Caucasian children, whereas EUC was more effective than PONI among non-Caucasian children. In the full sample, child health improvement was predicted by the severity of the child's depression and anxiety, and level of family conflict. Duration of child exposure to cognitive-behavioral treatment in PONI was related to greater improvement in overall child health, but other treatment parameters were unrelated to outcome.

CONCLUSIONS

These few significant relationships notwithstanding the findings indicate that the two treatments had robust effects on several outcomes and across selected child, parent, family, and treatment variables. The findings extend efforts to incorporate mental health services in pediatric practice.