Tracking Evidence Based Practice with Youth: Validity of the MATCH and Standard Manual Consultation Records

Using Adherence and Competence Measures Based on Practice Elements to Evaluate Treatment Fidelity for Two CBT Programs for Youth Anxiety

Measures designed to assess the quantity and quality of practices found across treatment programs for specific youth emotional or behavioral problems may be a good fit for evaluating treatment fidelity in effectiveness and implementation research. Treatment fidelity measures must demonstrate certain reliability and validity characteristics to realize this potential. This study examines the extent to which two observational measures, the Cognitive-Behavioral Treatment for Anxiety in Youth Adherence Scale (CBAY-A) and the CBAY Competence Scale (CBAY-C), can assess the quantity (the degree to which prescribed therapeutic techniques are delivered as intended) or quality (the competence with which prescribed techniques are delivered) of practices found in two distinct treatment programs for youth anxiety. Treatment sessions (N = 796) from 55 youth participants (M age = 9.89 years, SD = 1.71; 46% female; 55% White) with primary anxiety problems who participated in an effectiveness study were independently coded by raters who coded quantity, quality, and the youth-clinician alliance. Youth received one of three treatments: (a) standard (i.e., cognitive-behavioral therapy program), (b) modular (i.e., a cognitive-behavioral and parent-training program), and (c) usual clinical care. Interrater reliability for the CBAY-A items was good across the standard and modular conditions but mixed for the CBAY-C items. Across the standard and modular conditions, the CBAY-A Model subscale scores demonstrated evidence of construct validity, but the CBAY-C Model subscale scores showed mixed evidence. The results provide preliminary evidence that the CBAY-A can be used across different treatment programs but raise concerns about the generalizability of the CBAY-C.

Getting More Out of Clinical Documentation: Can Clinical Dashboards Yield Clinically Useful Information?

This study investigated coded data retrieved from clinical dashboards, which are decision-support tools that include a graphical display of clinical progress and clinical activities. Data were extracted from clinical dashboards representing 256 youth (M age = 11.9) from 128 practitioners who were trained in the Managing and Adapting Practice (MAP) system (Chorpita & Daleiden in BF Chorpita EL Daleiden 2014 Structuring the collaboration of science and service in pursuit of a shared vision. 43(2):323 338. 2014, Chorpita & Daleiden in BF Chorpita EL Daleiden 2018 Coordinated strategic action: Aspiring to wisdom in mental health service systems. 25(4):e12264. 2018) in 55 agencies across 5 regional mental health systems. Practitioners labeled up to 35 fields (i.e., descriptions of clinical activities), with the options of drawing from a controlled vocabulary or writing in a client-specific activity. Practitioners then noted when certain activities occurred during the episode of care. Fields from the extracted data were coded and reliability was assessed for Field Type, Practice Element Type, Target Area, and Audience (e.g., Caregiver Psychoeducation: Anxiety would be coded as Field Type = Practice Element; Practice Element Type = Psychoeducation; Target Area = Anxiety; Audience = Caregiver). Coders demonstrated moderate to almost perfect interrater reliability. On average, practitioners recorded two activities per session, and clients had 10 unique activities across all their sessions. Results from multilevel models showed that clinical activity characteristics and sessions accounted for the most variance in the occurrence, recurrence, and co-occurrence of clinical activities, with relatively less variance accounted for by practitioners, clients, and regional systems. Findings are consistent with patterns of practice reported in other studies and suggest that clinical dashboards may be a useful source of clinical information. More generally, the use of a controlled vocabulary for clinical activities appears to increase the retrievability and actionability of healthcare information and thus sets the stage for advancing the utility of clinical documentation.

Toward Efficient, Sustainable, and Scalable Methods of Treatment Characterization: An Investigation of Coding Clinical Practice from Chart Notes

PURPOSE

Chart notes provide a low-cost data source that could help characterize what occurs in treatment with sufficient precision to improve management of care. This study assessed the interrater reliability of treatment content coded from chart notes and evaluated its concordance with content coded from transcribed treatment sessions.

METHOD

Fifty randomly selected and digitally recorded treatment events were transcribed and coded for practice content. Independent coders then applied the same code system to chart notes for these same treatment events.

ANALYSIS

We measured reliability and concordance of practice occurrence and extensiveness at two levels of specificity: practices (full procedures) and steps (subcomponents of those procedures).

RESULTS

For chart notes, practices had moderate interrater reliability (M k = 0.50, M ICC = 0.56) and steps had moderate (M ICC = 0.74) to substantial interrater reliability (M k = 0.78). On average, 2.54 practices and 5.64 steps were coded per chart note and 4.53 practices and 13.10 steps per transcript. Across sources, ratings for 64% of practices and 41% of steps correlated significantly, with those with significant correlations generally demonstrating moderate concordance (practice M r = 0.48; step M r = 0.47). Forty one percent of practices and 34% of steps from transcripts were also identified in the corresponding chart notes.

CONCLUSION

Chart notes provide an accessible data source for evaluating treatment content, with different levels of specificity posing tradeoffs for validity and reliability, which in turn may have implications for chart note interfaces, training, and new metrics to support accurate, reliable, and efficient measurement of clinical practice.

Do specific modules of cognitive behavioral therapy for depression have measurable effects on youth internalizing symptoms? An idiographic analysis

OBJECTIVE

Understanding the efficacy of each module of cognitive behavioral therapy (CBT) may inform efforts to improve outcomes for youth depression, but effects of specific modules have been difficult to examine. Idiographic interrupted time series models offer a robust way to estimate module effects on an individual's symptoms. This study examined the association of specific CBT modules for depression on internalizing symptoms among depressed youths who received modular CBT in a randomized trial.

METHODS

Individual models were created for three youths who met study criteria. Youths completed weekly symptom reports, and clinicians completed records of modules delivered. First order auto-regressive models quantified the change in average internalizing symptom severity between pre- and post-module delivery.

RESULTS

All youths had 1-3 modules that were significantly associated with symptom reduction and 1-3 modules associated with deterioration. The 5 modules associated with improvement in at least one youth also lacked association (engagement, relaxation, cognitive reframing), or were associated with worsening (activity selection, parent psychoeducation) in others. Seven modules showed no measurable benefit, or detriment to any youth.

CONCLUSION

This study demonstrated that specific modules have measurable effects, but more work is needed to build an evidence base of specific module effects to inform treatment personalization for youth depression.

What is the Status of Multi-Informant Treatment Fidelity Research?

OBJECTIVE

The precise measurement of treatment fidelity (quantity and quality in the delivery of treatment strategies in an intervention) is essential for intervention development, evaluation, and implementation. Various informants are used in fidelity assessment (e.g., observers, practitioners [clinicians, teachers], clients), but these informants often do not agree on ratings. This scoping review aims to ascertain the state of science around multi-informant assessment of treatment fidelity.

METHOD

A literature search of articles published through December 2021 identified 673 articles. Screening reduced the number of articles to 44, and the final study set included 35 articles.

RESULTS

There was substantial variability across studies regarding study design, how fidelity was operationalized, and how reliability was defined and assessed. Most studies evaluated the agreement between independent observers and practitioner-report, though several other informant pairs were assessed. Overall, findings suggest that concordance across fidelity informants was low to moderate, with a few key exceptions.

CONCLUSIONS

It is difficult to draw clear conclusions about the degree to which single versus multiple informant assessment is needed to produce an accurate and complete picture of treatment fidelity. The field needs to take steps to determine how to leverage multi-informant assessment to accurately assess treatment fidelity.

How Do Treatment Protocols Affect the Use of Engagement Practices in Youth Mental Health Services?

PURPOSE

Treatment engagement poses challenges for youth mental health providers. With the expansion of evidence-based treatments (EBTs), providers face complex decisions regarding how to engage youth and families using available information sources. This study investigated how EBT protocols are associated with the selection and delivery of engagement practices.

METHOD

Twenty engagement practices were coded in a sample of digital recordings of early treatment sessions (N = 193) from the Child STEPs in California study, a randomized trial testing modular treatment and community-implemented treatment for youth mental health problems. Data were collected on which protocols mental health providers reportedly used to guide their sessions and the protocols in which they had received training. We examined which information sources (i.e., the guiding protocol, other protocols in training history, unspecified source) were associated with observed engagement practices.

RESULTS

In sessions guided by a protocol, most observed engagement practices were accounted for by the guiding protocol (p < .001), rather than protocols in training history or unspecified sources (p < .001). In sessions not guided by a protocol, most observed practices were accounted for by training history (p < .001). Practice frequency and extensiveness was generally greater when a protocol guided the session.

CONCLUSIONS

Inclusion in protocols is associated with the selection and delivery of engagement practices, but this strategy might be insufficient for supporting the use of the full range of engagement practices supported by evidence. Supports are needed that leverage the engagement evidence base to ensure that selected practices empirically fit the engagement needs of youth and families.

A Cognitive-Behavioral Formulation of Narcissistic Self-Esteem Dysregulation

Narcissistic personality disorder (NPD) is a commonly encountered diagnosis, affecting approximately 1%-6% of the population, with no evidence-based treatments. Recent scholarship has focused on self-esteem dysregulation as a key component of NPD: Excessively high expectations for oneself and how one should be treated leads to brittle self-esteem and maladaptive reactions to self-esteem threats. The current article builds on this formulation, introducing a cognitive-behavioral model of narcissistic self-esteem dysregulation that clinicians can use in providing a relatable model of change for their patients. Specifically, symptoms of NPD can be seen as a set of cognitive and behavioral habits that serve to regulate difficult emotions emerging from maladaptive beliefs and interpretations of self-esteem threats. This perspective renders narcissistic dysregulation amenable to cognitive-behavioral therapy (CBT) in which patients learn skills that help them gain awareness around these habitual reactions, reshape cognitive distortions, and engage in behavioral experiments that serve to transform maladaptive belief systems that consequently free them from symptomatic reactions. Here, we provide a precis of this formulation and examples of how CBT skills can be used to treat narcissistic dysregulation. We also discuss future research that could provide empirical support for the model and test the efficacy of CBT approaches to NPD. Conclusions focus on the notion that narcissistic self-esteem dysregulation likely varies continuously in the population and transdiagnostically across disorders. Greater insight into the cognitive-behavioral mechanisms of self-esteem dysregulation could foster tools for ameliorating distress both in people with NPD and the general populace.

Therapist Self-Report of Fidelity to Core Elements of Family Therapy for Adolescent Behavior Problems: Psychometrics of a Pragmatic Quality Indicator Tool

Therapist-report measures of evidence-based interventions have enormous potential utility as quality indicators in routine care; yet, few such tools have shown strong psychometric properties. This study describes reliability and validity characteristics of a therapist-report measure of family therapy techniques for treating adolescent conduct and substance use problems: Inventory of Therapy Techniques for Core Elements of Family Therapy (ITT-CEFT). Study participants included 31 staff therapists treating 68 adolescent clients in eight community-based mental health and substance use clinics. Therapists submitted ITT-CEFT checklists and companion audio recordings for 189 sessions. The ITT-CEFT contains 13 techniques identified as core elements of three manualized family therapy models that are empirically supported for the target group. Therapists also reported on their use of three motivational interventions, and independent observers coded the submitted recordings. ITT-CEFT factor validity was shown via confirmatory factor analyses of the tool's theoretical structure. Derived modules were: Family Engagement (four items; Cronbach's α = .72); Relational Orientation (five items; α = .74); and Interactional Change (four items; α = .66). Concurrent validity analyses showed fair-to-excellent therapist reliability compared to observer ratings (ICCs range .64-.75); they showed moderate therapist accuracy compared to observer mean scores, reflecting a tendency to overestimate delivery of the techniques. Discriminant validity analyses showed tool differentiation from motivational interventions. Results offer provisional evidence for the feasibility of using the therapist-report ITT-CEFT to anchor quality procedures for family therapy interventions in real-world settings.Trial Registration: The parent clinical trial is registered at www.ClinicalTrials.gov , ID: NCT03342872 (registration date: 11.10.17).

Alternate delivery platforms and implementation models for bringing evidence-based behavioral interventions to scale for youth facing adversity: a case study in West Africa

BACKGROUND

Youth Functioning and Organizational Success for West African Regional Development (Youth FORWARD) was launched as an implementation science collaboration focused on scaling out evidence-based mental health interventions for youth exposed to war and other adversities through novel delivery platforms. This implementation science case study examines the use of a collaborative team approach (CTA) as a scale-out strategy to foster the integration of an evidence-based group mental health intervention, the Youth Readiness Intervention, into youth employment programs tied to regional economic development in Sierra Leone.

METHODS

A case study methodology is used to explore the feasibility and acceptability of integrating an evidence-based intervention, the Youth Readiness Intervention (YRI), into youth entrepreneurship programs (ENTR) in Sierra Leone, facilitated by the CTA. The authors analyzed field notes logged during program implementation, 8 weeks of supervision notes, 20 interviews with agency leaders and front-line staff delivering the YRI within this alternate delivery platform. Quantitative dissemination and implementation interviews administered to youth, facilitators, and agency leaders were analyzed using descriptive statistics and mixed linear models. A linked Hybrid Type II effectiveness-implementation cluster randomized trial is evaluating the clinical effectiveness of the YRI within this delivery platform.

RESULTS

Extant data indicate the strong feasibility and acceptability of integrating the YRI into the ENTR program. Facilitators of integration of the YRI into the ENTR include mission alignment of the organizations with the delivery of psychosocial interventions, shared commitment to serving vulnerable youth, support from local District Youth Councils, and high interest from the youth served. Barriers include perceived competition between frontline organizations seeking funding for psychosocial interventions, and challenges in flexibility between donors and implementation partners operating in a fragile/post-conflict setting. The CTA was a feasible and acceptable strategy to support fidelity and quality improvement while scaling out the YRI.

CONCLUSIONS

Youth entrepreneurship and livelihood programs offer a promising mechanism for expanding the reach of evidence-based interventions to youth in fragile and post-conflict settings. Quality improvement and sustainment of evidence-based interventions are novel concepts in such settings. The CTA strategy institutionalizes the integration of an evidence-based intervention into youth entrepreneurship programs.

TRIAL REGISTRATION

NCT03603613 (phase 1 pilot, registered May 18, 2018) and NCT03542500 (phase 2 scale-out study, registered May 18, 2018).

In Search of the Common Elements of Clinical Supervision: A Systematic Review

The importance of clinical supervision for supporting effective implementation of evidence-based treatments (EBTs) is widely accepted; however, very little is known about which supervision practice elements contribute to implementation effectiveness. This systematic review aimed to generate a taxonomy of empirically-supported supervision practice elements that have been used in treatment trials and shown to independently predict improved EBT implementation. Supervision practice elements were identified using a two-phase, empirically-validated distillation process. In Phase I, a systematic review identified supervision protocols that had evidence of effectiveness based on (a) inclusion in one or more EBT trials, and (b) independent association with improved EBT implementation in one or more secondary studies. In Phase II, a hybrid deductive-inductive coding process was applied to the supervision protocols to characterize the nature and frequency of supervision practice elements across EBTs. Twenty-one of the 876 identified articles assessed the associations of supervision protocols with implementation or clinical outcomes, representing 13 separate studies. Coding and distillation of the supervision protocols resulted in a taxonomy of 21 supervision practice elements. The most frequently used elements were: reviewing supervisees' practice (92%; n = 12), clinical suggestions (85%; n = 11), behavioral rehearsal (77%; n = 10), elicitation (77%; n = 10), and fidelity assessment (77%; n = 10). This review identified supervision practice elements that could be targets for future research testing which elements are necessary and sufficient to support effective EBT implementation. Discrepancies between supervision practice elements observed in trials as compared to routine practice highlights the importance of research addressing supervision-focused implementation strategies.

Can a Single Measure Estimate Protocol Adherence for Two Psychosocial Treatments for Youth Anxiety Delivered in Community Mental Health Settings?

Treatment adherence measurement can be time and resource-intensive in clinical trials, so the ability to measure protocol adherence for two distinct treatment programs with a single measure may benefit the field. The present study sought to determine if the Therapy Process Observational Coding System - Revised Strategies Scale (TPOCS-RS) could assess protocol adherence to two youth treatment programs. Treatment sessions (N = 796) from 55 youth (M age = 9.89 years, SD = 1.71; range 7-15 years; 55.0% White; 46.0% female) with primary anxiety problems treatment by 39 clinicians (M age = 40.54 years, SD = 9.56; 50.0% White; 80.0% female) were independently scored by coders using observational treatment adherence and alliance measures. The youth received one of three treatments: (a) Standard (i.e., cognitive-behavioral treatment program), (b) Modular (i.e., a program with cognitive-behavioral and parent training components), or (c) Usual Care. Consultants filled out a self-report measure of protocol adherence within the Standard and Modular conditions. Interrater reliability, ICC(2,2) for the various items for the full sample ranged from .17 to .92 (M ICC = .67; SD = .17). Scores from a TPOCS-RS subscale that mapped onto the specific content of the treatment protocols used in the Standard and Modular conditions evidenced convergent validity with the consultant-report adherence measure and discriminant validity with the alliance measure. The model-specific TPOCS-RS subscales also discriminated between the Standard and Modular treatments and Usual Care. This study provides initial evidence that (a) the TPOCS-RS has utility in estimating protocol adherence in different treatment programs and (b) support the score validity of the self-report consultation records.

Factors Influencing Fidelity to a Calorie Posting Policy in Public Hospitals: A Mixed Methods Study

Background: Labelling menus with nutrition information has increasingly become an important obesity policy option. While much research to-date has focused on determining its effectiveness, few studies report the extent to which menu labelling is implemented as designed. The aim of this study was to explore factors influencing fidelity to a calorie posting policy in Irish acute public hospitals.

Methods: A mixed methods sequential explanatory study design was employed, with a nested case study for the qualitative component. Quantitative data on implementation fidelity at hospitals were analysed first and informed case sampling in the follow-on qualitative phase. Maximum variation sampling was used to select four hospitals with high and low levels of implementation and variation in terms of geographic location, hospital size, complexity of care provided and hospital type. Data were collected using structured observations, unstructured non-participant observations and in-depth semi-structured interviews. The Consolidated Framework for Implementation Research guided qualitative data collection and analysis. Using framework analysis, factors influencing implementation were identified. A triangulation protocol was used to integrate fidelity findings from multiple sources. Data on influencing factors and fidelity were then combined using joint displays for within and cross-case analysis.

Results: Quantitative fidelity data showed seven hospitals were categorised as low implementers and 28 hospitals were high implementers of the policy. Across the four hospitals selected as cases, qualitative analysis revealed factors influencing implementation and fidelity were multiple, and operated independently and in combination. Factors were related to the internal hospital environment (e.g., leadership support, access to knowledge and information, perceived importance of calorie posting implementation), external hospital environment (e.g., national policy, monitoring), features of the calorie posting policy (e.g., availability of supporting materials), and the implementation process (e.g., engaging relevant stakeholders). Integrated analysis of fidelity indicated a pattern of partial adherence to the calorie posting policy across the four hospitals. Across all hospitals, there was a consistent pattern of low adherence to calorie posting across all menu items on sale, low adherence to calorie information displayed per standard portion or per meal, low adherence to standardised recipes/portions, and inaccurate calorie information.

Conclusion: Efforts to maximise fidelity require multi-level, multi-component strategies in order to reduce or mitigate barriers and to leverage facilitators. Future research should examine the relative importance of calorie posting determinants and the association between implementation strategies and shifts in fidelity to intervention core components.

Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form

The Knowledge of Evidence-Based Services Questionnaire (KEBSQ) is an objective measure of therapist knowledge of practices derived from the evidence base for the treatment of youth psychopathology. However, the length of this measure (i.e., 40 items) and respondent demands associated with each item makes it burdensome for researchers and clinicians. This study developed and validated a Short Form of the KEBSQ using Item Response Theory measurement models. The Short Form consists of 17 items and generates two separate scores: Correct Endorsements and Correct Rejections. The Short Form was found to correlate highly with and perform similarly to the Full Form, providing preliminary validity evidence.

A Novel Approach to the Assessment of Fidelity to a Cognitive Behavioral Therapy for PTSD Using Clinical Worksheets: A Proof of Concept With Cognitive Processing Therapy

Fidelity monitoring is a critical indicator of psychotherapy quality and is central to successful implementation. A major barrier to fidelity in routine care is the lack of feasible, scalable, and valid measurement strategies. A reliable, low-burden fidelity assessment would promote sustained implementation of cognitive behavioral therapies (CBTs). The current study examined fidelity measurement for cognitive processing therapy (CPT) for posttraumatic stress disorder (PTSD) using clinical worksheets. External raters evaluated patient worksheets done as a part of treatment, both guided by the therapist and completed independently as homework. Results demonstrated that fidelity ratings from CPT session worksheets were feasible and efficient. Notably, they were strongly correlated with observer ratings of the fidelity of CPT strategies that were present on the worksheets. Agreement among ratings conducted by individuals with a range of experience with CPT was acceptable to high. There was not a main effect of therapist-guided, in-session worksheet ratings on PTSD symptom change. However, patient competence in completing worksheets independently was associated with greater PTSD symptom decline and in-session, therapist-guided worksheet completion was associated with larger symptom decreases among patients with high levels of competence. With further research and refinement, rating of worksheets may be an efficient way to examine therapist and patient skill in key CPT elements, and their interactions, compared to the gold standard of observer ratings of therapy video-recordings. Additional research is needed to determine if worksheets are an accurate and scalable alternative to gold standard observer ratings in settings in which time and resources are limited.

Creating a prepared mental health workforce: comparative illustrations of implementation strategies

BACKGROUND

Psychotherapy implementation must contend with the task of preparing a mental health workforce to provide the highest quality services to as much of a service population as possible, in high-income as well as low-to-middle income countries.

OBJECTIVE

We outline general challenges and solutions and investigate how well various implementation strategies would fit a clinical population.

METHODS

Using a data set from a prior cluster randomised trial with a clinically diverse population and 33 intervention practices, we presented multiple illustrations comparing the ability of different implementation strategies to serve youth and families with procedures in which service providers were trained.

FINDINGS

A series of survival functions demonstrated that many common implementation strategies are unlikely to create a prepared workforce, given the large and diverse number of practices needed to be mastered by providers.

CLINICAL IMPLICATIONS

'Benchmark' solutions that afforded superior coverage of the service population could be supported through paced learning approaches (ie, training interventions a little at a time) using extensible, modular intervention designs.

A Glimpse into the "Black Box": Which Elements of Consultation in an EBP are Associated with Client Symptom Change and Therapist Fidelity?

Background: Consultation is an implementation strategy that improves delivery and clinical outcomes for Cognitive Processing Therapy (CPT), an evidence-based practice (EBP) for posttraumatic stress disorder (PTSD). However, little is known about the specific components of consultation that influence the fidelity of treatment delivery or clinical outcomes. Methods: The current study examined whether specific activities performed during CPT consultation meetings were associated with better fidelity to the CPT protocol among 60 newly trained therapists or improved clinical outcomes among 135 clients treated by these therapists. Consultation activities that fall under three broad categories (discussion of the application of CPT to individual cases, review/feedback on fidelity, and technical difficulties) were measured by consultant checklists for each consultation session. Treatment fidelity (adherence to the protocol and competence of delivery) was rated by trained observers for a random sample of therapists' CPT sessions following consultation. The self-reported PTSD Checklist-IV assessed PTSD symptom change. Results: Multilevel regression analyses indicated that higher therapist consultation attendance predicted a greater decrease in their clients' PTSD symptoms and that attendance was not associated with observer-rated treatment fidelity. Discussion of the application of specific CPT strategies was the only consultation activity that was significantly associated with greater improvement in PTSD symptoms. Lastly, no consultation activities were significantly associated with treatment fidelity. Conclusions: Our findings suggest that specific consultation strategies such as emphasizing the discussion of the application of specific CPT strategies to individual cases during consultation meetings may be effective in improving the clinical outcomes of CPT.

Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children's Mental Health Services

Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also explored. Therapists and independent observers rated the extensiveness of therapist (n = 103) EBP strategy delivery in 680 community psychotherapy sessions in which six EBPs were used. Concordance between therapist- and observer-report of the extensiveness of therapist EBP strategy use was at least fair (ICC ≥ .40) for approximately half of the items. Greater therapist-observer concordance was observed in sessions where a structured EBP was delivered and in sessions where therapists reported being able to carry out planned activities. Findings highlighted conditions that may improve or hinder therapists' ability to report on their own EBP strategy delivery in a way that is consistent with independent observers. These results can help inform the development of pragmatic therapist-report measures of EBP strategy delivery and implementation efforts more broadly.

Turning Back to Treatment: The Effect of Attendance and Symptom Outcomes on Subsequent Service Use

This study explored whether post-treatment symptom severity moderated the association between session attendance during an initial treatment episode and subsequent mental health service use. Data on attendance, symptom severity, and service use were gathered from an effectiveness trial testing a modular treatment for youth anxiety, depression, disruptive behavior, and traumatic stress. Multilevel logistic regression analyses showed a significant interaction between attendance and post-treatment symptom severity on subsequent service use, such that attendance significantly predicted subsequent service use when post-treatment symptom severity was in the normal range. Implications regarding the influence of treatment engagement on future help-seeking are discussed.

Clinician-reported barriers to using exposure with response prevention in the treatment of paediatric obsessive-compulsive disorder

Exposure techniques are underutilised in the treatment of anxiety disorders in routine practice, but little is known about the use of exposure with response prevention (ERP) for OCD, particularly in youth. The current study aimed to examine the utilisation of ERP for paediatric OCD via an anonymous online survey completed by clinicians (N = 107). Specifically, we explored the association of clinician characteristics and OCD symptom subtypes with ERP use, as well as clinician-reported barriers to ERP implementation. The majority of clinicians reported commonly using ERP when treating youth with OCD, and rates of ERP use were highest among clinical psychologists. Clinician-held negative beliefs about exposure were significantly associated with lower ERP use. Additionally, clinicians reported being less likely to use ERP to treat hoarding symptoms and taboo obsessions, compared to other OCD symptom subtypes. The most commonly endorsed barriers to successful ERP implementation were aspects of the phenomenology of OCD (e.g. covert compulsions, frequently changing rituals) as opposed to general barriers (e.g. insufficient time during sessions). Overall, our findings suggest that OCD presents unique challenges for clinicians delivering exposure-based therapy. Training should address these OCD-specific obstacles in order to promote dissemination of ERP for youth with OCD.

Concordance between clinician, supervisor and observer ratings of therapeutic competence in CBT and treatment as usual: does clinician competence or supervisor session observation improve agreement?

BACKGROUND

Lowering the cost of assessing clinicians' competence could promote the scalability of evidence-based treatments such as cognitive behavioral therapy (CBT).

AIMS

This study examined the concordance between clinicians', supervisors' and independent observers' session-specific ratings of clinician competence in school-based CBT and treatment as usual (TAU). It also investigated the association between clinician competence and supervisory session observation and rater agreement.

METHOD

Fifty-nine school-based clinicians (90% female, 73% Caucasian) were randomly assigned to implement TAU or modular CBT for youth anxiety. Clinicians rated their confidence after each therapy session (n = 1898), and supervisors rated clinicians' competence after each supervision session (n = 613). Independent observers rated clinicians' competence from audio recordings (n = 395).

RESULTS

Patterns of rater discrepancies differed between the TAU and CBT groups. Correlations with independent raters were low across groups. Clinician competence and session observation were associated with higher agreement among TAU, but not CBT, supervisors and clinicians.

CONCLUSIONS

These results support the gold standard practice of obtaining independent ratings of adherence and competence in implementation contexts. Further development of measures and/or rater training methods for clinicians and supervisors is needed.

Unintended Consequences of Evidence-Based Treatment Policy Reform: Is Implementation the Goal or the Strategy for Higher Quality Care?

This study examined patterns of evidence-based treatment (EBT) delivery following a county-wide EBT reform initiative. Data were gathered from 60 youth and their 21 providers, who were instructed to deliver therapy as they normally would under the EBT initiative. Results showed limited applicability of county-supported EBTs to this service sample, and that most youth did not receive traditional delivery of EBTs. Findings suggest that it may be unrealistic to expect providers to deliver EBTs with fidelity with all clients, and that EBT implementation may be best thought of as a strategy for improving mental health services rather than a goal.

Investigating the agreement between the clinician and research diagnosis of attention deficit hyperactivity disorder and how it changes over time; a clinical cohort study

BACKGROUND

Attention Deficit Hyperactivity Disorder (ADHD) is a common reason for referral to Child and Adolescent Mental Health Services (CAMHS), but families experience long delays between first professional contact and diagnosis, which risks development of secondary impairments. This study explores the agreement between clinician and research diagnoses of ADHD among children attending CAMHS, and their access to interventions. From the limited literature, we anticipated fluctuation and delays, but no other study has focused prospectively on ADHD diagnoses.

METHODS

This was a secondary analysis of a cohort of children attending two CAMHS between 2006 and 2009. The sample included 288 consecutive referrals of children aged between 5 and 11 years. Parents and teachers completed the Development and Well-Being Assessment (DAWBA) when the child was recruited to the study, which provided the research diagnosis of ADHD from the baseline. Clinicians reported no, possible, or definite diagnosis of ADHD and interventions provided at 6-monthly intervals for up to 2 years while the child attended CAMHS. We assessed agreement between the diagnoses using Kendall's Tau-B.

RESULTS

Of the 101 children with a research diagnosis of ADHD, 26 received a definite clinician diagnosis during 2-year follow-up, and 47 received a possible clinician diagnosis. The chance-corrected agreement was poor at all time points (Kendall's Tau-B 0.14-0.48). Clinician diagnoses were unstable, particularly if possible ADHD was recorded at the initial assessment. Of those with a research diagnosis, 15 were prescribed medication and 11 were offered parent training.

CONCLUSIONS

The use of standardised diagnostic assessments could reduce diagnostic uncertainty and increase access to evidence-based interventions.

Community-Based Providers' Selection of Practices for Children and Adolescents With Comorbid Mental Health Problems

The goal of this study is to explore providers' patterns of implementation by investigating how community mental health providers selected therapy practice modules from a flexible, modular evidence-based treatment working with youths with comorbid mental health problems. Data were obtained from 57 youths, 5-15 years old, presenting with anxiety, depressive, and/or conduct problems and their 27 providers during their participation in an effectiveness trial involving a modular evidence-based treatment. Although all youths evidenced clinically elevated symptomatology in at least two problem areas, providers targeted youths' comorbid problems with only about half of their study cases. Practice modules indicated for youths' comorbid problems were typically used less frequently and with less depth relative to practice modules indicated for youths' principal clinical problem and were often transdiagnostic in nature (i.e., designed to target more than one problem area). To determine whether providers' decisions to target youths' comorbid problems were systematic, multilevel, logistic regression analyses were conducted and revealed that youths' pretreatment characteristics and time in therapy influenced providers' patterns of module selection. Providers tend to use, but not exploit, the flexibility allowed by modular EBTs and to focus treatment on youths' principal presenting problem. In addition, providers appear to make these practice choices in a systematic and rational manner, and whether and which choices are associated with improved outcomes is an important area of future study.

Fidelity monitoring in complex interventions: a case study of the WISE intervention

Background

Researchers face many decisions in developing a measurement tool and protocol for monitoring fidelity to complex interventions. The current study uses data evaluating a nutrition education intervention, Together, We Inspire Smart Eating (WISE), in a preschool setting to explore issues of source, timing, and frequency of fidelity monitoring.

Methods

The overall study from which these data are drawn was a pre/post design with an implementation-focused process evaluation. Between 2013 and 2016, researchers monitored fidelity to evidence-based components of the WISE intervention in 49 classrooms in two Southern states. Data collectors obtained direct assessment of fidelity on a monthly basis in study classrooms. Research staff requested that educators provide indirect assessment on a weekly basis. We used mean comparisons (t-tests), correlations (Pearson's r), and scatterplots to compare the direct and indirect assessments.

Results

No mean comparisons were statistically different. Correlations of direct and indirect assessments of the same component for the same month ranged between - 0.51 (p = 0.01) and 0.54 (p = 0.001). Scatterplots illustrate that negative correlations can be driven by individuals who are over reporting (i.e., self-report bias) and that near zero correlations approximate the ideal situation (i.e., both raters identify high fidelity).

Conclusion

Our findings illustrate that, on average, observed and self-reports may seem consistent despite weak correlations and individual cases of extreme over reporting by those implementing the intervention. The nature of the component to which fidelity is being monitored as well as the timing within the context of the intervention are important factors to consider when selecting the type of assessment and frequency of fidelity monitoring.

Trial registration

NCT03075085 Registered 20 February 2017. Trial registration corresponds to the funding that supported the writing of this manuscript, not the data collection. The original study was not a trial and was collected without registration. However, the data reported here provided foundational preliminary data for the trial.

Emergent Life Events: An In-Depth Investigation of Characteristics and Provider Responses during Youth Evidence-Based Treatment

Emergent life events (ELEs), or acute client stressors disclosed within psychotherapy sessions, are not addressed by many evidence-based psychosocial treatments (EBTs). Preliminary provider-report studies suggest that ELEs may interfere with effective EBT implementation. The present study offers a detailed, observational examination of ELEs and their impact on EBT within therapy sessions. Data were observationally coded from 274 sessions with 55 primarily low-income, Latino youth clients (58% male, ages 5-15) in the modular EBT condition (Modular Approach to Therapy for Children [MATCH]) of the Child STEPs California trial. The ELE Coding System-Revised was used to measure ELEs, their characteristics, and provider responses to ELEs, including provider adherence to MATCH. Interrater reliability was generally high. At least one ELE was identified in 13% of randomly selected sessions. ELEs ranged widely in content, and their characteristics did not cluster together. Providers responded more frequently to ELEs with non-EBT content (e.g., information gathering, empathy) than EBT content; use of the ELE as a "teaching moment" for EBT content was the least common response (40% of ELEs). Multilevel regression analyses revealed that compared to sessions without an ELE, ELE sessions were significantly associated with reduced provider adherence to MATCH. Within ELE sessions, higher client distress when discussing the ELE was associated with reduced provider adherence to MATCH, but only when ELE severity was high. Beyond provider report, observational measures indicate that ELEs are prevalent and unpredictable in community settings and disrupt EBT delivery. Findings can inform the development of structured ELE management procedures to enhance existing EBTs.

Therapist Attitudes Towards Evidence-Based Practice: A Joint Factor Analysis

Despite the accumulated research support for the use of evidence-based practices (EBPs) with youth, these treatment approaches remain underutilized in community settings. Therapist attitudes towards EBPs play a pivotal role in their adoption and implementation of these practices. The present investigation employs joint exploratory factor analysis to evaluate the structure of two measures of therapist attitudes, the Evidence-Based Practices Attitudes Scale and the Modified Practice Attitude Scale. Results suggest three factors including (a) importance of clinical experience over EBPs, (b) clinician openness to change, and (c) problems with EBPs. Recommendations are provided for future evaluation of therapist attitudes and associated characteristics.

Generalizability and Decision Studies of a Treatment Adherence Instrument

Observational measurement of treatment adherence has long been considered the gold standard. However, little is known about either the generalizability of the scores from extant observational instruments or the sampling needed. We conducted generalizability (G) and decision (D) studies on two samples of recordings from two randomized controlled trials testing cognitive–behavioral therapy for youth anxiety in two different contexts: research versus community. Two doctoral students independently coded 543 session recordings from 52 patients treated by 13 therapists. The initial G-study demonstrated that context accounted for a disproportionately large share of variance, so we conducted G- and D-studies for the two contexts separately. Results suggested that reliable cognitive–behavioral therapy adherence studies require at least 10 sessions per patient, assuming 12 patients per therapists and two coders—a challenging threshold even in well-funded research. Implications, including the importance of evaluating alternatives to observational measurement, are discussed.

VA residential treatment providers' use of two evidence-based psychotherapies for PTSD: Global endorsement versus specific components

OBJECTIVE

Despite a growing body of knowledge about the dissemination of evidence-based psychotherapies (EBPs), their actual use in clinical settings is not well understood. The purpose of the current study was to compare self-reported component use with global use for 2 EBPs for posttraumatic stress disorder (PTSD), prolonged exposure (PE), and cognitive processing therapy (CPT).

METHOD

Around 174 providers from 38 VA PTSD residential treatment programs were asked about both global use and component use of PE and CPT.

RESULTS

Among frequent users of these EBPs, component use was generally high, especially for low-intensity and nonspecific components. For each form of treatment, there were a small number of providers who reported using the treatment frequently but did not use most of the key components of the treatment.

CONCLUSIONS

These findings highlight the importance of understanding the modifications that providers make to EBPs and suggest the importance of flexibility within fidelity to these treatments. (PsycINFO Database Record

Distilling the Core Elements of Family Therapy for Adolescent Substance Use: Conceptual and Empirical Solutions

This article describes several barriers to widespread dissemination of manualized family-based treatments for adolescent substance use (ASU). We then offer a highly promising solution for adopting and sustaining family therapy in usual care: distilling the core practice elements of empirically validated family therapy models for ASU. We present a conceptual distillation of family therapy for ASU grounded in existing observational fidelity measures for three manualized models, a process that yielded four core elements: Family Engagement, Relational Reframing, Family Behavior Change, and Family Restructuring. We then introduce an innovative empirical method for distilling core elements that can serve as a template for rigorous distillation of other treatment approaches. Finally, we discuss how core elements can enhance family therapy services within the diverse workforce of usual care for ASU.

Behavioral Rehearsal for Analogue Fidelity: Feasibility in a State-Funded Children's Mental Health Initiative

A substantial number of evidence-based treatments (EBTs) are available, but are delivered infrequently in public mental health. To improve the quality of care, some states and systems have focused on EBT training; however, these efforts have rarely included objective measurement of clinician fidelity because of feasibility issues. The primary goal of the current study was evaluating the feasibility of the behavioral rehearsal (BR) method to assess "analogue fidelity" in a children's mental health quality improvement initiative. Results indicated low-but representative-clinician participation. Participants demonstrated greatest improvement at post-training with maintenance or decreases at 6-months (post-consultation). Implications for future use of BR are discussed.

A Digital Feedback System to Support Implementation of Measurement-Based Care by School-Based Mental Health Clinicians

Evidence supports the utility of measurement-based care (MBC) to improve youth mental health outcomes, but clinicians rarely engage in MBC practices. Digital measurement feedback systems (MFS) may reflect a feasible strategy to support MBC adoption and sustainment. This pilot study was initiated to evaluate the impact of a MFS and brief consultation supports to facilitate MBC uptake and sustainment among mental health clinicians in the education sector, the most common mental health service delivery setting for youth. Following an initial training in MBC, 14 clinicians were randomized to either a digital MFS and brief consultation supports or control. Baseline ratings of MBC attitudes, skill, and use were collected. In addition, daily assessment ratings tracked 2 core MBC practices (i.e., assessment tool administration, provision of feedback) over a 6-month follow-up period. Clinicians in the MFS condition demonstrated rapid increases in both MBC practices, whereas the control group did not significantly change. For clinicians in the MFS group, consultation effects were significant for feedback and approached significance for administration. Over the follow-up period, average decreases in the current study were moderate with only 1 of the 2 outcome variables (administration) decreasing significantly. Inspection of individual clinician trajectories revealed substantial within-group trend variation. MFS may represent an effective MBC implementation strategy beyond initial training, although individual clinician response is variable. Identifying feasible and impactful implementation strategies is critical given the ability of MBC to support precision health care.

Managing In-Session "Surprises:" Provider Responses to Emergent Life Events during Evidence-Based Treatment Implementation

This study aimed to: (1) pilot a psychotherapy coding system for provider responses to emergent life events (ELEs; unexpected events that have a significant negative impact on the client), (2) examine the impact of ELEs on evidence-based treatment (EBT) delivery in community settings. Raters coded 30 randomly-sampled EBT session recordings with and without reported ELEs. Inter-rater reliability and validity for the system were generally high. When an ELE occurred, providers were significantly less likely to deliver the EBT, and when they did, they rarely linked the EBT to the event. Findings highlight the potential for ELEs to disrupt EBT implementation.

Integrity of evidence-based practice: are providers modifying practice content or practice sequencing?

This study examined patterns of evidence-based treatment (EBT) implementation within community settings by evaluating integrity along separate dimensions of practice content (PC; a session included the prescribed procedure) and practice sequencing (a session occurred in the prescribed sequence) within a recent randomized effectiveness trial. We measured whether sessions showed integrity to PC and to flexible or linear practice sequences. Findings revealed that providers tended to incorporate content from the EBT protocol in most treatment sessions, but that the sequencing of the sessions was often modified, suggesting that providers are amenable to evidence-based procedures, but not necessarily their prescribed arrangement.

Choices in Treatment Integrity:Considering the Protocol and Consultant Recommendations in Child and Adolescent Therapy

Codifying a complex treatment intervention into a manual necessarily involves simplifying procedures, possibly resulting in developer and expert adaptations that align with but are not fully consistent with the manualized content. This study evaluated the integrity of an evidence-based treatment to two information sources (i.e., an expert consultant and the treatment manual) within youth community mental health settings. Consultant recommendations were compared with manual-prescribed therapy procedures, which identified instances of agreement between the sources as well as consultant adaptations. To determine clinicians' integrity to each source, clinician-reported therapy procedures were compared with consultant recommendations and manual prescriptions. Participants consisted of 30 clinicians, five consultants, and 78 youths (53% male; M age = 9.73 years, SD = 2.84; 81% Latino). Findings indicated that consultants generally recommended content from the manual but also made frequent adaptations that involved changing the sequence or problem focus of practices and adding additional content. Clinicians followed both information sources in a majority of their sessions when these sources agreed but were more likely to follow only one or neither source when they disagreed. Findings suggest that consultants often make adaptations to the manual and that clinicians are more likely to employ recommendations from the manual and/or consultant when the information sources are in agreement. Although it is as yet unclear whether consultant recommendations are effective when they differ from the prescriptions of the manual, results from this study indicate that information sources that supplement treatment manuals may benefit clinical decision making.

Factors associated with practitioners' use of exposure therapy for childhood anxiety disorders

The current study examines factors related to use of exposure therapy by clinicians who treat children with anxiety disorders. A sample of 331 therapists from a variety of backgrounds (i.e., social workers, doctoral psychologists, masters level counselors, and marriage and family therapists) completed a survey regarding use of exposure and other treatment techniques for childhood anxiety disorders, as well as beliefs about exposure and child resiliency. Although the majority of therapists endorsed a CBT orientation (81%) and use of CBT techniques, exposure therapy was rarely endorsed. Holding a PhD in psychology as well as more positive beliefs about exposure and child resiliency were associated with greater use of exposure. The results suggest that exposure-based therapy is rarely offered in community settings and that dissemination should focus on individual evidence-based principles and correcting therapist misconceptions.

A Statewide Common Elements Initiative for Children's Mental Health

Many evidence-based treatments (EBTs) for child and adolescent mental health disorders have been developed, but few are available in public mental health settings. This paper describes initial implementation outcomes for a state-funded effort in Washington State to increase EBT availability, via a common elements training and consultation approach focused on four major problem areas (anxiety, posttraumatic stress disorder, depression, and behavioral problems). Clinicians (N = 180) reported significant improvement in their ability to assess and treat all problem areas at post-consultation. Clinicians from organizations with a supervisor-level "EBT champion" had higher baseline scores on a range of outcomes, but many differences disappeared at post-consultation. Outcomes suggest that a common elements initiative, which includes training and consultation, may positively impact clinician-level outcomes and that having "in-house" EBT expertise may provide additional benefits.

Relationships between clinician-level attributes and fidelity-consistent and fidelity-inconsistent modifications to an evidence-based psychotherapy

BACKGROUND

Clinicians often modify evidence-based psychotherapies (EBPs) when delivering them in routine care settings. There has been little study of factors associated with or implications of modifications to EBP protocols. This paper differentiates between fidelity-consistent and fidelity-inconsistent modifications and it examines the potential influence of two clinician characteristics, training outcomes, and attitudes toward EBPs on fidelity-consistent and fidelity-inconsistent modifications of cognitive behavioral therapy in a sample of clinicians who had been trained to deliver these treatments for children or adults.

METHODS

Survey and coded interview data collected 2 years after completion of training programs in cognitive behavioral therapy were used to examine associations between successful or unsuccessful completion of training, clinician attitudes, and modifications. Modifications endorsed by clinicians were categorized as fidelity-consistent or fidelity-inconsistent and entered as outcomes into separate regression models, with training success and attitudes entered as independent variables.

RESULTS

Successful completion of a training program was associated with subsequent fidelity-inconsistent modifications but not fidelity-consistent modifications. Therapists who reported greater openness to using EBPs prior to training reported more fidelity-consistent modifications at follow-up, and those who reported greater willingness to adopt EBPs if they found them appealing were more likely to make fidelity-inconsistent modifications.

CONCLUSIONS

Implications of these findings for training, implementation, EBP sustainment, and future studies are discussed. Research on contextual and protocol-related factors that may impact decisions to modify EBPs will be an important future direction of study to complement to this research.

Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: correspondence between therapists and observers

Developing therapist-report fidelity tools to support quality delivery of evidence-based practices in usual care is a top priority for implementation science. This study tested the reliability and accuracy of two groups of community therapists who reported on their use of family therapy (FT) and motivational interviewing/cognitive-behavioral therapy (MI/CBT) interventions during routine treatment of inner-city adolescents with conduct and substance use problems. Study cases (n = 45) were randomized into two conditions: (a) Routine Family Therapy (RFT), consisting of a single site that featured family therapy as its standard of care for behavioral treatment; or (b) Treatment As Usual (TAU), consisting of five sites that featured non-family approaches. Therapists and trained observational raters provided FT and MI/CBT adherence ratings on 157 sessions (104 RFT, 53 TAU). Overall therapist reliability was adequate for averaged FT ratings (ICC = .66) but almost non-existent for MI/CBT (ICC = .06); moreover, both RFT and TAU therapists were more reliable in reporting on FT than on MI/CBT. Both groups of therapists overestimated the extent to which they implemented FT and MI/CBT interventions. Results offer support for the feasibility of using existing therapist-report methods to anchor quality assurance procedures for FT interventions in real-world settings, though not for MI/CBT.

More practice, less preach? the role of supervision processes and therapist characteristics in EBP implementation

Identifying predictors of evidence-based practice (EBP) use, such as supervision processes and therapist characteristics, may support dissemination. Therapists (N = 57) received training and supervision in EBPs to treat community-based youth (N = 136). Supervision involving modeling and role-play predicted higher overall practice use than supervision involving discussion, and modeling predicted practice use in the next therapy session. No therapist characteristics predicted practice use, but therapist sex and age moderated the supervision and practice use relation. Supervision involving discussion predicted practice use for male therapists only, and modeling and role-play in supervision predicted practice use for older, not younger, therapists.

Implementing an evidence-based practices training curriculum to U.S. Army child and family behavioral health providers

Military Children who have experienced multiple deployments of one or both parents are more likely to have emotional and behavioral difficulties compared with their civilian peers (e.g., Chandra et al., 2010 ). The U.S. Army Medical Command has tasked the Child, Adolescent and Family Behavioral Health Office (CAF-BHO) to develop programs to address the behavioral health (BH) needs of Army Children and Families. This article will describe the efforts of the CAF-BHO Training Section to disseminate and implement an evidence based practice training curriculum for Army Child and Family Behavioral Health Providers. Specifically, this article will detail: (a) the decision making strategy used to identify the training protocol for dissemination, (b) adaptations to the training program and treatment protocol to fit a Military context, and (c) efforts to implement, maximize and sustain utilization of evidence-based practices by Military BH providers over a large geographical area with limited resources.