Introduction to the Special Issue: Legislation Related to Children's Evidence-Based Practice

Validating a Pragmatic Measure of Evidence-Based Practice (EBP) Delivery: Therapist Reports of EBP Strategy Delivery and Associations with Child Outcome Trajectories

Pragmatic measures of evidence-based practice (EBP) implementation can support and evaluate implementation efforts. We examined the predictive validity of therapist reports of EBP strategy delivery for children's mental health outcomes. Data were obtained from 1,380 sessions with 248 children delivered by 76 therapists in two county systems. Children (Mage=11.8 years, SD = 3.7) presented with internalizing (52%), externalizing (27%), trauma (16%), and other (5%) concerns. Therapists reported their delivery of EBP strategies on a revised version of the EBP Concordant Care Assessment (ECCA; Brookman-Frazee, et al., Administration and Policy in Mental Health and Mental Health Services Research, 48, 155-170, 2021) that included 25 content (e.g., parenting, cognitive behavioral) and 12 technique strategies (e.g., modeling, practice/role-play). On average, 5.6 ECCA session reports (SD = 2.3) were obtained for each client, and caregivers reported symptoms on the Brief Problem Checklist (Chorpita, et al., Journal of Consulting and Clinical Psychology, 78(4), 526-536, 2010) at baseline, weekly over two months, and again at four months. Multilevel models examined whether the mean extensiveness of each EBP strategy predicted trajectories of child outcomes. More individual technique (6 of 12) than content strategies (1 of 25) were associated with outcome trajectories. For techniques, more extensive use of Performance Feedback and Live Coaching and less extensive use of Addressing Barriers were associated with greater declines in total symptoms, and more extensive use of Establishing/Reviewing Goals, Tracking/Reviewing Progress, and Assigning/Reviewing Homework was associated with declines in externalizing symptoms. For content, more extensive use of Cognitive Restructuring was associated with declines in total symptoms. In addition, higher average extensiveness ratings of the top content strategy across sessions was associated with greater declines in total and externalizing symptoms. Therapist-reported delivery of some EBP strategies showed evidence of predictive validity and may hold utility in indexing quality of care.

Effectiveness of training methods for delivery of evidence-based psychotherapies: a systematic review

BACKGROUND

Extensive efforts have been made to train mental health providers in evidence-based psychotherapies (EBPs); there is increasing attention focused on the methods through which providers are trained to deliver EBPs. Evaluating EBP training methods is an important step in determining which methods are most effective in increasing provider skill and improving client outcomes.

METHODS

We searched MEDLINE (Ovid) and PsycINFO for randomized controlled trials published from 1990 through June 2019 that evaluated EBP training methods to determine the effectiveness of EBP training modalities on implementation (provider and cost) and client outcomes. Eligible studies (N = 28) were evaluated for risk of bias, and the overall strength of evidence was assessed for each outcome. Data was extracted by a single investigator and confirmed by a second; risk of bias and strength of evidence were independently rated by two investigators and determined by consensus.

RESULTS

Overall, EBP training improved short-term provider satisfaction, EBP knowledge, and adherence compared to no training or self-study of training materials (low to moderate strength of evidence). Training in an EBP did not increase treatment adoption compared to no training or self-study. No specific active EBP training modality was found to consistently increase provider EBP knowledge, skill acquisition/adherence, competence, adoption, or satisfaction compared to another active training modality. Findings were mixed regarding the additive benefit of post-training consultation on these outcomes. No studies evaluated changes in provider outcomes with regards to training costs and few studies reported on client outcomes.

LIMITATIONS

The majority of included studies had a moderate risk of bias and strength of evidence for the outcomes of interest was generally low or insufficient. Few studies reported effect sizes. The ability to identify the most effective EBP training methods was limited by low strength of evidence for the outcomes of interest and substantial heterogeneity among studies.

CONCLUSIONS

EBP training may have increased short-term provider satisfaction, EBP knowledge, and adherence though not adoption. Evidence was insufficient on training costs and client outcomes. Future research is needed on EBP training methods, implementation, sustainability, client outcomes, and costs to ensure efforts to train providers in EBPs are effective, efficient, and durable.

TRIAL REGISTRATION

The protocol for this review is registered in PROSPERO (CRD42018093381).

Policy Levers to Promote Access to and Utilization of Children's Mental Health Services: A Systematic Review

Policies have potential to help families obtain behavioral healthcare for their children, but little is known about evidence for specific policy approaches. We reviewed evaluations of select policy levers to promote accessibility, affordability, acceptability, availability, or utilization of children's mental and behavioral health services. Twenty articles met inclusion criteria. Location-based policy levers (school-based services and integrated care models) were associated with higher utilization and acceptability, with mixed evidence on accessibility. Studies of insurance-based levers (mental health parity and public insurance) provided some evidence for affordability outcomes. We found no eligible studies of workforce development or telehealth policy levers, or of availability outcomes.

Leadership profiles associated with the implementation of behavioral health evidence-based practices for autism spectrum disorder in schools

Implementation of evidence-based practice (EBP) for autism spectrum disorder (ASD) in the education system is a public health priority. Leadership is a critical driver of EBP implementation but little is known about the types of leadership behaviors exhibited by school leaders and how this influences the context of EBP implementation, particularly for students with ASD. The objectives of this study were to determine (1) the leadership profiles of principals involved in EBP implementation for students with ASD and (2) how these leadership profiles related to school characteristics and implementation climate. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework was used to guide the design and analysis of this study. Participants (n = 296) included principals, teachers, and classroom support staff. They provided demographic information and completed the Multifactor Leadership Questionnaire and Implementation Climate Scale. Using latent profile analysis, a three-pattern solution was identified: Disengaged (6% of sample), Undifferentiated (23% of sample), and Optimal (71% of sample). Principals in schools with higher proportions of students with an individualized education program were more likely to be classified as Undifferentiated than Optimal. The Optimal group was associated with more positive implementation climate than the Undifferentiated or Disengaged groups. Findings suggest that leadership behaviors rated by principals and their staff involved in implementation of common autism EBPs can be meaningfully clustered into three discernible profiles that are shaped by organizational context and linked to strategic implementation climate. Our study findings have implications for leadership training and service delivery in schools by underscoring the critical nature of school leadership during implementation of EBPs for children with autism and the interplay between specific leadership behaviors and strategic implementation climate.

Comparing Agency Leader and Therapist Perspectives on Evidence-Based Practices: Associations with Individual and Organizational Factors in a Mental Health System-Driven Implementation Effort

Agency leaders and therapists are essential stakeholders in implementation of evidence-based practices (EBPs) within publicly-funded mental health services. Little is known about how these stakeholders differ in their perceptions of specific EBPs and which individual and organizational factors differentially influence these perceptions. Within the context of a system-driven implementation of multiple EBPs, survey data from 160 leaders and 720 therapists were examined to assess differences in perceptions of six EBPs. Findings indicated that leaders and therapists have unique perspectives and preferences regarding EBPs that are shaped by distinct sociodemographic and professional characteristics and aspects of organizational functioning.

From evidence to impact: Joining our best school mental health practices with our best implementation strategies

There is substantial research evidence for the effectiveness of school mental health strategies across problem areas, developmental levels, and the prevention-intervention spectrum. At the same time, it is clear that the education and mental health fields continue to struggle to apply this evidence at any level of scale. This commentary reflects on ways in which education-specific applications of implementation science principles - and explicit consideration of determinants of implementation success - may guide more consistent use of evidence in school mental health. After reviewing implementation determinants and strategies across multiple levels of effect (i.e., the outer setting, inner setting, individual, and intervention levels), the commentary goes on to recommend specific areas of needed attention in school mental health implementation efforts and research. These include a need to adapt interventions to better fit the context of schools, streamlining school mental health programs and practices to make them more implementable, and recognizing the critical role of assessment and selection of evidence-based interventions by school leaders. The commentary concludes by reflecting on the substantial opportunity provided by the education sector to both apply and advance implementation science.

Mixed-method analysis of program leader perspectives on the sustainment of multiple child evidence-based practices in a system-driven implementation

BACKGROUND

Understanding program leader perspectives on the sustainment of evidence-based practice (EBP) in community mental health settings is essential to improving implementation. To date, however, much of the literature has focused on direct service provider perspectives on EBP implementation. The aim of this mixed-method study was to identify factors associated with the sustainment of multiple EBPs within a system-driven implementation effort in children's mental health services.

METHODS

Data were gathered from 186 leaders at 59 agencies within the Los Angeles County Department of Mental Health who were contracted to deliver one of six EBPs within the Prevention and Early Intervention initiative.

RESULTS

Multi-level analyses of quantitative survey data (N = 186) revealed a greater probability of leader-reported EBP sustainment in large agencies and when leaders held more positive perceptions toward the EBP. Themes from semi-structured qualitative interviews conducted with a subset of survey participants (n = 47) expanded quantitative findings by providing detail on facilitating conditions in larger agencies and aspects of EBP fit that were perceived to lead to greater sustainment, including perceived fit with client needs, implementation requirements, aspects of the organizational workforce, availability of trainings, and overall therapist attitudes about EBPs.

CONCLUSIONS

Findings inform EBP implementation efforts regarding decisions around organizational-level supports and promotion of EBP fit.

Infusing Science into Politics and Policy: The Importance of Legislators as an Audience in Mental Health Policy Dissemination Research

Legislators (i.e., elected Senators and House Representatives at the federal- and state-level) are a critically important dissemination audience because they shape the architecture of the US mental health system through budgetary and regulatory decisions. In this Point of View, we argue that legislators are a neglected audience in mental health dissemination research. We synthesize relevant research, discuss its potential implications for dissemination efforts, identify challenges, and outline areas for future study.

Evidence-Based Program Service Deserts: A Geographic Information Systems (GIS) Approach to Identifying Service Gaps for State-Level Implementation Planning

The long term effects of untreated mental health need for individuals, families and society has prompted a number of federal policy statements encouraging the use of evidence-based programs (EBP) in children's healthcare. However, among other challenges of evidence-based practice implementation, states often do not know where to make investments based on population need. In this paper we present the use of a Geographic Information System approach to undertake a mental health needs assessment for Washington State. Our study found that this technology can be beneficially applied to conducting needs assessment for EBP implementation, and we provide recommendations for future applications.

Applying the Policy Ecology Framework to Philadelphia's Behavioral Health Transformation Efforts

Raghavan et al. (Implement Sci 3(26):1-9, 2008) proposed that effective implementation of evidence-based practices requires implementation strategies deployed at multiple levels of the "policy ecology," including the organizational, regulatory or purchaser agency, political, and social levels. However, much of implementation research and practice targets providers without accounting for contextual factors that may influence provider behavior. This paper examines Philadelphia's efforts to work toward an evidence-based and recovery-oriented behavioral health system, and uses the policy ecology framework to illustrate how multifaceted, multilevel implementation strategies can facilitate the widespread implementation of evidence-based practices. Ongoing challenges and implications for research and practice are discussed.

Measuring Sustainment of Multiple Practices Fiscally Mandated in Children's Mental Health Services

Examining sustainment of multiple evidence-based practices is crucial to understanding the processes and outcomes of system-driven implementation efforts that are increasingly common. This study used administrative claims data to characterize volume and penetration of six practices over 19 fiscal quarters following initial implementation within the context of a system-driven, fiscally mandated implementation effort in Los Angeles County Department of Mental Health's Prevention and Early Intervention services. Patterns of volume changes over time revealed an overall ramp up of practice claims over time, but variability in patterns of volume and penetration for each practice. Findings varied by the methods used to index and analyze volume and penetration. Furthermore, a number of client case-mix and therapist characteristics were associated with the volume of therapists' claims for each practice relative to their claims for the other practices.

The 4KEEPS study: identifying predictors of sustainment of multiple practices fiscally mandated in children's mental health services

Background

Research to date has largely focused on predictors of adoption and initial implementation of evidence-based practices (EBPs), yet sustained implementation is crucial to deliver a return on investments in dissemination. Furthermore, most studies focus on single EBPs, limiting opportunities to study the fit between practice characteristics EBPs and implementation contexts.

Methods/design

This observational study will characterize implementation sustainment and identify organizational and therapist characteristics that predict sustainment of multiple practices being implemented within a fiscal mandate in the largest public mental health system in the USA. Specific aims are to (1) characterize sustainment outcomes (volume/penetration, EBP concordant care); (2) use mixed methods to characterize inner context (agency- and therapist-level) factors and early implementation conditions; and (3) identify inner context factors and early implementation conditions that predict sustainment outcomes. This study will undertake original data collection and analysis of existing data sources to achieve its aims. Archived reports and documents will be used to characterize early implementation conditions in 102 agencies. Administrative claims data will be used to characterize volume and penetration outcomes over 8 years. Therapist and program manager surveys will be administered to characterize sustained EBP concordant care and inner context determinants of sustainment. An in-depth study in a subset of agencies will yield interview data and recordings of treatment sessions for validation of the EBP concordant care scale.

Discussion

This project will yield new understanding of whether and how multiple EBPs can be sustained in public mental health systems undergoing a policy-driven community implementation effort. We will produce generalizable models for characterizing sustainment, including feasible and flexible measurement of practice across multiple EBPs. The findings will inform the development of implementation interventions to promote sustained delivery of EBPs to maximize their public health impact.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-016-0388-4) contains supplementary material, which is available to authorized users.