Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety?

Barriers and facilitators of implementation of evidence-based interventions in children and young people's mental health care - a systematic review

BACKGROUND

Effective evidence-based practices (EBP) for children and young people's (CYP) mental health exist, however, there is low uptake in clinical practice and interventions do not always reach those in need. This review aimed to comprehensively identify and synthesise the barriers and facilitators to implementing EBP in CYP mental health care, mapped according to an implementation framework in order to make pragmatic recommendations for practitioners, commissioners and researchers.

METHODS

Following the PRISMA guidelines, an electronic search of PsycINFO, MEDLINE, CINAHL and Embase in 2021 yielded 1830 results. In total, 107 abstracts were screened, 57 of which were included for full-text review and 26 were included for data extraction and analysis.

RESULTS

We identified a number of organisational and clinician-level barriers which impede the implementation of evidence in 'real world' practice. Barriers included lack of access to funding; poor access to resources; clinician attitudes and flexibility of EBP. Facilitators included targeted funding and access to resources; supportive staff and leadership committed to innovation and skills in the EBP.

CONCLUSIONS

Although the process of translating research into practice is challenging and the solutions are not straightforward, we have identified a set of practical recommendations for ways in which implementation practices can improve. Researchers, commissioners, funders and practitioners can work together to improve the implementation of EBP in CYP mental health settings by ensuring funding is available, prioritise implementation beyond the end of trials, upskill staff on integrating EBP into care, ensure EBPs are flexible and are co-produced with service users.

REGISTRATION

This review was registered on PROSPERO international prospective register of systematic reviews (CRD42021252995).

Inter-organizational alignment and implementation outcomes in integrated mental healthcare for children and adolescents: a cross-sectional observational study

BACKGROUND

Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOA in general and implementation context factors are associated with implementation outcomes in integrated SBMH.

METHODS

SBMH clinicians from community-based organizations (CBOs; nclinician = 27) and their proximal student-support school staff (nschool = 99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes.

RESULTS

The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors.

CONCLUSIONS

This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

A multistep mediation model examining how consultation in prolonged exposure therapy affects PTSD treatment outcomes

Prolonged exposure therapy (PE) is an evidence-based psychotherapy (EBP) for posttraumatic stress disorder (PTSD) that is underutilized in the military health system. Previous research suggests that postworkshop consultation is important for successful implementation. However, little is known about how consultation may relate to EBP adoption or patient outcomes. The present study addressed these gaps by examining associations between consultation, provider self-efficacy, use of PE, and patient outcomes using a multistep mediation model. This study used data from Foa et al. (2020), a two-armed randomized implementation trial comparing two PE training models: standard training (workshop only) and extended training (workshop + 6-8 months of postworkshop expert consultation) at three U.S. Army sites. Participants were patients with PTSD (N = 242) receiving care from the participating providers (N = 103). Providers who received extended training reported greater PE self-efficacy compared to standard training providers, but self-efficacy was unrelated to their use of PE components or to patient outcomes. Extended training providers used more PE components and had superior patient outcomes than standard training providers, and patient outcomes were mediated by the use of PE components. To our knowledge, this is the first study to demonstrate that EBP consultation leads to improved clinical outcomes for patients through increased use of the EBP. PE adoption (i.e., use of PE components in therapy) was not explained by increases in self-efficacy among providers who received extended training. Therefore, future research should assess how other factors may influence provider behavior in implementing EBPs. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

A Longitudinal Analysis of Trajectories and Predictors of Fidelity Using the SafeCare Parenting Model

Evidence-based practice (EBP) fidelity, understood as the extent to which a program is implemented as the developers intended, is a key implementation variable which likely relates to consumer outcomes. However, studies that track fidelity longitudinally and at large scale are uncommon, and finding reliable predictors of fidelity has proven to be a complex challenge. Further, attitudes toward EBP are a potentially important predictor of fidelity, but results across the literature have been mixed. The purpose of the present study is to use data from the ongoing implementation and dissemination of the SafeCare model to better understand (1) the characteristics of SafeCare implementation fidelity trajectories, and (2) whether individual level factors predict differences in fidelity and fidelity trajectories, especially provider attitudes toward EBP. The analyses reported here include 14,778 observed fidelity sessions by 868 providers in 172 agencies. We use multilevel modeling to examine fidelity, fidelity trajectories over time, and several potential individual-level predictors of fidelity, including demographics, work history, and attitudes toward EBP. We found: (1) that SafeCare fidelity begins high at baseline (93.85% on average); (2) that SafeCare fidelity displays a statistically significant trend of positive linear growth, even among those with less positive attitudes; and (3) that positive attitudes are associated with slightly higher fidelity on average and at baseline, while negative attitudes are associated with slightly lower fidelity on average and at baseline. To our knowledge, this is the largest longitudinal analysis of EBP fidelity in a child welfare program to date, and our findings support the notion that intensive coaching supports which are titrated over time can be sufficient to ensure sustained high fidelity, at least in some cases. Further, these findings indicate that robust training and coaching processes can ensure high fidelity and fidelity growth even among providers with less positive attitudes toward EBP.

A roadmap to address stress in graduate students: How to develop and disseminate a student-led single-session evidence based intervention

Objective: This manuscript describes an evidence-based, student-led, single-session group intervention to support emotional wellbeing among graduate students. The present objective is to provide a roadmap for other universities. Participants: Key participants include clinical psychology graduate students (leader and workshop facilitators), faculty supervisor, representatives from receiving departments or schools, and institutional advocates. Methods: The two-hour workshop was based on four core transdiagnostic cognitive behavioral skills, including psychoeducation about emotions, mindful emotional awareness, cognitive flexibility, and behavior change. The workshop was designed and continues to be led by trained graduate students. Results: Key steps and lessons learned are presented for the exploration, preparation, implementation, and sustainment phases. Conclusions: This program has the potential to be flexibly replicated at other universities to assist with graduate student mental health. It provides unique supports for recipients and unique training opportunities for student facilitators.

Inter-Organizational Alignment and Implementation Outcomes in Integrated Mental Healthcare for Children and Adolescents: A Cross-Sectional Observational Study

Background

Integrated care involves care provided by a team of professionals, often in non-traditional settings. A common example worldwide is integrated school-based mental health (SBMH), which involves externally employed clinicians providing care at schools. Integrated mental healthcare can improve the accessibility and efficiency of evidence-based practices (EBPs) for vulnerable populations suffering from fragmented traditional care. However, integration can complicate EBP implementation due to overlapping organizational contexts, diminishing the public health impact. Emerging literature suggests that EBP implementation may benefit from the similarities in the implementation context factors between the different organizations in integrated care, which we termed inter-organizational alignment (IOA). This study quantitatively explored whether and how IOAs in general and implementation context factors are associated with implementation outcomes in integrated SBMH.

Methods

SBMH clinicians from community-based organizations (CBOs; nclinician=27) and their proximal student-support school staff (nschool=99) rated their schools and CBOs (clinician only) regarding general (organizational culture and molar climate) and implementation context factors (Implementation Climate and Leadership), and nine common implementation outcomes (e.g., treatment integrity, service access, acceptability). The levels of IOA were estimated by intra-class correlations (ICCs). We fitted multilevel models to estimate the standalone effects of context factors from CBOs and schools on implementation outcomes. We also estimated the 2-way interaction effects between CBO and school context factors (i.e., between-setting interdependence) on implementation outcomes.

Results

The IOA in general context factors exceeded those of implementation context factors. The standalone effects of implementation context factors on most implementation outcomes were larger than those of general context factors. Similarly, implementation context factors between CBOs and schools showed larger 2-way interaction effects on implementation outcomes than general context factors.

Conclusions

This study preliminarily supported the importance of IOA in context factors for integrated SBMH. The findings shed light on how IOA in implementation and general context factors may be differentially associated with implementation outcomes across a broad array of integrated mental healthcare settings.

Helping Educational Leaders Mobilize Evidence (HELM): The iterative redesign of the Leadership and Organizational Change for Implementation (LOCI) intervention for use in schools

Background

Few "intervention agnostic" strategies have been developed that can be applied to the broad array of evidence-based practices (EBPs) in schools. This paper describes two studies that reflect the initial iterative redesign phases of an effective leadership-focused implementation strategy-Leadership and Organizational Change for Implementation (LOCI)-to ensure its acceptability, feasibility, contextual appropriateness, and usability when used in elementary schools. Our redesigned strategy-Helping Educational Leaders Mobilize Evidence (HELM)-is designed to improve principals' use of strategic implementation leadership to support the adoption and high-fidelity delivery of a universal EBP to improve student outcomes.

Method

In Study 1, focus groups were conducted (n = 6) with 54 district administrators, principals, and teachers. Stakeholders provided input on the appropriateness of original LOCI components to maximize relevance and utility in schools. Transcripts were coded using conventional content analysis. Key themes referencing low appropriateness were summarized to inform LOCI adaptations. We then held a National Expert Summit (Study 2) with 15 research and practice experts. Participants provided feedback via a nominal group process (NGP; n = 6 groups) and hackathon (n = 4 groups). The research team rated each NGP suggestion for how actionable, impactful/effective, and feasible it was. We also coded hackathon notes for novel ideas or alignment with LOCI components.

Results

Study 1 suggestions included modifications to LOCI content and delivery. Study 2's NGP results revealed most recommendations to be actionable, impactful/effective, and feasible. Hackathon results surfaced two novel ideas (distributed leadership teams and leaders' knowledge to support educators EBP use) and several areas of alignment with LOCI components.

Conclusion

Use of these iterative methods informed the redesign of LOCI and the development of HELM. Because it was collaboratively constructed, HELM has the potential to be an effective implementation strategy to support the use of universal EBP in schools.

Therapists' Perceived Competence in Delivering Trauma-Focused Cognitive Behavioral Therapy During Statewide Learning Collaboratives

The learning collaborative (LC), a multi-component training and implementation model, is one promising approach to address the need for increased availability of trauma-focused evidence-based practices. The current study used data from four cohorts of a statewide LC on Trauma-Focused Cognitive Behavioral Therapy (TF-CBT) to 1) evaluate pre- to post-LC changes in therapists' perceived competence in delivering TF-CBT and 2) explore therapist and contextual factors related to therapists' perceived TF-CBT competence. Therapists (N = 237) completed pre- and post-LC measures of practice information, interprofessional collaboration, organizational climate, and TF-CBT knowledge, perceived competence, and use. Findings indicated therapists' perceived TF-CBT competence significantly increased, pre- to post-LC (d = 1.31), with greater use of trauma-focused practices at pre-training and more TF-CBT training cases completed predicting greater pre- to post-LC gains in perceived TF-CBT competence. These findings highlight the need to assist therapists in identifying and completing training cases to promote competence and implementation.

Study protocol for a multi-level cross-sectional study on the equitable reach and implementation of coordinated specialty care for early psychosis

BACKGROUND

Approximately 115,000 young adults will experience their first episode of psychosis (FEP) each year in the USA. Coordinated specialty care (CSC) for early psychosis is an evidence-based early intervention model that has demonstrated effectiveness by improving quality of life and reducing psychiatric symptoms for many individuals. Over the last decade, there has significant increase in the implementation of CSC programs throughout the USA. However, prior research has revealed difficulties among individuals and their family members accessing CSC. Research has also shown that CSC programs often report the limited reach of their program to underserved populations and communities (e.g., ethnoracial minorities, rural and low socioeconomic neighborhoods). Dissemination and implementation research focused on the equitable reach and implementation of CSC is needed to address disparities at the individual level.

METHODS

The proposed study will create a novel integrative multi-level geospatial database of CSC programs implemented throughout the USA that will include program-level data (e.g., geocoded location, capacity, setting, role availability), provider-level data (race, ethnicity, professional credentials), and neighborhood-level census data (e.g., residential segregation, ethnic density, area deprivation, rural-urban continua, public transit time). This database will be used to characterize variations in CSC programs by geographical location and examine the overall reach CSC programs to specific communities. The quantitative data will be combined with qualitative data from state administrators, providers, and service users that will inform the development of dissemination tools, such as an interactive dashboard, that can aid decision making.

DISCUSSION

Findings from this study will highlight the impact of outer contextual determinants on implementation and reach of mental health services, and will serve to inform the future implementation of CSC programs with a primary focus on equity.

Individual and organizational factors as predictors of early evidence-based practice adoption in Michigan high schools: Baseline data from an implementation trial

Background

Adolescents increasingly access mental health services in schools. School mental health professionals (SPs; school counselors, social workers, etc.) can offer evidence-based mental health practices (EBPs) in schools, which may address access gaps and improve clinical outcomes. Although some studies have assessed factors associated with EBP adoption in schools, additional research focusing on SP- and school-level factors is warranted to support EBP implementation as SPs' mental health delivery grows.

Methods

Baseline data were collected from SPs at Michigan high schools participating in a statewide trial to implement SP-delivered cognitive behavioral therapy (CBT) to students. Models examined factors associated with attitudes about EBPs, implementation climate, and implementation leadership, and their associations with CBT knowledge, training attendance, and pre-training CBT delivery.

Results

One hundred ninety-eight SPs at 107 schools (87%) completed a baseline survey. The mean Evidence-Based Practice Attitude Scale (EBPAS) total score was 2.9, and school-aggregated mean scores of the Implementation Climate Scale (ICS) and Implementation Leadership Scale (ILS) were 1.83 and 1.77, respectively, all on a scale ranging from 0 (low) to 4 (high). ICS and ILS scores were lower than typically reported in clinical settings, while EBPAS scores were higher. School characteristics were not significantly associated with EBPAS, ICS, or ILS scores, but scores did differ by SP role. Higher EBPAS scores were associated with more CBT knowledge (average marginal effect for 1 SD change [AME] = 0.15 points) and a higher probability of training completion (AME = 8 percentage points). Higher ICS scores were associated with a higher probability of pre-training CBT delivery (AME = 6 percentage points), and higher ILS scores were associated with higher probability of training completion (AME = 10 percentage points).

Conclusions

Our findings suggest that SPs' attitudes toward EBPs and organizational support were positively associated with early signs of implementation success. As schools increasingly fill the adolescent mental healthcare access gap, efforts to strengthen both provider attitudes toward EBP and strategic organizational factors supporting EBP delivery will be key to encouraging EBP uptake in schools.

Plain Language Summary

Schools are an important setting in which adolescents receive mental healthcare. We need to better understand how to implement evidence-based practices (EBPs) in this setting to improve student mental health. This study examined the attitudes and perceptions of school professionals (SPs) as key contributors to the implementation of a particular EBP, the delivery of cognitive behavioral therapy (CBT) in schools. The study found that implementation climate and leadership scores in participating schools were lower than scores typically reported in clinical settings, while scores for SP attitudes about EBP adoption were higher than typical scores in clinical settings. Results further suggest that SPs with more positive attitudes toward EBPs are more knowledgeable of CBT and more likely to complete a 1-day CBT training. We also found that higher implementation climate scores were associated with SPs reporting pre-training CBT delivery (although this association was not statistically significant), and more implementation leadership was associated with SPs completing the CBT training. These findings suggest that SP attitudes toward EBPs and organizational support in schools are positively associated with early signs of implementation success. Early, low-intensity efforts to (1) improve SP attitudes about mental health EBPs, and (2) increase schools' support for implementation may scaffold more intensive implementation efforts in schools down the road.

Implementation readiness for evidence-based autism practices in school systems

Background

The increase in the number of autistic children being identified has led to increased demand on public schools to provide high-quality services. Effectively scaling up evidence-based practice (EBP) use for autistic students is challenging, given the complicated organization of special education. Teachers have significant challenges implementing autism EBP with fidelity. Factors such as implementation leadership and climate and attitudes toward EBP are linked to successful EBP use and may vary at different levels of the education system. Examining mechanisms of successful implementation is a critical step to support scale-up.

Method

In this observational study, conducted from September 2018 to March 2020, California school personnel (n = 2273) at multiple levels of the system completed surveys related to implementation climate, leadership, and attitudes toward EBP. Data were collected throughout California at the Special Education Local Plan Areas, County Office of Education, and district and school levels from educators and administrators working in public schools supporting autistic students. Multi-level modeling was conducted to characterize implementation readiness.

Results

Overall, implementation climate and leadership scores are low across levels with regional levels rated more positively than districts or schools. Attitudes toward EBP were moderate, with those working in schools having the poorest ratings and specialists/trainers and related service providers (e.g., speech-language pathologists) having the highest ratings.

Conclusions

Outcomes provide a unique opportunity to compare implementation factors across organizational levels with a large, statewide sample. These data provide guidance for developing implementation interventions at multiple levels of the education system to increase readiness for effective scale-up of autism EBP in schools. Personnel and leaders at different organizational levels may need differentiated training targeting improved implementation climate and leadership. Personnel within districts and schools may experience a particular benefit from leadership support for EBP implementation.

Correspondence between practitioners’ self-assessment and independent motivational interviewing treatment integrity ratings

As evaluation of practitioners’ competence is largely based on self-report, accuracy in practitioners’ self-assessment is essential for ensuring high quality treatment-delivery. The aim of this study was to assess the relationship between independent observers’ ratings and practitioners’ self-reported treatment integrity ratings of Motivational interviewing (MI). Practitioners (N = 134) were randomized to two types of supervision [i.e., regular institutional group supervision, or individual telephone supervision based on the MI Treatment Integrity (MITI) code]. The mean age was 43.2 years (SD = 10.2), and 62.7 percent were females. All sessions were recorded and evaluated with the MITI, and the MI skills were self-assessed with a questionnaire over a period of 12 months. The associations between self-reported and objectively assessed MI skills were overall weak, but increased slightly from baseline to the 12-months assessment. However, the self-ratings from the group that received monthly objective feedback were not more accurate than those participating in regular group supervision. These results expand findings from previous studies and have important implications for assessment of practitioners’ treatment fidelity: Practitioners may learn to improve the accuracy of self-assessment of competence, but to ensure that patients receive intended care, adherence and competence should be assessed objectively.

Leading the charge in the education sector: development and validation of the School Implementation Leadership Scale (SILS)

Background

Strategic implementation leadership is a critical determinant of successful implementation, hypothesized to create a more supportive implementation climate conducive to the adoption and use of evidence-based practices. Implementation leadership behaviors may vary significantly across contexts, necessitating studies that examine the validity of established measurement tools in novel health service delivery sectors. The education sector is the most common site for delivering mental health services to children and adolescents in the USA, but research focused on implementation leadership in schools is in the early phases, and there is a need for adaptation and expansion of instruments in order to tailor to the school context. The current study adapted and validated the School Implementation Leadership Scale (SILS) (based on the Implementation Leadership Scale) in a sample of elementary school personnel from six school districts who were implementing one of two well-established prevention programs for supporting children’s mental health.

Methods

Participants were 441 public school teachers from 52 elementary schools in the Midwest and West Coast of the USA. Participants completed a survey that contained: (1) an adapted and expanded version of the SILS with additional items generated for four existing subscales as well as three new subscales (communication, vision/mission, and availability), and (2) additional tools to evaluate convergent and divergent validity (i.e., measures of general/molar leadership and teaching attitudes). Data underwent (1) examination of item characteristic curves to reduce items and ensure a pragmatic instrument, (2) confirmatory factor analyses to establish structural validity, and (3) evaluation of convergent and divergent validity.

Results

Item reduction analyses resulted in seven subscales of three items each. Results indicated acceptable fit for a seven-factor structural model (CFI = .995, TLI = .99, RMSEA = .07, SRMR = 0.02). Second-order factor loadings were high (λ = .89 to .96), suggesting that the SILS subscales comprise a higher-order implementation leadership factor. All subscales demonstrated good inter-item reliability (α = .91–.96). Convergent and divergent validity results were generally as hypothesized, with moderate to high correlations between SILS subscales and general leadership, moderate correlations with teaching attitudes, and low correlations with school demographics.

Conclusions

Overall, results provided strong structural, convergent, and divergent validity evidence for the 21-item, 7-factor SILS instrument. Implications for the measurement of implementation leadership in schools are discussed, as well as strategies to support leaders to enhance their strategic behaviors related to the implementation of mental health prevention programs (e.g., adaptation of existing leadership-focused implementation strategies).

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-022-01222-7.

Practice-driven research for statewide scale up: Implementation outcomes of the California Autism Professional Training and Information Network

Scaling up the use of evidence-based practice (EBP) for autism across service sectors and regions has presented a considerable challenge indicating a clear need for continued development. The California Autism Professional Training and Information Network (CAPTAIN) integrates implementation drivers into specific procedures and methodology as an implementation strategy to support statewide scale up. The current study was designed to evaluate the impact of CAPTAIN on provider-level outcomes including attitude toward, and knowledge, fidelity and use of autism EBPs, and overall classroom quality. Overall, results indicated variability across measures, with some significant differences between CAPTAIN-trained and non-CAPTAIN-trained providers. CAPTAIN-trained providers reported more openness to EBP. Significantly more CAPTAIN-trained direct service providers reported collecting fidelity of implementation data (χ2(2, N = 1515) = 10.95, p = 0.004), collecting student data (χ2(2, N = 1509) = 14.19, p = 0.001), and reported using their primary EBP with “most or all students” (χ2(2, N = 1514) = 11.41, p = 0.003) than providers not trained by CAPTAIN. In summary, these preliminary findings show promise for the efficacy of the CAPTAIN model to increase dissemination and implementation of EBP at the classroom level.

Lay abstract

Supporting use of evidence-based practice in public service programs for autistic individuals is critical. The California Autism Professional Training and Information Network (CAPTAIN) brings together best practices from intervention and implementation research to support scale up of autism services. The current study was designed to evaluate the impact of CAPTAIN on provider-level outcomes including attitude toward, knowledge, fidelity, and use of autism EBPs and overall classroom quality. Overall, results indicated variability across measures, with some significant differences between CAPTAIN-trained and non-CAPTAIN-trained providers. These preliminary findings show promise for the efficacy of the CAPTAIN model to increase dissemination and implementation of EBP at the classroom level.

Tailored isn't always better: Impact of standardized versus tailored training on intention to use measurement-based care

Background: Brief educational trainings are often used for disseminating and implementing evidence-based practices (EBPs). However, many accessible trainings are ubiquitously standardized. Tailored training focused on modifying individual or contextual factors that may hinder EBP implementation is recommended, but there is a dearth of research comparing standardized versus tailored training. This study sought to: (a) assess the impact of MBC training on clinician intention to use measurement-based care (MBC); (b) compare the effect of standardized versus tailored training on clinician intention to MBC; and (c) identify clinician-level predictors of intention. Methods: Clinicians (n = 152) treating adult clients with depression at 12 community mental health clinics were randomized to either tailored or standardized MBC training. Clinic-specific barriers and facilitators were used to inform training content and structure tailoring. Linear mixed modeling tested the association between training condition and post-training intention to use MBC, as well as hypothesized individual-level predictors of post-training intention (e.g., age, gender). Results: Clinician intention pre- and post-training increased across training conditions (B = 0.38, t = -5.95, df = 36.99, p < .01, Cohen's d = 0.58). Results of linear mixed modeling procedures suggest no significant difference in clinician intention between conditions post-training (B = -0.03, SE = .19, p > .05, Cohen's d = .15). Only baseline intention emerged as a predictor of post-training intention (B = 0.39, SE = .05, p < .05). Conclusions: These findings suggest the additional effort to tailor training may not yield incremental benefit over standardized training, at least in the short term. As a result, implementation efforts may be able to reserve time and finances for other elements of implementation beyond the training component.

Plain Language Summary

Educational training is a common approach for enhancing knowledge about research-supported mental health treatments. However, these trainings are often not tailored to meet the needs of the trainees, and there is insufficient evidence about whether tailoring might improve the impact of training compared to a one-size-fits-all, standard version. This study compared the impact of a tailored versus standard training on mental health clinician's intentions to use measurement-based (MBC) care for monitoring treatment progress for clients with depression. Study results indicated that intention to use MBC improved for clinicians receiving both the tailored and standard training after training completion. There were no differences in intention to use MBC care when the two types of training were compared. These study findings suggest that tailoring, which may require substantial time and effort, may not be a necessary step to improve the short-term impact of educational trainings.

A mixed-methods study to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings

It has been well-documented that the degree to which interventions are implemented with fidelity in typical service settings has varied. Frequently, interventions are developed and tested in highly controlled or early adopter settings. Less attention has been given to what implementation looks like in usual care, and which factors promote practitioners' ability to implement with fidelity. Individuals and organizations implementing interventions in the real world receive varying levels of external supports and may apply a new intervention unaided. The purpose of this mixed-methods study was to explore factors that support implementation as intended in local community agencies. In the quantitative phase of this study, 32 case planners implementing Family Connections (FC), a child maltreatment preventive intervention, completed a survey about their perceptions of practitioner and organizational factors related to fidelity. The survey data were connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner's ability to implement an intervention in usual care. The quantitative and qualitative results suggest supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioners' ability to learn and use FC. The quantitative results suggest that the level of education was positively associated with fidelity and perceptions of the intervention's limitations may be negatively related to implementation. Additional components that influence implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

Plain Language Abstract

This mixed-methods study sought to understand the impact of practitioner and organizational factors on fidelity of a child maltreatment prevention intervention in community-based settings. The study first asked case planners about their perceptions of practitioner and organizational factors related to fidelity through an online survey. This survey was connected to case-level fidelity scores to understand the relationship between perceptions and fidelity. The qualitative phase of this study involved further exploration with nine case planner interviews and two separate focus groups with supervisors and agency leadership. The results of this study suggest that supervision is a key contributor to a practitioner’s ability to implement a maltreatment prevention intervention. Both methods of the study suggest that various aspects of supervision, including supervisors’ perseverance, proactiveness, knowledge, availability, and skill reinforcement are important components of enhancing a practitioner's ability to learn and use the intervention. Additional components that influence the implementation for future research emerged from the qualitative phase related to system expectations and policies, individual practitioner attributes, and characteristics of the intervention.

The relationship between first-level leadership and inner-context and implementation outcomes in behavioral health: a scoping review

BACKGROUND

First-level leadership is uniquely positioned to support evidence-based practice (EBP) implementation for behavioral health due to first-level leaders' access to and relationship with service providers. First-level leaders are individuals who directly supervise and manage frontline employees who do not manage others. However, first-level leadership is underrepresented in existing reviews of the impact of leadership on EBP implementation. This review describes the relationship between first-level leadership and implementation determinants and outcomes.

METHODS

A scoping review was performed to synthesize the literature on the relationship between first-level leadership and inner-context and implementation outcomes. A literature search was conducted in PubMed, Eric, PsycINFO, CINAHL, Scopus, and Web of Science. To be eligible, studies had to examine first-level leadership, be conducted in settings providing behavioral health services, and examine the relationship between first-level leadership and an implementation or inner-context outcome. Data extraction and synthesis were performed to describe study characteristics, leader-outcome relationships, and overlap in leadership frameworks.

RESULTS

Twenty-one records met our inclusion criteria. Studies primarily relied on observational designs and were often cross-sectional. Studies more often examined general leadership rather than leadership strategically focused on EBP implementation (i.e., strategic implementation leadership). Our findings suggest that several forms of first-level leadership are inconsistently related to a broad set of implementation determinants, with infrequent examination of specific implementation outcomes. The broad set of implementation determinants studied, limited number of replications, and inconsistent findings have resulted in sparse evidence for any specific leadership-outcome relationship. The greatest accumulation of evidence exists for general leadership's positive relationship with providers' EBP attitudes, most notably in the form of transformational leadership. This was followed by evidence for strategic implementation leadership facilitating general implementation. Our synthesis revealed moderate conceptual overlap of strategic implementation leadership behaviors described in the theory of implementation leadership and theory of middle managers' role in implementation.

CONCLUSIONS

Our findings suggest that first-level leadership may play an important role in shaping implementation determinants and outcomes, but consistent empirical support is sparse and confidence dampened by methodological issues. To advance the field, we need studies that adopt stronger methodological rigor, address the conceptual overlap in leadership frameworks, examine a broader set of implementation outcomes, and examine conditions under which leadership impacts implementation.

TRIAL REGISTRATION

This review was not registered.

An Exploratory Study of a Training Team-Coordinated Approach to Implementation

Background

It is now widely understood that successful implementation of evidence-based treatments is facilitated by several favorable conditions (e.g., community buy-in, invested agency leadership). However, strategies for supporting agencies in promoting these conditions have been examined to a lesser extent. In this exploratory study, the implementation support procedures of Parenting with Love and Limits (PLL), an evidence-informed family treatment for child/adolescent behavior problems in which the training team follows structured procedures to help coordinate implementation support activities, are illustrated, and their preliminary effectiveness examined.

Methods

PLL documents and communication records between PLL and n = 23 sites across the U.S. that initiated PLL pre-implementation activities were reviewed. In addition, implementation activities completed for each agency were entered into the Stages of Implementation Completion (SIC) dashboard.

Results

The prescriptive nature of the PLL implementation support protocol was illustrated through descriptions of procedural documents and case examples. Quantitative analyses revealed that, among the 23 sites that began pre-implementation, 9 discontinued, with a trend toward sites in metropolitan areas being more likely to discontinue than those in less populous areas. In addition, the 14 sites that launched PLL demonstrated a high amount of consistency in activities, with sites in the sustainability phase completing an average of 86% of implementation behaviors.

Conclusions

Training team-coordination of implementation activities may be one promising approach for supporting agencies in completing tasks to facilitate successful uptake of evidence-supported interventions. In turn, sustained implementation of evidence-supported treatments could allow communities to benefit from practice innovations to a greater extent.

A Preliminary Investigation of Provider Attitudes Toward a Transdiagnostic Treatment: Outcomes from Training Workshops with the Unified Protocol

Evidence-based psychological treatments (EBPTs) for common mental health conditions are efficacious but remain underutilized in clinical service settings. Novel transdiagnostic and modular approaches that treat several disorders simultaneously promise to address common barriers to the dissemination and implementation of traditional EBPTs. Despite the promise that transdiagnostic treatments hold, the claims that these interventions can be more easily disseminated and implemented have not been widely tested. The present study examined whether a transdiagnostic treatment, the Unified Protocol (UP), addresses some barriers to dissemination and implementation for clinicians. Exploratory aims of the current study were to examine the effects of a UP introductory training workshop on clinician attitudes and behaviors by: (1) evaluating UP knowledge and treatment delivery, (2) determining relationships between clinician characteristics and their knowledge acquisition, satisfaction with UP, and UP penetration, and (3) exploring clinicians' perceptions of the UP's characteristics utilizing mixed methods. Workshop participants showed a good understanding of UP treatment concepts following training, and over a third of survey respondents reported use of the intervention 6-months after training. Positive attitudes toward EBPTs and fewer years of clinical practice were associated with greater satisfaction with the UP. Clinicians held positive views of the UP's flexibility and relative advantage over standard EBPTs but held negative views toward the manual's design and packaging. Overall, our findings suggest that clinicians may view transdiagnostic treatments such as the UP favorably and may consider them appealing over standard EBPTs. However, barriers associated with traditional EBPTs may extend to transdiagnostic treatments like the UP.

The Association Between Clinician and Perceived Organizational Factors with Early Fidelity to Cognitive Processing Therapy for Posttraumatic Stress Disorder in a Randomized Controlled Implementation Trial

A common metric for determining implementation success is the measurement of clinician adherence to, and competence in, delivering a psychotherapy. The present study examined clinician and organizational factors as predictors of early adherence and competence among 78 clinicians delivering cognitive processing therapy (CPT), an evidence-based psychotherapy (EBP) for posttraumatic stress disorder, in a randomized controlled implementation trial. Results indicated that clinicians' willingness to adopt an EBP if required to do so was significantly associated with early adherence and competence in CPT delivery. Level of clinician education was significantly associated with early competence in delivering CPT. Organizational factors did not predict early adherence or competence. Implications of the findings are discussed.

Changes in Community Clinicians' Attitudes and Competence following a Transdiagnostic Cognitive Behavioral Therapy Training

Background

Although the literature suggest that attitudes toward evidence-based practices (EBPs) are associated with provider use of EBPs, less is known about the association between attitudes and how competently EBPs are delivered. This study examined how initial attitudes and competence relate to improvements in attitudes and competence following EBP training.

Methods

Program evaluation data was collected during implementation of an EBP in a large community mental health network. Clinicians (N=891) received intensive training in cognitive behavioral therapy followed by six months of consultation. Attitudes were assessed using the Evidence-Based Practice Attitude Scale, and competence was assessed using the Cognitive Therapy Rating Scale. Data were analyzed by fitting three latent change score models to examine the relationship between changes in attitudes and competence across the training and within its two phases (workshop phase, consultation phase).

Results

Latent change models identified significant improvement in attitudes (Ms _latent change≥1.03, SEs≤ 0.18, zs≥6.55, ps< .001) and competence (Ms _{latent change} ≥14.16, SEs≤3.10, zs≥2.82, ps<.001) across the full training and in each phase. Higher pre-workshop attitudes predicted significantly greater change in competence in the workshop and across training (bs≥1.62, SEs≤0.90, z≥1.09, p<.04, β≥0.10); however, contrary to our hypothesis, post-workshop attitudes did not significantly predict change in competence in consultation (b=1.62, SE=0.86, z=1.87, p=.06, β=0.09). Change in attitudes and change in competence in the training period and within the two phases were not significantly correlated.

Conclusions

Results indicate that pre-training attitudes about EBPs present a target for implementation interventions, given their relation to changes in both attitudes and competence throughout training. Following participation in in itial training workshops, other factors such as subjective norms, implementation culture, or system-level policy shifts may be more predictive of change in competence through consultation.

Confirmatory factor analysis of the Evidence-Based Practice Attitude Scale (EBPAS) in a large and representative Swedish sample: is the use of the total scale and subscale scores justified?

Background

There is a call for valid and reliable instruments to evaluate implementation of evidence-based practices (EBP). The 15-item Evidence-Based Practice Attitude Scale (EBPAS) measures attitude toward EBP, incorporating four lower-order factor subscales (Appeal, Requirements, Openness, and Divergence) and a Total scale (General Attitudes). It is one of a few measures of EBP attitudes evaluated for its psychometric properties. The reliability of the Total scale has been repeatedly supported, but also the multidimensionality of the inventory. However, whether all of the items contribute to the EBPAS Total beyond their subscales has yet to be demonstrated. In addition, the Divergence subscale has been questioned because of its low correlation with the other subscales and low inter-item correlations. The EBPAS is widely used to tailor and evaluate implementation efforts, but a Swedish version has not yet been validated. This study aimed to contribute to the development and cross-validation of the EBPAS by examining the factor structure of t a Swedish-language version in a large sample of mental health professionals.

Methods

The EBPAS was translated into Swedish and completed by 570 mental health professionals working in child and adolescent psychiatry settings spread across Sweden. The factor structure was examined using first-order, second-order and bifactor confirmatory factor analytic (CFA) models.

Results

Results suggested adequate fit for all CFA models. The EBPAS Total was strongly supported in the Swedish version. Support for the hierarchical second-order model was also strong, while the bifactor model gave mixed support for the subscales. The Openness and Requirements subscales came out best, while there were problems with both the Appeal (e.g. not different from the General Attitudes factor) and the Divergence subscales (e.g. low reliability).

Conclusions

Overall, the psychometric properties were on par with the English version and the total score appears to be a valid measure of general attitudes towards EBP. This is the first study supporting this General Attitudes factor based on a bifactor model. Although comparatively better supported in this Swedish sample, we conclude that the use of the EBPAS subscale scores may result in misleading conclusions. Practical implications and future directions are discussed.

Identifying the organizational innovation-specific capacity needed for exposure therapy

BACKGROUND

Current approaches to increasing the rates of clinician use of exposure therapy for anxiety disorders in community settings are limited. Research underscores the importance of addressing contextual variables to facilitate clinician use of evidence-based practices; however, no studies have identified the innovation-specific organizational capacity necessary to implement exposure therapy. Such work is critical to ensure that treatment-seeking individuals with anxiety receive effective care.

METHODS

We used a two-step process to identify the innovation-specific organizational capacity necessary to deliver exposure. First, 24 leaders of specialty anxiety clinics in the United States (50% female, mean [M]age = 47.7 years) completed a survey about the organizational innovation-specific capacity (e.g., policies and procedures) they employ to support their providers in delivering exposure therapy. Second, 19 community clinicians (79% female, M age = 42.9 years) reported on the extent to which these characteristics were present in their settings.

RESULTS

In Step 1, specialty clinic leaders unanimously endorsed six organizational characteristics as essential and five as important within the areas of organizational policies, supervisory support, and peer clinician support. These characteristics were present in more than 90% of specialty clinics. In Step 2, therapists in community clinics reported these characteristics were minimally present in their organizations.

CONCLUSIONS

Specialty clinic leaders exhibited consensus on the innovation-specific organizational capacity necessary to implement exposure therapy. Identified characteristics were largely absent from community clinics. Developing fiscal, policy, or organizational strategies that enhance the organizational capacity within community settings may improve the patients' access to effective treatment for anxiety disorders.

Provider Readiness and Adaptations of Competency Drivers During Scale-Up of the Family Check-Up

We used provider (n = 112) data that staff at the agency disseminating the Family Check-Up (FCU; REACH Institute) collected to profile provider diversity in community settings and to examine whether provider profiles are related to implementation fidelity. Prior to FCU training, REACH Institute staff administered the FCU Provider Readiness Assessment (PRA), a provider self-report measure that assesses provider characteristics previously linked with provider uptake of evidence-based interventions. We conducted a latent class analysis using PRA subscale scores as latent class indicators. Results supported four profiles: experienced high readiness (ExHR), experienced low readiness (ExLR), moderate experience (ME), and novice. The ExHR class was higher than all other classes on: (1) personality variables (i.e., agreeableness, conscientiousness, openness, extraversion); (2) evidence-based practice attitudes; (3) work-related enthusiasm and engagement; and (4) their own well-being. The ExHR class was also higher than ExLR and ME classes on clinical flexibility. The ME class was lowest of all classes on conscientiousness, supervision, clinical flexibility, work-related enthusiasm and engagement, and well-being. During the FCU certification process, FCU Consultants rated providers' fidelity to the model. Twenty-three of the 112 providers that completed the PRA also participated in certification. We conducted follow-up regression analyses using fidelity data for these 23 providers to explore associations between probability of class membership and fidelity. The likelihood of being in the ExHR class was related to higher FCU fidelity, whereas the likelihood of being in the ExLR class was related to lower fidelity. We discuss how provider readiness assessment data can be used to guide the adaptation of provider selection, training, and consultation in community settings.

Adapting parent engagement strategies for an evidence-based parent-mediated intervention for young children at risk for autism spectrum disorder

AIMS

Aims included (a) characterizing provider feedback on parent engagement strategies integrated into a parent-mediated intervention for toddlers at risk for autism spectrum disorder (ASD) and (b) identifying provider characteristics that predict attitudes about parent engagement strategies.

METHODS

A mixed method approach was utilized, including gathering quantitative data via survey (breadth) and collecting qualitative data via interview (depth). Acceptability, utility, appropriateness, sustainment, generalizability, and perceived effectiveness were examined. Fourteen agency leaders and 24 therapists provided input.

RESULTS

Providers perceived the integration of parent engagement strategies as having a positive impact on implementation. Providers considered the strategies to be acceptable, appropriate, and effective, though barriers of time and complexity were noted. Provider characteristics did not consistently predict attitudes about the engagement strategies.

CONCLUSIONS

Incorporating parent engagement strategies into parent-mediated interventions for ASD is well-received by providers and may improve quality of service delivery for families served in early intervention for ASD.

Adaption and pilot implementation of an autism executive functioning intervention in children's mental health services: a mixed-methods study protocol

BACKGROUND

Youth with autism spectrum disorder (ASD) represent a growing population with significant service needs. Prominent among these needs are high rates of co-occurring psychiatric conditions that contribute to increased functional impairments and often necessitate mental health services. Executive functioning deficits are associated with ASD as well as common co-occurring conditions (e.g., attention-deficit/hyperactivity disorder) and an evidence-based intervention has been developed and tested to address executive functioning within the school context. There is an urgent need to implement indicated evidence-based interventions for youth with ASD receiving care in community mental health settings. Interventions that optimally "fit" the mental health services context as well as the complex and co-occurring mental health needs of these youth have the potential to improve key clinical outcomes for this high priority population.

METHODS

This mixed-methods developmental study will apply the Exploration, Preparation, Implementation, Sustainment implementation framework and a community-academic partnership approach to systematically adapt and test an evidence-based executive functioning intervention for youth with ASD for delivery in community mental health settings. Specific aims are to (1) conduct a need and context assessment to inform the systematic adaptation an executive functioning evidence-based intervention; (2) systematically adapt the clinical intervention and develop a corresponding implementation plan, together entitled "Executive Functioning for Enhancing Community-based Treatment for ASD," (EFFECT for ASD); and (3) conduct a feasibility pilot test of EFFECT for ASD in community mental health settings.

DISCUSSION

Tailoring evidence-based interventions for delivery in community-based mental health services for youth with ASD has the potential to increase quality of care and improve child outcomes. Results from the current study will serve as the foundation for large-scale hybrid implementation and effectiveness trials and a generalizable approach for different service systems of care and clinical populations.

TRIAL REGISTRATION

Clinicaltrials.gov, NCT04295512.

A Case Study of Implementing Grant-Funded Integrated Care in a Community Mental Health Center

The US government funds integrated care demonstration projects to decrease health disparities for individuals with serious mental illness. Drawing on the Exploration Preparation Implementation Sustainability (EPIS) implementation framework, this case study of a community mental health clinic describes implementation barriers and sustainability challenges with grant-funded integrated care. Findings demonstrate that integrated care practices evolve during implementation and the following factors influenced sustainability: workforce rigidity, intervention clarity, policy and funding congruence between the agency and state/federal regulations, on-going support and training in practice application, and professional institutions. Implementation strategies for primary care integration within CMHCs include creating a flexible workforce, shared definition of integrated care, policy and funding congruence, and on-going support and training.

Attitudes Toward Family-Based Treatment Impact Therapists' Intent to Change Their Therapeutic Practice for Adolescent Anorexia Nervosa

Community-based clinicians who treat patients with eating disorders rarely use empirically-supported treatments, and research demonstrates that clinicians make significant modifications when implementing family-based treatment (FBT) for anorexia nervosa. This study examined clinician attitudes toward FBT and explored the extent to which attitudes predicted intent to shift practices following training in FBT. Clinicians (N = 129) completed a standardized training in FBT for AN, either a two-day introductory training (n = 99) or a one-day "advanced" training (n = 30). Linear regressions were used to examine the association between therapists' attitudes toward FBT and their intent to use strategies consistent with FBT in the future, adjusting for pre-training use of strategies. Providers reported very positive attitudes toward evidence-based practices in general and moderately positive attitudes toward FBT. There were no significant differences between "novice" and "advanced" providers on attitudes toward evidence-based practices or FBT (ps > .10). For the subset of providers attending their first training in FBT, more positive attitudes toward FBT significantly predicted greater intent to use FBT-consistent strategies (p = .004), and more positive attitudes toward evidence-based practice significantly predicted lesser intent to use FBT-inconsistent strategies (p = .009). This study suggests that both general attitudes toward evidence-based practice and specific attitudes toward FBT may impact implementation. Future research might examine whether a brief intervention to improve attitudes toward FBT might increase the likelihood of seeking expert consultation post-training.

Individual and organizational characteristics predicting intervention use for children with autism in schools

LAY ABSTRACT

Several interventions or treatment programs have been shown through research to improve social outcomes for children with autism, but they often are not used in schools. This study examined individual (school personnel) and organizational (school level) factors associated with the use of a research-informed social engagement intervention, Remaking Recess, for children with autism in elementary schools. A total of 28 school personnel from 12 schools in 5 districts in the northeastern United States participated. Schools were randomly assigned to (1) training in Remaking Recess only or (2) training in Remaking Recess with implementation support (tailored strategies to remove barriers to increase use of Remaking Recess). School personnel rated their attitudes about research-informed interventions, organizational readiness (school's readiness to use a research-informed intervention), and fidelity or the degree to which an intervention is used as it was originally designed. Observers rated school personnel's fidelity at baseline (pre-intervention training) and exit (post-intervention training). The results suggest that self-rated fidelity was lower when staff perceived the use of Remaking Recess was required; however, observer-rated fidelity was lower when staff rated Remaking Recess as appealing. In addition, self-rated fidelity was higher when there was a sufficient number of staff, positive individual growth, and organizational adaptability. The results also indicated higher observer-rated fidelity when staff perceived positive influence over their coworkers. The results suggest that both individual (attitudes) and organizational (influence, staffing, growth, adaptability) characteristics may affect implementation success in schools. A collaborative decision-making approach for evidence-based practice use is recommended.

Leadership profiles associated with the implementation of behavioral health evidence-based practices for autism spectrum disorder in schools

Implementation of evidence-based practice (EBP) for autism spectrum disorder (ASD) in the education system is a public health priority. Leadership is a critical driver of EBP implementation but little is known about the types of leadership behaviors exhibited by school leaders and how this influences the context of EBP implementation, particularly for students with ASD. The objectives of this study were to determine (1) the leadership profiles of principals involved in EBP implementation for students with ASD and (2) how these leadership profiles related to school characteristics and implementation climate. The Exploration, Preparation, Implementation, Sustainment (EPIS) framework was used to guide the design and analysis of this study. Participants (n = 296) included principals, teachers, and classroom support staff. They provided demographic information and completed the Multifactor Leadership Questionnaire and Implementation Climate Scale. Using latent profile analysis, a three-pattern solution was identified: Disengaged (6% of sample), Undifferentiated (23% of sample), and Optimal (71% of sample). Principals in schools with higher proportions of students with an individualized education program were more likely to be classified as Undifferentiated than Optimal. The Optimal group was associated with more positive implementation climate than the Undifferentiated or Disengaged groups. Findings suggest that leadership behaviors rated by principals and their staff involved in implementation of common autism EBPs can be meaningfully clustered into three discernible profiles that are shaped by organizational context and linked to strategic implementation climate. Our study findings have implications for leadership training and service delivery in schools by underscoring the critical nature of school leadership during implementation of EBPs for children with autism and the interplay between specific leadership behaviors and strategic implementation climate.

Potential Reduction of Symptoms With the Use of Persuasive Systems Design Features in Internet-Based Cognitive Behavioral Therapy Programs for Children and Adolescents With Anxiety: A Realist Synthesis

BACKGROUND

Internet-based cognitive behavioral therapy (iCBT) for children and adolescents is a persuasive system that combines 3 major components to therapy-therapeutic content, technological features, and interactions between the user and program-intended to reduce users' anxiety symptoms. Several reviews report the effectiveness of iCBT; however, iCBT design and delivery components differ widely across programs, which raise important questions about how iCBT effects are produced and can be optimized.

OBJECTIVE

The objective of this study was to review and synthesize the iCBT literature using a realist approach with a persuasive systems perspective to (1) document the design and delivery components of iCBT and (2) generate hypotheses as to how these components may explain changes in anxiety symptoms after completing iCBT.

METHODS

A multi-strategy search identified published and gray literature on iCBT for child and adolescent anxiety up until June 2019. Documents that met our prespecified inclusion criteria were appraised for relevance and methodological rigor. Data extraction was guided by the persuasive systems design (PSD) model. The model describes 28 technological design features, organized into 4 categories that help users meet their health goals: primary task support, dialogue support, system credibility support, and social support. We generated initial hypotheses for how PSD (mechanisms) and program delivery (context of use) features were linked to symptom changes (outcomes) across iCBT programs using realist and meta-ethnographic techniques. These hypothesized context-mechanism-outcome configurations were refined during analysis using evidence from the literature to improve their explanatory value.

RESULTS

A total of 63 documents detailing 15 iCBT programs were included. A total of six iCBT programs were rated high for relevance, and most studies were of moderate-to-high methodological rigor. A total of 11 context-mechanism-outcome configurations (final hypotheses) were generated. Configurations primarily comprised PSD features from the primary task and dialogue support categories. Several key PSD features (eg, self-monitoring, simulation, social role, similarity, social learning, and rehearsal) were consistently reported in programs shown to reduce anxiety; many features were employed simultaneously, suggesting synergy when grouped. We also hypothesized the function of PSD features in generating iCBT impacts. Adjunct support was identified as an important aspect of context that may have complemented certain PSD features in reducing users' anxiety.

CONCLUSIONS

This synthesis generated context-mechanism-outcome configurations (hypotheses) about the potential function, combination, and impact of iCBT program components thought to support desired program effects. We suggest that, when delivered with adjunct support, PSD features may contribute to reduced anxiety for child and adolescent users. Formal testing of the 11 configurations is required to confirm their impact on anxiety-based outcomes. From this we encourage a systematic and deliberate approach to iCBT design and evaluation to increase the pool of evidence-based interventions available to prevent and treat children and adolescents with anxiety.

Understanding the Organizational Implementation Context of Schools: A Qualitative Study of School District Administrators, Principals, and Teachers

Key features of the school environment can have a significant impact on teachers' effective use of evidence-based practices (EBP), yet implementation-specific organizational constructs have rarely been studied in the education sector. This study examined three aspects of the organizational implementation context (implementation leadership, climate, and citizenship behavior), which have been conceptualized and validated in other service settings. Focus groups with central office administrators, principals, and teachers were conducted to understand the applicability and conceptual boundaries of these organizational constructs in schools. Focus group transcripts were coded, and the results indicated both similarities and differences in their conceptualizations of implementation leadership, climate, and citizenship behavior in school. The data indicated that: (1) implementation leadership was largely present in schools with the addition of Distributed Leadership; (2) two implementation climate constructs were most clearly present (i.e., Focus on EBP and Educational Support for EBP) and two additional constructs (i.e., Existing Support to Deliver EBP and Prioritization of EBP) emerged as part of this construct; and (3) implementation citizenship behavior (Helping Others and Keeping Informed) was consistently acknowledged across schools and two new components emerged (i.e., Information Sharing and Observation/Feedback). Recommendations to researchers and community stakeholders are discussed.

The influence of a blended, theoretically-informed pre-implementation strategy on school-based clinician implementation of an evidence-based trauma intervention

BACKGROUND

Individual-level implementation determinants, such as clinician attitudes, commonly influence the successful adoption of evidence-based practices, but few explicit strategies have been tested with regard to their ability to impact these key mechanisms of change. This paper reports on an initial test of a blended, theoretically informed pre-implementation strategy designed to target malleable individual-level determinants of behavior change. Beliefs and Attitudes for Successful Implementation in Schools (BASIS) is a brief and pragmatic pre-implementation strategy that uses strategic education, social influence techniques, and group-based motivational interviewing to target implementation attitudes, perceived social norms, perceived behavioral control, and behavioral intentions to implement among mental health clinicians working in the education sector.

METHODS

As part of a pilot trial, 25 school mental health clinicians were randomized to BASIS (n = 12) or an attention control placebo (n = 13), with both conditions receiving training and consultation in an evidence-based intervention for youth experiencing trauma (the Cognitive Behavioral Intervention for Trauma in Schools). Theorized mechanisms of change (attitudes, perceived social norms, perceived behavioral control, and behavioral intentions) were assessed at baseline, post-training, and 4-month follow-up. Clinician participation in post-training consultation and intervention adoption were also tracked.

RESULTS

A series of regression models and independent sample t tests indicated that BASIS had significant, medium to large effects on the majority of its proximal mechanisms from baseline to post-training. BASIS was also associated with a greater latency between initial training in the intervention and discontinuation of participation in post-training consultation, with clinicians in the BASIS condition persisting in consultation for an average of 134 days versus 32 days for controls, but this difference was not statistically significant. At 4-month follow-up, most differences in the theorized mechanisms had attenuated, and approximately the same small number of BASIS clinicians adopted the trauma intervention as controls.

CONCLUSION

Findings suggest that the brief BASIS pre-implementation strategy had a significant influence on its proximal mechanisms of change, but that these changes did not persist over time or translate into adoption of the trauma intervention. Implications for theory refinement, revisions to the BASIS protocol, and next steps for research surrounding individual-level implementation strategies are discussed.

TRIAL REGISTRATION

ClinicalTrials.gov Identifier: NCT03791281 . Registered 31 December 2018-Retrospectively registered.

Advancing the state-level tracking of evidence-based practices: a case study

Background

Despite a sustained focus by policymakers and researchers on improving the standard of clinical care in public mental health services, the use of evidence-based practice remains low. Among other challenges, this reflects the difficulty of translating clinical research into useable policy that can be feasibly funded and monitored by state or large healthcare systems.

Case presentation

In this paper we present a case study of Washington State’s strategy for monitoring the use of clinical elements at the session level for all Medicaid-funded children’s mental health services. The implementation of this strategy reflects policy actions to promote effective practice while also actively influencing multiple other levels of the implementation ecology. The approach is informed by the Policy Ecology Framework, the Consolidated Framework for Implementation Research, the evidence-based policymaking literature, and common ontology and clinical elements models.

Conclusions

We found the strategy developed in Washington State to be a feasible method of collecting session level information about the use of effective clinical mental health practices. In addition, the approach appears to be having influence on multiple layers of the implementation ecology that could be explored through further study.

Annual Research Review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field

BACKGROUND

Scientists have developed evidence-based interventions that improve the symptoms and functioning of youth with psychiatric disorders; however, these interventions are rarely used in community settings. Eliminating this research-to-practice gap is the purview of implementation science, the discipline devoted to the study of methods to promote the use of evidence-based practices in routine care.

METHODS

We review studies that have tested factors associated with implementation in child psychology and psychiatry, explore applications of social science theories to implementation, and conclude with recommendations to advance implementation science through the development and testing of novel, multilevel, causal theories.

RESULTS

During its brief history, implementation science in child psychology and psychiatry has documented the implementation gap in routine care, tested training approaches and found them to be insufficient for behavior change, explored the relationships between variables and implementation outcomes, and initiated randomized controlled trials to test implementation strategies. This research has identified targets related to implementation (e.g., clinician motivation, organizational culture) and demonstrated the feasibility of activating these targets through implementation strategies. However, the dominant methodological approach has been atheoretical and predictive, relying heavily on a set of variables from heuristic frameworks.

CONCLUSIONS

Optimizing the implementation of effective treatments in community care for youth with psychiatric disorders is a defining challenge of our time. This review proposes a new direction focused on developing and testing integrated causal theories. We recommend implementation scientists: (a) move from observational studies of implementation barriers and facilitators to trials that include causal theory; (b) identify a core set of implementation determinants; (c) conduct trials of implementation strategies with clear targets, mechanisms, and outcomes; (d) ensure that behaviors that are core to EBPs are clearly defined; and (e) agree upon standard measures. This agenda will help fulfill the promise of evidence-based practice for improving youth behavioral health.

Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools: a cross-sectional observational study

BACKGROUND

Children with autism receive most of their intervention services in public schools, but implementation of evidence-based practices (EBPs) for autism varies. Studies suggest that individual (attitudes) and organizational characteristics (implementation leadership and climate) may influence providers' use of EBPs, but research is relatively limited in this area. This study examined individual and organizational factors associated with implementation of three EBPs-discrete trial training, pivotal response training, and visual schedules-for children with autism in special education classrooms in public elementary schools.

METHODS

Participants included 67 autism support teachers and 85 other classroom staff from 52 public elementary schools in the northeastern United States. Participants reported their attitudes toward EBPs (e.g., intuitive appeal, willingness if required, openness, and divergence), implementation leadership and climate of their school, and the frequency with which they deliver each of three EBPs. Linear regression was used to estimate the association of attitudes about EBPs with organizational characteristics and intensity of EBP use. Demographic covariates with a bivariate association with EBP use significant at p < .20 were entered into the adjusted models.

RESULTS

There were significant findings for only one EBP, discrete trial training. Teachers who reported higher perceived divergence (perceived difference of usual practice with academically developed or research-based practices) between EBPs and current practices used less discrete trial training (f2 = .18), and teachers who reported higher appeal (willingness to adopt EBPs given their intuitive appeal) of EBPs used more discrete trial training (f2 = .22). No organizational factors were significantly associated with implementation with any of the three EBPs.

CONCLUSIONS

Attitudes toward EBPs may affect teachers' decisions to use EBPs; however, implementation leadership and climate did not predict EBP use. Future implementation efforts ought to consider the type of EBP and its fit within the context in terms of the EBP's similarities to and differences from existing practices and programs in the setting. Implementation strategies that target individual attitudes about EBPs may be warranted in public schools.

Training transfer: a systematic review of the impact of inner setting factors

Consistent with Baldwin and Ford's model (Pers Psychol 41(1):63-105, 1988), training transfer is defined as the generalization of learning from a training to everyday practice in the workplace. The purpose of this review was to examine the influence of work-environment factors, one component of the model hypothesized to influence training transfer within behavioral health. An electronic literature search guided by the Consolidated Framework for Implementation Research's inner setting domain was conducted was conducted on Medline OVID, Medline EMBASE, and PsycINFO databases. Of 9184 unique articles, 169 full-text versions of articles were screened for eligibility, yielding 26 articles meeting inclusion criteria. Results from the 26 studies revealed that overall, having more positive networks and communication, culture, implementation climate, and readiness for implementation can facilitate training transfer. Although few studies have examined the impact of inner setting factors on training transfer, these results suggest organizational context is important to consider with training efforts. These findings have important implications for individuals in the broader health professions educational field.

Measuring Youth Residential Care Provider Attitudes

Abstract. In this study the 15-item Evidence-Based Practices Attitude Scale (EBPAS), a measure designed to assess attitudes toward the adoption of EBPs, was collected from administrators of residential care facilities for youth in the United States. As the EBPAS was administered to a different sample for which it was originally developed (i.e., community mental health), we conducted a Confirmatory Factor Analysis (CFA) to investigate if its factor structure was maintained. Results confirm the factor structure of the EBPAS as a valid measure of attitude toward evidence-based practice among youth residential care providers. Limitations and areas of future research are discussed.

Measuring Constructs of the Consolidated Framework for Implementation Research in the Context of Increasing Colorectal Cancer Screening in Federally Qualified Health Center

OBJECTIVE

To operationalize constructs from each of the Consolidated Framework for Implementation Research domains and to present psychometric properties within the context of evidence-based approaches for promoting colorectal cancer screening in federally qualified health centers (FQHCs).

METHODS

Data were collected from FQHC clinics across seven states. A web-based Staff Survey and a Clinic Characteristics Survey were completed by staff and leaders (n = 277) from 59 FQHCs.

RESULTS

Internal reliability of scales was adequate ranging from 0.62 for compatibility to 0.88 for other personal attributes (openness). Intraclass correlations for the scales indicated that 2.4 percent to 20.9 percent of the variance in scale scores occurs within clinics. Discriminant validity was adequate at the clinic level, with all correlations less than 0.75. Convergent validity was more difficult to assess given lack of hypothesized associations between factors expected to predict implementation.

CONCLUSIONS

Our results move the field forward by describing initial psychometric properties of constructs across CFIR domains.

Linking molar organizational climate and strategic implementation climate to clinicians' use of evidence-based psychotherapy techniques: cross-sectional and lagged analyses from a 2-year observational study

Background

Behavioral health organizations are characterized by multiple organizational climates, including molar climate, which encompasses clinicians’ shared perceptions of how the work environment impacts their personal well-being, and strategic implementation climate, which includes clinicians’ shared perceptions of the extent to which evidence-based practice implementation is expected, supported, and rewarded by the organization. Theory suggests these climates have joint, cross-level effects on clinicians’ implementation of evidence-based practice and that these effects may be long term (i.e., up to 2 years); however, no empirical studies have tested these relationships. We hypothesize that molar climate moderates implementation climate’s concurrent and long-term relationships with clinicians’ use of evidence-based practice such that strategic implementation climate will have its most positive effects when it is accompanied by a positive molar climate.

Methods

Hypotheses were tested using data collected from 235 clinicians in 20 behavioral health organizations. At baseline, clinicians reported on molar climate and implementation climate. At baseline and at a 2-year follow-up, all clinicians who were present in the organizations reported on their use of cognitive-behavioral psychotherapy techniques, an evidence-based practice for youth psychiatric disorders. Two-level mixed-effects regression models tested whether baseline molar climate and implementation climate interacted in predicting clinicians’ evidence-based practice use at baseline and at 2-year follow-up.

Results

In organizations with more positive molar climates at baseline, higher levels of implementation climate predicted increased evidence-based practice use among clinicians who were present at baseline and among clinicians who were present in the organizations at 2-year follow-up; however, in organizations with less positive molar climates, implementation climate was not related to clinicians’ use of evidence-based practice at either time point.

Conclusions

Optimizing clinicians’ implementation of evidence-based practice in behavioral health requires attention to both molar climate and strategic implementation climate. Strategies that focus exclusively on implementation climate may not be effective levers for behavior change if the organization does not also engender a positive molar climate. These findings have implications for the development of implementation theory and effective implementation strategies.

How Intervention and Implementation Characteristics Relate to Community Therapists' Attitudes Toward Evidence-Based Practices: A Mixed Methods Study

Therapists' perceptions toward evidence-based practices (EBPs) are important in implementation efforts, however little is known about characteristics of EBPs associated with more positive attitudes. This mixed-methods study examined how intervention and implementation characteristics of six EBPs related to therapist attitudes. Quantitative analysis of 793 cross-sectional surveys revealed that therapists endorsed more positive attitudes toward EBPs with (1) prescribed session content and order and (2) required consultation. Associations between these intervention and implementation characteristics and attitudes were not moderated by therapist experience or emotional exhaustion. Qualitative analyses complemented quantitative findings, indicating that "structure" was appealing for interventions and that therapists felt supported by consultation.

Navigating the development and dissemination of internet cognitive behavioral therapy (iCBT) for anxiety disorders in children and young people: A consensus statement with recommendations from the #iCBTLorentz Workshop Group

Initial internet-based cognitive behavioral therapy (iCBT) programs for anxiety disorders in children and young people (CYP) have been developed and evaluated, however these have not yet been widely adopted in routine practice. The lack of guidance and formalized approaches to the development and dissemination of iCBT has arguably contributed to the difficulty in developing iCBT that is scalable and sustainable beyond academic evaluation and that can ultimately be adopted by healthcare providers. This paper presents a consensus statement and recommendations from a workshop of international experts in CYP anxiety and iCBT (#iCBTLorentz Workshop Group) on the development, evaluation, engagement and dissemination of iCBT for anxiety in CYP.

Effectiveness of a multi-level implementation strategy for ASD interventions: study protocol for two linked cluster randomized trials

BACKGROUND

The Centers for Disease Control (2018) estimates that 1 in 59 children has autism spectrum disorder, and the annual cost of ASD in the U.S. is estimated to be $236 billion. Evidence-based interventions have been developed and demonstrate effectiveness in improving child outcomes. However, research on generalizable methods to scale up these practices in the multiple service systems caring for these children has been limited and is critical to meet this growing public health need. This project includes two, coordinated studies testing the effectiveness of the Translating Evidence-based Interventions (EBI) for ASD: Multi-Level Implementation Strategy (TEAMS) model. TEAMS focuses on improving implementation leadership, organizational climate, and provider attitudes and motivation in order to improve two key implementation outcomes-provider training completion and intervention fidelity and subsequent child outcomes. The TEAMS Leadership Institute applies implementation leadership strategies and TEAMS Individualized Provider Strategies for training applies motivational interviewing strategies to facilitate provider and organizational behavior change.

METHODS

A cluster randomized implementation/effectiveness Hybrid, type 3, trial with a dismantling design will be used to understand the effectiveness of TEAMS and the mechanisms of change across settings and participants. Study #1 will test the TEAMS model with AIM HI (An Individualized Mental Health Intervention for ASD) in publicly funded mental health services. Study #2 will test TEAMS with CPRT (Classroom Pivotal Response Teaching) in education settings. Thirty-seven mental health programs and 37 school districts will be randomized, stratified by county and study, to one of four groups (Standard Provider Training Only, Standard Provider Training + Leader Training, Enhanced Provider Training, Enhanced Provider Training + Leader Training) to test the effectiveness of combining standard, EBI-specific training with the two TEAMS modules individually and together on multiple implementation outcomes. Implementation outcomes including provider training completion, fidelity (coded by observers blind to group assignment) and child behavior change will be examined for 295 mental health providers, 295 teachers, and 590 children.

DISCUSSION

This implementation intervention has the potential to increase quality of care for ASD in publicly funded settings by improving effectiveness of intervention implementation. The process and modules will be generalizable to multiple service systems, providers, and interventions, providing broad impact in community services.

TRIAL REGISTRATION

This study is registered with Clinicaltrials.gov ( NCT03380078 ). Registered 20 December 2017, retrospectively registered.

Community-Sourced Intervention Programs: Review of Submissions in Response to a Statewide Call for "Promising Practices"

This study was initiated to add to the nascent literature on locally-grown intervention programs in the youth mental health, child welfare, and juvenile justice service sectors, many of which demonstrate practice-based or community-defined evidence, but may not have been subjected to empirical evaluation. Characteristics of applications submitted in response to three public calls for additions to an inventory of research-supported intervention programs were reviewed on evidence for effectiveness, the use of key quality assurance (QA) elements (e.g., clearly specified training or integrity monitoring procedures), and cultural specificity. Findings indicate that four QA processes were identified in approximately half of all submissions: a specific initial training process, the existence of intervention integrity measures, routine outcome monitoring, and ongoing support post-training. An initial training process and integrity measurement were more commonly described among programs determined to have greater research evidence for their effectiveness. Overall, cultural elements were described relatively infrequently and most often reflected surface-level program delivery characteristics (e.g., offering services in languages other than English). Discussion is focused on the alignment of submitted programs with the larger literatures focused on implementation science and cultural competence.

The Consistencies and Vagaries of the Washington State Inventory of Evidence-Based Practice: The Definition of "Evidence-Based" in a Policy Context

As states increasingly establish the importance of evidence-based practice through policy and funding mandates, the definition of evidence-based practice can have a significant impact on investment decisions. Not meeting established criteria can mean a loss of funding for established programs and the implementation disruption of programs without a strong research base. Whether the definition of "evidence-based" is influenced by these high stakes contexts is an interesting question that can inform the larger field about the value and utility of evidence-based practice lists/inventories for disseminating knowledge. In this paper we review the development of the Washington State Inventory of Evidence-Based, Research-Based and Promising Practices as a case study for the process of defining evidence-based practice in a policy context. As part of this study we also present a comparison of other well-known evidence-based practice inventories and examine consistencies and differences in the process of identifying and developing program ratings.

The impact of inter-organizational alignment (IOA) on implementation outcomes: evaluating unique and shared organizational influences in education sector mental health

BACKGROUND

Integrated healthcare delivered by work groups in nontraditional service settings is increasingly common, yet contemporary implementation frameworks typically assume a single organization-or organizational unit-within which system-level processes influence service quality and implementation success. Recent implementation frameworks predict that inter-organizational alignment (i.e., similarity in values, characteristics, activities related to implementation across organizations) may facilitate the implementation of evidence-based practices (EBP), but few studies have evaluated this premise. This study's aims examine the impact of overlapping organizational contexts by evaluating the implementation contexts of externally employed mental health clinicians working in schools-the most common integrated service delivery setting for children and adolescents. Aim 1 is to estimate the effects of unique intra-organizational implementation contexts and combined inter-organizational alignment on implementation outcomes. Aim 2 is to examine the underlying mechanisms through which inter-organizational alignment facilitates or hinders EBP implementation.

METHODS/DESIGN

This study will conduct sequential, exploratory mixed-methods research to evaluate the intra- and inter-organizational implementation contexts of schools and the external community-based organizations that most often employ school-based mental health clinicians, as they relate to mental health EBP implementation. Aim 1 will involve quantitative surveys with school-based, externally-employed mental health clinicians, their supervisors, and proximal school-employed staff (total n = 120 participants) to estimate the effects of each organization's general and implementation-specific organizational factors (e.g., climate, leadership) on implementation outcomes (fidelity, acceptability, appropriateness) and assess the moderating role of the degree of clinician embeddedness in the school setting. Aim 2 will explore the mechanisms through which inter-organizational alignment influences implementation outcomes by presenting the results of Aim 1 surveys to school-based clinicians (n = 30) and conducting semi-structured qualitative interviews. Qualitative data will be evaluated using an integrative inductive and deductive approach.

DISCUSSION

The study aims are expected to identify intra- and inter-organizational constructs that are most instrumental to EBP implementation success in school-based integrated care settings and illuminate mechanisms that may account for the influence of inter-organizational alignment. In addition to improving school-based mental health, these findings will spur future implementation science that considers the relationships across organizations and optimize the capacity of implementation science to guide practice in increasingly complex systems of care.

Training Community-Based Clinicians in Parent-Child Interaction Therapy: The Interaction Between Expert Consultation and Caseload

Professional psychologists are increasingly encouraged to utilize evidence-based treatments (EBTs), and therefore have a need to participate and provide the most efficient training methods for these treatments. Multicomponent trainings, which commonly include ongoing support, are more effective than brief methods such as 1-day workshops or reading treatment manuals. The present study examined the effectiveness of 1 form of ongoing support, consultation, as part of a multicomponent training protocol. Thirty-two community-based clinicians were trained in Parent-Child Interaction Therapy (PCIT) as part of a statewide implementation effort, and data were collected on clinician and implementation outcomes at pre-, mid-, and posttraining. Simple and multiple linear regression analyses were conducted to predict posttraining knowledge, skill, acceptability, and feasibility, as well as to examine clinician variables that might moderate these relations. Greater consultation call attendance significantly predicted higher posttraining skill; however, this association was qualified by a significant interaction with PCIT caseload. Implications for training guidelines are discussed.