Supporting implementation: the role of community development teams to build infrastructure

Scale-up of Global Child and Youth Mental Health Services: A Scoping Review

Numerous influential policy and scientific bodies are calling for more rapid advances in the scale-up of child and youth mental health services (CYMHS). A number of CYMHS innovations hold promise for advancing scale-up but little is known about how real-world efforts are progressing. We conducted a scoping review to identify promising approaches to CYMHS scale-up across the globe. Searches were completed in six databases (Academic Search Complete, CINAHL, MEDLINE, PsychInfo, PubMed, and Web of Science). Article selection and synthesis were conducted in accordance to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR) checklist. A second search focused on low-and-middle-income countries (LMIC) was conducted based on the Cochrane Library recommended search filters of the World Bank listed LMIC countries. Authors used a double coding strategy during the title/abstract and full-text review. Twenty-eight articles meeting the eligibility criteria were identified that described 22 initiatives (in 11 different countries). Our review found the majority of published scale-up studies in CYMHS were not informed by scale-up frameworks in design or reporting. The methods and outcomes used in the identified articles were highly variable and limited our ability to draw conclusions about comparative effectiveness although promising approaches emerged. Successes and failures identified in our review largely reflect consensus in the broader literature regarding the need for strategies to better navigate the complexities of system and policy implementation while ensuring CYMHS interventions fit local contexts.

Implementing a pharmacist-integrated collaborative model of medication treatment for opioid use disorder in primary care: study design and methodological considerations

BACKGROUND

Pharmacists remain an underutilized resource in the treatment of opioid use disorder (OUD). Although studies have engaged pharmacists in dispensing medications for OUD (MOUD), few studies have evaluated collaborative care models in which pharmacists are an active, integrated part of a primary care team offering OUD care.

METHODS

This study seeks to implement a pharmacist integrated MOUD clinical model (called PrIMO) and evaluate its feasibility, acceptability, and impact across four diverse primary care sites. The Consolidated Framework for Implementation Research is used as an organizing framework for study development and interpretation of findings. Implementation Facilitation is used to support PrIMO adoption. We assess the primary outcome, the feasibility of implementing PrIMO, using the Stages of Implementation Completion (SIC). We evaluate the acceptability and impact of the PrIMO model at the sites using mixed-methods and combine survey and interview data from providers, pharmacists, pharmacy technicians, administrators, and patients receiving MOUD at the primary care sites with patient electronic health record data. We hypothesize that it is feasible to launch delivery of the PrIMO model (reach SIC Stage 6), and that it is acceptable, will positively impact patient outcomes 1 year post model launch (e.g., increased MOUD treatment retention, medication regimen adherence, service utilization for co-morbid conditions, and decreased substance use), and will increase each site's capacity to care for patients with MOUD (e.g., increased number of patients, number of prescribers, and rate of patients per prescriber).

DISCUSSION

This study will provide data on a pharmacist-integrated collaborative model of care for the treatment of OUD that may be feasible, acceptable to both site staff and patients and may favorably impact patients' access to MOUD and treatment outcomes.

TRIAL REGISTRATION

The study was registered on Clinicaltrials.gov (NCT05310786) on April 5, 2022, https://www.

CLINICALTRIALS

gov/study/NCT05310786?id=NCT05310786&rank=1.

Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Background

Dissemination and implementation frameworks provide the scaffolding to explore the effectiveness of evidence-based practices (EBPs) targeting process of care and organizational outcomes. Few instruments, like the stages of implementation completion (SIC) examine implementation fidelity to EBP adoption and how organizations differ in their approach to implementation. Instruments to measure organizational competency in the utilization of implementation strategies are lacking.

Method

An iterative process was utilized to adapt the SIC to the NIATx implementation strategies. The new instrument, NIATx-SIC, was applied in a randomized controlled trial involving 53 addiction treatment agencies in Washington state to improve agency co-occurring capacity. NIATx-SIC data were reported by state staff and external facilitators and through participating agency documentation. Proportion and duration scores for each stage and phase of the NIATx-SIC were calculated for each agency. Competency was assessed using the NIATx fidelity tool. Comparisons of proportion, duration, and NIATx activities completed were determined using independent sample t-tests by agency competency level.

Results

The NIATx-SIC distinguished between agencies achieving competency (n  =  23) and those not achieving competency (n  =  26). Agencies achieving competency completed a greater proportion of implementation phase activities and had a significantly longer Stage 7 duration. These agencies participated in significantly more individual and group coaching calls, attended more in-person meetings, implemented more change projects, and spent approximately 64 more days, on average, engaging in all NIATx activities.

Conclusions

Organizational participation in dissemination and implementation research requires a significant investment of staff resources. The inability of an organization to achieve competency when utilizing a set of implementation strategies waste an opportunity to institutionalize knowledge of how to apply implementation strategies to future change efforts. The NIATx-SIC provides evidence that competency is not an attribute of the organization but rather a result of the application of the NIATx implementation strategies to improve agency co-occurring capacity.

Trial Registration

ClinicalTrials.gov, NCT03007940. Registered January 2, 2017, https://clinicaltrials.gov/ct2/show/NCT03007940.

Accounting for Context in Randomized Trials after Assignment

Many preventive trials randomize individuals to intervention condition which is then delivered in a group setting. Other trials randomize higher levels, say organizations, and then use learning collaboratives comprised of multiple organizations to support improved implementation or sustainment. Other trials randomize or expand existing social networks and use key opinion leaders to deliver interventions through these networks. We use the term contextually driven to refer generally to such trials (traditionally referred to as clustering, where groups are formed either pre-randomization or post-randomization - i.e., a cluster-randomized trial), as these groupings or networks provide fixed or time-varying contexts that matter both theoretically and practically in the delivery of interventions. While such contextually driven trials can provide efficient and effective ways to deliver and evaluate prevention programs, they all require analytical procedures that take appropriate account of non-independence, something not always appreciated. Published analyses of many prevention trials have failed to take this into account. We discuss different types of contextually driven designs and then show that even small amounts of non-independence can inflate actual Type I error rates. This inflation leads to rejecting the null hypotheses too often, and erroneously leading us to conclude that there are significant differences between interventions when they do not exist. We describe a procedure to account for non-independence in the important case of a two-arm trial that randomizes units of individuals or organizations in both arms and then provides the active treatment in one arm through groups formed after assignment. We provide sample code in multiple programming languages to guide the analyst, distinguish diverse contextually driven designs, and summarize implications for multiple audiences.

Stepped implementation-to-target: a study protocol of an adaptive trial to expand access to addiction medications

BACKGROUND

In response to the US opioid epidemic, significant national campaigns have been launched to expand access to `opioid use disorder (MOUD). While adoption has increased in general medical care settings, specialty addiction programs have lagged in both reach and adoption. Elevating the quality of implementation strategy, research requires more precise methods in tailoring strategies rather than a one-size-fits-all-approach, documenting participant engagement and fidelity to the delivery of the strategy, and conducting an economic analysis to inform decision making and policy. Research has yet to incorporate all three of these recommendations to address the challenges of implementing and sustaining MOUD in specialty addiction programs.

METHODS

This project seeks to recruit 72 specialty addiction programs in partnership with the Washington State Health Care Authority and employs a measurement-based stepped implementation-to-target approach within an adaptive trial design. Programs will be exposed to a sequence of implementation strategies of increasing intensity and cost: (1) enhanced monitoring and feedback (EMF), (2) 2-day workshop, and then, if outcome targets are not achieved, randomization to either internal facilitation or external facilitation. The study has three aims: (1) evaluate the sequential impact of implementation strategies on target outcomes, (2) examine contextual moderators and mediators of outcomes in response to the strategies, and (3) document and model costs per implementation strategy. Target outcomes are organized by the RE-AIM framework and the Addiction Care Cascade.

DISCUSSION

This implementation project includes elements of a sequential multiple assignment randomized trial (SMART) design and a criterion-based design. An innovative and efficient approach, participating programs only receive the implementation strategies they need to achieve target outcomes. Findings have the potential to inform implementation research and provide key decision-makers with evidence on how to address the opioid epidemic at a systems level.

TRIAL REGISTRATION

This trial was registered at ClinicalTrials.gov (NCT05343793) on April 25, 2022.

Knowledge and attitudes of Implementation Support Practitioners-Findings from a systematic integrative review

BACKGROUND

It requires thoughtful planning and work to successfully apply and sustain research-supported interventions like healthcare treatments, social support, or preventive programs in practice. Implementation support practitioners (ISPs) such as facilitators, technical assistance providers, knowledge brokers, coaches or consultants may be involved to actively support the implementation process. This article presents knowledge and attitudes ISPs bring to their work.

METHODS

Building on a previously developed program logic, a systematic integrative review was conducted. Literature was sourced by searching nine electronic data bases, organizational websites, and by launching a call for publications among selected experts and social media. Article screening was performed independently by two researchers, and data from included studies were extracted by members of the research team and quality-assured by the lead researcher. The quality of included RCTs was assessed based on a framework by Hodder and colleagues. Thematic Analysis was used to capture information on knowledge and attitudes of ISPs across the included studies. Euler diagrams and heatmaps were used to present the results.

RESULTS

Results are based on 79 included studies. ISPs reportedly displayed knowledge about the clinical practice they work with, implementation / improvement practice, the local context, supporting change processes, and facilitating evidence-based practice in general. In particular, knowledge about the intervention to be implemented and its target population, specific improvement / implementation methods and approaches, organizational structures and sensitivities, training, and characteristics of (good) research was described in the literature. Seven themes describing ISPs' attitudes were identified: 1) professional, 2) motivated / motivating / encouraging / empowering, 3) empathetic / respectful / sensitive, 4) collaborative / inclusive, 5) authentic, 6) creative / flexible / innovative / adaptive, and 7) frank / direct / honest. Pertaining to a professional attitude, being responsive and focused were the most prevalent indicators across included publications.

CONCLUSION

The wide range and complexity of knowledge and attitudes found in the literature calls for a comprehensive and systematic approach to collaboratively develop a professional role for ISPs across disciplines. Embedding the ISP role in different health and social welfare settings will enhance implementation capacities considerably.

Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers

Background

Economic evaluations frequently are utilized to compare the value of different interventions in medicine and health in concrete terms. Implementation science also would benefit from the incorporation of economic evaluations, but such studies are rare in the literature. The National Cancer Institute has supported a special collection of articles focusing on economic evaluations in implementation science. Even when interventions are supported by substantial evidence, they are implemented infrequently in the field. Implementation costs are important determinants for whether organizational decision-makers choose to adopt an intervention and whether the implementation process is successful. Economic evaluations, such as cost-effectiveness analyses, can help organizational decision-makers choose between implementation approaches for evidence-based interventions by accounting for costs and succinctly presenting cost/benefit tradeoffs.

Main text

This manuscript presents a discussion of important considerations for incorporating economic evaluations into implementation science. First, the distinction between intervention and implementation costs is presented, along with an explanation of why the comprehensive representation of implementation costs is elusive. Then, the manuscript describes how economic evaluations in implementation science may differ from those in medicine and health intervention studies, especially in terms of determining the perspectives and outcomes of interest. Finally, referencing a scale-up trial of an evidence-based behavioral health intervention, concrete case examples of how cost data can be collected and used in economic evaluations targeting implementation, rather than clinical outcomes, are described.

Conclusions

By gaining a greater understanding of the costs and economic impact associated with different implementation approaches, organizational decision-makers will have better transparency for future replication and scale-up. The use of economic evaluations can help to advance this understanding and provide researchers, purveyors or third-party intermediaries, and organizational decision-makers with essential information to facilitate implementation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00295-1.

Sustainment of Integrated Care in Addiction Treatment Settings: Primary Outcomes From a Cluster-Randomized Controlled Trial

OBJECTIVE

Integrated treatment services are the gold standard for addressing co-occurring mental and substance use disorders, yet they are not readily available. The Network for the Improvement of Addiction Treatment (NIATx) was hypothesized to be an effective strategy to implement and sustain integrated mental health and substance use care in addiction treatment programs. This study examined sustainment of integrated services for up to 2 years after the active implementation phase.

METHODS

The effectiveness of NIATx strategies to implement and sustain integrated services was evaluated by using a cluster-randomized, waitlist control group design. Forty-nine addiction treatment organizations were randomly assigned to either NIATx1 (active implementation strategy) or NIATx2 (waitlist control). The Dual Diagnosis Capability in Addiction Treatment Index was used to evaluate organizations' capability to provide integrated care. The NIATx Stages of Implementation Completion scale was used to assess participation in and adherence to the NIATx implementation process. Linear mixed-effects modeling was used to evaluate changes from baseline to end of the sustainment period.

RESULTS

Both cohorts sustained their capability to provide integrated treatment services. Both groups achieved successful implementation and sustained integrated services to a similar degree, regardless of sustainment year. Sustainment did not vary as a function of NIATx adherence.

CONCLUSIONS

The delivery of integrated treatment services was sustained for 2 years after receipt of active implementation support. Future research should consider how contextual factors may predict, mediate, and moderate sustainment outcomes.

Implementation of the Wolverine Mental Health Program, Part 1: Adoption Phase

Residential treatment facilities (RTFs) are a first-line treatment option for juvenile justice-involved youth. However, RTFs rarely offer evidence-based interventions for youth with internalizing or externalizing mental health problems. Wolverine Human Services (WHS) is one of the first RTFs in the nation to implement cognitive-behavioral therapy (CBT) to enhance mental health care for their youth. This study outlines the preimplementation phase of a 5-year collaborative CBT implementation effort among WHS, the Beck Institute, and an implementation science research team. The preimplementation phase included a needs assessment across two sites of WHS to identify and prioritize barriers to CBT implementation. Of the 76 unique barriers, 23 were prioritized as important and feasible to address. Implementation teams, consisting of clinician and staff champions and opinion leaders, worked across 8 months to deploy 10 strategies from a collaboratively designed blueprint. Upon reevaluation of the needs assessment domains, all prioritized barriers to CBT implementation were removed and WHS's readiness for CBT implementation was enhanced. This study serves as a model of a preimplementation process that can be employed to enhance the potential for successful evidence-based practice implementation in youth RTFs.

Improving Medication Access within Integrated Treatment for Individuals with Co-Occurring Disorders in Substance Use Treatment Agencies

BACKGROUND

The best approach to provide comprehensive care for individuals with co-occurring disorders (CODs) related to substance use and mental health is to address both disorders through an integrated treatment approach. However, only 25% of behavioral health agencies offer integrated care and less than 7% of individuals who need integrated treatment receive it. A project used a cluster-randomized waitlist control group design to evaluate the effectiveness of Network for the Improvement of Addiction Treatment (NIATx) implementation strategies to improve access to addiction and psychotropic medications.

METHODS

This study represents a secondary analysis of data from the NIATx project. Forty-nine agencies were randomized to Cohort1 (active implementation group, receiving the NIATx strategy [n=25]) or Cohort2 (waitlist control group [n=24]). Data were collected at three time points (Baseline, Year1 and Year2). A two-level (patient within agency) multinomial logistic regression model investigated the effects of implementation strategy condition on one of four medication outcomes: both medication types, only psychotropic medication, only addiction medication, or neither medication type. A per-protocol analysis included time, NIATx fidelity, and agency focus as predictors.

RESULTS

The intent-to-treat analysis found a statistically significant change in access to addiction versus neither medication, but Cohort1 compared to Cohort2 at Year1 showed no differences. Changes were associated with the experimental intervention and occurred in the transition from Year 1 to Year 2, where greater increases were seen for agencies in Cohort2 versus Cohort1. The per-protocol analysis showed increased access to both medications and addiction medications from pre- to post-intervention for agencies in both cohorts; however, differences in change between high- and low-implementation agencies were not significant.

CONCLUSIONS

Access to integrated services for people with CODs is a long-standing problem. NIATx implementation strategies had limited effectiveness in improving medication access for individuals with CODs. Implementation strategy adherence is associated with increased medication access.

Establishing cross-systems collaborations for implementation: protocol for a longitudinal mixed methods study

Background

Cross-system interventions can help integrate services across different service delivery systems but require organizations to establish strong collaborative relationships for implementation. Contingency theory suggests that the effectiveness of different collaborative strategies (i.e. specific ways organizations align operations and services) varies by context. This paper describes a study of different strategies for fostering collaboration between child welfare and substance abuse treatment agencies and the conditions under which they are effective for implementation. We also describe the development and piloting of the Collaborating Across Systems for Program Implementation (CASPI) tool—a decision-making guide intended to help researchers and organizational leaders identify and use appropriate collaborative strategies for their context.

Methods/design

This multisite longitudinal, mixed methods study, leverages a naturally occurring implementation initiative -- in up to 17 Ohio counties -- to implement Ohio START (Sobriety Treatment and Reducing Trauma). START is a child welfare model that requires strong collaboration with local substance use treatment organizations to promote integrated services. During the first two years, we will identify collaborative strategies associated with improved START implementation (penetration and fidelity) and service delivery outcomes (timeliness), given system, and organizational features. We will conduct a convergent mixed methods study drawing on worker surveys, agency documents, administrative data, formal partner agreements, and group interviews. Data will be integrated and analyzed using Qualitative Comparative Analysis (QCA). To develop the CASPI, an expert panel comprised of implementation experts, and community stakeholders will convene to synthesize our findings and develop contents (including a decision tree). During the final year of the study, we will assess the acceptability, appropriateness, and feasibility of the CASPI in a randomized vignette experiment, and a pilot-test with 3 child welfare agencies that have not yet implemented START.

Discussion

Our results will lay the groundwork for a larger controlled trial that will test the CASPI’s effectiveness for supporting effective and efficient implementation of cross-system interventions like START. The CASPI is expected to help leaders and researchers select and use collaboration strategies tailored to their context and be applicable in a wide range of settings including rural communities. Our work also advances system-level implementation strategies.

Trial registration

NCT03931005, Registered April 29, 2019.

The value of implementation frameworks: Using the active implementation frameworks to guide system-wide implementation of Collaborative Problem Solving

In the last decade, many implementation frameworks have emerged that consolidate the research on implementation science, guiding purveyors and service agencies in improving implementation of evidence-based practices (EBPs). In this paper, we describe how the purveyor of one EBP utilized the active implementation frameworks (AIFs) to define and standardize strategies for site-wide implementation. We illustrate what implementation looked like before and after using AIFs to understand implementation, as well as some ways in which using the AIFs helped the purveyor identify, and then overcome, barriers to implementation. This paper provides a model for others who seek to use AIFs to guide their implementation practices, or more broadly, an illustration of how to use any implementation framework to ensure best practices in implementation.

Advancing successful implementation of task-shifted mental health care in low-resource settings (BASIC): protocol for a stepped wedge cluster randomized trial

BACKGROUND

The mental health treatment gap-the difference between those with mental health need and those who receive treatment-is high in low- and middle-income countries. Task-shifting has been used to address the shortage of mental health professionals, with a growing body of research demonstrating the effectiveness of mental health interventions delivered through task-shifting. However, very little research has focused on how to embed, support, and sustain task-shifting in government-funded systems with potential for scale up. The goal of the Building and Sustaining Interventions for Children (BASIC) study is to examine implementation policies and practices that predict adoption, fidelity, and sustainment of a mental health intervention in the education sector via teacher delivery and the health sector via community health volunteer delivery.

METHODS

BASIC is a Hybrid Type II Implementation-Effectiveness trial. The study design is a stepped wedge, cluster randomized trial involving 7 sequences of 40 schools and 40 communities surrounding the schools. Enrollment consists of 120 teachers, 120 community health volunteers, up to 80 site leaders, and up to 1280 youth and one of their primary guardians. The evidence-based mental health intervention is a locally adapted version of Trauma-focused Cognitive Behavioral Therapy, called Pamoja Tunaweza. Lay counselors are trained and supervised in Pamoja Tunaweza by local trainers who are experienced in delivering the intervention and who participated in a Train-the-Trainer model of skills transfer. After the first sequence completes implementation, in-depth interviews are conducted with initial implementing sites' counselors and leaders. Findings are used to inform delivery of implementation facilitation for subsequent sequences' sites. We use a mixed methods approach including qualitative comparative analysis to identify necessary and sufficient implementation policies and practices that predict 3 implementation outcomes of interest: adoption, fidelity, and sustainment. We also examine child mental health outcomes and cost of the intervention in both the education and health sectors.

DISCUSSION

The BASIC study will provide knowledge about how implementation of task-shifted mental health care can be supported in government systems that already serve children and adolescents. Knowledge about implementation policies and practices from BASIC can advance the science of implementation in low-resource contexts.

TRIAL REGISTRATION

Trial Registration: ClinicalTrials.gov Identifier: NCT03243396. Registered 9th August 2017, https://clinicaltrials.gov/ct2/show/NCT03243396.

Change in Collaborative Ties in a Children's Mental Health Services Network: A Clique Perspective

Effective mental health service delivery networks are organized around clusters of closely coordinated provider organizations, or organizational cliques. Yet, research overlooks conditions that influence clique change and implications for aligning services. This study examines changes in organizational cliques within a regional children's behavioral health service delivery network over a two-year period characterized by substantial funding increases. Results from quantitative network survey data, and administrative data gathered from 22 non-profit organizations show that cliques expanded and overlapped two years after a funding influx. Cliques aligning organizations with more distinct services experienced greater tie churn, and may be vulnerable to change.

Is the Concept of Self-Regulation Useful for Supporting Effective Implementation in Community Settings?

The literature and utility of self-regulation extends beyond individuals; a critical factor for successful and sustainable implementation of evidence-based programs in a community setting may be the capacity of teams to self-regulate implementation processes. The conceptual foundation of this proposal is explored and definitions of the five dimensions of self-regulation for implementation processes are provided. Practice examples illustrate how the provision of external implementation support to build self-regulatory capacity among implementation teams adopting and scaling-up EBPs in the local community setting has shaped and refined the proposed definitions to better reflect the work on-the-ground. The role of external implementation support providers in developing implementation team self-regulation is explored and practice strategies to promote self-regulation are provided. Implications and directions for future research are discussed.

Implementation capital: merging frameworks of implementation outcomes and social capital to support the use of evidence-based practices

BACKGROUND

Although there is growing recognition that the implementation of evidence-based practices is a social process, the conceptualization of social capital in implementation frameworks often conflates bonding and bridging social capital. This conflation makes it difficult to concretely operationalize social capital and limits the concept's utility for explaining implementation outcomes.

DISCUSSION

We propose a new framework of implementation capital that merges an existing conceptual framework of implementation outcomes with an existing operational framework of social capital. First, we review a conceptual framework of implementation outcomes, which includes the acceptability, appropriateness, adoption, feasibility, fidelity, cost, penetration, and sustainability of evidence-based practices. Second, we describe an operational framework of social capital that grounds bonding and bridging social capital in the structure of implementers' social networks. Third, we bring these two frameworks together to create a merged framework of implementation capital that shows how specific aspects of social capital can support specific implementation outcomes. Implementation outcomes of acceptability, appropriateness, and adoption are linked to bonding social capital through mechanisms of trust and norm enforcement, while outcomes of feasibility and fidelity are linked to bridging social capital through mechanisms of increased access to information and resources. Additionally, setting-level implementation outcomes of cost, penetration, and sustainability are associated with small worldliness at the setting level, which simultaneously optimizes both bonding and bridging social capital in a setting.

CONCLUSION

The implementation capital framework is helpful because it separates two distinct forms of social capital-bonding and bridging-that are often conflated in the implementation literature, and offers concrete ways to operationalize them by examining the structure of implementers' social networks and the networks of their settings. This framework offers specific guidance about how individual and setting networks might be shifted to support implementation outcomes.

A Mixed Methods Study of the Stages of Implementation for an Evidence-Based Trauma Intervention in Schools

A mixed methods study was conducted to examine the implementation process of 26 urban school-based mental health clinics that took part in a training and implementation support program for an evidence-based school trauma intervention. Implementation process was observed using the Stages of Implementation Completion (SIC) measure. Qualitative interviews were conducted with clinic leaders in order to gain insight into clinic processes related to the SIC. Results showed that almost all of the clinics engaged in some activities related to pre-implementation (engagement, feasibility, and readiness), but only 31% of the sites formally started delivering the program to youth. Completing more pre-implementation activities, particularly those related to readiness, predicted program start-up. Qualitative analysis comparing those that implemented the program to those that did not revealed critical differences in decision-making processes, leadership strategies, and the presence of local champions for the program. This study documented the patterns of clinic behavior that occurs as part of large-scale training efforts, suggests some unique challenges that occur in schools, and highlights the importance of engaging in particular implementation activities (i.e., readiness planning, stakeholder consensus and planning meetings) as part of program start-up. Findings indicate that pre-implementation and readiness-related consultation should be employed as part of broad-scale implementation and training efforts.

Using Goal Achievement Training in juvenile justice settings to improve substance use services for youth on community supervision

BACKGROUND

The link between substance use and involvement in the juvenile justice system has been well established. Justice-involved youth tend to have higher rates of drug use than their non-offending peers. At the same time, continued use can contribute to an elevated risk of recidivism, which leads to further, and oftentimes more serious, involvement with the juvenile justice system. Because of these high rates of use, the juvenile justice system is well positioned to help identify youth with substance use problems and connect them to treatment. However, research has found that only about 60% of juvenile probation agencies screen all youth for substance involvement, and even fewer provide comprehensive assessment or help youth enroll in substance use treatment.

METHOD

This paper describes an integrated training curriculum that was developed to help juvenile justice agencies improve their continuum of care for youth probationers with substance use problems. Goal Achievement Training (GAT) provides a platform for continuous quality improvement via two sessions delivered onsite to small groups of staff from juvenile justice and behavioral health agencies. In the first session, participants are taught to identify goals and goal steps for addressing identified areas of unmet need (i.e., screening, assessment, and linkage to treatment services). In the second session, participants learn principles and strategies of data-driven decision-making for achieving these goals. This paper highlights GAT as a model for the effective implementation of cost-efficient training strategies designed to increase self-directed quality improvement activities that can be applied to any performance domain within juvenile justice settings. Efforts to monitor implementation fidelity of GAT within the specific context of the juvenile justice settings are highlighted.

DISCUSSION

Challenges to setting the stage for process improvement generally, as well as specific hurdles within juvenile justice settings are discussed, as are next steps in disseminating findings regarding the fidelity to and effectiveness of GAT in this unique context.

TRIAL REGISTRATION

Clinical Trials Registration number - NCT02672150 .

Alpha test results for a Housing First eLearning strategy: the value of multiple qualitative methods for intervention design

Background

Detailed descriptions of implementation strategies are lacking, and there is a corresponding dearth of information regarding methods employed in implementation strategy development. This paper describes methods and findings related to the alpha testing of eLearning modules developed as part of the Housing First Technical Assistance and Training (HFTAT) program's development. Alpha testing is an approach for improving the quality of a product prior to beta (i.e., real world) testing with potential applications for intervention development.

Methods

Ten participants in two cities tested the modules. We collected data through (1) a structured log where participants were asked to record their experiences as they worked through the modules; (2) a brief online questionnaire delivered at the end of each module; and (3) focus groups.

Results

The alpha test provided useful data related to the acceptability and feasibility of eLearning as an implementation strategy, as well as identifying a number of technical issues and bugs. Each of the qualitative methods used provided unique and valuable information. In particular, logs were the most useful for identifying technical issues, and focus groups provided high quality data regarding how the intervention could best be used as an implementation strategy.

Conclusions

Alpha testing was a valuable step in intervention development, providing us an understanding of issues that would have been more difficult to address at a later stage of the study. As a result, we were able to improve the modules prior to pilot testing of the entire HFTAT. Researchers wishing to alpha test interventions prior to piloting should balance the unique benefits of different data collection approaches with the need to minimize burdens for themselves and participants.

Use of Research Evidence and Implementation of Evidence-Based Practices in Youth-Serving Systems

Although the effectiveness of interventions for prevention and treatment of mental health and behavioral problems in abused and neglected youth is demonstrated through the accumulation of evidence through rigorous and systematic research, it is uncertain whether use of research evidence (URE) by child-serving systems leaders increases the likelihood of evidence- based practice (EBP) implementation and sustainment. Information on URE was collected from 151 directors and senior administrators of child welfare, mental health and juvenile justice systems in 40 California and 11 Ohio counties participating in an RCT of the use of community development teams (CDTs) to scale up implementation of Treatment Foster Care Oregon over a 3 year period (2010-12). Separate multivariate models were used to assess independent effects of evidence acquisition (input), evaluation (process), application (output), and URE in general (SIEU Total) on two measures of EBP implementation, highest stage reached and proportion of activities completed at pre-implementation, implementation and sustainment phases. Stage of implementation and proportion of activities completed in the implementation and sustainment phases were independently associated with acquisition of evidence and URE in general. Participation in CDTs was significantly associated with URE in general and acquisition of research evidence in particular. Implementation of EBPs for treatment of abused and neglected youth does appear to be associated with use of research evidence, especially during the later phases.

Active involved community partnerships: co-creating implementation infrastructure for getting to and sustaining social impact

Active involved community partnerships (AICPs) are essential to co-create implementation infrastructure and translate evidence into real-world practice. Across varied forms, AICPs cultivate community and tribal members as agents of change, blending research and organizational knowledge with relationships, context, culture, and local wisdom. Unlike selective engagement, AICPs enable active involvement of partners in the ongoing process of implementation and sustainability. This includes defining the problem, developing solutions, detecting practice changes, aligning organizational supports, and nurturing shared responsibility, accountability, and ownership for implementation. This paper builds on previously established active implementation and scaling functions by outlining key AICP functions to close the research-practice gap. Part of a federal initiative, California Partners for Permanency (CAPP) integrated AICP functions for implementation and system change to reduce disproportionality and disparities in long-term foster care. This paper outlines their experience defining and embedding five AICP functions: (1) relationship-building; (2) addressing system barriers; (3) establishing culturally relevant supports and services; (4) meaningful involvement in implementation; and (5) ongoing communication and feedback for continuous improvement. Planning for social impact requires the integration of AICP with other active implementation and scaling functions. Through concrete examples, authors bring multilevel AICP roles to life and discuss implications for implementation research and practice.

Automatic classification of communication logs into implementation stages via text analysis

BACKGROUND

To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes.

RESULTS

We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage.

CONCLUSIONS

This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

Sustained Implementation of Evidence-based Programs in Disadvantaged Communities: A Conceptual Framework of Supporting Factors

This paper presents a review of the empirical literature for studies evaluating factors that facilitate and create barriers to sustained program implementation in disadvantaged communities. It outlines study methodology and sustainment outcomes and proposes a conceptual model that involves implementation sustainment support for providers delivering evidence-based health and family services in disadvantaged communities. Sustained program implementation in the community setting is a significant issue as only 43% of studies reported successfully sustained programs. The review identified 18 factors that facilitate success and create barriers to program sustainment. The factors are synthesized into three themes; program characteristics, workplace capacity, and process and interaction factors. The majority of factors map onto commonly cited sustainability influences in implementation science. However, there was an additional focus for studies included in this review on the importance of factors such as program burden, program familiarity and perceived competence in program skills, workplace support for the program, staff mobility and turnover, supervision and peer support, and ongoing technical assistance. The need to use a conceptual framework and develop measures to guide and evaluate capacity building in EBP implementation and sustainment in low-resource community settings is highlighted.

The full translational spectrum of prevention science: facilitating the transfer of knowledge to practices and policies that prevent behavioral health problems

A broad-span, six-stage translational prevention model is presented, extending from the basic sciences-taking a multi-level systems approach, including the neurobiological sciences-through to globalization. The application of a very wide perspective of translation research from basic scientific discovery to international policy change promises to elicit sustainable, population-level reductions in behavioral health disorders. To illustrate the conceptualization and actualization of a program of translational prevention research, we walk through each stage of research to practice and policy using an exemplar, callous-unemotional (CU) traits. Basic science has identified neurobiological, psychophysiological, behavioral, contextual, and experiential differences in this subgroup, and yet, these findings have not been applied to the development of more targeted intervention. As a result, there are currently no programs considered especially effective for CU traits, likely because they do not specifically target underlying mechanisms. To prevent/reduce the prevalence of conduct disorder, it is critical that we transfer existing knowledge to subsequent translational stages, including intervention development, implementation, and scaling. And eventually, once resulting programs have been rigorously evaluated, replicated, and adapted across cultural, ethnic, and gender groups, there is potential to institutionalize them as well as call attention to the special needs of this population. In this paper, we begin to consider what resources and changes in research perspectives are needed to move along this translational spectrum.

Transforming community prevention systems for sustained impact: embedding active implementation and scaling functions

Traditional efforts to translate evidence-based prevention strategies to communities, at scale, have not often produced socially significant outcomes or the local capacity needed to sustain them. A key gap in many efforts is the transformation of community prevention systems to support and sustain local infrastructure for the active implementation, scaling, and continuous improvement of effective prevention strategies. In this paper, we discuss (1) the emergence of applied implementation science as an important type 3-5 translational extension of traditional type 2 translational prevention science, (2) active implementation and scaling functions to support the full and effective use of evidence-based prevention strategies in practice, (3) the organization and alignment of local infrastructure to embed active implementation and scaling functions within community prevention systems, and (4) policy and practice implications for greater social impact and sustainable use of effective prevention strategies.

Are inner context factors related to implementation outcomes in cognitive-behavioral therapy for youth anxiety?

Among the challenges facing the mental health field are the dissemination and implementation of evidence-based practices. The present study investigated the relationships between inner context variables (i.e., adopter characteristics and individual perceptions of intra-organizational factors) and two implementation outcomes-independently rated therapist fidelity on a performance-based role-play (i.e., adherence and skill) and self-reported penetration of cognitive behavioral therapy for youth anxiety following training. A significant relationship was found between inner context variables and fidelity. Specifically, adopter characteristics were associated with adherence and skill; individual perceptions of intra-organizational factors were associated with adherence. Inner context variables were not associated with penetration. Future directions are discussed.

Development and testing of an implementation strategy for a complex housing intervention: protocol for a mixed methods study

BACKGROUND

There is currently a lack of scientifically designed and tested implementation strategies. Such strategies are particularly important for highly complex interventions that require coordination between multiple parts to be successful. This paper presents a protocol for the development and testing of an implementation strategy for a complex intervention known as the Housing First model (HFM). Housing First is an evidence-based practice for chronically homeless individuals demonstrated to significantly improve a number of outcomes.

METHODS/DESIGN

Drawing on practices demonstrated to be useful in implementation and e-learning theory, our team is currently adapting a face-to-face implementation strategy so that it can be delivered over a distance. Research activities will be divided between Chicago and Central Indiana, two areas with significantly different barriers to HFM implementation. Ten housing providers (five from Chicago and five from Indiana) will be recruited to conduct an alpha test of each of four e-learning modules as they are developed. Providers will be requested to keep a detailed log of their experience completing the modules and participate in one of two focus groups. After refining the modules based on alpha test results, we will test the strategy among a sample of four housing organizations (two from Chicago and two from Indiana). We will collect and analyze both qualitative and quantitative data from administration and staff. Measures of interest include causal factors affecting implementation, training outcomes, and implementation outcomes.

DISCUSSION

This project is an important first step in the development of an evidence-based implementation strategy to increase scalability and impact of the HFM. The project also has strong potential to increase limited scientific knowledge regarding implementation strategies in general.

Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial

Background

Much is to be learned about what implementation strategies are the most beneficial to communities attempting to adopt evidence-based practices. This paper presents outcomes from a randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) in child public service systems in California and Ohio, including child welfare, juvenile justice, and mental health.

Methods

Fifty-one counties were assigned randomly to one of two different implementation strategies (Community Development Teams (CDT) or independent county implementation strategy (IND)) across four cohorts after being matched on county characteristics. We compared these two strategies on implementation process, quality, and milestone achievements using the Stages of Implementation Completion (SIC) (Implement Sci 6(1):1–8, 2011).

Results

A composite score for each county, combining the final implementation stage attained, the number of families served, and quality of implementation, was used as the primary outcome. No significant difference between CDT and IND was found for the composite measure. Additional analyses showed that there was no evidence that CDT increased the proportion of counties that started-up programs (i.e., placed at least one family in MTFC). For counties that did implement MTFC, those in the CDT condition served over twice as many youth during the study period as did IND. Of the counties that successfully achieved program start-up, those in the CDT condition completed the implementation process more thoroughly, as measured by the SIC. We found no significant differences by implementation condition on the time it took for first placement, achieving competency, or number of stages completed.

Conclusions

This trial did not lead to higher rates of implementation or faster implementation but did provide evidence for more robust implementation in the CDT condition compared to IND implementation once the first family received MTFC services. This trial was successful from a design perspective in that no counties dropped out, even though this study took place during an economic recession. We believe that this methodologic approach of measurement utilizing the SIC, which is comprised of the three dimensions of quality, quantity, and timing, is appropriate for a wide range of implementation and translational studies.

Trial registration

Trial ID: NCT00880126 (ClinicalTrials.gov).

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-014-0134-8) contains supplementary material, which is available to authorized users.

Mental health interventions in schools 1: Mental health interventions in schools in high-income countries

Mental health services embedded within school systems can create a continuum of integrative care that improves both mental health and educational attainment for children. To strengthen this continuum, and for optimum child development, a reconfiguration of education and mental health systems to aid implementation of evidence-based practice might be needed. Integrative strategies that combine classroom-level and student-level interventions have much potential. A robust research agenda is needed that focuses on system-level implementation and maintenance of interventions over time. Both ethical and scientific justifications exist for integration of mental health and education: integration democratises access to services and, if coupled with use of evidence-based practices, can promote the healthy development of children.

Consultation as an implementation strategy for evidence-based practices across multiple contexts: unpacking the black box

There is great interest in the dissemination and implementation of evidence-based treatments and practices for children across schools and community mental health settings. A growing body of literature suggests that the use of one-time workshops as a training tool is ineffective in influencing therapist behavior and patient outcomes and that ongoing expert consultation and coaching is critical to actual uptake and quality implementation. Yet, we have very limited understanding of how expert consultation fits into the larger implementation support system, or the most effective consultation strategies. This commentary reviews the literature on consultation in child mental health, and proposes a set of core consultation functions, processes, and outcomes that should be further studied in the implementation of evidence-based practices for children.

Exploration and Adoption of Evidence-based Practice by US Child Welfare Agencies

OBJECTIVE

To examine the extent to which child welfare agencies adopt new practices and to determine the barriers to and facilitators of adoption of new practices.

METHODS

Data came from telephone interviews with the directors of the 92 public child welfare agencies that constituted the probability sample for the first National Survey of Child and Adolescent Well-being (NSCAWI). In a semi-structured 40 minute interview administered by a trained Research Associate, agency directors were asked about agency demographics, knowledge of evidence-based practices, use of technical assistance and actual use of evidence-based practices.. Of the 92 agencies, 83 or 90% agreed to be interviewed.

RESULTS

Agencies reported that the majority of staff had a BA degree (53.45%) and that they either paid for (52.6%) or provided (80.7%) continuing education. Although agencies routinely collect standardized child outcomes (90%) they much less frequently collect measures of child functioning (30.9%). Almost all agencies (94%) had started a new program or practice but only 24.8% were evidence-based and strategies used to explore new programs or practices usually involved local or state contracts. Factors that were associated with program success included internal support for the innovation (27.3%), and an existing evidence base (23.5%).

CONCLUSIONS

Directors of child welfare agencies frequently institute new programs or practices but they are not often evidence-based. Because virtually all agencies provide some continuing education adding discussions of evidence-based programs/practices may spur adaption. Reliance on local and state colleagues to explore new programs and practices suggests that developing well informed social networks may be a way to increase the spread of evidence0based practices.

Influence network linkages across implementation strategy conditions in a randomized controlled trial of two strategies for scaling up evidence-based practices in public youth-serving systems

BACKGROUND

Given the importance of influence networks in the implementation of evidence-based practices and interventions, it is unclear whether such networks continue to operate as sources of information and advice when they are segmented and disrupted by randomization to different implementation strategy conditions. The present study examines the linkages across implementation strategy conditions of social influence networks of leaders of youth-serving systems in 12 California counties participating in a randomized controlled trial of community development teams (CDTs) to scale up use of an evidence-based practice.

METHODS

Semi-structured interviews were conducted with 38 directors, assistant directors, and program managers of county probation, mental health, and child welfare departments. A web-based survey collected additional quantitative data on information and advice networks of study participants. A mixed-methods approach to data analysis was used to create a sociometric data set (n = 176) to examine linkages between treatment and standard conditions.

RESULTS

Of those network members who were affiliated with a county (n = 137), only 6 (4.4%) were directly connected to a member of the opposite implementation strategy condition; 19 (13.9%) were connected by two steps or fewer to a member of the opposite implementation strategy condition; 64 (46.7%) were connected by three or fewer steps to a member of the opposite implementation strategy condition. Most of the indirect steps between individuals who were in different implementation strategy conditions were connections involving a third non-county organizational entity that had an important role in the trial in keeping the implementation strategy conditions separate. When these entities were excluded, the CDT network exhibited fewer components and significantly higher betweenness centralization than did the standard condition network.

CONCLUSION

Although the integrity of the RCT in this instance was not compromised by study participant influence networks, RCT designs should consider how influence networks may extend beyond boundaries established by the randomization process in implementation studies.

TRIAL REGISTRATION

NCT00880126.

Advances in bridging research and practice: introduction to the second special issue on the interactive system framework for dissemination and implementation

The need for new ways to bridge the gap between research and practice is clear; the use of evidence-based prevention programs and implementation with fidelity in practice are strikingly limited. The Interactive Systems Framework for Dissemination and Implementation (ISF) was created to help bridge research and practice by specifying the systems and processes required to support dissemination and implementation of evidence-based programs, processes, practices, and policies. The ISF identifies three key systems necessary for this process which include the Synthesis and Translation System, the Support System, and the Delivery System. The ISF was featured in a special issue of the American Journal of Community Psychology in 2008. This special issue extends that work by including both researchers who have applied an ISF lens to aspects of their current work and researchers who have proactively applied the ISF in a process that goes across the various systems of the ISF, i.e., Synthesis and Translation, Support, and Delivery. Content areas include: children's mental health, teen pregnancy prevention, HIV prevention, violence prevention, heart disease and stroke prevention, breast cancer prevention, and substance abuse prevention. In this introductory article, we provide a brief description of the history of the ISF and a summary of the articles in the special issue.