Three collaborative models for scaling up evidence-based practices

Scale-up of Global Child and Youth Mental Health Services: A Scoping Review

Numerous influential policy and scientific bodies are calling for more rapid advances in the scale-up of child and youth mental health services (CYMHS). A number of CYMHS innovations hold promise for advancing scale-up but little is known about how real-world efforts are progressing. We conducted a scoping review to identify promising approaches to CYMHS scale-up across the globe. Searches were completed in six databases (Academic Search Complete, CINAHL, MEDLINE, PsychInfo, PubMed, and Web of Science). Article selection and synthesis were conducted in accordance to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews (PRISMA-ScR) checklist. A second search focused on low-and-middle-income countries (LMIC) was conducted based on the Cochrane Library recommended search filters of the World Bank listed LMIC countries. Authors used a double coding strategy during the title/abstract and full-text review. Twenty-eight articles meeting the eligibility criteria were identified that described 22 initiatives (in 11 different countries). Our review found the majority of published scale-up studies in CYMHS were not informed by scale-up frameworks in design or reporting. The methods and outcomes used in the identified articles were highly variable and limited our ability to draw conclusions about comparative effectiveness although promising approaches emerged. Successes and failures identified in our review largely reflect consensus in the broader literature regarding the need for strategies to better navigate the complexities of system and policy implementation while ensuring CYMHS interventions fit local contexts.

Interventions targeting the mental health and wellbeing of care-experienced children and young people in higher-income countries: Evidence map and systematic review

BACKGROUND

The mental health and wellbeing of care-experienced children and young people (i.e. foster care, kinship care, residential care) is poorer than non-care-experienced populations. The Care-experienced cHildren and young people's Interventions to improve Mental health and wEll-being outcomes Systematic review (CHIMES) aimed to synthesise the international evidence base for interventions targeting subjective wellbeing, mental health and suicide amongst care-experienced young people aged ≤ 25 years.

METHODS

For the first phase of the review, we constructed an evidence map identifying key clusters and gaps in interventions and evaluations. Studies were identified through 16 electronic databases and 22 health and social care websites, in addition to expert recommendations, citation tracking and screening of relevant systematic reviews. We charted interventions and evaluations with a summary narrative, tables and infographics.

RESULTS

In total, 64 interventions with 124 associated study reports were eligible. The majority of study reports were from the USA (n = 77). Interventions primarily targeted children and young people's skills and competencies (n = 9 interventions), the parental functioning and practices of carers (n = 26), or a combination of the two (n = 15). While theoretically under-specified, interventions were largely informed by theories of Attachment, Positive Youth Development, and Social Learning Theory. Current evaluations prioritised outcomes (n = 86) and processes (n = 50), with a paucity of study reports including theoretical descriptions (n = 24) or economic evaluations (n = 1). Interventions most frequently targeted outcomes related to mental, behavioural or neurodevelopmental disorders, notably total social, emotional and behavioural problems (n = 48 interventions) and externalising problem behaviours (n = 26). There were a limited number of interventions targeting subjective wellbeing or suicide-related outcomes.

CONCLUSIONS

Future intervention development might focus on structural-level intervention theories and components, and target outcomes related to subjective wellbeing and suicide. In accordance with current methodological guidance for intervention development and evaluation, research needs to integrate theoretical, outcome, process and economic evaluation in order to strengthen the evidence base.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020177478.

The role of implementation organizations in scaling evidence-based psychosocial interventions

BACKGROUND

To bring evidence-based interventions (EBIs) to individuals with behavioral health needs, psychosocial interventions must be delivered at scale. Despite an increasing effort to implement effective treatments in communities, most individuals with mental health and behavioral problems do not receive EBIs. We posit that organizations that commercialize EBIs play an important role in disseminating EBIs, particularly in the USA. The behavioral health and implementation industry is growing, bringing the implementation field to an important inflection point: how to scale interventions to improve access while maintaining EBI effectiveness and minimizing inequities in access to psychosocial intervention.

MAIN BODY

We offer a first-hand examination of five illustrative organizations specializing in EBI implementation: Beck Institute for Cognitive Behavioral Therapy; Incredible Years, Inc.; the PAXIS Institute; PracticeWise, LLC; and Triple P International. We use the Five Stages of Small Business Growth framework to organize themes. We discuss practical structures (e.g., corporate structures, intellectual property agreements, and business models) and considerations that arise when trying to scale EBIs including balancing fidelity and reach of the intervention. Business models consider who will pay for EBI implementation and allow organizations to scale EBIs.

CONCLUSION

We propose research questions to guide scaling: understanding the level of fidelity needed to maintain efficacy, optimizing training outcomes, and researching business models to enable organizations to scale EBIs.

Adapting the stages of implementation completion to an evidence-based implementation strategy: The development of the NIATx stages of implementation completion

Background

Dissemination and implementation frameworks provide the scaffolding to explore the effectiveness of evidence-based practices (EBPs) targeting process of care and organizational outcomes. Few instruments, like the stages of implementation completion (SIC) examine implementation fidelity to EBP adoption and how organizations differ in their approach to implementation. Instruments to measure organizational competency in the utilization of implementation strategies are lacking.

Method

An iterative process was utilized to adapt the SIC to the NIATx implementation strategies. The new instrument, NIATx-SIC, was applied in a randomized controlled trial involving 53 addiction treatment agencies in Washington state to improve agency co-occurring capacity. NIATx-SIC data were reported by state staff and external facilitators and through participating agency documentation. Proportion and duration scores for each stage and phase of the NIATx-SIC were calculated for each agency. Competency was assessed using the NIATx fidelity tool. Comparisons of proportion, duration, and NIATx activities completed were determined using independent sample t-tests by agency competency level.

Results

The NIATx-SIC distinguished between agencies achieving competency (n  =  23) and those not achieving competency (n  =  26). Agencies achieving competency completed a greater proportion of implementation phase activities and had a significantly longer Stage 7 duration. These agencies participated in significantly more individual and group coaching calls, attended more in-person meetings, implemented more change projects, and spent approximately 64 more days, on average, engaging in all NIATx activities.

Conclusions

Organizational participation in dissemination and implementation research requires a significant investment of staff resources. The inability of an organization to achieve competency when utilizing a set of implementation strategies waste an opportunity to institutionalize knowledge of how to apply implementation strategies to future change efforts. The NIATx-SIC provides evidence that competency is not an attribute of the organization but rather a result of the application of the NIATx implementation strategies to improve agency co-occurring capacity.

Trial Registration

ClinicalTrials.gov, NCT03007940. Registered January 2, 2017, https://clinicaltrials.gov/ct2/show/NCT03007940.

Stepped implementation-to-target: a study protocol of an adaptive trial to expand access to addiction medications

BACKGROUND

In response to the US opioid epidemic, significant national campaigns have been launched to expand access to `opioid use disorder (MOUD). While adoption has increased in general medical care settings, specialty addiction programs have lagged in both reach and adoption. Elevating the quality of implementation strategy, research requires more precise methods in tailoring strategies rather than a one-size-fits-all-approach, documenting participant engagement and fidelity to the delivery of the strategy, and conducting an economic analysis to inform decision making and policy. Research has yet to incorporate all three of these recommendations to address the challenges of implementing and sustaining MOUD in specialty addiction programs.

METHODS

This project seeks to recruit 72 specialty addiction programs in partnership with the Washington State Health Care Authority and employs a measurement-based stepped implementation-to-target approach within an adaptive trial design. Programs will be exposed to a sequence of implementation strategies of increasing intensity and cost: (1) enhanced monitoring and feedback (EMF), (2) 2-day workshop, and then, if outcome targets are not achieved, randomization to either internal facilitation or external facilitation. The study has three aims: (1) evaluate the sequential impact of implementation strategies on target outcomes, (2) examine contextual moderators and mediators of outcomes in response to the strategies, and (3) document and model costs per implementation strategy. Target outcomes are organized by the RE-AIM framework and the Addiction Care Cascade.

DISCUSSION

This implementation project includes elements of a sequential multiple assignment randomized trial (SMART) design and a criterion-based design. An innovative and efficient approach, participating programs only receive the implementation strategies they need to achieve target outcomes. Findings have the potential to inform implementation research and provide key decision-makers with evidence on how to address the opioid epidemic at a systems level.

TRIAL REGISTRATION

This trial was registered at ClinicalTrials.gov (NCT05343793) on April 25, 2022.

Using economic evaluations in implementation science to increase transparency in costs and outcomes for organizational decision-makers

Background

Economic evaluations frequently are utilized to compare the value of different interventions in medicine and health in concrete terms. Implementation science also would benefit from the incorporation of economic evaluations, but such studies are rare in the literature. The National Cancer Institute has supported a special collection of articles focusing on economic evaluations in implementation science. Even when interventions are supported by substantial evidence, they are implemented infrequently in the field. Implementation costs are important determinants for whether organizational decision-makers choose to adopt an intervention and whether the implementation process is successful. Economic evaluations, such as cost-effectiveness analyses, can help organizational decision-makers choose between implementation approaches for evidence-based interventions by accounting for costs and succinctly presenting cost/benefit tradeoffs.

Main text

This manuscript presents a discussion of important considerations for incorporating economic evaluations into implementation science. First, the distinction between intervention and implementation costs is presented, along with an explanation of why the comprehensive representation of implementation costs is elusive. Then, the manuscript describes how economic evaluations in implementation science may differ from those in medicine and health intervention studies, especially in terms of determining the perspectives and outcomes of interest. Finally, referencing a scale-up trial of an evidence-based behavioral health intervention, concrete case examples of how cost data can be collected and used in economic evaluations targeting implementation, rather than clinical outcomes, are described.

Conclusions

By gaining a greater understanding of the costs and economic impact associated with different implementation approaches, organizational decision-makers will have better transparency for future replication and scale-up. The use of economic evaluations can help to advance this understanding and provide researchers, purveyors or third-party intermediaries, and organizational decision-makers with essential information to facilitate implementation.

Supplementary Information

The online version contains supplementary material available at 10.1186/s43058-022-00295-1.

Scaling up a community-led health promotion initiative: Lessons learned and promising practices from the Healthy Weights for Children Project

The increase in overweight and obesity among children has emerged as an important public health issue. This trend has highlighted the need for accessible and novel approaches to support healthy weights for children and their families to prevent childhood obesity. The purpose of this article is to describe the iterative development and scale-up of a community-led, national-level project to promote healthy weights among Canadian children and families who may be experiencing vulnerabilities. In this project, the Healthy Together program was designed to engage families in an interactive program to support healthy lifestyles. The program also provides a platform for creating supportive environments for healthful lifestyles through practice and policy change. Based on a process evaluation, we describe the iterative development of Healthy Together from Phase 1 through 3 to shed light on processes shaping implementation and scale-up of the program. Lessons learned during each phase were used to refine the program and further expansion. Indicators of successful scale-up include the Healthy Together program's cross-jurisdictional reach and promising evaluation results in real-world conditions. The practice-based program scaling approach provides practical guidance for planning and implementing similar health promotion programs in diverse communities.

Sustainment of Trauma-Focused and Evidence-Based Practices Following Learning Collaborative Implementation

Given the need to develop and validate effective implementation models that lead to sustainable improvements, we prospectively examined changes in attitudes, behaviors, and perceived organizational support during and after statewide Community-Based Learning Collaboratives (CBLCs) promoting trauma-focused evidence-based practices (EBPs). Participants (N = 857; i.e., 492 clinicians, 218 brokers, and 139 senior leaders) from 10 CBLCs completed surveys pre- and post-CBLC; a subsample (n = 146) completed a follow-up survey approximately two years post-CBLC. Results indicated (a) medium, sustained increases in clinician-reported use of trauma-focused EBPs, (b) medium to large, sustained increases in perceived organizational support for trauma-focused EBPs, and (c) trivial to small, sustained increases in perceived organizational support for EBPs broadly. In contrast, clinician-reported overall attitudes towards EBPs decreased to a trivial degree pre- to post-CBLC, but then increased to a small, statistically significant degree from post-CBLC to follow-up. Notably, the degree of perceived improvements in organizational support for general and trauma-focused EBPs varied by professional role. Findings suggest the CBLC implementation strategies may both increase and sustain provider practices and organizational support towards EBPs, particularly those EBPs a CBLC explicitly targets.

Improving Medication Access within Integrated Treatment for Individuals with Co-Occurring Disorders in Substance Use Treatment Agencies

BACKGROUND

The best approach to provide comprehensive care for individuals with co-occurring disorders (CODs) related to substance use and mental health is to address both disorders through an integrated treatment approach. However, only 25% of behavioral health agencies offer integrated care and less than 7% of individuals who need integrated treatment receive it. A project used a cluster-randomized waitlist control group design to evaluate the effectiveness of Network for the Improvement of Addiction Treatment (NIATx) implementation strategies to improve access to addiction and psychotropic medications.

METHODS

This study represents a secondary analysis of data from the NIATx project. Forty-nine agencies were randomized to Cohort1 (active implementation group, receiving the NIATx strategy [n=25]) or Cohort2 (waitlist control group [n=24]). Data were collected at three time points (Baseline, Year1 and Year2). A two-level (patient within agency) multinomial logistic regression model investigated the effects of implementation strategy condition on one of four medication outcomes: both medication types, only psychotropic medication, only addiction medication, or neither medication type. A per-protocol analysis included time, NIATx fidelity, and agency focus as predictors.

RESULTS

The intent-to-treat analysis found a statistically significant change in access to addiction versus neither medication, but Cohort1 compared to Cohort2 at Year1 showed no differences. Changes were associated with the experimental intervention and occurred in the transition from Year 1 to Year 2, where greater increases were seen for agencies in Cohort2 versus Cohort1. The per-protocol analysis showed increased access to both medications and addiction medications from pre- to post-intervention for agencies in both cohorts; however, differences in change between high- and low-implementation agencies were not significant.

CONCLUSIONS

Access to integrated services for people with CODs is a long-standing problem. NIATx implementation strategies had limited effectiveness in improving medication access for individuals with CODs. Implementation strategy adherence is associated with increased medication access.

A pragmatic randomised controlled trial of the fostering changes programme

BACKGROUND

Many looked after young people in Wales are cared for by foster or kinship carers, usually as a consequence of maltreatment or developmentally traumatising experiences within a family context. Confidence in Care is a pragmatic unblinded individually randomised controlled parallel group trial evaluating a training programme to improve foster carer self-efficacy, when compared to usual care.

OBJECTIVE

To determine whether group-based training improves foster carer self-efficacy.

PARTICIPANTS AND SETTING

Participants are foster carers, currently looking after children aged 2+ years for at least 12 weeks. Carers from households where one or more carer had previously attended the training were not eligible. Sixteen local authorities and three independent fostering providers in Wales took part.

METHODS

The primary outcome measure was the Carer Efficacy Questionnaire assessed at 12 months. Secondary outcomes included the Strengths and Difficulties Questionnaire, Quality of Attachment Questionnaire, Carer Defined Problems Scale, Carer Coping Strategies, placement moves.

RESULTS

312 consented foster carers were allocated to FC (n = 204) or usual care (n = 108) group. 65.3 % of FC group participants attended sufficient training sessions (8/12, including sessions three and four). There were no differences in carer-reported self-efficacy at 12 months (adjusted difference in means (95 % CI): -0.19 (-1.38 to 1.00)). Small differences in carer-reported child behaviour difficulties and carer coping strategies over time favoured the intervention but these effects diminished from three to 12 months. No other intervention effects were observed.

CONCLUSIONS

Although well-received by participants, training was associated with small and mostly short-term benefit for trial secondary outcomes.

Design and baseline data of a randomized trial comparing two methods for scaling-up an occupational sun protection intervention

Background

Successful methods for scaling-up evidence-based programs are needed to prevent skin cancer among adults who work outdoors in the sun.

Methods

A randomized trial is being conducted comparing two methods of scaling-up the Sun Safe Workplaces (SSW) intervention. Departments of transportation (DOTs) from 21 U.S. states are participating and their 138 regional districts were randomized following baseline assessment. In districts assigned to the in-person method (n = 46), project staff meets personally with managers, conducts trainings for employees, and provides printed materials. In districts assigned to the digital method (n = 92), project staff conduct these same activities virtually, using conferencing technology, online training, and electronic materials. Delivery of SSW in both groups was tailored to managers' readiness to adopt occupational sun safety. Posttesting will assess manager's support for and use of SSW and employees' sun safety. An economic evaluation will explore whether the method that uses digital technology results in lower implementation of SSW but is more cost-effective relative to the in-person method.

Results

The state DOTs range in size from 997 to 18,415 employees. At baseline, 1113 managers (49.0%) completed the pretest (91.5% male, 91.1% white, 19.77 years on the job, 66.5% worked outdoors; and 24.4% had high-risk skin types). They were generally supportive of occupational sun safety. A minority reported that the employer had a written policy, half reported training, and two-thirds, messaging on sun protection.

Conclusions

Digital methods are available that may make scale-up of SSW cost-effective in a national distribution to nearly half of the state DOTs.

Trial registration: The ClinicalTrials.gov registration number is NCT03278340.

Practitioners' Use and Evaluation of Transdiagnostic Youth Psychotherapy Years After Training and Consultation Have Ended

We examined practitioners' use of the transdiagnostic Modular Approach to Therapy for Children (MATCH) 7 years after learning MATCH for a clinical trial. The practitioners (N = 29; M_age = 52.10, SD = 12.29, 86% women, 97% white) reported using MATCH with 55% of their caseload; use of the various MATCH modules ranged from 39 to 70%. Use was positively associated with amount of MATCH experience in the trial, perceived effectiveness, and ease of implementation. Patterns of specific module use did not consistently match strength of prior evidence (e.g., exposure was least used of the anxiety modules), suggesting challenges for implementation science.

"Creative Financing": Funding Evidence-Based Interventions in Human Service Systems

Implementation and sustainment of evidence-based interventions (EBIs) is influenced by outer (e.g., broader environments in which organizations operate) and inner (e.g., organizations, their administrators, and staff) contexts. One important outer-context element that shapes the inner context is funding, which is complex and unpredictable. There is a dearth of knowledge on how funding arrangements affect sustainment of EBIs in human service systems and the organizations delivering them, including child welfare and behavioral health agencies. This study uses qualitative interview and focus group data with stakeholders at the system, organizational, and provider levels from 11 human service systems in two states to examine how stakeholders strategically negotiate diverse and shifting funding arrangements over time. Study findings indicate that, while diverse funding streams may contribute to flexibility of organizations and possible transformations in the human service delivery environment, a dedicated funding source for EBIs is crucial to their successful implementation and sustainment.

State Strategies for Enhancing Access and Quality in Systems of Care for Youth with Complex Behavioral Health Needs

The investigators conducted a qualitative study with state administrators and policymakers about the financing and policies that structure public systems of care for children with complex behavioral healthcare needs. The objective was to characterize diverse strategies states employed to enhance funding for, access to, and quality and effectiveness of home and community-based services (HCBS) for these youth. States report using a wide variety of creative solutions while navigating the unique needs and constraints of their political and economic environments. Findings can benefit public officials, researchers, and advocates by advancing knowledge-sharing of public policy-making and resourceful problem-solving.

Improving Substance Use Services for Juvenile Justice-Involved Youth: Complexity of Process Improvement Plans in a Large Scale Multi-site Study

Despite the high prevalence of substance use disorders among juvenile offenders, most do not receive services. System-level process improvement plans to address unmet service needs can be optimized by combining data-driven decisions and facilitated meetings with behavioral health stakeholders. This paper operationalizes and analyzes the level of specified complexity among process improvement plans evident within 36 juvenile probation and drug courts across 7 states. To inform more effective implementation strategies, this analysis identifies and prioritizes promising courses of agency enhancement toward addressing unmet substance use needs.

Using NIATx strategies to implement integrated services in routine care: a study protocol

Background

Access to integrated services for individuals with co-occurring substance use and mental health disorders is a long-standing public health issue. Receiving integrated treatment services are both more effective and preferred by patients and families versus parallel or fragmented care. National policy statements and expert consensus guidelines underscore the benefits of integrated treatment. Despite decades of awareness, adequate treatment for individuals with co-occurring substance use and mental health disorders occurs infrequently. The underlying disease burden associated with alcohol, illicit and prescription drug problems, as well as mental health disorders, such as depression, posttraumatic stress disorder and schizophrenia, is substantial.

Methods

This cluster randomized controlled trial (RCT) is designed to determine if the multi-component Network for the Improvement of Addiction Treatment (NIATx) strategies are effective in implementing integrated services for persons with co-occurring substance use and mental health disorders. In this study, 50 behavioral health programs in Washington State will be recruited and then randomized into one of two intervention arms: 1) NIATx implementation strategies, including coaching and learning sessions over a 12-month intervention period to implement changes targeting integrated treatment services; or 2) wait-list control. Primary outcome measures include: 1) fidelity - a standardized organizational assessment of integrated services (Dual Diagnosis in Addiction Treatment [DDCAT] Index); and 2) penetration - proportion of patients screened and diagnosed with co-occurring disorders, proportion of eligible patients receiving substance use and mental health services, and psychotropic or substance use disorder medications. Barriers and facilitators, as determinants of implementation outcomes, will be assessed using the Consolidated Framework for Implementation Research (CFIR) Index. Fidelity to and participation in NIATx strategies will be assessed utilizing the NIATx Fidelity Scale and Stages of Implementation Completion (SIC).

Discussion

This study addresses an issue of substantial public health significance: the gap in access to an evidence-based practice for integrated treatment for individuals with co-occurring mental health and substance use disorders. The study utilizes rigorous and reproducible quantitative approaches to measuring implementation determinants and strategies, and may address a longstanding gap in the quality of care for persons with co-occurring disorders.

Trial registration

ClinicalTrials.gov NCT03007940. Registered 02 January 2017 – Retrospectively Registered

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3241-4) contains supplementary material, which is available to authorized users.

Using Goal Achievement Training in juvenile justice settings to improve substance use services for youth on community supervision

BACKGROUND

The link between substance use and involvement in the juvenile justice system has been well established. Justice-involved youth tend to have higher rates of drug use than their non-offending peers. At the same time, continued use can contribute to an elevated risk of recidivism, which leads to further, and oftentimes more serious, involvement with the juvenile justice system. Because of these high rates of use, the juvenile justice system is well positioned to help identify youth with substance use problems and connect them to treatment. However, research has found that only about 60% of juvenile probation agencies screen all youth for substance involvement, and even fewer provide comprehensive assessment or help youth enroll in substance use treatment.

METHOD

This paper describes an integrated training curriculum that was developed to help juvenile justice agencies improve their continuum of care for youth probationers with substance use problems. Goal Achievement Training (GAT) provides a platform for continuous quality improvement via two sessions delivered onsite to small groups of staff from juvenile justice and behavioral health agencies. In the first session, participants are taught to identify goals and goal steps for addressing identified areas of unmet need (i.e., screening, assessment, and linkage to treatment services). In the second session, participants learn principles and strategies of data-driven decision-making for achieving these goals. This paper highlights GAT as a model for the effective implementation of cost-efficient training strategies designed to increase self-directed quality improvement activities that can be applied to any performance domain within juvenile justice settings. Efforts to monitor implementation fidelity of GAT within the specific context of the juvenile justice settings are highlighted.

DISCUSSION

Challenges to setting the stage for process improvement generally, as well as specific hurdles within juvenile justice settings are discussed, as are next steps in disseminating findings regarding the fidelity to and effectiveness of GAT in this unique context.

TRIAL REGISTRATION

Clinical Trials Registration number - NCT02672150 .

Defining the external implementation context: an integrative systematic literature review

Background

Proper implementation of evidence-based interventions is necessary for their full impact to be realized. However, the majority of research to date has overlooked facilitators and barriers existing outside the boundaries of the implementing organization(s). Better understanding and measurement of the external implementation context would be particularly beneficial in light of complex health interventions that extend into and interact with the larger environment they are embedded within. We conducted a integrative systematic literature review to identify external context constructs likely to impact implementation of complex evidence-based interventions.

Methods

The review process was iterative due to our goal to inductively develop the identified constructs. Data collection occurred in four primary stages: (1) an initial set of key literature across disciplines was identified and used to inform (2) journal and (3) author searches that, in turn, informed the design of the final (4) database search. Additionally, (5) we conducted citation searches of relevant literature reviews identified in each stage. We carried out an inductive thematic content analysis with the goal of developing homogenous, well-defined, and mutually exclusive categories.

Results

We identified eight external context constructs: (1) professional influences, (2) political support, (3) social climate, (4) local infrastructure, (5) policy and legal climate, (6) relational climate, (7) target population, and (8) funding and economic climate.

Conclusions

This is the first study to our knowledge to use a systematic review process to identify empirically observed external context factors documented to impact implementation. Comparison with four widely-utilized implementation frameworks supports the exhaustiveness of our review process. Future work should focus on the development of more stringent operationalization and measurement of these external constructs.

Electronic supplementary material

The online version of this article (10.1186/s12913-018-3046-5) contains supplementary material, which is available to authorized users.

Variation in Research Designs Used to Test the Effectiveness of Dissemination and Implementation Strategies: A Review

Background

The need for optimal study designs in dissemination and implementation (D&I) research is increasingly recognized. Despite the wide range of study designs available for D&I research, we lack understanding of the types of designs and methodologies that are routinely used in the field. This review assesses the designs and methodologies in recently proposed D&I studies and provides resources to guide design decisions.

Methods

We reviewed 404 study protocols published in the journal Implementation Science from 2/2006 to 9/2017. Eligible studies tested the efficacy or effectiveness of D&I strategies (i.e., not effectiveness of the underlying clinical or public health intervention); had a comparison by group and/or time; and used ≥1 quantitative measure. Several design elements were extracted: design category (e.g., randomized); design type [e.g., cluster randomized controlled trial (RCT)]; data type (e.g., quantitative); D&I theoretical framework; levels of treatment assignment, intervention, and measurement; and country in which the research was conducted. Each protocol was double-coded, and discrepancies were resolved through discussion.

Results

Of the 404 protocols reviewed, 212 (52%) studies tested one or more implementation strategy across 208 manuscripts, therefore meeting inclusion criteria. Of the included studies, 77% utilized randomized designs, primarily cluster RCTs. The use of alternative designs (e.g., stepped wedge) increased over time. Fewer studies were quasi-experimental (17%) or observational (6%). Many study design categories (e.g., controlled pre-post, matched pair cluster design) were represented by only one or two studies. Most articles proposed quantitative and qualitative methods (61%), with the remaining 39% proposing only quantitative. Half of protocols (52%) reported using a theoretical framework to guide the study. The four most frequently reported frameworks were Consolidated Framework for Implementing Research and RE-AIM (n = 16 each), followed by Promoting Action on Research Implementation in Health Services and Theoretical Domains Framework (n = 12 each).

Conclusion

While several novel designs for D&I research have been proposed (e.g., stepped wedge, adaptive designs), the majority of the studies in our sample employed RCT designs. Alternative study designs are increasing in use but may be underutilized for a variety of reasons, including preference of funders or lack of awareness of these designs. Promisingly, the prevalent use of quantitative and qualitative methods together reflects methodological innovation in newer D&I research.

A Sequential Implementation Model for Workforce Development: A Case Study of Medical Residency Training for Substance Use Concerns

PROBLEM

In graduate medical education, residency programs are often educationally isolated from each other, with varying needs and patient populations, so strategies are needed when attempting to implement training in evidence-based practices across multiple residencies.

INTERVENTION

Using implementation science as a guide, we adapted a community development model to sequentially implement an evidence-based intervention, Screening, Brief Intervention, and Referral to Treatment (SBIRT) for alcohol and drug use problems, across internal medicine, pediatrics, emergency medicine, psychiatry, and obstetrics and gynecology residency programs.

CONTEXT

A grant-funded "executive" team coordinated the implementation, enrolled a new residency program annually, and served as the consultative team to span the programs. The team was attentive to aligning implementation with the needs of each program. To assist in planning, the team included a program champion 1 year prior to SBIRT implementation to provide the opportunity to develop resources and work with peers. We evaluated this model through an implementation science lens using a case study approach that included interviews and quantitative tabulation of products and resident perceptions.

OUTCOME

We successfully instituted SBIRT training in all 5 residency programs through the use of a supported local champion model. Teams developed 90 curricular products and had 57 presentations and publications. Residents reported satisfaction with the SBIRT activities. Champions reported that SBIRT was a useful approach and that they gained valuable knowledge and relationships from working with the executive team when designing learning materials appropriate for their residency. Champions successfully incorporated SBIRT into routine clinical practice.

LESSONS LEARNED

Having a strong team to support subsequent SBIRT champions was essential for implementation. The champions needed financial support to have the necessary time to implement training. The strategy of building a peer network across programs was critical for sustaining the implementation. Collecting and sharing resources aided the champions in developing their materials.

The Effect of a Change Agent on Use of Evidence-Based Mental Health Practices

Children's service systems are faced with a critical need to disseminate evidence-based mental health interventions. Despite the proliferation of comprehensive implementation models, little is known about the key active processes in effective implementation strategies. This proof of concept study focused on the effect of change agent interactions as conceptualized by Rogers' diffusion of innovation theory on providers' (N = 57) use of a behavioral intervention in a child welfare agency. An experimental design compared use for providers randomized to training as usual or training as usual supplemented by change agent interactions after the training. Results indicate that the enhanced condition increased use of the intervention, supporting the positive effect of change agent interactions on use of new practices. Change agent types of interaction may be a key active process in implementation strategies following training.

Juvenile Justice-Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS): a cluster randomized trial targeting system-wide improvement in substance use services

Background

The purpose of this paper is to describe the Juvenile Justice—Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) study, a cooperative implementation science initiative involving the National Institute on Drug Abuse, six research centers, a coordinating center, and Juvenile Justice Partners representing seven US states. While the pooling of resources across centers enables a robust implementation study design involving 36 juvenile justice agencies and their behavioral health partner agencies, co-producing a study protocol that has potential to advance implementation science, meets the needs of all constituencies (funding agency, researchers, partners, study sites), and can be implemented with fidelity across the cooperative can be challenging. This paper describes (a) the study background and rationale, including the juvenile justice context and best practices for substance use disorders, (b) the selection and use of an implementation science framework to guide study design and inform selection of implementation components, and (c) the specific study design elements, including research questions, implementation interventions, measurement, and analytic plan.

Methods/design

The JJ-TRIALS primary study uses a head-to-head cluster randomized trial with a phased rollout to evaluate the differential effectiveness of two conditions (Core and Enhanced) in 36 sites located in seven states. A Core strategy for promoting change is compared to an Enhanced strategy that incorporates all core strategies plus active facilitation. Target outcomes include improvements in evidence-based screening, assessment, and linkage to substance use treatment.

Discussion

Contributions to implementation science are discussed as well as challenges associated with designing and deploying a complex, collaborative project.

Trial registration

NCT02672150.

Promoting development and uptake of health innovations: The Nose to Tail Tool

Introduction

Health sector management is increasingly complex as new health technologies, treatments, and innovative service delivery strategies are developed. Many of these innovations are implemented prematurely, or fail to be implemented at scale, resulting in substantial wasted resources.  

Methods

A scoping review was conducted to identify articles that described the scale up process conceptually or that described an instance in which a healthcare innovation was scaled up. We define scale up as the expansion and extension of delivery or access to an innovation for all end users in a jurisdiction who will benefit from it.

Results

Sixty nine articles were eligible for review. Frequently described stages in the innovation process and contextual issues that influence progress through each stage were mapped. 16 stages were identified: 12 deliberation and 4 action stages. Included papers suggest that innovations progress through stages of maturity and the uptake of innovation depends on the innovation aligning with the interests of 3 critical stakeholder groups (innovators, end users and the decision makers) and is also influenced by 3 broader contexts (social and physical environment, the health system, and the regulatory, political and economic environment). The 16 stages form the rows of the Nose to Tail Tool (NTT) grid and the 6 contingency factors form columns. The resulting stage-by-issue grid consists of 72 cells, each populated with cell-specific questions, prompts and considerations from the reviewed literature.

Conclusion

We offer a tool that helps stakeholders identify the stage of maturity of their innovation, helps facilitate deliberative discussions on the key considerations for each major stakeholder group and the major contextual barriers that the innovation faces. We believe the NTT will help to identify potential problems that the innovation will face and facilitates early modification, before large investments are made in a potentially flawed solution.

The full translational spectrum of prevention science: facilitating the transfer of knowledge to practices and policies that prevent behavioral health problems

A broad-span, six-stage translational prevention model is presented, extending from the basic sciences-taking a multi-level systems approach, including the neurobiological sciences-through to globalization. The application of a very wide perspective of translation research from basic scientific discovery to international policy change promises to elicit sustainable, population-level reductions in behavioral health disorders. To illustrate the conceptualization and actualization of a program of translational prevention research, we walk through each stage of research to practice and policy using an exemplar, callous-unemotional (CU) traits. Basic science has identified neurobiological, psychophysiological, behavioral, contextual, and experiential differences in this subgroup, and yet, these findings have not been applied to the development of more targeted intervention. As a result, there are currently no programs considered especially effective for CU traits, likely because they do not specifically target underlying mechanisms. To prevent/reduce the prevalence of conduct disorder, it is critical that we transfer existing knowledge to subsequent translational stages, including intervention development, implementation, and scaling. And eventually, once resulting programs have been rigorously evaluated, replicated, and adapted across cultural, ethnic, and gender groups, there is potential to institutionalize them as well as call attention to the special needs of this population. In this paper, we begin to consider what resources and changes in research perspectives are needed to move along this translational spectrum.

Designs for Evaluating the Community-Level Impact of Comprehensive Prevention Programs: Examples from the CDC Centers of Excellence in Youth Violence Prevention

This article discusses the opportunities and challenges of developing research designs to evaluate the impact of community-level prevention efforts. To illustrate examples of evaluation designs, we describe six projects funded by the Centers for Disease Control and Prevention to evaluate multifaceted approaches to reduce youth violence in high-risk communities. Each of these projects was designed to evaluate the community-level impact of multiple intervention strategies to address individual and contextual factors that place youth at risk for violent behavior. Communities differed across projects in their setting, size, and how their boundaries were defined. Each project is using multiple approaches to compare outcomes in one or more intervention communities to those in comparison communities. Five of the projects are using comparative interrupted time-series designs to compare outcomes in an intervention community to matched comparison communities. A sixth project is using a multiple baseline design in which the order and timing of intervention activities is randomized across three communities. All six projects are also using regression point displacement designs to compare outcomes within intervention communities to those within broader sets of similar communities. Projects are using a variety of approaches to assess outcomes including archival records, surveys, and direct observations. We discuss the strengths and weaknesses of the designs of these projects and illustrate the challenges of designing high-quality evaluations of comprehensive prevention approaches implemented at the community level.

Achieving successful evidence-based practice implementation in juvenile justice: The importance of diagnostic and evaluative capacity

Evidence-based programs (EBPs) are an increasingly visible aspect of the treatment landscape in juvenile justice. Research demonstrates that such programs yield positive returns on investment and are replacing more expensive, less effective options. However, programs are unlikely to produce expected benefits when they are not well-matched to community needs, not sustained and do not reach sufficient reach and scale. We argue that achieving these benchmarks for successful implementation will require states and county governments to invest in data-driven decision infrastructure in order to respond in a rigorous and flexible way to shifting political and funding climates. We conceptualize this infrastructure as diagnostic capacity and evaluative capacity: Diagnostic capacity is defined as the process of selecting appropriate programing and evaluative capacity is defined as the ability to monitor and evaluate progress. Policy analyses of Washington State, Pennsylvania and Louisiana's program implementation successes are used to illustrate the benefits of diagnostic and evaluate capacity as a critical element of EBP implementation.

Practitioners' views and use of evidence-based treatment: positive attitudes but missed opportunities in children's services

The extent evidence-based treatments (EBTs) are used in clinical practice within the Australian therapeutic child welfare sector is unknown. In this study, we investigated practitioners' knowledge, attitudes, and use of EBT when providing interventions to children and families and how the intended outcomes of interventions are evaluated. Practitioners (N = 112) from 41 non-government organizations were surveyed and reported few barriers to implementing EBTs and positive attitudes. While just over half the practitioners surveyed provided an accurate definition of EBT, 72 % of practitioners reported using EBTs in their clinical practice. Of those, 88 % reported modifying the EBT, however interventions were rarely evaluated systematically. Implications for the use of EBTs, how they are modified, and the role of systematic evaluation are discussed.

Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial

Background

Much is to be learned about what implementation strategies are the most beneficial to communities attempting to adopt evidence-based practices. This paper presents outcomes from a randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) in child public service systems in California and Ohio, including child welfare, juvenile justice, and mental health.

Methods

Fifty-one counties were assigned randomly to one of two different implementation strategies (Community Development Teams (CDT) or independent county implementation strategy (IND)) across four cohorts after being matched on county characteristics. We compared these two strategies on implementation process, quality, and milestone achievements using the Stages of Implementation Completion (SIC) (Implement Sci 6(1):1–8, 2011).

Results

A composite score for each county, combining the final implementation stage attained, the number of families served, and quality of implementation, was used as the primary outcome. No significant difference between CDT and IND was found for the composite measure. Additional analyses showed that there was no evidence that CDT increased the proportion of counties that started-up programs (i.e., placed at least one family in MTFC). For counties that did implement MTFC, those in the CDT condition served over twice as many youth during the study period as did IND. Of the counties that successfully achieved program start-up, those in the CDT condition completed the implementation process more thoroughly, as measured by the SIC. We found no significant differences by implementation condition on the time it took for first placement, achieving competency, or number of stages completed.

Conclusions

This trial did not lead to higher rates of implementation or faster implementation but did provide evidence for more robust implementation in the CDT condition compared to IND implementation once the first family received MTFC services. This trial was successful from a design perspective in that no counties dropped out, even though this study took place during an economic recession. We believe that this methodologic approach of measurement utilizing the SIC, which is comprised of the three dimensions of quality, quantity, and timing, is appropriate for a wide range of implementation and translational studies.

Trial registration

Trial ID: NCT00880126 (ClinicalTrials.gov).

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-014-0134-8) contains supplementary material, which is available to authorized users.

The effect of waiting time on youth engagement to evidence based treatments

Prolonged waiting times to receive mental health services are common and may have negative consequences. This study examines the relationship between waiting time and treatment engagement among 2,054 youth referred to an evidence based treatment (EBT). Findings indicate that families are more likely to refuse services if they face longer waiting times. Families exposed to a prolonged waiting time were also more likely to drop out prematurely from Functional Family Therapy, but this relationship was not significant among youth receiving Multisystemic Therapy. Implications for EBT implementation and strategies for engaging families are discussed.

MTFC for High Risk Adolescent Girls: A Comparison of Outcomes in England and the United States

The current study examined 12-month outcomes for girls enrolled in an implementation trial of Multidimensional Treatment Foster Care (MTFC) in England. In addition to examining changes from pre-treatment to post-treatment, we also compared results for girls enrolled in the England implementation trial to girls enrolled in two randomized controlled trials (RCTs) of MTFC in the United States (US). The England MTFC sample included 58 girls in foster care between the ages of 12 and 16 years. The US MTFC intervention samples included 81 girls between the ages of 13 and 17 years who were referred to out-of-home care due to chronic delinquency. Results indicated improvement in offending, violent behavior, risky sexual behavior, self-harm, and school activities for girls enrolled in the England implementation trial. The effect sizes of these results were similar to those obtained in the US RCTs, with the exception of substance use which showed significant decreases for girls enrolled in the US RCTs, but not for girls enrolled in the England implementation trial. These results, in combination with other cross-cultural findings, support the notion that MTFC might be relevant across US and European cultures.

Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review

OBJECTIVE

Although there has been a dramatic increase in the number of evidence-based practices (EBPs) to improve child and adolescent mental health, the poor uptake of these EBPs has led to investigations of factors related to their successful dissemination and implementation. The purpose of this systematic review was to identify key findings from empirical studies examining the dissemination and implementation of EBPs for child and adolescent mental health.

METHOD

Of 14,247 citations initially identified, 73 articles drawn from 44 studies met inclusion criteria. The articles were classified by implementation phase (exploration, preparation, implementation, and sustainment) and specific implementation factors examined. These factors were divided into outer (i.e., system level) and inner (i.e., organizational level) contexts.

RESULTS

Few studies used true experimental designs; most were observational. Of the many inner context factors that were examined in these studies (e.g., provider characteristics, organizational resources, leadership), fidelity monitoring and supervision had the strongest empirical evidence. Albeit the focus of fewer studies, implementation interventions focused on improving organizational climate and culture were associated with better intervention sustainment as well as child and adolescent outcomes. Outer contextual factors such as training and use of specific technologies to support intervention use were also important in facilitating the implementation process.

CONCLUSIONS

The further development and testing of dissemination and implementation strategies is needed to more efficiently move EBPs into usual care.

Implementing an evidence-based practices training curriculum to U.S. Army child and family behavioral health providers

Military Children who have experienced multiple deployments of one or both parents are more likely to have emotional and behavioral difficulties compared with their civilian peers (e.g., Chandra et al., 2010 ). The U.S. Army Medical Command has tasked the Child, Adolescent and Family Behavioral Health Office (CAF-BHO) to develop programs to address the behavioral health (BH) needs of Army Children and Families. This article will describe the efforts of the CAF-BHO Training Section to disseminate and implement an evidence based practice training curriculum for Army Child and Family Behavioral Health Providers. Specifically, this article will detail: (a) the decision making strategy used to identify the training protocol for dissemination, (b) adaptations to the training program and treatment protocol to fit a Military context, and (c) efforts to implement, maximize and sustain utilization of evidence-based practices by Military BH providers over a large geographical area with limited resources.

Building a science of partnership-focused research: forging and sustaining partnerships to support child mental health prevention and services research

Building on growing interest in translational research, this paper provides an overview of a special issue of Administration and Policy in Mental Health and Mental Health Service Research, which is focused on the process of forging and sustaining partnerships to support child mental health prevention and services research. We propose that partnership-focused research is a subdiscipline of translational research which requires additional research to better refine the theoretical framework and the core principles that will guide future research and training efforts. We summarize some of the major themes across the eight original articles and three commentaries included in the special issue. By advancing the science of partnership-focused research we will be able to bridge the gap between child mental health prevention and services research and practice.

Partnerships for the design, conduct, and analysis of effectiveness, and implementation research: experiences of the prevention science and methodology group

What progress prevention research has made comes through strategic partnerships with communities and institutions that host this research, as well as professional and practice networks that facilitate the diffusion of knowledge about prevention. We discuss partnership issues related to the design, analysis, and implementation of prevention research and especially how rigorous designs, including random assignment, get resolved through a partnership between community stakeholders, institutions, and researchers. These partnerships shape not only study design, but they determine the data that can be collected and how results and new methods are disseminated. We also examine a second type of partnership to improve the implementation of effective prevention programs into practice. We draw on social networks to studying partnership formation and function. The experience of the Prevention Science and Methodology Group, which itself is a networked partnership between scientists and methodologists, is highlighted.