The stages of implementation completion for evidence-based practice: protocol for a mixed methods study

Implementation and sustainment of diverse practices in a large integrated health system: a mixed methods study

BACKGROUND

One goal of health systems seeking to evolve into learning health systems is to accelerate the implementation and sustainment of evidence-based practices (EBPs). As part of this evolution, the Veterans Health Administration (VHA) developed the Innovation Ecosystem, which includes the Diffusion of Excellence (DoE), a program that identifies and diffuses Gold Status Practices (GSPs) across facilities. The DoE hosts an annual "Shark Tank" competition in which leaders bid on the opportunity to implement a GSP with 6 months of implementation support. Over 750 diverse practices were submitted in cohorts 2 and 3 of Shark Tank; 23 were designated GSPs and were implemented in 31 VA networks or facilities. As part of a national evaluation of the DoE, we identified factors contributing to GSP implementation and sustainment.

METHODS

Our sequential mixed methods evaluation of cohorts 2 and 3 of Shark Tank included semi-structured interviews with at least one representative from 30/31 implementing teams (N = 78/105 people invited) and survey responses from 29/31 teams (N = 39/47 invited). Interviews focused on factors influencing implementation and future sustainment. Surveys focused on sustainment 1.5-2 years after implementation. The Consolidated Framework for Implementation Research (CFIR) informed data collection and directed content analysis. Ordinal scales were developed inductively to rank implementation and sustainment outcomes.

RESULTS

Over 50% of teams (17/30) successfully implemented their GSP within the 6-month implementation period. Despite extensive implementation support, significant barriers related to centralized decision-making, staffing, and resources led to partial (n = 6) or no (n = 7) implementation for the remaining teams. While 12/17 initially successful implementation teams reported sustained use of their GSP, over half of the initially unsuccessful teams (n = 7/13) also reported sustained GSP use 1.5 years after the initial implementation period. When asked at 6 months, 18/27 teams with complete data accurately anticipated their future sustainability based on reported sustainment an average of 1.5 years later.

CONCLUSIONS

Most teams implemented within 6 months and/or sustained their GSP 1.5 years later. High levels of implementation and sustainment across diverse practices and teams suggest that VHA's DoE is a successful large-scale model of diffusion. Team predictions about sustainability after the first 6 months of implementation provide a promising early assessment and point of intervention to increase sustainability.

Engaging stakeholders in the adaptation of the Connect for Health pediatric weight management program for national implementation

Background

Connect for Health is an evidence-based weight management program with clinical- and family-facing components for delivery in pediatric primary care for families of children ages 2 to 12 years. We used the Consolidated Framework for Implementation Research (CFIR) to guide formative work prior to national implementation. The purpose of this study was to describe the process and results of stakeholder engagement and program adaptation.

Methods

We used mixed qualitative and quantitative methods to iteratively adapt and optimize the program by assessing needs and perspectives of clinicians and parents, as well as contextual barriers, facilitators, and organizational readiness for the uptake of the proposed program tools and implementation strategies. We conducted interviews with primary care clinicians from four health care organizations in Boston, MA; Denver, CO; and Greenville, SC, and used principles of immersion-crystallization for qualitative analyses. We also conducted surveys of parents of children with a body mass index ≥ 85th percentile.

Results

We reached thematic saturation after 52 clinician interviews. Emergent themes representing the CFIR domains of intervention characteristics, outer and inner setting, and process included (1) importance of evidence-based clinical decision support tools that integrate into the workflow and do not extend visit time, (2) developing resources that respond to family’s needs, (3) using multimodal delivery options for family resources, (4) addressing childhood obesity while balancing competing demands, (5) emphasizing patient care rather than documentation and establishing sustainability plans, and (6) offering multiple training methods that incorporate performance feedback. Of the parents surveyed (n = 400), approximately 50% were Spanish-speaking and over 75% reported an annual income < $50,000. Parents affirmed the importance of addressing weight management during well-child visits, being provided with referrals and resources, and offering multiple methods for resource delivery. Decisions about program modifications were made at the program and healthcare-system level and based on stakeholder engagement findings. Modifications included cultural, geographic, and target audience adaptations, as well as varied resource delivery options.

Conclusions

To ensure the fit between the Connect for Health program and national implementation settings, adaptations were systematically made through engagement of clinician and parent stakeholders to support adoption, sustainability, and health outcomes.

Trial registration

NCT04042493

Applying the Exploration Preparation Implementation Sustainment (EPIS) Framework to the Kigali Imbereheza Project for Rwandan Adolescents Living With HIV

BACKGROUND

Sub-Saharan African adolescents living with HIV face challenges to antiretroviral therapy (ART) adherence. Poor mental health drives nonadherence but can be improved with cognitive behavioral therapy (CBT). CBT delivered by peers may strengthen effects while building capacity for sustainment in low-income countries. This case study retrospectively applied the Exploration Preparation Implementation Sustainment framework to characterize the execution of the Kigali Imbereheza Project, a 2-arm individually randomized group controlled trial of Trauma-Informed Adherence-Enhanced CBT (TI-CBTe) delivered by Rwandan youth leaders (YLs) to adolescents living with HIV.

METHODS

YL (n = 14, 43% female, M = 22.71 years) had confirmed HIV and self-reported ART adherence >95%. Participants (n = 356, 51% female, M = 16.78 years) living with HIV were randomized to TI-CBTe or usual care. Two YLs co-led TI-CBTe sessions over 2 months for a total of 12 hours, while other YL observed and rated fidelity. Participants reported on YL competence. Additional data evaluated feasibility, acceptability, uptake, and fidelity.

RESULTS

In the Exploration phase, focus groups, stakeholder meetings, and individual interviews revealed strong consensus for delivering TI-CBT to reduce adolescent depression and trauma and improve ART adherence. In the Preparation phase, curriculum revisions were made, YLs were successfully trained, and a cascading supervision model was established. In the Implementation phase, YL delivered TI-CBTe with close monitoring and supervision. Findings revealed strong feasibility, acceptability, uptake, and fidelity, increasing the likelihood of Sustainment.

CONCLUSIONS

Exploration Preparation Implementation Sustainment can guide implementation planning and delivery and evaluate implementation outcomes.

Implementing Teledermatology for Rural Veterans: An Evaluation Using the RE-AIM Framework

Introduction: Few systematic evaluations of implementing teledermatology programs in large health care systems exist. We conducted a longitudinal evaluation of a U.S. Department of Veterans Affairs (VA) initiative to expand asynchronous consultative teledermatology services for rural veterans. Methods: The reach, effectiveness, adoption, implementation, and maintenance framework guided the evaluation, which included analysis of quantitative VA administrative data as well as an online survey completed by participating facilities. The first 2 years of the program were compared with the year before the start of funding. Results: Sixteen hub facilities expanded teledermatology's reach over the 2-year period, increasing the number of referral spoke sites, unique patients served, and teledermatology encounters. Effectiveness was reflected as teledermatology constituted an increasing fraction of dermatology activity and served more remotely located patients. Adoption through defined stages of implementation progressed as facilities engaged in a variety of strategies to enhance teledermatology implementation, and facilitators and barriers were identified. Program maintenance was assessed by Program Sustainability Index scores, which reflected the importance of executive support, and ongoing concerns about staffing and longitudinal funding. Discussion: Enabling hubs to create solutions that best fit their needs and culture likely increased reach and effectiveness. Important facilitators included organizational leadership and encouraging communication between stakeholders before and during the intervention. Conclusions: A systematic analysis of teledermatology implementation to serve rural sites in VA documented a high degree of implementation and sustainability as well as areas for improvement.

Stage-based implementation of immediate postpartum long-acting reversible contraception using a reproductive justice framework

The immediate postpartum period is a favorable, safe, and effective time to provide long-acting reversible contraceptives, yet it is not available widely. We describe an innovative hospital-based approach to immediate postpartum long-acting reversible contraceptives that includes (1) an emphasis on multidisciplinary teambuilding and identification of champions, (2) a focus on the use of implementation science at every stage of the process to develop a systematic and replicable strategy, and (3) an imperative to apply a reproductive justice framework to immediate postpartum long-acting reversible contraceptive implementation. Our model was developed with the use of implementation science best practices. Implementation teams comprised of diverse stakeholders were formed and included champions to promote progress. Our team assessed the implementation context for immediate postpartum long-acting reversible contraceptives and used the findings to develop a readiness assessment for hospitals. A stage-based implementation checklist was then developed to outline necessary infrastructure to support an immediate postpartum long-acting reversible contraceptive initiative. A reproductive justice lens guided planning and implementation. The 3 innovative aspects of our implementation process resulted in a systematic, multidisciplinary, and culturally appropriate model for immediate postpartum long-acting reversible contraceptives that can be replicated across hospitals. Implementation teams and champions moved the work forward at each hospital, and 3 of the 5 participating hospitals moved beyond the exploration stage of implementation during the engagement. Patient education materials and provider training incorporated person-centered and reproductive justice frameworks. Our hope is to continue to partner with stakeholders to better understand how our efforts to support hospital provision of immediate postpartum long-acting reversible contraceptives can increase reproductive health equity rather than perpetuate disparity.

Scaling Implementation of Collaborative Care for Depression: Adaptation of the Stages of Implementation Completion (SIC)

Tools to monitor implementation progress could facilitate scale-up of effective treatments. Most treatment for depression, a common and disabling condition, is provided in primary care settings. Collaborative Care Management (CoCM) is an evidence-based model for treating common mental health conditions, including depression, in this setting; yet, it is not widely implemented. The Stages of Implementation Completion (SIC) was adapted for CoCM and piloted in eight rural primary care clinics serving adults challenged by low-income status. The CoCM-SIC accurately assessed implementation effectiveness and detected site variations in performance, suggesting key implementation activities to aid future scale-ups of CoCM for diverse populations.

Improving the implementation and sustainment of evidence-based practices in community mental health organizations: a study protocol for a matched-pair cluster randomized pilot study of the Collaborative Organizational Approach to Selecting and Tailoring Implementation Strategies (COAST-IS)

Background

Implementing and sustaining evidence-based programs with fidelity may require multiple implementation strategies tailored to address multi-level, context-specific barriers and facilitators. Ideally, selecting and tailoring implementation strategies should be guided by theory, evidence, and input from relevant stakeholders; however, methods to guide the selection and tailoring of strategies are not well-developed. There is a need for more rigorous methods for assessing and prioritizing implementation determinants (barriers and facilitators) and linking implementation strategies to determinants. The Collaborative Organizational Approach to Selecting and Tailoring Implementation Strategies (COAST-IS) is an intervention designed to increase the effectiveness of evidence-based practice implementation and sustainment. COAST-IS will enable organizational leaders and clinicians to use Intervention Mapping to select and tailor implementation strategies to address their site-specific needs. Intervention Mapping is a multi-step process that incorporates theory, evidence, and stakeholder perspectives to ensure that implementation strategies effectively address key determinants of change.

Methods

COAST-IS will be piloted with community mental health organizations that are working to address the needs of children and youth who experience trauma-related emotional or behavioral difficulties by engaging in a learning collaborative to implement an evidence-based psychosocial intervention (trauma-focused cognitive behavioral therapy). Organizations will be matched and then randomized to participate in the learning collaborative only (control) or to receive additional support through COAST-IS. The primary aims of this study are to (1) assess the acceptability, appropriateness, feasibility, and perceived utility of COAST-IS; (2) evaluate the organizational stakeholders' fidelity to the core elements of COAST-IS; and (3) demonstrate the feasibility of testing COAST-IS in a larger effectiveness trial.

Discussion

COAST-IS is a systematic method that integrates theory, evidence, and stakeholder perspectives to improve the effectiveness and precision of implementation strategies. If effective, COAST-IS has the potential to improve the implementation and sustainment of a wide range of evidence-based practices in mental health and other sectors.

Trial registration

This study was registered in ClinicalTrials.gov (NCT03799432) on January 10, 2019 (last updated August 5, 2019).

Development, implementation, and evaluation of a novel guideline engine for pediatric patients with severe traumatic brain injury: a study protocol

BACKGROUND

Severe traumatic brain injury (TBI) is a leading cause of death and disability for children. The Brain Trauma Foundation released evidence-based guidelines, a series of recommendations regarding care for pediatric patients with severe TBI. Clinical evidence suggests that adoption of guideline-based care improves outcomes in patients with severe TBI. However, guideline implementation has not been systematic or consistent in clinical practice. There is also a lack of information about implementation strategies that are effective given the nature of severe TBI care and the complex environment in the intensive care unit (ICU). Novel technology-based strategies may be uniquely suited to the fast-paced, transdisciplinary care delivered in the ICU, but such strategies must be carefully developed and evaluated to prevent unintended consequences within the system of care. This challenge presents a unique opportunity for intervention to more appropriately implement guideline-based care for pediatric patients with severe TBI.

METHODS

This mixed-method study will develop a novel technology-based bedside guideline engine (the implementation strategy) to facilitate uptake of evidence-based guidelines (the intervention) for management of severe TBI. Group model building and systems dynamics will inform the guideline engine design, and bedside functionality will be initially assessed through patient simulation. Using the Promoting Action on Research Implementation in Health Services (PARIHS) framework, we will determine the feasibility of incorporating the guideline engine in the ICU. Study participants will include pediatric patients with severe TBI and providers at three trauma centers. Quantitative data will include measures of guideline engine acceptance and organizational readiness for change. Qualitative data will include semi-structured interviews from clinicians. We will test the feasibility of incorporating the guideline engine in "real life practice" in preparation for a future clinical trial that will assess clinical and implementation outcomes, including feasibility, acceptability, and adoption of the guideline engine.

DISCUSSION

This study will lead to the development and feasibility testing of an adaptable strategy for implementing guideline-based care for severe TBI, a strategy that meets the needs of individual critical care environments and patients. A future study will test the adaptability and impact of the bedside guideline engine in a randomized clinical trial.

Conceptualizing and measuring sustainability of prevention programs, policies, and practices

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what "sustainability" means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

Conceptualizing and measuring sustainability of prevention programs, policies, and practices

A large knowledge gap exists regarding the measurement of sustainability of evidence-based prevention programs for mental and behavioral health. We interviewed 45 representatives of 10 grantees and 9 program officers within 4 Substance Abuse and Mental Health Services Administration prevention grant initiatives to identify experiences with implementation and sustainability barriers and facilitators; what "sustainability" means and what it will take to sustain their programs; and which Consolidated Framework for Implementation Research (CFIR) elements are important for sustainability. Lists of sustainability determinants and outcomes were then compiled from each data set and compared with one another. Analysis of themes from interviews and free lists revealed considerable overlap between sustainability determinants and outcomes. Four sustainability elements were identified by all three data sets (ongoing coalitions, collaborations, and networks and partnerships; infrastructure and capacity to support sustainability; community need for program; and ongoing evaluation of performance and outcomes), and 11 elements were identified by two of three data sets (availability of funding; consistency with organizational culture; evidence of positive outcomes; development of a plan for implementation and sustainment; presence of a champion; institutionalization and integration of program; institutional support and commitment; community buy-in and support; program continuity; supportive leadership; and opportunities for staff training). All but one of the CFIR domain elements (pressure from other states, tribes, or communities) were endorsed as important to sustainability by 50% or more of participants. It may be more important to implement a standardized process of eliciting determinants and outcomes of sustainability than to implement a single standardized instrument.

A Longitudinal Mixed Method Approach for Assessing Implementation Context and Process Factors: Comparison of Three Sites from a Housing First Implementation Strategy Pilot

BACKGROUND

Implementation science's focus on establishing implementation strategy effectiveness has overshadowed the need to understand differential performance of such strategies under various conditions. Methods allowing for assessment between implementation context and process can help address this gap. This paper provides a detailed description of a mixed method procedure for assessing factors related to the implementation context and process intersection, which was developed as part of the pilot study of the Housing First Technical Assistance and Training (HFTAT) Program, a multifaceted strategy designed to support Housing First model implementation.

METHODS

The HFTAT was pilot tested among a sample of three organizations. Our mixed method approach combines two tools often used in implementation research-the Stages of Implementation Completion and the Consolidated Framework for Implementation Research-in a novel way. Several stages to analysis were completed, starting with a separate analysis of data pertaining to each measure and then two levels of mixed method analysis.

RESULTS

The approach provided a better understanding of the issues that impacted the implementation guided by the HFTAT, suggesting: (1) individual determinants seemed to have a bigger impact based on the number of SIC phases they affected, (2) implementation context and process were connected through climate-related factors in the inner setting that made the sites more or less responsive to addressing identified barriers, and (3) there is a need to better assess context factors to identify areas where implementation drivers should be better targeted to facilitate change, and this is supported by prior research.

CONCLUSIONS

Understanding the underlying factors impacting a setting's performance related to a specific implementation strategy has potential to improve decision-making and optimize future implementation efforts. The approach likely be as successful combining the SIC with other determinant frameworks and should be utilized at the onset of an implementation project to maximize its usefulness.

Implementing risk stratification to the treatment of adolescent substance use among youth involved in the juvenile justice system: protocol of a hybrid type I trial

BACKGROUND

Youth involved in the juvenile justice system (YIJJ) have high rates of substance use problems; however, rates of YIJJ engagement in substance use services is low. Barriers to service engagement include lack of appropriate screening and connection to services by the juvenile justice system, as well as lack of resources for delivering evidence-based treatment in community-based settings. To address these barriers, this paper describes a protocol for a type 1 hybrid design to (1) implement universal substance use screening for YIJJ; (2) implement and evaluate the feasibility and effectiveness of a brief, three-session substance use interventions based in motivational interviewing for youth with mild/moderate substance use: Teen Intervene (an individual-based intervention); (3) implement ENCOMPASS, an evidence-based substance use intervention based in motivational enhancement and cognitive behavioral therapy for youth with severe substance use; and (4) evaluate facilitators and barriers to implementing these interventions for mild to severe substance use among YIJJ in community mental health centers (CMHC).

METHODS/DESIGN

Using a hybrid type 1 clinical effectiveness-implementation design, we will collaborate with CMHCs and juvenile justice in two rural Indiana counties. Guided by the EPIS (exploration, preparation, implementation, sustainability) framework, we will measure factors that affect implementation of substance use screening in juvenile justice and implementation of substance use interventions in CMHCs utilizing self-reports and qualitative interviews with juvenile justice and CMHC staff pre- and post-implementation. YIJJ with mild/moderate substance use will receive a brief interventions and YIJJ with severe substance use will receive ENCOMPASS. We will measure the effectiveness of a brief and comprehensive intervention by assessing changes in substance use across treatment. We anticipate recruiting 160 YIJJ and their caregivers into the study. We will assess intervention outcomes utilizing baseline, 3-, and 6-month assessments.

DISCUSSION

Findings have the potential to improve screening and intervention services for YIJJ.

Testing an implementation strategy bundle on adoption and sustainability of evidence to optimize physical function in community-dwelling disabled and older adults in a Medicaid waiver: a multi-site pragmatic hybrid type III protocol

Background

In partnership with a state Medicaid home and community-based waiver program, this study tests implementation strategies for adoption and sustainability of an evidence-based intervention to support disabled and older adults who have difficulty with physical function and daily living tasks. A multi-level implementation strategy bundle will be directed at relationship, coalition, and team building; readiness to implement, leadership, and clinician attitude toward evidence assessments; intervention and facilitation training; interdisciplinary coordination; facilitation; and audit and feedback to support practice change.

Methods

Knowledge-to-Action model underpins this 2-arm, 3-year pragmatic mixed method randomized hybrid type III trial in 18 waiver program sites in Michigan. Data will be collected on sites, 775 clinicians (registered nurses, occupational therapists, social workers), and 15,000 disabled and older adults. Consolidated Framework for Implementation Research guides examination of site, clinician, and beneficiary characteristics; clinician attitude and self-efficacy; leadership and readiness to implement; and intervention impact on beneficiary outcomes. Sites will be randomized to either usual waiver care with internal facilitation of the bundle of implementation strategies or usual waiver care with both internal and external facilitation of the bundle. Primary outcomes are site-level adoption and sustainability over 12 months, and intervention effects on these outcomes are hypothesized to be mediated by clinicians’ attitude and self-efficacy. At the beneficiary level, by addressing the individual’s capabilities and home environment, the intervention is hypothesized to improve secondary outcomes of activities of daily living, pain, depression, falls, emergency department visits, and hospitalizations. Baseline site readiness and leadership and stages of implementation at 6 months will be explored as potential moderators. Linear mixed effects models will be used to test intervention effects on primary outcomes, with bias-correcting analytic strategy in mediation analyses. Generalized linear mixed effects modeling will be employed for the analysis of intervention effects on secondary outcomes.

Discussion

Synthesizing findings within and across the sites, we will specify how leadership, readiness for change, and level of facilitation enhance capacity for adoption and sustainability of an evidence-based intervention in an under-resourced Medicaid setting that cares for disabled and older adults.

Trial registration

ClinitalTrials.gov, NCT03634033. Registered 16 August 2018.

Electronic supplementary material

The online version of this article (10.1186/s13012-019-0907-1) contains supplementary material, which is available to authorized users.

Costs of implementing and sustaining enhanced collaborative care programs involving community partners

BACKGROUND

Collaborative care is an evidence-based program for treating depression in primary care. We sought to expand this model by recruiting clinics interested in incorporating community partners (i.e., community-based organizations (CBO) and/or family members) in the care team. Seven sites implemented evidence-based collaborative care programs with community partners while collecting information on costs of implementing and sustaining programs.

METHODS

Sites retrospectively collected data on planning and implementation costs with technical assistance from study researchers. Sites also prospectively collected cost of care activities over a 1-month period once the program was implemented to determine resources needed to sustain programs. Personnel salary costs were adjusted, adding 30% for benefits and 30% for administrative overhead.

RESULTS

The programs implemented varied considerably in staffing, involvement of care partners, and allocation of costs. Total planning and implementation costs varied from $39,280 to $60,575. The largest implementation cost category involved workflow development and ranged from $16,325 to $31,375 with the highest costs in this category attributed to the most successful implementation among clinic-CBO programs. Following implementation, cost per patient over the 1-month period ranged from $154 to $544. Ongoing strategic decision-making and administrative costs, which were included in cost of care, ranged from $284 to $2328 for the month.

CONCLUSIONS

Sites implemented collaborative care through differing partnerships, staffing, and related costs. Costs to implement and sustain programs developed in partnership are often not collected but are crucial to understanding financial aspects of developing sustainable partnerships. Assessing such costs is feasible and can inform future partnership efforts.

Effects of an evidence-based practice education program using multifaceted interventions: a quasi-experimental study with undergraduate nursing students

BACKGROUND

Although Evidence-Based Practice (EBP) should be introduced early on in nursing education to develop students' independence and self-learning ability, there are few such courses for undergraduate nursing students in Korea. This study examined the effects of the EBP education program for undergraduate nursing students (EBP-EPUNS) on nursing students' knowledge, skills, attitudes, competencies, and future use of EBP.

METHODS

A quasi-experimental study design with pre-test, intervention, and post-test was used. The participants were 44 nursing students (experimental: 22, control: 22). A 20-h long EBP-EPUNS consisting of 5-step EBP components was provided through 8 sessions spread across 4 weeks.

RESULTS

An independent t-test and a repeated-measures ANOVA showed that the experimental group had statistically significant higher post-test scores on EBP knowledge (p < 0.001), skills (p < 0.001), attitudes (p < 0.001), competencies (p < 0.001), future use of EBP (p = 0.001), and critical thinking (p < 0.001), compared to the control group.

CONCLUSION

The EBP education program was effective in improving the knowledge, skills, attitudes, competencies, and future use of EBP among nursing students. Hence, we recommend the EBP education program as a general education course for undergraduate nursing students to promote needed proficiency in EBP.

The implementation of a team training intervention for school mental health: Lessons learned

Children obtain more mental health services through schools than through any other system. In urban, low-resource schools, mental health care often is provided by teams of contracted community mental health workers. Implementation of intended services may struggle in the context of challenges related to team functioning. Team Strategies and Tools to Enhance Performance and Patient Safety (TeamSTEPPS) is an efficacious approach for improving team functioning in health care. In collaboration with stakeholders, we adapted TeamSTEPPS for school mental health teams and pilot-tested it in 3 schools participating in an ongoing implementation of cognitive-behavioral therapy. In total, 3 teams randomized to receive TeamSTEPPS were compared with 3 teams who did not participate in TeamSTEPPS. Feasibility and acceptability of the adapted TeamSTEPPS and the impact on team skills and behavior were assessed through qualitative interviews and field notes and quantitatively over the course of 1 school year. In this article, we describe the process of adapting and implementing TeamSTEPPS. In addition to providing the researchers' perspective, we illustrate participant perspectives using qualitative data when possible. Key challenges included leader and staff turnover, logistical barriers (e.g., difficulty securing private space for qualitative interviews in schools), and navigating the protection of participant rights and autonomy given that prospective participants were employed by an agency with a vested interest in their participation. Concrete suggestions for overcoming challenges are provided to guide future research. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

A Mixed Methods Study of the Stages of Implementation for an Evidence-Based Trauma Intervention in Schools

A mixed methods study was conducted to examine the implementation process of 26 urban school-based mental health clinics that took part in a training and implementation support program for an evidence-based school trauma intervention. Implementation process was observed using the Stages of Implementation Completion (SIC) measure. Qualitative interviews were conducted with clinic leaders in order to gain insight into clinic processes related to the SIC. Results showed that almost all of the clinics engaged in some activities related to pre-implementation (engagement, feasibility, and readiness), but only 31% of the sites formally started delivering the program to youth. Completing more pre-implementation activities, particularly those related to readiness, predicted program start-up. Qualitative analysis comparing those that implemented the program to those that did not revealed critical differences in decision-making processes, leadership strategies, and the presence of local champions for the program. This study documented the patterns of clinic behavior that occurs as part of large-scale training efforts, suggests some unique challenges that occur in schools, and highlights the importance of engaging in particular implementation activities (i.e., readiness planning, stakeholder consensus and planning meetings) as part of program start-up. Findings indicate that pre-implementation and readiness-related consultation should be employed as part of broad-scale implementation and training efforts.

Development and testing of the Measure of Innovation-Specific Implementation Intentions (MISII) using Rasch measurement theory

Background

Implementation is proposed to be a multiphase, multilevel process. After a period of exploration, an adoption decision is made, typically at the upper management or policy level. Nevertheless, movement through each of the subsequent phases of the implementation process involves clinicians or providers at the individual level to adopt the innovation and then change their behavior to use/deliver the innovation. Multiple behavioral change theories propose that intentions are a critical determinant of implementation behavior. However, there is a need for the development and testing of pragmatic measures of providers’ intentions to use a specific innovation or evidence-based practice (EBP).

Methods

Nine items were developed to assess providers’ intentions to use a specific innovation or EBP. Motivational interviewing was the EBP in the study. Items were administered, as part of larger survey, to 179 providers across 38 substance use disorder treatment (SUDT) programs within five agencies in California, USA. Rasch analysis was conducted using RUMM2030 software to assess the items, their overall fit to the Rasch model, the response scale used, individual item fit, differential item functioning (DIF), and person separation.

Results

Following a stepwise process, the scale was reduced from nine items to three items to increase the feasibility and acceptability of the scale while maintaining suitable psychometric properties. The three-item unidimensional scale showed good person separation (PSI = .872), no disordering of thresholds, and no evidence of uniform or non-uniform DIF. Rasch analysis supported the viability of the scale as a measure of implementation intentions.

Conclusions

The Measure of Innovation-Specific Implementation Intentions (MISII) is a sound measure of providers’ intentions to use a specific innovation or EBP. Future evaluation of convergent, divergent, and predictive validity are needed. The study also demonstrates the value of Rasch analysis for testing the psychometric properties of pragmatic implementation measures.

Overcoming challenges to dissemination and implementation of research findings in under-resourced countries

Louis Pasteur once commented on the happiness that a scientist finds when, besides making a discovery, study results find practical application. Where health status is poor and resources are limited, finding such applications is a necessity, not merely a joy.

Dissemination, or the distribution of new knowledge gained through research, is essential to the ethical conduct of research. Further, when research is designed to improve health, dissemination is critical to the development of evidence-based medicine and the adoption of evidence-supported interventions and improved practice patterns within specific settings. When dissemination is lacking, research may be considered a waste of resources and a useless pursuit unable to influence positive health outcomes.

Effective translation of the findings of health research into policy and the practice of medicine has been slow in many countries considered low or lower middle-income (as defined by the World Bank). This is because such countries often have health care systems that are under-resourced (e.g., lacking personnel or facilities) and thus insufficiently responsive to health needs of their populations. However, implementation research has produced many tools and strategies that can prompt more effective and timelier application of research findings to real world situations.

A conscientious researcher can find many suggestions for improving the integration of research evidence into practice. First and foremost, the truthful reporting of results is emphasized as essential because both studies with desirable findings as well those with less than ideal results can provide new and valuable knowledge. Consideration in advance of the audience likely to be interested in study findings can result in suitable packaging and targeted communication of results. Other strategies for avoiding the barriers that can negatively impact implementation of research evidence include the early involvement of stakeholders as research is being designed and discussion before initiation of proposed research with those who will be affected by it. It is also important to recognize the role of education and training for ensuring the skills and knowledge needed for not only the conduct of high quality research but also for the meaningful promotion of results and application of research findings to achieve intended purposes.

Implementing an Evidence-Based Intervention for Children in Europe: Evaluating the Full-Transfer Approach

This study evaluated the implementation outcomes of GenerationPMTO, an evidence-based parenting intervention for child and adolescent behavior problems, in three European countries. The implementation approach was full transfer, in which purveyors train a first generation (G1) of practitioners; adopting sites assume oversight, training, certification, and fidelity assessment for subsequent generations (Forgatch & DeGarmo, 2011; Forgatch & Gewirtz, 2017). Three hundred therapists participated in trainings in GenerationPMTO in Iceland, Denmark, and the Netherlands. Data are from the implementation's initiation in each country through 2016, resulting in 6 generations in Iceland, 8 in Denmark, and 4 in the Netherlands. Therapist fidelity was measured at certification with an observation-based tool, the Fidelity of Implementation Rating System (Knutson, Forgatch, Rains, & Sigmarsdóttir, 2009). Candidates in all generations achieved fidelity scores at or above the required standard. Certification fidelity scores were evaluated for G1 candidates, who were trained by the purveyor, and subsequent generations trained by the adopting implementation site. In each country, certification fidelity scores declined for G2 candidates compared with G1 and recovered to G1 levels for subsequent generations, partially replicating findings from a previous Norwegian study (Forgatch & DeGarmo, 2011). Recovery to G1 levels of fidelity scores was obtained in Iceland and the Netherlands by G3; in Denmark, the recovery was obtained by G5. The mean percentage of certification in each country was more than 80%; approximately 70% of certified therapists remained active in 2017. Findings support full transfer as an effective implementation approach with long-term sustainability and fidelity.

Agency Leaders' Assessments of Feasibility and Desirability of Implementation of Evidence-Based Practices in Youth-Serving Organizations Using the Stages of Implementation Completion

Background: This study examined influences on the decisions of administrators of youth-serving organizations to initiate and proceed with implementation of an evidence-based practice (EBP).

Methods: Semi-structured interviews, developed using the Stages of Implementation Completion (SIC) as a framework, were conducted with 19 agency chief executive officers and program directors of 15 organizations serving children and adolescents.

Results: Agency leaders' self-assessments of implementation feasibility and desirability prior to implementation (Pre-implementation) were influenced by intervention affordability, feasibility, requirements, validity, reliability, relevance, cost savings, positive outcomes, and adequacy of information; availability of funding, support from sources external to the agency, and adequacy of technical assistance; and staff availability and attitudes toward innovation in general and EBPs in particular, organizational capacity, fit between the EBP and agency mission and capacity, prior experience with implementation, experience with seeking evidence, and developing consensus. Assessments during the Implementation phase included intervention flexibility and requirements; availability of funding, adequacy of training and technical assistance, and getting sufficient and appropriate referrals; and staffing and implementing with fidelity. Assessments during the Sustainment phase included intervention costs and benefits; availability of funding, support from sources outside of the agency, and need for the EBP; and the fit between the EBP and the agency mission.

Discussion: The results point to opportunities for using agency leader models to develop strategies to facilitate implementation of evidence-based and innovative practices for children and adolescents. The SIC provides a standardized framework for guiding agency leader self-assessments of implementation.

Measuring Costs to Community-Based Agencies for Implementation of an Evidence-Based Practice

Healthcare reform has led to an increase in dissemination of evidence-based practices. Cost is frequently cited as a significant yet rarely studied barrier to dissemination of evidence-based practices and the associated improvements in quality of care. This study describes an approach to measuring the incremental, unreimbursed costs in staff time and direct costs to community-based clinics implementing an evidence-based practice through participating in a learning collaborative. Initial implementation costs exceeding those for providing "treatment as usual" were collected for ten clinics implementing trauma-focused cognitive behavioral therapy through participation in 10-month learning collaboratives. Incremental implementation costs of these ten community-based clinic teams averaged the equivalent of US$89,575 (US$ 2012). The most costly activities were training, supervision, preparation time, and implementation team meetings. Recommendations are made for further research on implementation costs, dissemination of evidence-based practices, and implications for researchers and policy makers.

Using implementation tools to design and conduct quality improvement projects for faster and more effective improvement

Purpose

The purpose of this paper is to enable improvers to use recent knowledge from implementation science to carry out improvement changes more effectively. It also highlights the importance of converting research findings into practical tools and guidance for improvers so as to make research easier to apply in practice.

Design/methodology/approach

This study provides an illustration of how a quality improvement (QI) team project can make use of recent findings from implementation research so as to make their improvement changes more effective and sustainable. The guidance is based on a review and synthesis of improvement and implementation methods.

Findings

The paper illustrates how research can help a quality project team in the phases of problem definition and preparation, in design and planning, in implementation, and in sustaining and spreading a QI. Examples of the use of different ideas and methods are cited where they exist.

Research limitations/implications

The example is illustrative and there is little limited experimental evidence of whether using all the steps and tools in the one approach proposed do enable a quality team to be more effective. Evidence supporting individual guidance proposals is cited where it exists.

Practical implications

If the steps proposed and illustrated in the paper were followed, it is possible that quality projects could avoid waste by ensuring the conditions they need for success are in place, and sustain and spread improvement changes more effectively.

Social implications

More patients could benefit more quickly from more effective implementation of proven interventions.

Originality/value

The paper is the first to describe how improvement and implementation science can be combined in a tangible way that practical improvers can use in their projects. It shows how QI project teams can take advantage of recent advances in improvement and implementation science to make their work more effective and sustainable.

Standards for Reporting Implementation Studies (StaRI): explanation and elaboration document

OBJECTIVES

Implementation studies are often poorly reported and indexed, reducing their potential to inform the provision of healthcare services. The Standards for Reporting Implementation Studies (StaRI) initiative aims to develop guidelines for transparent and accurate reporting of implementation studies.

METHODS

An international working group developed the StaRI guideline informed by a systematic literature review and e-Delphi prioritisation exercise. Following a face-to-face meeting, the checklist was developed iteratively by email discussion and critical review by international experts.

RESULTS

The 27 items of the checklist are applicable to the broad range of study designs employed in implementation science. A key concept is the dual strands, represented as 2 columns in the checklist, describing, on the one hand, the implementation strategy and, on the other, the clinical, healthcare or public health intervention being implemented. This explanation and elaboration document details each of the items, explains the rationale and provides examples of good reporting practice.

CONCLUSIONS

Previously published reporting statements have been instrumental in improving reporting standards; adoption by journals and authors may achieve a similar improvement in the reporting of implementation strategies that will facilitate translation of effective interventions into routine practice.

An Overview of Research and Evaluation Designs for Dissemination and Implementation

The wide variety of dissemination and implementation designs now being used to evaluate and improve health systems and outcomes warrants review of the scope, features, and limitations of these designs. This article is one product of a design workgroup that was formed in 2013 by the National Institutes of Health to address dissemination and implementation research, and whose members represented diverse methodologic backgrounds, content focus areas, and health sectors. These experts integrated their collective knowledge on dissemination and implementation designs with searches of published evaluations strategies. This article emphasizes randomized and nonrandomized designs for the traditional translational research continuum or pipeline, which builds on existing efficacy and effectiveness trials to examine how one or more evidence-based clinical/prevention interventions are adopted, scaled up, and sustained in community or service delivery systems. We also mention other designs, including hybrid designs that combine effectiveness and implementation research, quality improvement designs for local knowledge, and designs that use simulation modeling.

Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: a cluster randomized trial study protocol

BACKGROUND

Evidence-based practice (EBP) implementation represents a strategic change in organizations that requires effective leadership and alignment of leadership and organizational support across organizational levels. As such, there is a need for combining leadership development with organizational strategies to support organizational climate conducive to EBP implementation. The leadership and organizational change for implementation (LOCI) intervention includes leadership training for workgroup leaders, ongoing implementation leadership coaching, 360° assessment, and strategic planning with top and middle management regarding how they can support workgroup leaders in developing a positive EBP implementation climate.

METHODS

This test of the LOCI intervention will take place in conjunction with the implementation of motivational interviewing (MI) in 60 substance use disorder treatment programs in California, USA. Participants will include agency executives, 60 program leaders, and approximately 360 treatment staff. LOCI will be tested using a multiple cohort, cluster randomized trial that randomizes workgroups (i.e., programs) within agency to either LOCI or a webinar leadership training control condition in three consecutive cohorts. The LOCI intervention is 12 months, and the webinar control intervention takes place in months 1, 5, and 8, for each cohort. Web-based surveys of staff and supervisors will be used to collect data on leadership, implementation climate, provider attitudes, and citizenship. Audio recordings of counseling sessions will be coded for MI fidelity. The unit of analysis will be the workgroup, randomized by site within agency and with care taken that co-located workgroups are assigned to the same condition to avoid contamination. Hierarchical linear modeling (HLM) will be used to analyze the data to account for the nested data structure.

DISCUSSION

LOCI has been developed to be a feasible and effective approach for organizations to create a positive climate and fertile context for EBP implementation. The approach seeks to cultivate and sustain both effective general and implementation leadership as well as organizational strategies and support that will remain after the study has ended. Development of a positive implementation climate for MI should result in more positive service provider attitudes and behaviors related to the use of MI and, ultimately, higher fidelity in the use of MI.

TRIAL REGISTRATION

This study is registered with Clinicaltrials.gov ( NCT03042832 ), 2 February 2017, retrospectively registered.

Tracking implementation strategies: a description of a practical approach and early findings

BACKGROUND

Published descriptions of implementation strategies often lack precision and consistency, limiting replicability and slowing accumulation of knowledge. Recent publication guidelines for implementation strategies call for improved description of the activities, dose, rationale and expected outcome(s) of strategies. However, capturing implementation strategies with this level of detail can be challenging, as responsibility for implementation is often diffuse and strategies may be flexibly applied as barriers and challenges emerge. We describe and demonstrate the development and application of a practical approach to identifying implementation strategies used in research and practice that could be used to guide their description and specification.

METHODS

An approach to tracking implementation strategies using activity logs completed by project personnel was developed to facilitate identification of discrete strategies. This approach was piloted in the context of a multi-component project to improve children's access to behavioural health services in a county-based child welfare agency. Key project personnel completed monthly activity logs that gathered data on strategies used over 17 months. Logs collected information about implementation activities, intent, duration and individuals involved. Using a consensus approach, two sets of coders categorised each activity based upon Powell et al.'s (Med Care Res Rev 69:123-57, 2012) taxonomy of implementation strategies.

RESULTS

Participants reported on 473 activities, which represent 45 unique strategies. Initial implementation was characterised by planning strategies followed by educational strategies. After project launch, quality management strategies predominated, suggesting a progression of implementation over time. Together, these strategies accounted for 1594 person-hours, many of which were reported by the leadership team that was responsible for project design, implementation and oversight.

CONCLUSIONS

This approach allows for identifying discrete implementation strategies used over time, estimating dose, describing temporal ordering of implementation strategies, and pinpointing the major implementation actors. This detail could facilitate clear reporting of a full range of implementation strategies, including those that may be less observable. This approach could lead to a more nuanced understanding of what it takes to implement different innovations, the types of strategies that are most useful during specific phases of implementation, and how implementation strategies need to be adaptively applied throughout the course of a given initiative.

Tracking implementation strategies: a description of a practical approach and early findings

BACKGROUND

Published descriptions of implementation strategies often lack precision and consistency, limiting replicability and slowing accumulation of knowledge. Recent publication guidelines for implementation strategies call for improved description of the activities, dose, rationale and expected outcome(s) of strategies. However, capturing implementation strategies with this level of detail can be challenging, as responsibility for implementation is often diffuse and strategies may be flexibly applied as barriers and challenges emerge. We describe and demonstrate the development and application of a practical approach to identifying implementation strategies used in research and practice that could be used to guide their description and specification.

METHODS

An approach to tracking implementation strategies using activity logs completed by project personnel was developed to facilitate identification of discrete strategies. This approach was piloted in the context of a multi-component project to improve children's access to behavioural health services in a county-based child welfare agency. Key project personnel completed monthly activity logs that gathered data on strategies used over 17 months. Logs collected information about implementation activities, intent, duration and individuals involved. Using a consensus approach, two sets of coders categorised each activity based upon Powell et al.'s (Med Care Res Rev 69:123-57, 2012) taxonomy of implementation strategies.

RESULTS

Participants reported on 473 activities, which represent 45 unique strategies. Initial implementation was characterised by planning strategies followed by educational strategies. After project launch, quality management strategies predominated, suggesting a progression of implementation over time. Together, these strategies accounted for 1594 person-hours, many of which were reported by the leadership team that was responsible for project design, implementation and oversight.

CONCLUSIONS

This approach allows for identifying discrete implementation strategies used over time, estimating dose, describing temporal ordering of implementation strategies, and pinpointing the major implementation actors. This detail could facilitate clear reporting of a full range of implementation strategies, including those that may be less observable. This approach could lead to a more nuanced understanding of what it takes to implement different innovations, the types of strategies that are most useful during specific phases of implementation, and how implementation strategies need to be adaptively applied throughout the course of a given initiative.

Pebbles, rocks, and boulders: The implementation of a school-based social engagement intervention for children with autism

Few evidence-based practices, defined as the use of empirically supported research and clinical expertise for children with autism, have been successfully implemented and sustained in schools. This study examined the perspectives of school personnel ( n = 39) on implementing a social engagement intervention for children with autism. Semi-structured interviews, informed by the Domitrovich et al. (2008) framework, were conducted. Participants were asked about (1) school factors that affect the general implementation of evidence-based practices, (2) their specific experiences implementing the social engagement intervention, and (3) barriers to and facilitators of implementing the social engagement intervention. Data were analyzed using an integrated approach. General (e.g. implementation process, leadership, support, and staff) and intervention-specific (e.g. staff, barriers, and facilitators) implementation themes were identified. These findings suggest that a variety of factors should be considered when implementing evidence-based practices in schools and that implementing social engagement interventions for children with autism may require additional specific support for implementation.

Automatic classification of communication logs into implementation stages via text analysis

BACKGROUND

To improve the quality, quantity, and speed of implementation, careful monitoring of the implementation process is required. However, some health organizations have such limited capacity to collect, organize, and synthesize information relevant to its decision to implement an evidence-based program, the preparation steps necessary for successful program adoption, the fidelity of program delivery, and the sustainment of this program over time. When a large health system implements an evidence-based program across multiple sites, a trained intermediary or broker may provide such monitoring and feedback, but this task is labor intensive and not easily scaled up for large numbers of sites. We present a novel approach to producing an automated system of monitoring implementation stage entrances and exits based on a computational analysis of communication log notes generated by implementation brokers. Potentially discriminating keywords are identified using the definitions of the stages and experts' coding of a portion of the log notes. A machine learning algorithm produces a decision rule to classify remaining, unclassified log notes.

RESULTS

We applied this procedure to log notes in the implementation trial of multidimensional treatment foster care in the California 40-county implementation trial (CAL-40) project, using the stages of implementation completion (SIC) measure. We found that a semi-supervised non-negative matrix factorization method accurately identified most stage transitions. Another computational model was built for determining the start and the end of each stage.

CONCLUSIONS

This automated system demonstrated feasibility in this proof of concept challenge. We provide suggestions on how such a system can be used to improve the speed, quality, quantity, and sustainment of implementation. The innovative methods presented here are not intended to replace the expertise and judgement of an expert rater already in place. Rather, these can be used when human monitoring and feedback is too expensive to use or maintain. These methods rely on digitized text that already exists or can be collected with minimal to no intrusiveness and can signal when additional attention or remediation is required during implementation. Thus, resources can be allocated according to need rather than universally applied, or worse, not applied at all due to their cost.

Effectiveness of Implementation of Electronic Malaria Information System as the National Malaria Surveillance System in Thailand

Background

In moving toward malaria elimination, one strategy is to implement an active surveillance system for effective case management. Thailand has developed and implemented the electronic Malaria Information System (eMIS) capturing individualized electronic records of suspected or confirmed malaria cases.

Objective

The main purpose of this study was to determine how well the eMIS improves the quality of Thailand’s malaria surveillance system. In particular, the focus of the study was to evaluate the effectiveness of the eMIS in terms of the system users’ perception and the system outcomes (ie, quality of data) regarding the management of malaria patients.

Methods

A mixed-methods technique was used with the framework based on system effectiveness attributes: data quality, timeliness, simplicity, acceptability, flexibility, stability, and usefulness. Three methods were utilized: data records review, survey of system users, and in-depth interviews with key stakeholders. From the two highest endemic provinces, paper forms matching electronic records of 4455 noninfected and 784 malaria-infected cases were reviewed. Web-based anonymous questionnaires were distributed to all 129 eMIS data entry staff throughout Thailand, and semistructured interviews were conducted with 12 management-level officers.

Results

The eMIS is well accepted by system users at both management and operational levels. The data quality has enabled malaria personnel to perform more effective prevention and control activities. There is evidence of practices resulting in inconsistencies and logical errors in data reporting. Critical data elements were mostly completed, except for a few related to certain dates and area classifications. Timeliness in reporting a case to the system was acceptable with a delay of 3-4 days. The evaluation of quantitative and qualitative data confirmed that the eMIS has high levels of simplicity, acceptability, stability, and flexibility.

Conclusions

Overall, the system implemented has achieved its objective. The results of the study suggested that the eMIS helps improve the quality of Thailand’s malaria surveillance system. As the national malaria surveillance system, the eMIS’s functionalities have provided the malaria staff working at the point of care with close-to-real-time case management data quality, covering case detection, case investigation, drug compliance, and follow-up visits. Such features has led to an improvement in the quality of the malaria control program; the government officials now have quicker access to both individual and aggregated data to promptly react to possible outbreak. The eMIS thus plays one of the key roles in moving toward the national goal of malaria elimination by the next decade.

Implementation and evaluation of linked parenting models in a large urban child welfare system

During the past decade, there have been increased efforts to implement evidence-based practices into child welfare systems to improve outcomes for children in foster care and their families. In this paper, the implementation and evaluation of a policy-driven large system-initiated reform is described. Over 250 caseworkers and supervisors were trained and supported to implement two evidence-based parent focused interventions in five private agencies serving over 2,000 children and families. At the request of child welfare system leaders, a third intervention was developed and implemented to train the social work workforce to use evidence-based principles in everyday interactions with caregivers (including foster, relative, adoptive, and biological parents). In this paper, we describe the policy context and the targeted outcomes of the reform. We discuss the theory of the interventions and the logistics of how they were linked to create consistency and synergy. Training and ongoing consultation strategies used are described as are some of the barriers and opportunities that arose during the implementation. The strategy for creating a path to sustainability is also discussed. The reform effort was evaluated using both qualitative and quantitative methods; the evaluation design, research questions and preliminary results are provided.

Evaluation of two implementation strategies in 51 child county public service systems in two states: results of a cluster randomized head-to-head implementation trial

Background

Much is to be learned about what implementation strategies are the most beneficial to communities attempting to adopt evidence-based practices. This paper presents outcomes from a randomized implementation trial of Multidimensional Treatment Foster Care (MTFC) in child public service systems in California and Ohio, including child welfare, juvenile justice, and mental health.

Methods

Fifty-one counties were assigned randomly to one of two different implementation strategies (Community Development Teams (CDT) or independent county implementation strategy (IND)) across four cohorts after being matched on county characteristics. We compared these two strategies on implementation process, quality, and milestone achievements using the Stages of Implementation Completion (SIC) (Implement Sci 6(1):1–8, 2011).

Results

A composite score for each county, combining the final implementation stage attained, the number of families served, and quality of implementation, was used as the primary outcome. No significant difference between CDT and IND was found for the composite measure. Additional analyses showed that there was no evidence that CDT increased the proportion of counties that started-up programs (i.e., placed at least one family in MTFC). For counties that did implement MTFC, those in the CDT condition served over twice as many youth during the study period as did IND. Of the counties that successfully achieved program start-up, those in the CDT condition completed the implementation process more thoroughly, as measured by the SIC. We found no significant differences by implementation condition on the time it took for first placement, achieving competency, or number of stages completed.

Conclusions

This trial did not lead to higher rates of implementation or faster implementation but did provide evidence for more robust implementation in the CDT condition compared to IND implementation once the first family received MTFC services. This trial was successful from a design perspective in that no counties dropped out, even though this study took place during an economic recession. We believe that this methodologic approach of measurement utilizing the SIC, which is comprised of the three dimensions of quality, quantity, and timing, is appropriate for a wide range of implementation and translational studies.

Trial registration

Trial ID: NCT00880126 (ClinicalTrials.gov).

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-014-0134-8) contains supplementary material, which is available to authorized users.