Sustaining an Evidence-Based Program Over Time: Moderators of Sustainability and the Role of the Getting to Outcomes® Implementation Support Intervention

Adaptive Designs in Implementation Science and Practice: Their Promise and the Need for Greater Understanding and Improved Communication

The promise of adaptation and adaptive designs in implementation science has been hindered by the lack of clarity and precision in defining what it means to adapt, especially regarding the distinction between adaptive study designs and adaptive implementation strategies. To ensure a common language for science and practice, authors reviewed the implementation science literature and found that the term adaptive was used to describe interventions, implementation strategies, and trial designs. To provide clarity and offer recommendations for reporting and strengthening study design, we propose a taxonomy that describes fixed versus adaptive implementation strategies and implementation trial designs. To improve impact, (a) futureimplementation studies should prespecify implementation strategy core functions that in turn can be taught to and replicated by health system/community partners, (b) funders should support exploratory studies that refine and specify implementation strategies, and (c) investigators should systematically address design requirements and ethical considerations (e.g., randomization, blinding/masking) with health system/community partners.

Interventions to prevent or cease electronic cigarette use in children and adolescents

BACKGROUND

The prevalence of e-cigarette use has increased globally amongst children and adolescents in recent years. In response to the increasing prevalence and emerging evidence about the potential harms of e-cigarettes in children and adolescents, leading public health organisations have called for approaches to address increasing e-cigarette use. Whilst evaluations of approaches to reduce uptake and use regularly appear in the literature, the collective long-term benefit of these is currently unclear.

OBJECTIVES

The co-primary objectives of the review were to: (1) evaluate the effectiveness of interventions to prevent e-cigarette use in children and adolescents (aged 19 years and younger) with no prior use, relative to no intervention, waitlist control, usual practice, or an alternative intervention; and (2) evaluate the effectiveness of interventions to cease e-cigarette use in children and adolescents (aged 19 years and younger) reporting current use, relative to no intervention, waitlist control, usual practice, or an alternative intervention. Secondary objectives were to: (1) examine the effect of such interventions on child and adolescent use of other tobacco products (e.g. cigarettes, cigars types, and chewing tobacco); and (2) describe the unintended adverse effects of the intervention on individuals (e.g. physical or mental health of individuals), or on organisations (e.g. intervention displacement of key curricula or learning opportunities for school students) where such interventions are being implemented.

SEARCH METHODS

We searched CENTRAL, Ovid MEDLINE, Ovid Embase, Ovid PsycINFO, EBSCO CINAHL, and Clarivate Web of Science Core Collection from inception to 1 May 2023. Additionally, we searched two trial registry platforms (WHO International Clinical Trials Registry Platform; US National Institutes of Health Ongoing Trials Register ClinicalTrials.gov), Google Scholar, and the reference lists of relevant systematic reviews. We contacted corresponding authors of articles identified as ongoing studies.

SELECTION CRITERIA

We included randomised controlled trials (RCTs), including cluster-RCTs, factorial RCTs, and stepped-wedge RCTs. To be eligible, the primary targets of the interventions must have been children and adolescents aged 19 years or younger. Interventions could have been conducted in any setting, including community, school, health services, or the home, and must have sought to influence children or adolescent (or both) e-cigarette use directly. Studies with a comparator of no intervention (i.e. control), waitlist control, usual practice, or an alternative intervention not targeting e-cigarette use were eligible. We included measures to assess the effectiveness of interventions to: prevent child and adolescent e-cigarette use (including measures of e-cigarette use amongst those who were never-users); and cease e-cigarette use (including measures of e-cigarette use amongst children and adolescents who were e-cigarette current-users). Measures of e-cigarette use included current-use (defined as use in the past 30 days) and ever-use (defined as any lifetime use).

DATA COLLECTION AND ANALYSIS

Two review authors independently screened the titles and abstracts of references, with any discrepancies resolved through consensus. Pairs of review authors independently assessed the full-text articles for inclusion in the review. We planned for two review authors to independently extract information from the included studies and assess risk of bias using the Cochrane RoB 2 tool. We planned to conduct multiple meta-analyses using a random-effects model to align with the co-primary objectives of the review. First, we planned to pool interventions to prevent child and adolescent e-cigarette use and conduct two analyses using the outcome measures of 'ever-use' and 'current-use'. Second, we planned to pool interventions to cease child and adolescent e-cigarette use and conduct one analysis using the outcome measure of 'current-use'. Where data were unsuitable for pooling in meta-analyses, we planned to conduct a narrative synthesis using vote-counting approaches and to follow the Cochrane Handbook for Systematic Reviews of Interventions and the Synthesis Without Meta-analysis (SWiM) guidelines.

MAIN RESULTS

The search of electronic databases identified 7141 citations, with a further 287 records identified from the search of trial registries and Google Scholar. Of the 110 studies (116 records) evaluated in full text, we considered 88 to be ineligible for inclusion for the following reasons: inappropriate outcome (27 studies); intervention (12 studies); study design (31 studies); and participants (18 studies). The remaining 22 studies (28 records) were identified as ongoing studies that may be eligible for inclusion in a future review update. We identified no studies with published data that were eligible for inclusion in the review.

AUTHORS' CONCLUSIONS

We identified no RCTs that met the inclusion criteria for the review, and as such, there is no evidence available from RCTs to assess the potential impact of interventions targeting children and adolescent e-cigarette use, tobacco use, or any unintended adverse effects. Evidence from studies employing other trial designs (e.g. non-randomised) may exist; however, such studies were not eligible for inclusion in the review. Evidence from studies using non-randomised designs should be examined to guide actions to prevent or cease e-cigarette use. This is a living systematic review. We search for new evidence every month and update the review when we identify relevant new evidence. Please refer to the Cochrane Database of Systematic Reviews for the current status of this review.

Improving Sexual Assault and Sexual Harassment Prevention from the Bottom-up: a Pilot of Getting To Outcomes in the US Military

While the Department of Defense (DoD) has given increased attention and priority to preventing sexual assault and sexual harassment (SA/SH), it remains a problem. To build its prevention capacity, DoD piloted Getting To Outcomes® (GTO®) from 2019 to 2022 at 10 military installations. GTO is an evidence-based planning and implementation support that has been used in many civilian contexts but has only recently been adapted for military SA/SH. The purpose of this study was to describe GTO use, identify its benefits and challenges, and discuss lessons the GTO effort yielded for prevention more broadly using a framework of organizational and program-level capacities needed for successful prevention in the military context, called the Prevention Evaluation Framework (PEF). GTO was piloted with 10 military installations ("sites") representing all Military Services, plus the Coast Guard and National Guard. GTO is comprised of a written guide, training, and ongoing coaching. The pilot's goal was for each site to use GTO to implement a SA/SH prevention program twice. Participants from each site were interviewed and data was collected on GTO steps completed, whether GTO spurred new evaluation activities and collaborations, and the degree of leadership support for GTO. Most sites completed all GTO steps at least once. Interviews showed that DoD participants believe GTO improved prevention understanding, planning, and evaluation capacity; strengthened confidence in chosen programs; and helped sites tailor programs to the military context. Barriers were the complexity of GTO, DoD personnel turnover, and the disruption that the COVID pandemic caused in sexual assault prevention program delivery. Many respondents were unsure if they would continue all of GTO after the coaching ended, but many believed they would continue at least some parts. According to the PEF, the GTO pilot revealed several additional prevention system gaps (e.g., need for leadership support) and changes needed to GTO (e.g., stronger leader and champion engagement), to support quality prevention. The military and other large organizations will need to focus on these issues to ensure prevention implementation and evaluation are conducted with quality.

Developing a sustainability readiness strategy for health systems: Toolkit, interactive tools, and virtual support system

While the literature strongly supports the need for sustainability of evidence-based interventions (EBIs), we present a review of the literature that indicates only three articles discuss a health-focused sustainability strategy. The aims of our sustainability readiness strategy (SRS) are to increase infrastructure capacity and EBI advocacy to impact the level of sustainability readiness. In this article, we describe the development of an evidence-informed promising practice sustainability readiness strategy (SRS) with three evidence-based components. This strategy: 1) is based on an adaptation of the Getting To Outcomes® (GTO) evidence-based implementation process, 2) includes a logic model with documented evidence of the connection between targeted readiness factors and sustainability outcomes, and 3) describes resources considered necessary to support implementation of the readiness strategy, namely a step-by-step Toolkit, Excel™ Tools, webinar coaching and evaluation guides, and a coaching and evaluation training guide. The national SRS survey results are presented. Lessons learned and future dissemination and implementation plans are described.

Getting to implementation: Adaptation of an implementation playbook

Introduction

Implementation strategies supporting the translation of evidence into practice need to be tailored and adapted for maximum effectiveness, yet the field of adapting implementation strategies remains nascent. We aimed to adapt "Getting To Outcomes"® (GTO), a 10-step implementation playbook designed to help community-based organizations plan and evaluate behavioral health programs, into "Getting To Implementation" (GTI) to support the selection, tailoring, and use of implementation strategies in health care settings.

Methods

Our embedded evaluation team partnered with operations, external facilitators, and site implementers to employ participatory methods to co-design and adapt GTO for Veterans Health Administration (VA) outpatient cirrhosis care improvement. The Framework for Reporting Adaptations and Modifications to Evidenced-based Implementation Strategies (FRAME-IS) guided documentation and analysis of changes made pre- and post-implementation of GTI at 12 VA medical centers. Data from multiple sources (interviews, observation, content analysis, and fidelity tracking) were triangulated and analyzed using rapid techniques over a 3-year period.

Results

Adaptations during pre-implementation were planned, proactive, and focused on context and content to improve acceptability, appropriateness, and feasibility of the GTI playbook. Modifications during and after implementation were unplanned and reactive, concentrating on adoption, fidelity, and sustainability. All changes were collaboratively developed, fidelity consistent at the level of the facilitator and/or implementer.

Conclusion

GTO was initially adapted to GTI to support health care teams' selection and use of implementation strategies for improving guideline-concordant medical care. GTI required ongoing modification, particularly in steps regarding team building, context assessment, strategy selection, and sustainability due to difficulties with step clarity and progression. This work also highlights the challenges in pragmatic approaches to collecting and synthesizing implementation, fidelity, and adaptation data.

Trial registration

This study was registered on ClinicalTrials.gov (Identifier: NCT04178096).

Scaling Up Implementation Support for Violence Prevention and Resilience Promotion in the Air Force

Closing the gap between research and practice requires that organizations can consistently incorporate new ideas and best practices. The Department of the Air Force (DAF) Integrated Resilience Directorate is leading a coordinated effort to increase the adoption of evidence-based violence prevention and resilience promotion programs across the entire Department. To support this effort, DAF is using Getting To Outcomes (GTO), an evidence-based implementation support that helps organizations plan, implement, and self-evaluate programs. Thus, the DAF is not only scaling up prevention programs but is also embarking on the largest scale-up of GTO to date. The study team trained personnel from every Air Force installation across the world to use GTO for their programs. Quantitative findings from training questionnaires and ratings of implementation plans as well as qualitative results from resilience personnel interviews suggest some lessons learned for scaling up implementation support. This study builds on established implementation science frameworks for scaling up interventions by identifying critical tasks and unique supports needed to scale up evidence-based prevention. Results suggest GTO helped DAF scale-up prevention across the Department, and that establishing leadership buy-in, simplifying evidence-based program selection and adaptation, monitoring implementation and outcomes, and creating dedicated prevention practitioner roles are critical tasks to support scale-up of evidence-based prevention. Unique supports needed to scale-up evidence-based prevention include multitiered learning systems; integrated tools that improve access to data and evolving evidence; prevention personnel with skills in program management, adaptation, and evaluation; timely crosscutting data; continuous learning to support sustainability; and leadership buy-in.

Do the Expert Recommendations for Implementing Change (ERIC) strategies adequately address sustainment?

Background

Sustainability science is an emerging area within implementation science. There is limited evidence regarding strategies to best support the continued delivery and sustained impact of evidence-based interventions (EBIs). To build such evidence, clear definitions, and ways to operationalize strategies specific and/or relevant to sustainment are required. Taxonomies and compilations such as the Expert Recommendations for Implementing Change (ERIC) were developed to describe and organize implementation strategies. This study aimed to adapt, refine, and extend the ERIC compilation to incorporate an explicit focus on sustainment. We also sought to classify the specific phase(s) of implementation when the ERIC strategies could be considered and applied.

Methods

We used a two-phase iterative approach to adapt the ERIC. This involved: (1) adapting through consensus (ERIC strategies were mapped against barriers to sustainment as identified via the literature to identify if existing implementation strategies were sufficient to address sustainment, needed wording changes, or if new strategies were required) and; (2) preliminary application of this sustainment-explicit ERIC glossary (strategies described in published sustainment interventions were coded against the glossary to identify if any further amendments were needed). All team members independently reviewed changes and provided feedback for subsequent iterations until consensus was reached. Following this, and utilizing the same consensus process, the Exploration, Preparation, Implementation and Sustainment (EPIS) Framework was applied to identify when each strategy may be best employed across phases.

Results

Surface level changes were made to the definitions of 41 of the 73 ERIC strategies to explicitly address sustainment. Four additional strategies received deeper changes in their definitions. One new strategy was identified: Communicate with stakeholders the continued impact of the evidence-based practice. Application of the EPIS identified that at least three-quarters of strategies should be considered during preparation and implementation phases as they are likely to impact sustainment.

Conclusion

A sustainment-explicit ERIC glossary is provided to help researchers and practitioners develop, test, or apply strategies to improve the sustainment of EBIs in real-world settings. Whilst most ERIC strategies only needed minor changes, their impact on sustainment needs to be tested empirically which may require significant refinement or additions in the future.

Getting To Implementation (GTI)-Teach: A seven-step approach for teaching the fundamentals of implementation science

Introduction:

Implementation Science (IS) is a complex and rapidly evolving discipline, posing challenges for educators. We developed, implemented, and evaluated a novel, pragmatic approach to teach IS.

Methods:

Getting To Implementation (GTI)-Teach was developed as a seven-step educational model to guide students through the process of developing, conducting, and sustaining an IS research project. During the four-week online course, students applied the steps to self-selected implementation problems. Students were invited to complete two online post-course surveys to assess course satisfaction and self-reported changes in IS knowledge and relevance of GTI-Teach Steps to their work. Results were summarized using descriptive statistics; self-reported post-course changes in IS knowledge were compared using paired t-tests.

Results:

GTI-Teach was developed to include seven Steps: 1. Define the implementation problem; 2. Conceptualize the problem; 3. Prioritize implementation barriers and facilitators; 4. Select and tailor implementation strategies; 5. Design an implementation study; 6. Evaluate implementation; 7. Sustain implementation. Thirteen students, ranging in experience from medical students to full professors, enrolled in and completed the first GTI-Teach course. Of the seven students (54%) completing an end-of course survey, six (86%) were very satisfied with the course. Ten students (77%) responded to the tailored, 6-month post-course follow-up survey. They retrospectively reported a significant increase in their knowledge across all steps of GTI-Teach (1.3–1.8 points on a 5-point Likert scale) and rated each of the Steps as highly relevant to their work.

Conclusions:

GTI-Teach is a seven-step model for teaching IS fundamentals that students reported increased their knowledge and was relevant to their work.