RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review

Implementation of a Transitional Care Model for Stroke: Perspectives From Frontline Clinicians, Administrators, and COMPASS-TC Implementation Staff

BACKGROUND AND OBJECTIVES

Stroke is a chronic, complex condition that disproportionally affects older adults. Health systems are evaluating innovative transitional care (TC) models to improve outcomes in these patients. The Comprehensive Post-Acute Stroke Services (COMPASS) Study, a large cluster-randomized pragmatic trial, tested a TC model for patients with stroke or transient ischemic attack discharged home from the hospital. The implementation of COMPASS-TC in complex real-world settings was evaluated to identify successes and challenges with integration into the clinical workflow.

RESEARCH DESIGN AND METHODS

We conducted a concurrent process evaluation of COMPASS-TC implementation during the first year of the trial. Qualitative data were collected from 4 sources across 19 intervention hospitals. We analyzed transcripts from 43 conference calls with hospital clinicians, individual and group interviews with leaders and clinicians from 9 hospitals, and 2 interviews with the COMPASS-TC Director of Implementation using iterative thematic analysis. Themes were compared to the domains of the RE-AIM framework.

RESULTS

Organizational, individual, and community factors related to Reach, Adoption, and Implementation were identified. Organizational readiness was an additional key factor to successful implementation, in that hospitals that were not "organizationally ready" had more difficulty addressing implementation challenges.

DISCUSSION AND IMPLICATIONS

Multifaceted TC models are challenging to implement. Facilitators of implementation were organizational commitment and capacity, prioritizing implementation of innovative delivery models to provide comprehensive care, being able to address challenges quickly, implementing systems for tracking patients throughout the intervention, providing clinicians with autonomy and support to address challenges, and adequately resourcing the intervention.

CLINICAL TRIAL REGISTRATION

NCT02588664.

What Is Dissemination and Implementation Science?: An Introduction and Opportunities to Advance Behavioral Medicine and Public Health Globally

There has been a well-documented gap between research (e.g., evidence-based programs, interventions, practices, policies, guidelines) and practice (e.g., what is routinely delivered in real-world community and clinical settings). Dissemination and implementation (D&I) science has emerged to address this research-to-practice gap and accelerate the speed with which translation and real-world uptake and impact occur. In recent years, there has been tremendous development in the field and a growing global interest, but much of the introductory literature has been U.S.-centric. This piece provides an introduction to D&I science and summarizes key concepts and progress of the field for a global audience, provides two case studies that highlight examples of D&I research globally, and identifies opportunities and innovations for advancing the field of D&I research globally.

A Qualitative Study to Understand Stakeholders' Views About the Fruits & Veggies (FNV) Social Marketing Campaign to Promote Fruit and Vegetable Consumption in the United States

BACKGROUND

In 2015, the Partnership for a Healthier America (PHA) piloted a branded social marketing communications initiative called the FNV (Fruits & Veggies) Campaign to promote the sale and consumption of fruits and vegetables among mom and teen target audiences in 2 California and Virginia pilot markets. After the first year, FNV was expanded to multiple markets across the United States through diverse partnerships. Factors, resources, and strategies that contribute to the effectiveness and sustainability of the campaign need to be identified.

OBJECTIVE

The purpose of this qualitative study was to explore diverse stakeholders' views and expectations about the FNV Campaign design, reach, effectiveness, and impact and prospects for expansion, scale-up, and sustainability.

DESIGN

Semistructured interviews were conducted between July and October 2016 with stakeholders to elicit their perspectives and expectations of the FNV Campaign.

PARTICIPANTS/SETTING

A purposive sample of diverse stakeholders (n = 22) affiliated with national, state, and local industry and government and nongovernmental organizations.

ANALYSIS

Data were recorded, transcribed, input into NVivo 11 software (QSR International. Melbourne, Australia), coded and organized thematically using a conceptual framework that examined campaign design, reach, effectiveness, adoption, implementation, expansion, scaling up, and sustainability.

RESULTS

Opportunities, challenges, and expectations of stakeholders were reported for the FNV Campaign related to engaging future partners from different levels of community, government, and industry; conducting and disseminating findings from multifaceted evaluations; ensuring campaign effectiveness; and developing and improving campaign elements for target audiences and partner adoption. Conducting formative research, partner, and audience engagement, implementation, and evaluation are essential to the ensure effectiveness and sustainability of the FNV Campaign.

CONCLUSIONS

Insights from this study can inform the strategic use of social marketing campaign elements to facilitate the effectiveness and sustainability of the FNV Campaign as it expands and is scaled up across markets nationwide.

Testing a self-directed lifestyle intervention among veterans: The D-ELITE pragmatic clinical trial

Nearly half of Veterans have obesity, fueling chronic diseases. The Department of Veterans Affairs (VA) offers an evidence-based behavioral weight management intervention called MOVE!, mostly delivered through in-person group sessions. Few eligible Veterans participate due to factors like distance and preferences, mirroring barriers in the general population. Practical alternatives to standard in-person programs are needed to improve access and engagement. A self-directed lifestyle intervention called D-ELITE-delivered through pre-recorded videos by DVD or online streaming-previously efficacious in a general primary care population, may provide such an alternative. This pragmatic clinical trial will evaluate whether D-ELITE improves weight and general health status among Veterans with obesity, relative to VA usual care. The yearlong intervention includes one orientation by phone, supplemental lifestyle coaching primarily via technology-based messages, 12 DVD or online streaming sessions over 3 months, and continued self-directed weight management for months 4-12. Participants use MyFitnessPal.com or paper booklets for self-monitoring weight, diet, and physical activity. Follow-up assessments at 12 and 24 months are administered by mail or phone. The study hypothesis is that compared with usual care, D-ELITE will lead to greater improvements in 12-month weight loss, per VA electronic health records, and general physical health status, assessed using the self-reported SF-12 physical composite score. We will also explore D-ELITE's effects on secondary biometric (e.g., HbA1c) and intermediate (e.g., diet) outcomes, reach, and budget impact. If effective, D-ELITE will offer a potentially scalable, low-cost alternative to VA's existing weight loss interventions by mitigating barriers presented by distance and technology.

Kindergarten to 12th Grade School-Based Nutrition Interventions: Putting Past Recommendations Into Practice

School-based nutrition interventions are used to improve dietary habits of schoolchildren and reverse trends on obesity. This article reports on kindergarten through 12th grade nutrition interventions published between 2009 and 2018 compared with interventions published between 2000 and 2008 based on (1) behaviorally focused, (2) multicomponent, (3) healthful food/school environment (4) family involvement, (5) self-assessments, (6) quantitative evaluation, (7) community involvement, (8) ethnic/heterogeneous groups, (9) multimedia technology, and (10) sequential and sufficient duration. These 10 recommendations help guide educators, researchers, and nutritionists on more effective nutrition interventions. Future use of implementation science to determine potential drivers of adoption, nonadoption, and effectiveness of the recommendations is encouraged.

Defining gaps in pre-exposure prophylaxis delivery for pregnant and post-partum women in high-burden settings using an implementation science framework

Pregnancy is a high-risk period for HIV acquisition in African women, and pregnant women who become acutely infected with HIV account for up to a third of vertical HIV transmission cases in African settings. To protect women and eliminate vertical transmission, WHO recommends offering oral pre-exposure prophylaxis (PrEP) based on tenofovir to HIV-negative pregnant and post-partum women with a substantial risk of HIV acquisition. PrEP implementation for pregnant and post-partum women lags behind implementation for other high-risk populations. Unique considerations for PrEP implementation arise during pregnancy and post partum, including the integration of provider training with clinical delivery and monitoring of PrEP exposure and outcomes within existing maternal health systems, yet scarce implementation data are available to generate evidence in this context.

A multicenter trial of a shared DECision Support Intervention for Patients offered implantable Cardioverter-DEfibrillators: DECIDE-ICD rationale, design, Medicare changes, and pilot data

Shared decision making (SDM) facilitates delivery of medical therapies that are in alignment with patients' goals and values. Medicare national coverage decision for several interventions now includes SDM mandates, but few have been evaluated in nationwide studies. Based upon a detailed needs assessment with diverse stakeholders, we developed pamphlet and video patient decision aids (PtDAs) for implantable cardioverter/defibrillator (ICD) implantation, ICD replacement, and cardiac resynchronization therapy with defibrillation to help patients contemplate, forecast, and deliberate their options. These PtDAs are the foundation of the Multicenter Trial of a Shared Decision Support Intervention for Patients Offered Implantable Cardioverter-Defibrillators (DECIDE-ICD), a multicenter, randomized trial sponsored by the National Heart, Lung, and Blood Institute aimed at understanding the effectiveness and implementation of an SDM support intervention for patients considering ICDs. Finalization of a Medicare coverage decision mandating the inclusion of SDM for new ICD implantation occurred shortly after trial initiation, raising novel practical and statistical considerations for evaluating study end points. METHODS/DESIGN: A stepped-wedge randomized controlled trial was designed, guided by the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) planning and evaluation framework using an effectiveness-implementation hybrid type II design. Six electrophysiology programs from across the United States will participate. The primary effectiveness outcome is decision quality (defined by knowledge and values-treatment concordance). Patients with heart failure who are clinically eligible for an ICD are eligible for the study. Target enrollment is 900 participants. DISCUSSION: Study findings will provide a foundation for implementing decision support interventions, including PtDAs, with patients who have chronic progressive illness and are facing decisions involving invasive, preference-sensitive therapy options.

STEPS to Care: Translating an Evidence-Informed HIV Care Coordination Program Into a Field-Tested Online Practice Improvement Toolkit

Increasing care engagement is essential to meet HIV prevention goals and achieve viral suppression. It is difficult, however, for agencies to establish the systems and practice improvements required to ensure coordinated care, especially for clients with complex health needs. We describe the theory-driven, field-informed transfer process used to translate key components of the evidence-informed Ryan White Part A New York City Care Coordination Program into an online practice improvement toolkit, STEPS to Care (StC), with the potential to support broader dissemination. Informed by analyses of qualitative and quantitative data collected from eight agencies, we describe our four phases: (1) review of StC strategies and key elements, (2) translation into a three-part toolkit: Care Team Coordination, Patient Navigation, and HIV Self-Management, (3) pilot testing, and (4) toolkit refinement for national dissemination. Lessons learned can guide the translation of evidence-informed strategies to online environments, a needed step to achieve wide-scale implemention.

State of Mind Ireland-Higher Education: A Mixed-Methods Longitudinal Evaluation of a Positive Mental Health Intervention

Objective: This study evaluates the impact of the State of Mind Ireland-Higher Education (SOMI-HE) Mental Fitness intervention on student wellbeing, resilience, and physical activity (PA) participation. Design: A mixed-methods research design, comprising of a self-report questionnaire, and semi-structured focus group interviews at pre, post and follow-up phases were employed. Participants were a sample of 134 higher education students (29% male: 71% female; mean age range 18 to 25 years old). The quantitative outcome measures of wellbeing, resilience and PA data were analysed using SPSS version 26.0, (IBM, Armonk, NY, USA) with appropriate statistical analysis. Qualitative data were analysed using thematic analysis to capture the long-term outcomes and impact of the intervention. Results: The results indicate a significant intervention effect on participants’ wellbeing (t (120) = −4.27, p < 0.001), PA levels (t (126) = 3.91, p < 0.001) and motivational readiness for exercise change (χ2 (1, n = 131) = 6.9, p < 0.009 (2–sided). Qualitative findings suggest a sustained long-term increase in PA and resilience skills for positive mental health, and reduced stigma and barriers to positive mental health. Conclusion: The findings demonstrate the effectiveness of the SOMI-HE evidence-based intervention, and beneficial outcomes of a salutary approach to higher education student mental health.

Health Promoting School Interventions in Latin America: A Systematic Review Protocol on the Dimensions of the RE-AIM Framework

Understanding the dimensions of internal and external validities (e.g., using the RE-AIM model: Reach, Effectiveness/Efficacy, Adoption, Implementation, and Maintenance) of school interventions is important to guide research and practice in this context. The aim of this systematic review protocol is to synthesize evidence on the RE-AIM dimensions in interventions based on the Health Promoting School (HPS) approach from the World Health Organization (WHO) in Latin America. Studies of interventions based on HPS-WHO that were carried out in Latin America involving the population of 5 to 18-year-olds will be eligible. Searches in nine electronic databases, a study repository, the gray literature, and the retrieved articles' reference lists will be performed, without year or publication language limits. Study selection and data extraction will be conducted by independent researchers. Data on intervention implementation will be summarized in categories of HPS-WHO actions: (1) school curriculum, (2) changes in the social and/or physical environment of schools, and (3) actions with families and the community. A previously validated tool will be used to summarize the information on the dimensions of the RE-AIM model. The strengths and limitations of the included studies will be evaluated using the Critical Appraisal Skills Program (CASP) tool, and the confidence level of evidence will be assessed according to the GRADE CERQual tool.

Evaluating the Implementation of Injury Prevention Strategies in Rugby Union and League: A Systematic Review using the RE-AIM Framework

Rugby (union and league) has come under intense scrutiny due to its injury risk. Various interventions have been introduced to protect players from injury, with many deemed efficacious and advocated for use across various worldwide contexts. However, their implementation is less clear. The objective of this systematic review was to determine whether injury prevention interventions in rugby have evaluated their 'reach', 'effectiveness', 'adoption', 'implementation' and 'maintenance' as per the RE-AIM Multi-Dimension Item Checklist. Six electronic databases were searched in November 2019. Inclusion criteria included: English language, peer-reviewed journal article, original research, field-based rugby code, prospective intervention. Of the 4253 studies identified, 74 met the full inclusion criteria. Protective equipment, predominately mouthguards, was the intervention of interest in 44 studies. Other interventions included multimodal national injury prevention programmes, law changes and neuromuscular training programmes. 'Effectiveness' was the highest scoring RE-AIM dimension (55%), followed by 'reach' (26%). All other RE-AIM dimensions scored below 20%. Research currently focuses on determining intervention 'effectiveness'. For injury prevention strategies to have their desired impact, there must be a shift to address all determinants associated with implementation. Consideration should be given to how this can be achieved by adopting specific reporting checklists, research frameworks and study designs.

Using Digital Technology to Reduce the Prevalence of Mental Health Disorders in Populations: Time for a New Approach

Digital technology, which includes the collection, analysis, and use of data from a variety of digital devices, has the potential to reduce the prevalence of disorders and improve mental health in populations. Among the many advantages of digital technology is that it allows preventive and clinical interventions, both of which are needed to reduce the prevalence of mental health disorders, to be feasibly integrated into health care and community delivery systems and delivered at scale. However, the use of digital technology also presents several challenges, including how systems can manage and implement interventions in a rapidly changing digital environment and handle critical issues that affect population-wide outcomes, including reaching the targeted population, obtaining meaningful levels of uptake and use of interventions, and achieving significant outcomes. We describe a possible solution, which is to have an outcome optimization team that focuses on the dynamic use of data to adapt interventions for populations, while at the same time, addressing the complex relationships among reach, uptake, use, and outcome. We use the example of eating disorders in young people to illustrate how this solution could be implemented at scale. We also discuss system, practitioner-related, and other issues related to the adaptation of such an approach. Digital technology has great potential for facilitating the reduction of mental illness rates in populations. However, achieving this goal will require the implementation of new approaches. As a solution, we argue for the need to create outcome optimization teams, tasked with integrating data from various sources and using advanced data analytics and new designs to develop interventions/strategies to increase reach, uptake, use/engagement, and outcomes for both preventive and treatment interventions.

Integration of Mobile Health Into Sickle Cell Disease Care to Increase Hydroxyurea Utilization: Protocol for an Efficacy and Implementation Study

BACKGROUND

Hydroxyurea prevents disease complications among patients with sickle cell disease (SCD). Although its efficacy has been endorsed by the National Health Lung and Blood Institute evidence-based guidelines, its adoption is low, both by patients with SCD and providers. Mobile health (mHealth) apps provide benefits in improving medication adherence and self-efficacy among patients with chronic diseases and have facilitated prescription among medical providers. However, mHealth has not been systematically tested as a tool to increase hydroxyurea adherence nor has the combination of mHealth been assessed at both patient and provider levels to increase hydroxyurea utilization.

OBJECTIVE

This study aims to increase hydroxyurea utilization through a combined two-level mHealth intervention for both patients with SCD and their providers with the goals of increasing adherence to hydroxyurea among patients and improve hydroxyurea prescribing behavior among providers.

METHODS

We will test the efficacy of 2 mHealth interventions to increase both patient and provider utilization and knowledge of hydroxyurea in 8 clinical sites of the NHLBI-funded Sickle Cell Disease Implementation Consortium (SCDIC). The patient mHealth intervention, InCharge Health, includes multiple components that address memory, motivation, and knowledge barriers to hydroxyurea use. The provider mHealth intervention, Hydroxyurea Toolbox (HU Toolbox), addresses the clinical knowledge barriers in prescribing and monitoring hydroxyurea. The primary hypothesis is that among adolescents and adults with SCD, adherence to hydroxyurea, as measured by the proportion of days covered (the ratio of the number of days the patient is covered by the medication to the number of days in the treatment period), will increase by at least 20% after 24 weeks of receiving the InCharge Health app, compared with their adherence at baseline. As secondary objectives, we will (1) examine the change in health-related quality of life, acute disease complications, perceived health literacy, and perceived self-efficacy in taking hydroxyurea among patients who use InCharge Health and (2) examine potential increases in the awareness of hydroxyurea benefits and risks, appropriate prescribing, and perceived self-efficacy to correctly administer hydroxyurea therapy among SCD providers between baseline and 9 months of using the HU Toolbox app. We will measure the reach, adoption, implementation, and maintenance of both the InCharge Health and the HU Toolbox apps using the reach, effectiveness, adoption, implementation, and maintenance framework and qualitatively evaluate the implementation of both mHealth interventions.

RESULTS

The study is currently enrolling study participants. Recruitment is anticipated to be completed by mid-2021.

CONCLUSIONS

If this two-level intervention, that is, the combined use of InCharge Health and HU Toolbox apps, demonstrates efficacy in increasing adherence to hydroxyurea and prescribing behavior in patients with SCD and their providers, respectively, both apps will be offered to other institutions outside the SCDIC through a future large-scale implementation-effectiveness study.

TRIAL REGISTRATION

ClinicalTrials.gov NCT04080167; https://clinicaltrials.gov/ct2/show/NCT04080167.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/16319.

Doing What We Know, Knowing What to Do: Californians Linking Action with Science for Prevention of Breast Cancer (CLASP-BC)

Given the lack of progress in breast cancer prevention, the California Breast Cancer Research Program (CBCRP) plans to apply current scientific knowledge about breast cancer to primary prevention at the population level. This paper describes the first phase of Californians Linking Action with Science for Prevention of Breast Cancer (CLASP-BC). The foci of Phase 1 are building coalitions and coalition capacity building through community engagement in community-based participatory research (CBPR) and dissemination and implementation (D&I) research training. Based on the successful implementation and evaluation of Phase 1, the foci of Phase 2 (presented separately in this special issue) will be to translate the California Breast Cancer Prevention Plan overarching goal and specific intervention goals for 23 breast cancer risk and protective factors strategies into evidence-informed interventions (EIIs) that are disseminated and implemented across California. CLASP-BC is designed to identify, disseminate and implement high-impact, population-based prevention approaches by funding large scale EIIs, through multi-jurisdictional actions, with the intent to decrease the risk of breast cancer and other chronic diseases (sharing common risk factors), particularly among racial/ethnic minorities and medically underserved populations in California.

Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings

Purpose

Patient-reported outcome and experience measures (PROMs/PREMs) are well established in research for many health conditions, but barriers persist for implementing them in routine care. Implementation science (IS) offers a potential way forward, but its application has been limited for PROMs/PREMs.

Methods

We compare similarities and differences for widely used IS frameworks and their applicability for implementing PROMs/PREMs through case studies. Three case studies implemented PROMs: (1) pain clinics in Canada; (2) oncology clinics in Australia; and (3) pediatric/adult clinics for chronic conditions in the Netherlands. The fourth case study is planning PREMs implementation in Canadian primary care clinics. We compare case studies on barriers, enablers, implementation strategies, and evaluation.

Results

Case studies used IS frameworks to systematize barriers, to develop implementation strategies for clinics, and to evaluate implementation effectiveness. Across case studies, consistent PROM/PREM implementation barriers were technology, uncertainty about how or why to use PROMs/PREMs, and competing demands from established clinical workflows. Enabling factors in clinics were context specific. Implementation support strategies changed during pre-implementation, implementation, and post-implementation stages. Evaluation approaches were inconsistent across case studies, and thus, we present example evaluation metrics specific to PROMs/PREMs.

Conclusion

Multilevel IS frameworks are necessary for PROM/PREM implementation given the complexity. In cross-study comparisons, barriers to PROM/PREM implementation were consistent across patient populations and care settings, but enablers were context specific, suggesting the need for tailored implementation strategies based on clinic resources. Theoretically guided studies are needed to clarify how, why, and in what circumstances IS principles lead to successful PROM/PREM integration and sustainability.

Electronic supplementary material

The online version of this article (10.1007/s11136-020-02564-9) contains supplementary material, which is available to users.

A longitudinal implementation evaluation of a physical activity program for cancer survivors: LIVESTRONG® at the YMCA

Purpose

Increased physical activity (PA) levels in cancer survivors are associated with decreased risk of recurrence and mortality as well as additional positive health outcomes. PA interventions have shown to be efficacious, though many lack translation to and sustainability in community settings. We used dimensions of the RE-AIM framework to evaluate LIVESTRONG® at the YMCA, a nation-wide community-based PA program for cancer survivors delivered at Ys.

Methods

This was a longitudinal study design using national LIVESTRONG at the YMCA data compiled between 2010 and 2018. Data is from all YMCAs who deliver LIVESTRONG at the YMCA, submitted by Program Directors to the YMCA-USA. We assessed reach (number of participants), adoption (associations offering the program), implementation (conducting 3 fidelity checks), and organizational level maintenance (associations recently offering program). We also examined relationships between organizational characteristics (years of program existence and association area household income) and program implementation factors with member conversion rates.

Results

As of 2018, LIVESTRONG at the YMCA has reached 62,044 survivors and 245 of the 840 (29.2%) of Y associations have adopted the program. Among the adopters, 91% were aware of fidelity checks; implementation of observational (62.3%), goal setting (49.9%), and functional (64.6%) checklists varied. Most (95.1%) adopters reported offering ≥ 1 LIVESTRONG session per year (organizational-level maintenance) and a facility-level mean membership conversion percentage of 46.9 ± 31.2%. Fewer years implementing the program and higher association area household income were significantly associated with a greater membership conversion rate vs their comparison. In a multiple regression model controlling for organizational characteristics, conducting the fidelity checks independently (observational, β = 8.41; goal-setting, β = 9.70; and functional, β = 9.61) and collectively (β = 10.82; 95% CI 5.90–16.80) was positively associated with higher membership conversion rates.

Conclusions

LIVESTRONG at the YMCA, in its early years, has shown promise for high reach, while adoption at more associations could be facilitated. Implementing fidelity checks along with organizational characteristics were associated with membership conversion rate. Identification of association-level strategies to increase reach, adoption, implementation, and maintenance may increase the impact of this community-based PA program.

Characterizing evolving frameworks: issues from Esmail et al. (2020) review

There are complex issues in understanding and categorizing implementation science theories, models, and frameworks. Systematic reviews of these models are important undertakings for synthesizing current knowledge. The issues involved are even more challenging when reviewing a large number of frameworks and when some of the frameworks have evolved significantly over time. This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework was described in the recent Esmail (2020) review and identifies four mischaracterizations. This is followed by a more general discussion of how advances or extensions of frameworks after an original source publication or influential review tend to be overlooked. We discuss why inadvertent mischaracterization of what a framework is and is not, and what it can and cannot be used for, can have deleterious consequences. Finally, we suggest initial ideas about what could be done to prevent or alleviate some of these problems by reviewers, framework developers, and scholars at large.

Enhancing Success of Medicare's Shared Decision Making Mandates Using Implementation Science: Examples Applying the Pragmatic Robust Implementation and Sustainability Model (PRISM)

The Centers for Medicare and Medicaid Services (CMS) has mandated shared decision making (SDM) using patient decision aids for three conditions (lung cancer screening, atrial fibrillation, and implantable defibrillators). These forward-thinking approaches are in response to a wealth of efficacy data demonstrating that decision aids can improve patient decision making. However, there has been little focus on how to implement these approaches in real-world practice. This article demonstrates how using an implementation science framework may help programs understand multilevel challenges and opportunities to improve adherence to the CMS mandates. Using the PRISM (Pragmatic Robust Implementation and Sustainability Model) framework, we discuss general challenges to implementation of SDM, issues specific to each mandate, and how to plan for, enhance, and assess SDM implementation outcomes. Notably, a theme of this discussion is that successful implementation is context-specific and to truly have successful and sustainable changes in practice, context variability, and adaptation to context must be considered and addressed.

Global PRoMiSe (Perioperative Recommendations for Medication Safety): protocol for a mixed-methods study

INTRODUCTION

Medication errors (MEs), which occur commonly in the perioperative period, have the potential to cause patient harm or death. Many published recommendations exist for preventing perioperative MEs; however, many of these recommendations conflict and are often not applicable to middle-income and low-income countries. The goal of this study is to develop and disseminate consensus-based recommendations for perioperative medication safety that are tailored to country income level.

METHODS AND ANALYSIS

The primary site of this mixed-methods study is Massachusetts General Hospital/Harvard Medical School. Participants include a minimum of 108 international medication safety experts, 27 from each of the World Bank's four country income groups (high, upper-middle, lower-middle and low-income). Using the Delphi method, participants will rate the appropriateness of candidate medication safety recommendations by completing online surveys using RedCAP. We will use Condorcet ranking methods to prioritise the final recommendations for each country income group. We will execute a comprehensive dissemination strategy for the recommendations across each country income group. Finally, we will conduct semistructured interviews with our participants to evaluate the initial adoption and implementation of the recommendations in each country income group.

ETHICS AND DISSEMINATION

This study was approved by the Human Research Committee/Institutional Review Board at Partners Healthcare (2019P003567). Findings will be published in peer-reviewed journals and presented at local and international conferences.

TRIAL REGISTRATION NUMBER

NCT04240301.

A Mixed Methods Evaluation of a Digital Intervention to Improve Sedentary Behaviour Across Multiple Workplace Settings

BACKGROUND

Prolonged sedentary behaviour (SB) is associated with risk of chronic diseases. Digital interventions in SB require mixed method evaluations to understand potential for impact in real-world settings. In this study, the RE-AIM QuEST evaluation framework will be used to understand the potential of a digital health promotion application which targets reducing and breaking up SB across multiple workplace settings.

METHODS

Four companies and 80 employees were recruited to use a digital application. Questionnaires were used to measure SB, and additional health and work-related outcomes at baseline, one month, three month and six month follow-up. Qualitative data was collected through focus groups with employees and interviews with stakeholders. Questionnaire data was analysed using Wilcoxon Sign Rank tests and qualitative data was thematically analysed.

RESULTS

The digital application significantly increased standing time at one month for the total group and transitions per hour in one of the companies. Facilitators and barriers were identified across RE-AIM.

CONCLUSIONS

Addressing the barriers which have been identified, while maintaining the positive attributes will be critical to producing an effective digital application which also has the potential for impact in the real world.

Nurse Educators Establish and Evaluate Community Coalition to Promote Statewide Advance Care Planning Through RE-AIM

The purpose of this quality improvement project was to evaluate a statewide initiative promoting Advance Care Planning (ACP) to educate and support multidisciplinary ACP educators and provide tools to start ACP conversations in a predominantly rural state of the Upper Midwest. Individual objectives were to (1) motivate people of different professions and backgrounds to support the vision and (2) implement a system to educate and maintain a pipeline of ACP educators in appropriate methodologies to enable ACP in distant communities. The Advance Care Planning: Quality Conversations coalition was formed in 2015 to improve health care across the life span. The Reach-Effectiveness-Adoption-Implementation-Maintenance framework was applied to evaluate the project. Outcome variables were measured before, during, and after program implementation through service statistics and a questionnaire. Participation in the coalition's membership team between September 2015 and September 2019 ranged from 18 to 36 with a median of 27 and mode of 27. At least 20 different professions were represented. The coalition provided funds for educating 9 ACP instructors and 180 facilitators according to the Respecting Choices-First Steps ACP program. The coalition's mission has generated sustained interest for 4 years. Key elements and obstacles to implementing a statewide coalition were identified.

Impetus to change: a multi-site qualitative exploration of the national audit of dementia

BACKGROUND

National audit is a key strategy used to improve care for patients with dementia. Audit and feedback has been shown to be effective, but with variation in how much it improves care. Both evidence and theory identify active ingredients associated with effectiveness of audit and feedback. It is unclear to what extent national audit is consistent with evidence- and theory-based audit and feedback best practice.

METHODS

We explored how the national audit of dementia is undertaken in order to identify opportunities to enhance its impact upon the improvement of care for people with dementia. We undertook a multi-method qualitative exploration of the national audit of dementia at six hospitals within four diverse English National Health Service organisations. Inductive framework analysis of 32 semi-structured interviews, documentary analysis (n = 39) and 44 h of observations (n = 36) was undertaken. Findings were presented iteratively to a stakeholder group until a stable description of the audit and feedback process was produced.

RESULTS

Each organisation invested considerable resources in the audit. The audit results were dependent upon the interpretation by case note reviewers who extracted the data. The national report was read by a small number of people in each organisation, who translated it into an internal report and action plan. The internal report was presented at specialty- and organisation-level committees. The internal report did not include information that was important to how committee members collectively decided whether and how to improve performance. Participants reported that the national audit findings may not reach clinicians who were not part of the specialty or organisation-level committees.

CONCLUSIONS

There is considerable organisational commitment to the national audit of dementia. We describe potential evidence- and theory-informed enhancements to the enactment of the audit to improve the local response to performance feedback in the national audit. The enhancements relate to the content and delivery of the feedback from the national audit provider, support for the clinicians leading the organisational response to the feedback, and the feedback provided within the organisation.

Pragmatic Application of the RE-AIM Framework to Evaluate the Implementation of Tobacco Cessation Programs Within NCI-Designated Cancer Centers

Tobacco cessation after cancer diagnosis leads to better patient outcomes. However, tobacco treatment services are frequently unavailable in cancer care settings, and multilevel implementation challenges can impede uptake of new programs. The National Cancer Institute (NCI) dedicated Cancer Moonshot funding through the Cancer Center Cessation Initiative (C3I) for NCI-Designated Cancer Centers to implement or enhance the implementation of tobacco treatment services. We examined a pragmatic application of the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) to evaluate tobacco treatment programs implemented within Cancer Centers funded through C3I. Using three C3I-funded Centers as examples, we describe how each RE-AIM construct was operationalized to evaluate the implementation of a wide range of cessation services (e.g., tobacco use screening, counseling, Quitline referral, pharmacotherapy) in this heterogeneous group of cancer care settings. We discuss the practical challenges encountered in assessing RE-AIM constructs in real world situations, including using the electronic health record (EHR) to aid in assessment. Reach and effectiveness evaluation required that Centers define the setting(s) where cessation services were implemented (to determine the “denominator”), enumerate the patient population, report current patient tobacco use, patient engagement in tobacco treatment, and 6-month cessation outcomes. To reduce site heterogeneity, increase data accuracy, and reduce burden, reach was frequently captured via standardized EHR enhancements that improved the identification of current smokers and tobacco treatment referrals. Effectiveness was determined by cessation outcomes (30-day point prevalence abstinence at 6-months post-engagement) assessed through a variety of data collection approaches. Adoption was measured by the characteristics and proportion of targeted cancer care settings and clinicians engaged in cessation service delivery. Implementation was assessed by examining the delivery of tobacco screening assessments and intervention components across sites, and provider-level implementation consistency. Maintenance assessments identified whether tobacco treatment services continued in the setting after implementation and documented the sustainability plan and organizational commitment to continued delivery. In sum, this paper demonstrates a pragmatic approach to using RE-AIM as an evaluation framework that yields relevant outcomes on common implementation metrics across widely differing tobacco treatment approaches and settings.

Evaluating a multicomponent program to improve hypertension control in Guatemala: study protocol for an effectiveness-implementation cluster randomized trial

BACKGROUND

Hypertension is a major risk factor for cardiovascular disease (CVD). Despite advances in hypertension prevention and treatment, the proportion of patients who are aware, treated and controlled is low, particularly in low-income and middle-income countries (LMICs). We will evaluate an adapted version of a multilevel and multicomponent hypertension control program in Guatemala, previously proven effective and feasible in Argentina. The program components are: protocol-based hypertension treatment using a standardized algorithm; team-based collaborative care; health provider education; health coaching sessions; home blood pressure monitoring; blood pressure audit; and feedback.

METHODS

Using a hybrid type 2 effectiveness-implementation design, we will evaluate clinical and implementation outcomes of the multicomponent program in Guatemala over an 18-month period. Through a cluster randomized trial, we will randomly assign 18 health districts to the intervention arm and 18 to enhanced usual care across five departments, enrolling 44 participants per health district and 1584 participants in total. The clinical outcomes are (1) the difference in the proportion of patients with controlled hypertension (< 130/80 mmHg) between the intervention and control groups at 18 months and (2) the net change in systolic and diastolic blood pressure from baseline to 18 months. The context-enhanced Reach, Efficacy, Adoption, Implementation, Maintenance (RE-AIM)/Practical Robust Implementation and Sustainability Model (PRISM) framework will guide the evaluation of the implementation at the level of the patient, provider, and health system. Using a mixed-methods approach, we will evaluate the following implementation outcomes: acceptability, adoption, feasibility, fidelity, adaptation, reach, sustainability, and cost-effectiveness.

DISCUSSION

We will disseminate the study findings, and promote scale up and scale out of the program, if proven effective. This study will generate urgently needed data on effective, adoptable, and sustainable interventions and implementation strategies to improve hypertension control in Guatemala and other LMICs.

TRIAL REGISTRATION

ClinicalTrials.gov: NCT03504124. Registered on 20 April 2018.

Integrating Economic Evaluation and Implementation Science to Advance the Global HIV Response

BACKGROUND

Numerous cost-effectiveness analyses have indicated good value for money from a wide array of interventions for treatment and prevention of HIV/AIDS. There is limited evidence, however, regarding how cost-effectiveness information contributes to better decision-making around investment and action in the global HIV response.

METHODS

We review challenges for economic evaluation relevant to the global HIV response and consider how the practice of cost-effectiveness analysis could integrate approaches and insights from implementation science to enhance the impact and efficiency of HIV investments.

RESULTS

In light of signals that cost-effectiveness analyses may be vulnerable to systematic bias toward overly optimistic conclusions, we emphasize two priorities for advancing the field of economic evaluation in HIV/AIDS and more broadly in global health: (1) systematic reevaluation of the cost-effectiveness literature with reference to ex-post empirical evidence on costs and effects in real-world programs and (2) development and adoption of good-practice guidelines for incorporating implementation and delivery aspects into economic evaluations. Toward the latter aim, we propose an integrative approach that focuses on comparative evaluation of strategies, which specify both technologies/interventions as well as the delivery platforms, complementary interventions, and actions needed to increase coverage, quality, and uptake of those technologies/interventions. Specific recommendations draw on several existing implementation science models that provide systematic frameworks for understanding implementation barriers and enablers, designing and choosing specific implementation and policy actions, and evaluating outcomes.

DISCUSSION

These preliminary steps aimed at bridging the divide between economic evaluation and implementation science can help to advance the practice of economic evaluation toward a science of comparative strategy evaluation.

If you build it, will they come? Evaluation of a co-located exercise clinic and cancer treatment centre using the RE-AIM framework

OBJECTIVE

Employ the Reach, Effectiveness, Adoption, Implementation, Maintenance framework to evaluate the effectiveness of a co-located exercise clinic model in increasing access to exercise for people undergoing cancer treatment in a private clinic in Western Australia.

METHODS

This retrospective evaluation utilised a mixed-method approach to gather feedback from key stakeholder groups involved with the exercise clinic. Questionnaires and workout summary sheets were gathered from 237 exercise clinic participants over the 50-month evaluation period. These were supplemented by survey results from 119 patients who received cancer treatment at the facility, and semi-structured interviews from seven radiation oncologists, eight nurses, and three accredited exercise physiologists involved with the exercise clinic.

RESULTS

The co-located clinic demonstrated positive outcomes related to effectiveness and adoption. Participant feedback indicated satisfaction with the exercise programming (effectiveness), and clinicians were receptive to referring patients to the clinic (adoption). However, no clear implementation or maintenance plan was employed and overall reach (12%) remained suboptimal throughout the evaluation period.

CONCLUSION

Co-locating an exercise clinic into a treatment facility does not in itself overcome the logistical challenges of providing integrated exercise services to people during cancer treatment. To enhance its utilisation, an implementation plan needs to accompany the intervention.

Landscape of HIV Implementation Research Funded by the National Institutes of Health: A Mapping Review of Project Abstracts

In 2019, the requisite biomedical and behavioral interventions to eliminate new HIV infections exist. "Ending the HIV Epidemic" now becomes primarily a challenge of will and implementation. This review maps the extent to which implementation research (IR) has been integrated into HIV research by reviewing the recent funding portfolio of the NIH. We searched NIH RePORTER for HIV and IR-related research projects funded from January 2013 to March 2018. The 4629 unique studies identified were screened using machine learning and manual methods. 216 abstracts met the eligibility criteria of HIV and IR. Key study characteristics were then abstracted. NIH currently funds HIV studies that are either formally IR (n = 109) or preparatory for IR (n = 107). Few (13%) projects mentioned a guiding implementation model, theory, or framework, and only 56% of all studies explicitly mentioned measuring an implementation outcome. Considering the study aims along an IR continuum, 18 (8%) studies examined barriers and facilitators, 43 (20%) developed implementation strategies, 46 (21%) piloted strategies, 73 (34%) tested a single strategy, and 35 (16%) compared strategies. A higher proportion of formal IR projects involved established interventions (e.g., integrated services) compared to newer interventions (e.g., pre-exposure prophylaxis). Prioritizing HIV-related IR in NIH and other federal funding opportunity announcements and expanded training in implementation science could have a substantial impact on ending the HIV pandemic. This review serves as a baseline by which to compare funding patterns and the sophistication of IR in HIV research over time.

Making Implementation Science More Rapid: Use of the RE-AIM Framework for Mid-Course Adaptations Across Five Health Services Research Projects in the Veterans Health Administration

Introduction: Implementation science frameworks have helped advance translation of research to practice. They have been widely used for planning and post-hoc evaluation, but seldom to inform and guide mid-course adjustments to intervention and implementation strategies.

Materials and Methods: This study developed an innovative methodology using the RE-AIM framework and related tools to guide mid-course assessments and adaptations across five diverse health services improvement projects in the Veterans Health Administration (VA). Using a semi-structured guide, project team members were asked to assess the importance of and progress on each RE-AIM dimension (i.e., reach, effectiveness, adoption, implementation, maintenance) at the current phase of their project. Based on these ratings, each team identified one or two RE-AIM dimensions for focused attention. Teams developed proximal goals and implementation strategies to improve progress on their selected dimension(s). A follow-up meeting with each team occurred approximately 6 weeks after the goal setting meeting to evaluate the usefulness of the iterative process. Results were evaluated using both descriptive quantitative analyses and qualitative assessments from interviews and meeting notes.

Results: A median of seven team members participated in the two meetings. Qualitative and descriptive data revealed that the process was feasible, understandable and useful to teams in adjusting their interventions and implementation strategies. The RE-AIM dimensions identified as most important were adoption and effectiveness, and the dimension that had the largest gap between importance and rated progress was reach. The dimensions most frequently selected for improvement were reach and adoption. Examples of action plans were summarizing stakeholder interviews for leadership, revising exclusion criteria, and conducting in-service trainings. Follow-up meetings indicated that teams found the process very useful and were able to implement the action plans they set.

Discussion: The iterative use of RE-AIM to support adjustments during project implementation proved feasible and useful across diverse projects in the VA setting. Building on this and related examples, future research should replicate these findings and further develop the methodology, as well as explore the optimal frequency and timing for these iterative applications of RE-AIM. More generally, greater focus on more rapid and iterative use of implementation science frameworks is encouraged to facilitate successful translation of research to practice.

The Faith, Activity, and Nutrition (FAN) Dissemination and Implementation Study: 24-Month Organizational Maintenance in a Countywide Initiative

Introduction: Despite the important role that faith-based organizations can play in eliminating health disparities, few studies have focused on organizational change and maintenance of interventions in this setting, making their long-term impact unknown. This study reports 24-month maintenance of the Faith, Activity, and Nutrition (FAN) program in a southeastern county. Previously reported findings of reach, adoption, implementation, and effectiveness are also summarized.

Methods: Church coordinators from 35 intervention churches (97% predominantly African American) located in a rural, medically underserved county in South Carolina were interviewed at baseline (2015), and 12- and 24-months post-training regarding implementation of physical activity (PA) and healthy eating (HE) components of the FAN program. Guided by the RE-AIM framework, organizational maintenance was defined as church coordinator-reported 24-month implementation of the four FAN components (providing opportunities, setting guidelines/policies, sharing messages, engaging pastor). Repeated measures analyses (mixed models) examined change in implementation over time. Churches were also classified as maintainers, non-sustained implementers, and low implementers for each FAN component. Statistical analyses were conducted in 2019.

Results: Church coordinators reported significantly greater implementation of both PA and HE FAN components at 12 and 24 months compared to baseline (medium to large effects). The percentage of churches classified as maintainers ranged from 21 to 42 and 27 to 94% across PA and HE components, respectively. Most churches (58% for PA, 97% for HE) were maintaining at least one FAN component at 24 months.

Conclusions: These promising findings position FAN well for the national implementation study now underway.

Trial Registration: This study is registered at www.clinicaltrials.gov NCT02868866.

An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of "maintenance" within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and "evolvability" across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

Supporting Workers to Sit Less and Move More Through the Web-Based BeUpstanding Program: Protocol for a Single-Arm, Repeated Measures Implementation Study

BACKGROUND

The web-based BeUpstanding Champion Toolkit was developed to support work teams in addressing the emergent work health and safety issue of excessive sitting. It provides a step-by-step guide and associated resources that equip a workplace representative-the champion-to adopt and deliver the 8-week intervention program (BeUpstanding) to their work team. The evidence-informed program is designed to raise awareness of the benefits of sitting less and moving more, build a supportive culture for change, and encourage staff to take action to achieve this change. Work teams collectively choose the strategies they want to implement and promote to stand up, sit less, and move more, with this bespoke and participative approach ensuring the strategies are aligned with the team's needs and existing culture. BeUpstanding has been iteratively developed and optimized through a multiphase process to ensure that it is fit for purpose for wide-scale implementation.

OBJECTIVE

The study aimed to describe the current version of BeUpstanding, and the methods and protocol for a national implementation trial.

METHODS

The trial will be conducted in collaboration with five Australian workplace health and safety policy and practice partners. Desk-based work teams from a variety of industries will be recruited from across Australia via partner-led referral pathways. Recruitment will target sectors (small business, rural or regional, call center, blue collar, and government) that are of priority to the policy and practice partners. A minimum of 50 work teams will be recruited per priority sector with a minimum of 10,000 employees exposed to the program. A single-arm, repeated-measures design will assess the short-term (end of program) and long-term (9 months postprogram) impacts. Data will be collected on the web via surveys and toolkit analytics and by the research team via telephone calls with champions. The Reach, Effectiveness, Adoption, Implementation, and Maintenance Framework will guide the evaluation, with assessment of the adoption/reach of the program (the number and characteristics of work teams and participating staff), program implementation (completion by the champion of core program components), effectiveness (on workplace sitting, standing, and moving), and maintenance (sustainability of changes). There will be an economic evaluation of the costs and outcomes of scaling up to national implementation, including intervention affordability and sustainability.

RESULTS

The study received funding in June 2018 and the original protocol was approved by institutional review board on January 9, 2017, with national implementation trial consent and protocol amendment approved March 12, 2019. The trial started on June 12, 2019, with 48 teams recruited as of December 2019.

CONCLUSIONS

The implementation and multimethod evaluation of BeUpstanding will provide the practice-based evidence needed for informing the potential broader dissemination of the program.

TRIAL REGISTRATION

Australian New Zealand Clinical Trials Registry ACTRN12617000682347; https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372843&isReview=true.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/15756.

Translating evidence-based parenting programs for primary care: Stakeholder recommendations for sustainable implementation

AIMS

To translate evidence-based programs (EBP) for a new setting, attention must be given to the characteristics of the intervention and the local setting, as well as evidence that is compelling to decision-makers. This paper describes the history of a partnership and stakeholder recommendations to inform the adaptation of an EBP for primary care.

METHODS

We established a community advisory board (CAB) consisting of stakeholders with expertize in primary care delivery. A thematic analysis was conducted with fieldnotes and transcriptions from CAB meetings and regular meetings with participating clinics.

RESULTS

We found that (a) parenting programs with a focus on behavioral and physical health are appropriate for this setting, (b) variability in the structure of primary care means implementation must be tailorable, and (c) financial and organizational outcomes are compelling for decision-makers.

CONCLUSION

Factors related to the content and structure of evidence-based programs are uniquely related to distinct implementation outcomes of interest to key stakeholders.

The state of the evidence for voluntary home-based human immunodeficiency virus counseling and testing in the community: A proposed model for evidence translation

BACKGROUND

Home-based voluntary human immunodeficiency virus (HIV) counseling and testing (HBVCT) plays a significant role in reducing HIV-related risk-taking behaviors. Adopting evidence-based interventions is essential, but few conceptual models exist to guide the development, implementation, and evaluation of these interventions.

AIMS

Our proposed model for evidence translation based on evidence review describes the implementation process of HBVCT in the community.

MATERIALS AND METHODS

Our study adopted the translating research into practice model, which incorporates information needs about the implementation, planning, and execution required for consideration by clinicians. Thirteen systematic reviews published in the English language from January 1, 2000 to February 9, 2020 were retrieved and reviewed from four electronic databases and journals, including EMBASE, PubMed, JBI Database of Systematic Reviews and Implementation Reports, and Cochrane Library.

RESULTS

The analysis of the review papers based on the phenomenon of interest, results, and the population revealed some distinctions in the number of socioecological levels used by this evidence synthesis. These levels of factors include individual, interpersonal, organizational, community, and public policies. HBVCT is an essential component of HIV prevention programs and a critical entry point for adequate care and treatment.

DISCUSSION

This collaborative model demonstrates the application of research to a real-world health care setting.

RE-AIM Evaluation Plan for Washington State Innovation Models Project

The State of Washington received a State Innovation Models (SIM) $65 million award from the federal Centers for Medicare & Medicaid Services to improve population health and quality of care and reduce the growth of health care costs in the entire state, which has over 7 million residents. SIM is a "complex intervention" that implements several interacting components in a complex, decentralized health system to achieve goals, which poses challenges for evaluation. Our purpose is to present the state-level evaluation methods for Washington's SIM, a 3-year intervention (2016-2018). We apply the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) evaluation framework to structure our evaluation. We create a conceptual model and a plan to use multiple and mixed methods to study SIM performance in the RE-AIM components from a statewide, population-based perspective.

Planning for Implementation Success Using RE-AIM and CFIR Frameworks: A Qualitative Study

Background: RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) and CFIR (Consolidated Framework for Implementation Research) dissemination and implementation frameworks define theory-based domains associated with the adoption, implementation and maintenance of evidence-based interventions. Used together, the two frameworks identify metrics for evaluating implementation success, i.e., high reach and effectiveness resulting in sustained practice change (RE-AIM), and modifiable factors that explain and enhance implementation outcomes (CFIR). We applied both frameworks to study the implementation planning process for a technology-delivered asthma care intervention called Breathewell within an integrated care organization. The goal of the Breathewell intervention is to increase the efficiency of delivering resource-intensive asthma care services. Methods: We reviewed historical documents (i.e., meeting agendas; minutes) from 14 months of planning to evaluate alignment of implementation team priorities with RE-AIM domains. Key content was extracted and analyzed on topics, frequency and amount of discussion within each RE-AIM domain. Implementation team members were interviewed using questions adapted from the CFIR Interview Guide Tool to focus their reflection on the process and contextual factors considered during pre-implementation planning. Documents and transcripts were initially coded using RE-AIM domain definitions, and recoded using CFIR constructs, with intent to help explain how team decisions and actions can contribute to adoption, implementation and maintenance outcomes. Results: Qualitative analysis of team documents and interviews demonstrated strong alignment with the RE-AIM domains: Reach, Effectiveness, and Implementation; and with the CFIR constructs: formal inclusion of provider and staff stakeholders in implementation planning, compatibility of the intervention with workflows and systems, and alignment of the intervention with organizational culture. Focus on these factors likely contributed to RE-AIM outcomes of high implementation fidelity. However, team members expressed low confidence that Breathewell would be adopted and maintained post-trial. A potential explanation was weak alignment with several CFIR constructs, including tension for change, relative priority, and leadership engagement that contribute to organizational receptivity and motivation to sustain change. Conclusions: While RE-AIM provides a practical framework for planning and evaluating practice change interventions to assure their external validity, CFIR explains why implementation succeeded or failed, and when used proactively, identifies relevant modifiable factors that can promote or undermine adoption, implementation, and maintenance.

ImPlementation REsearCh to DEvelop Interventions for People Living with HIV (the PRECluDE consortium): Combatting chronic disease comorbidities in HIV populations through implementation research

Antiretroviral therapy (ART) prevented premature mortality and improved the quality of life among people living with the human immunodeficiency virus (PLWH), such that now more than half of PLWH in the United States are 50 years of age and older. Increased longevity among PLWH has resulted in a significant rise in chronic, comorbid diseases. However, the implementation of guideline-based interventions for preventing, treating, and managing such age-related, chronic conditions among the HIV population is lacking. The PRECluDE consortium supported by the Center for Translation Research and Implementation Science at the National Heart, Lung, and Blood Institute catalyzes implementation research on proven-effective interventions for co-occurring heart, lung, blood, and sleep diseases and conditions among PLWH. These collaborative research studies use novel implementation frameworks with HIV, mental health, cardiovascular, and pulmonary care to advance comprehensive HIV and chronic disease healthcare in a variety of settings and among diverse populations.

Adaptation for Regulatory Application: A Content Analysis of FDA Risk Evaluation and Mitigation Strategies Assessment Plans (2014-2018) Using RE-AIM

Background: Risk Evaluation and Mitigation Strategies (REMS) are safety programs that U.S. Food and Drug Administration can require to ensure a drug's benefits outweigh its risks and can be considered public health interventions. FDA's 2019 draft Guidance for Industry on REMS Assessments encourages the development of “novel methods for assessing REMS [to] help advance the science of post-market assessment of effectiveness of risk mitigation strategies.”

Objective: To characterize REMS assessment plans using RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework and identify areas for advancing methods for evaluating REMS programs. RE-AIM was selected for its wide application evaluating the translation of scientific advances into practice for public health impact.

Methods: A content analysis of REMS assessment plans (N = 18) and measures(n = 540) was conducted for REMS programs approved by FDA between 1/1/2014–12/31/2018. Eligibility criteria were: a new drug application or biologic license application, included FDA-mandated mitigation strategies called elements to assure safe use (ETASU), and represented a single product REMS program. Assessment plans were collected from publicly available regulatory approval letters from REMS@FDA website. Blinded reviewers categorized each REMS assessment measure to a RE-AIM dimension, adjudicated their application (average IRR 75%), and refined the adapted dimensions' definitions. Dimensions were also mapped to REMS Assessment guidance categories.

Results: The median number of assessment measures per REMS assessment plan was 31 (IQR: 21–36). Frequency of measures per RE-AIM criteria per REMS program was: Reach (median = 2; IQR: 2–4); Effectiveness (median = 2.5; IQR:1–4); Adoption (median = 3.5; IQR: 2–5); Implementation (median = 18; IQR: 15–24); Maintenance (median = 0; IQR: 0–1). Adoption (among prescriber, health system agents of implementation) was more commonly assessed than Reach (population-attributable number of patients affected). Assessment of heterogeneity of Adoption and Reach was generally absent. Implementation assessment measures were most common among drugs requiring evidence of safe-use conditions before dispensing or administering the drug. Patient-level Effectiveness and Maintenance assessments were most common among drugs requiring patient monitoring.

Discussion: Implementation science frameworks, such as RE-AIM, can be applied to characterize REMS assessment measures and identify opportunities for standardizing and strengthening their evaluation. Methods to measure Maintenance are needed to provide real-world evidence of REMS integration into the healthcare system.

Advancing health equity through CTSA programs: Opportunities for interaction between health equity, dissemination and implementation, and translational science

Dissemination and implementation (D&I) science is dedicated to studying how to effectively translate and apply research in real-world contexts. There has been increasing interest in health equity within the D&I field to ensure the equitable implementation of evidence-based programs/practices across a range of diverse populations and settings. At the same time, health equity researchers recognize the potential of D&I science to promote the more widespread dissemination, implementation, and sustainment of evidence-based interventions to address health inequities. The National Center for Accelerating Clinical and Translational Science Clinical and Translational Science Award (CTSA) Program has been a champion for community engagement and translational scholarship in its mission to improve individual and population health. The overall CTSA infrastructure and resources within and among CTSA hubs are well-equipped to facilitate a health equity focus to D&I across the phases of translational research. This paper proposes a framework that demonstrates the interaction and opportunities between health equity and D&I science and highlights how CTSAs can support and facilitate wider efforts in translational research with a focus on equitable D&I.

Scale-Up and Scale-Out of a Gender-Sensitized Weight Management and Healthy Living Program Delivered to Overweight Men via Professional Sports Clubs: The Wider Implementation of Football Fans in Training (FFIT)

Increasing prevalence of obesity poses challenges for public health. Men have been under-served by weight management programs, highlighting a need for gender-sensitized programs that can be embedded into routine practice or adapted for new settings/populations, to accelerate the process of implementing programs that are successful and cost-effective under research conditions. To address gaps in examples of how to bridge the research to practice gap, we describe the scale-up and scale-out of Football Fans in Training (FFIT), a weight management and healthy living program in relation to two implementation frameworks. The paper presents: the development, evaluation and scale-up of FFIT, mapped onto the PRACTIS guide; outcomes in scale-up deliveries; and the scale-out of FFIT through programs delivered in other contexts (other countries, professional sports, target groups, public health focus). FFIT has been scaled-up through a single-license franchise model in over 40 UK professional football clubs to 2019 (and 30 more from 2020) and scaled-out into football and other sporting contexts in Australia, Canada, New Zealand, England and other European countries. The successful scale-up and scale-out of FFIT demonstrates that, with attention to cultural constructions of masculinity, public health interventions can appeal to men and support them in sustainable lifestyle change.

The Invested in Diabetes Study Protocol: a cluster randomized pragmatic trial comparing standardized and patient-driven diabetes shared medical appointments

Background

Shared medical appointments (SMAs) have been shown to be an efficient and effective strategy for providing diabetes self-management education and self-management support. SMA features vary and it is not known which features are most effective for different patients and practice settings. The Invested in Diabetes study tests the comparative effectiveness of SMAs with and without multidisciplinary care teams and patient topic choice for improving patient-centered and clinical outcomes related to diabetes.

Methods

This study compares the effectiveness of two SMA approaches using the Targeted Training for Illness Management (TTIM) curriculum. Standardized SMAs are led by a health educator with a set order of TTIM topics. Patient-driven SMAs are delivered collaboratively by a multidisciplinary care team (health educator, medical provider, behavioral health provider, and a peer mentor); patients select the order and emphasis on TTIM topics. Invested in Diabetes is a cluster randomized pragmatic trial involving approximately 1440 adult patients with type 2 diabetes. Twenty primary care practices will be randomly assigned to either standardized or patient-driven SMAs. A mixed-methods evaluation will include quantitative (practice- and patient-level data) and qualitative (practice and patient interviews, observation) components. The primary patient-centered outcome is diabetes distress. Secondary outcomes include autonomy support, self-management behaviors, clinical outcomes, patient reach, and practice-level value and sustainability.

Discussion

Practice and patient stakeholder input guided protocol development for this pragmatic trial comparing SMA approaches. Implementation strategies from the enhanced Replicating Effective Programs framework will help ensure practices maintain fidelity to intervention protocols while tailoring workflows to their settings. Invested in Diabetes will contribute to the literature on chronic illness management and implementation science using the RE-AIM model.

Trial registration

ClinicalTrials.gov, NCT03590041. Registered on 5 July 2018.

National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

The Golden Compass Program: Overview of the Initial Implementation of a Comprehensive Program for Older Adults Living with HIV

The population with HIV is aging and has unique health needs. We present findings from an evaluation of the geriatric-HIV program, Golden Compass, at San Francisco General Hospital. We used the implementation science framework, RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) to guide the evaluation and used quantitative and qualitative methods to assess RE-AIM dimensions. From January 2017 to June 2018, 198 adults age ≥50 years participated in the program, with an estimated reach of 17%. Providers and patients indicated high acceptability of the program and were satisfied with clinics and classes. Colocation of services, specific pharmacy and geriatric assessments, and social support from classes were valued (effectiveness). Provider adoption was high, and the program was implemented as originally designed. Areas for improvement included challenges of framing aging services to patients. Future efforts will focus on expanding the reach of the program and examining long-term outcomes.

Quantitative approaches for the evaluation of implementation research studies

Implementation research necessitates a shift from clinical trial methods in both the conduct of the study and in the way that it is evaluated given the focus on the impact of implementation strategies. That is, the methods or techniques to support the adoption and delivery of a clinical or preventive intervention, program, or policy. As strategies target one or more levels within the service delivery system, evaluating their impact needs to follow suit. This article discusses the methods and practices involved in quantitative evaluations of implementation research studies. We focus on evaluation methods that characterize and quantify the overall impacts of an implementation strategy on various outcomes. This article discusses available measurement methods for common quantitative implementation outcomes involved in such an evaluation-adoption, fidelity, implementation cost, reach, and sustainment-and the sources of such data for these metrics using established taxonomies and frameworks. Last, we present an example of a quantitative evaluation from an ongoing randomized rollout implementation trial of the Collaborative Care Model for depression management in a large primary healthcare system.

Should We Scale-Up? A Mixed Methods Process Evaluation of an Intervention Targeting Sedentary Office Workers Using the RE-AIM QuEST Framework

Background: Interventions targeting a reduction in sedentary behaviour in office workers need to be scaled-up to have impact. In this study, the RE-AIM QuEST framework was used to evaluate the potential for further implementation and scale-up of a consultation based workplace intervention which targeted both the reduction, and breaking up of sitting time. Methods: To evaluate the Springfield College sedentary behaviour intervention across multiple RE-AIM QuEST indicators; intervention participant, non-participant (employees who did not participate) and key informant (consultation delivery team; members of the research team and stakeholders in workplace health promotion) data were collected using interviews, focus groups and questionnaires. Questionnaires were summarized using descriptive statistics and interviews and focus groups were transcribed verbatim, and thematically analysed. Results: Barriers to scale-up were: participant burden of activity monitoring; lack of management support; influence of policy; flexibility (scheduling/locations); time and cost. Facilitators to scale up were: visible leadership; social and cultural changes in the workplace; high acceptability; existing health and wellbeing programmes; culture and philosophy of the participating college. Conclusions: There is potential for scale-up, however adaptations will need to be made to address the barriers to scale-up. Future interventions in office workers should evaluate for scalability during the pilot phases of research.