Pragmatic Applications of RE-AIM for Health Care Initiatives in Community and Clinical Settings

Pragmatic Application of the RE-AIM Framework to Evaluate the Implementation of Tobacco Cessation Programs Within NCI-Designated Cancer Centers

Tobacco cessation after cancer diagnosis leads to better patient outcomes. However, tobacco treatment services are frequently unavailable in cancer care settings, and multilevel implementation challenges can impede uptake of new programs. The National Cancer Institute (NCI) dedicated Cancer Moonshot funding through the Cancer Center Cessation Initiative (C3I) for NCI-Designated Cancer Centers to implement or enhance the implementation of tobacco treatment services. We examined a pragmatic application of the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) to evaluate tobacco treatment programs implemented within Cancer Centers funded through C3I. Using three C3I-funded Centers as examples, we describe how each RE-AIM construct was operationalized to evaluate the implementation of a wide range of cessation services (e.g., tobacco use screening, counseling, Quitline referral, pharmacotherapy) in this heterogeneous group of cancer care settings. We discuss the practical challenges encountered in assessing RE-AIM constructs in real world situations, including using the electronic health record (EHR) to aid in assessment. Reach and effectiveness evaluation required that Centers define the setting(s) where cessation services were implemented (to determine the “denominator”), enumerate the patient population, report current patient tobacco use, patient engagement in tobacco treatment, and 6-month cessation outcomes. To reduce site heterogeneity, increase data accuracy, and reduce burden, reach was frequently captured via standardized EHR enhancements that improved the identification of current smokers and tobacco treatment referrals. Effectiveness was determined by cessation outcomes (30-day point prevalence abstinence at 6-months post-engagement) assessed through a variety of data collection approaches. Adoption was measured by the characteristics and proportion of targeted cancer care settings and clinicians engaged in cessation service delivery. Implementation was assessed by examining the delivery of tobacco screening assessments and intervention components across sites, and provider-level implementation consistency. Maintenance assessments identified whether tobacco treatment services continued in the setting after implementation and documented the sustainability plan and organizational commitment to continued delivery. In sum, this paper demonstrates a pragmatic approach to using RE-AIM as an evaluation framework that yields relevant outcomes on common implementation metrics across widely differing tobacco treatment approaches and settings.

Making Implementation Science More Rapid: Use of the RE-AIM Framework for Mid-Course Adaptations Across Five Health Services Research Projects in the Veterans Health Administration

Introduction: Implementation science frameworks have helped advance translation of research to practice. They have been widely used for planning and post-hoc evaluation, but seldom to inform and guide mid-course adjustments to intervention and implementation strategies.

Materials and Methods: This study developed an innovative methodology using the RE-AIM framework and related tools to guide mid-course assessments and adaptations across five diverse health services improvement projects in the Veterans Health Administration (VA). Using a semi-structured guide, project team members were asked to assess the importance of and progress on each RE-AIM dimension (i.e., reach, effectiveness, adoption, implementation, maintenance) at the current phase of their project. Based on these ratings, each team identified one or two RE-AIM dimensions for focused attention. Teams developed proximal goals and implementation strategies to improve progress on their selected dimension(s). A follow-up meeting with each team occurred approximately 6 weeks after the goal setting meeting to evaluate the usefulness of the iterative process. Results were evaluated using both descriptive quantitative analyses and qualitative assessments from interviews and meeting notes.

Results: A median of seven team members participated in the two meetings. Qualitative and descriptive data revealed that the process was feasible, understandable and useful to teams in adjusting their interventions and implementation strategies. The RE-AIM dimensions identified as most important were adoption and effectiveness, and the dimension that had the largest gap between importance and rated progress was reach. The dimensions most frequently selected for improvement were reach and adoption. Examples of action plans were summarizing stakeholder interviews for leadership, revising exclusion criteria, and conducting in-service trainings. Follow-up meetings indicated that teams found the process very useful and were able to implement the action plans they set.

Discussion: The iterative use of RE-AIM to support adjustments during project implementation proved feasible and useful across diverse projects in the VA setting. Building on this and related examples, future research should replicate these findings and further develop the methodology, as well as explore the optimal frequency and timing for these iterative applications of RE-AIM. More generally, greater focus on more rapid and iterative use of implementation science frameworks is encouraged to facilitate successful translation of research to practice.

An Extension of RE-AIM to Enhance Sustainability: Addressing Dynamic Context and Promoting Health Equity Over Time

RE-AIM is a widely adopted, robust implementation science (IS) framework used to inform intervention and implementation design, planning, and evaluation, as well as to address short-term maintenance. In recent years, there has been growing focus on the longer-term sustainability of evidence-based programs, policies and practices (EBIs). In particular, investigators have conceptualized sustainability as the continued health impact and delivery of EBIs over a longer period of time (e.g., years after initial implementation) and incorporated the complex and evolving nature of context. We propose a reconsideration of RE-AIM to integrate recent conceptualizations of sustainability with a focus on addressing dynamic context and promoting health equity. In this Perspective, we present an extension of the RE-AIM framework to guide planning, measurement/evaluation, and adaptations focused on enhancing sustainability. We recommend consideration of: (1) extension of "maintenance" within RE-AIM to include recent conceptualizations of dynamic, longer-term intervention sustainability and "evolvability" across the life cycle of EBIs, including adaptation and potential de-implementation in light of changing and evolving evidence, contexts, and population needs; (2) iterative application of RE-AIM assessments to guide adaptations and enhance long-term sustainability; (3) explicit consideration of equity and cost as fundamental, driving forces that need to be addressed across RE-AIM dimensions to enhance sustainability; and (4) use or integration of RE-AIM with other existing frameworks that address key contextual factors and examine multi-level determinants of sustainability. Finally, we provide testable hypotheses and detailed research questions to inform future research in these areas.

Using Integrated Research-Practice Partnerships to Move Evidence-Based Principles Into Practice

The implementation of evidence-based physical activity interventions is improved when integrated research-practice partnerships are used. These partnerships consider both research- and practice-based evidence that moves beyond only assessing program efficacy. Our novel hypothesis is that integrated research-practice partnerships may lead to interventions that are practical and effective, reach more participants, and are more likely to be sustained in practice.

Ten recommendations for using implementation frameworks in research and practice

BACKGROUND

Recent reviews of the use and application of implementation frameworks in implementation efforts highlight the limited use of frameworks, despite the value in doing so. As such, this article aims to provide recommendations to enhance the application of implementation frameworks, for implementation researchers, intermediaries, and practitioners.

DISCUSSION

Ideally, an implementation framework, or multiple frameworks should be used prior to and throughout an implementation effort. This includes both in implementation science research studies and in real-world implementation projects. To guide this application, outlined are ten recommendations for using implementation frameworks across the implementation process. The recommendations have been written in the rough chronological order of an implementation effort; however, we understand these may vary depending on the project or context: (1) select a suitable framework(s), (2) establish and maintain community stakeholder engagement and partnerships, (3) define issue and develop research or evaluation questions and hypotheses, (4) develop an implementation mechanistic process model or logic model, (5) select research and evaluation methods (6) determine implementation factors/determinants, (7) select and tailor, or develop, implementation strategy(s), (8) specify implementation outcomes and evaluate implementation, (9) use a framework(s) at micro level to conduct and tailor implementation, and (10) write the proposal and report. Ideally, a framework(s) would be applied to each of the recommendations. For this article, we begin by discussing each recommendation within the context of frameworks broadly, followed by specific examples using the Exploration, Preparation, Implementation, Sustainment (EPIS) framework.

SUMMARY

The use of conceptual and theoretical frameworks provides a foundation from which generalizable implementation knowledge can be advanced. On the contrary, superficial use of frameworks hinders being able to use, learn from, and work sequentially to progress the field. Following the provided ten recommendations, we hope to assist researchers, intermediaries, and practitioners to improve the use of implementation science frameworks.

Couple-based collaborative management model of type 2 diabetes mellitus for community-dwelling older adults in China: protocol for a hybrid type 1 randomized controlled trial

BACKGROUND

China's limited health care resources cannot meet the needs of chronic disease treatment and management of its rapid growing ageing population. The improvement and maintenance of patient's self-management is essential to disease management. Given disease management mainly occurs in the context of family, this study proposes to validate a Couple-based Collaborative Management Model of chronic diseases that integrates health professionals and family supporters; such as to empower the couples with disease management knowledge and skills, and to improve the couples' health and quality of life.

METHODS

The proposed study will validate a couple-based collaborative management model of Type 2 Diabetes Mellitus (T2DM) in a community-based multicenter, two-arm, randomized controlled trial of block design in Guangzhou, China. Specifically, 194 T2DM patients aged ≥55 and their partners recruited from community health care centers will be randomized at the patient level for each center at a 1:1 ratio into the couple-based intervention arm and the individual-based control arm. For the intervention arm, both the patients and their spouses will receive four-weekly structured group education & training sessions and 2 months of weekly tailored behavior change boosters; while these interventions will be only provided to the patients in the control group. Behavior change incentives will be targeted at the couples or only at the patient respectively. Treatment effects on patients' hemoglobin, spouses' quality of life, alongside couples' behavior outcomes will be compared between arms. Study implementation will be evaluated considering its Reach, Effectiveness, Adoption, Implementation and Maintenance following the RE-AIM framework.

DISCUSSION

This study will generate a model of effective collaboration between community health professionals and patients' family, which will shield light on chronic disease management strategy for the increasing ageing population.

TRIAL REGISTRATION

Chinese Clinical Trial Registry, ChiCTR1900027137, Registered 1st Nov. 2019.

Challenges in Reaching Medicaid and Medicare Enrollees in a Mailed Fecal Immunochemical Test Program

BeneFIT was a demonstration project that worked with a Medicaid/Medicare health plan to implement a mailed fecal immunochemical test (FIT) program. The goal was to reach age-eligible enrollees who were due for colorectal cancer (CRC) screening and prompt them to complete a FIT. One health insurance plan collaborated with six federally qualified health centers (FQHCs) in Oregon. Reach was defined as the percent of eligible individuals overdue for CRC screening who were mailed a FIT in 2016. We examined patient-level factors associated with reach, using multivariable log binomial regression and FIT completion rates at 6 months. The health plan identified 3386 age-eligible members overdue for CRC screening. Of these, 2615 (77.2%) were reached (mailed FIT kits) and 771 (22.8%) were not; 478 (14.1%) because they were not considered to be clinic patients and 290 (8.6%) because of mailing issues. Patient-level factors associated with not being reached were: being male, being Medicaid-insured (vs. Medicare), and having no primary care visits (vs. 4+ visits) in the last year. Among all enrollees identified as overdue for CRC screening, FIT completion rates at 6 months were 14.8% overall and 18.5% in the subgroup reached. In a mailed FIT program, a health insurance plan attempted to reach as many enrollees overdue for CRC screening as possible, however 22.8% were not mailed a FIT. Additional efforts are needed to ensure that the hardest to reach enrollees can participate in CRC screening.

Re-thinking Physical Activity Programs for Older Brazilians and the Role of Public Health Centers: A Randomized Controlled Trial Using the RE-AIM Model

Background: Explored the role of public health centers in the delivery of physical activity programs to older Brazilians.

Methods: Total of 114 older adults (81% women) from public health centers across the city of Florianopolis, Brazil, were randomized into three groups: behavior change group (n = 36), traditional exercise group (n = 52), and control group (n = 26). The behavioral change group included 12 weekly meetings (2 h each). The traditional exercise group offered a 12-week exercise class. Individuals in the control group participated only in measurements. Program evaluation included a mixed-methods approach following the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance). Trained interviewers conducted 12 focus groups and 32 interviews with participants in the program, professionals delivering the programs, community health workers, and local and city administrators overseeing public health centers. Participants completed health, quality of life, and fitness assessments at four time points.

Results: The study reached 11.5% of the eligible population in the community. Older adults' resistance to change and limited understanding of behavior change science by public health center staff hindered program reach. Physician encouraging patient participation and personal invitations by community health workers were perceived as favorable factors. Results of program effectiveness and maintenance suggest that behavior change strategies may be better suited than traditional exercise classes for decreasing sedentary time and increasing moderate-to-vigorous physical activity, as well as improving participants' quality of life. Only 14% of public health centers in the city adopted the programs. Heavy workload of health educators delivering the programs and limited physical space for program delivery were barriers for adoption. The fidelity of program delivery was high and indicates that the programs are culturally-appropriate for the Brazilian context and feasible for implementation by local health educators.

Conclusions: Our findings support the potential for dissemination of behavior change and traditional exercise programs to older adults through public health centers in Brazil.

REBEC: RBR-9pkxn2 (retrospectively registered) Register April 20, 2019.

Planning for Implementation Success Using RE-AIM and CFIR Frameworks: A Qualitative Study

Background: RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) and CFIR (Consolidated Framework for Implementation Research) dissemination and implementation frameworks define theory-based domains associated with the adoption, implementation and maintenance of evidence-based interventions. Used together, the two frameworks identify metrics for evaluating implementation success, i.e., high reach and effectiveness resulting in sustained practice change (RE-AIM), and modifiable factors that explain and enhance implementation outcomes (CFIR). We applied both frameworks to study the implementation planning process for a technology-delivered asthma care intervention called Breathewell within an integrated care organization. The goal of the Breathewell intervention is to increase the efficiency of delivering resource-intensive asthma care services. Methods: We reviewed historical documents (i.e., meeting agendas; minutes) from 14 months of planning to evaluate alignment of implementation team priorities with RE-AIM domains. Key content was extracted and analyzed on topics, frequency and amount of discussion within each RE-AIM domain. Implementation team members were interviewed using questions adapted from the CFIR Interview Guide Tool to focus their reflection on the process and contextual factors considered during pre-implementation planning. Documents and transcripts were initially coded using RE-AIM domain definitions, and recoded using CFIR constructs, with intent to help explain how team decisions and actions can contribute to adoption, implementation and maintenance outcomes. Results: Qualitative analysis of team documents and interviews demonstrated strong alignment with the RE-AIM domains: Reach, Effectiveness, and Implementation; and with the CFIR constructs: formal inclusion of provider and staff stakeholders in implementation planning, compatibility of the intervention with workflows and systems, and alignment of the intervention with organizational culture. Focus on these factors likely contributed to RE-AIM outcomes of high implementation fidelity. However, team members expressed low confidence that Breathewell would be adopted and maintained post-trial. A potential explanation was weak alignment with several CFIR constructs, including tension for change, relative priority, and leadership engagement that contribute to organizational receptivity and motivation to sustain change. Conclusions: While RE-AIM provides a practical framework for planning and evaluating practice change interventions to assure their external validity, CFIR explains why implementation succeeded or failed, and when used proactively, identifies relevant modifiable factors that can promote or undermine adoption, implementation, and maintenance.

ImPlementation REsearCh to DEvelop Interventions for People Living with HIV (the PRECluDE consortium): Combatting chronic disease comorbidities in HIV populations through implementation research

Antiretroviral therapy (ART) prevented premature mortality and improved the quality of life among people living with the human immunodeficiency virus (PLWH), such that now more than half of PLWH in the United States are 50 years of age and older. Increased longevity among PLWH has resulted in a significant rise in chronic, comorbid diseases. However, the implementation of guideline-based interventions for preventing, treating, and managing such age-related, chronic conditions among the HIV population is lacking. The PRECluDE consortium supported by the Center for Translation Research and Implementation Science at the National Heart, Lung, and Blood Institute catalyzes implementation research on proven-effective interventions for co-occurring heart, lung, blood, and sleep diseases and conditions among PLWH. These collaborative research studies use novel implementation frameworks with HIV, mental health, cardiovascular, and pulmonary care to advance comprehensive HIV and chronic disease healthcare in a variety of settings and among diverse populations.

National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions

The National Working Group on RE-AIM Planning and Evaluation Framework (herein Workgroup) was established in 2004 to support the application of the framework and advance dissemination and implementation science (D&I). Workgroup members developed and disseminated products and resources (and continue to do so) to advocate for consistent application of RE-AIM and allow for cross study comparisons. The purpose of this paper is to summarize key Workgroup activities, products, and services (e.g., webinars, consultations, planning tools) and enhance bidirectional communication between the Workgroup and RE-AIM users. The ultimate goal of this work is to serve as a forum for dissemination to improve the balance between RE-AIM user demand (needs) and the currently limited RE-AIM Workgroup supply (consultation and resources) to demonstrate and expand the utility of RE-AIM as a D&I planning and evaluation framework. A summary of resources is provided as well as specific examples of how the Workgroup has been responsive to user needs.

The Golden Compass Program: Overview of the Initial Implementation of a Comprehensive Program for Older Adults Living with HIV

The population with HIV is aging and has unique health needs. We present findings from an evaluation of the geriatric-HIV program, Golden Compass, at San Francisco General Hospital. We used the implementation science framework, RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) to guide the evaluation and used quantitative and qualitative methods to assess RE-AIM dimensions. From January 2017 to June 2018, 198 adults age ≥50 years participated in the program, with an estimated reach of 17%. Providers and patients indicated high acceptability of the program and were satisfied with clinics and classes. Colocation of services, specific pharmacy and geriatric assessments, and social support from classes were valued (effectiveness). Provider adoption was high, and the program was implemented as originally designed. Areas for improvement included challenges of framing aging services to patients. Future efforts will focus on expanding the reach of the program and examining long-term outcomes.

Unraveling the Ethical, Legal, and Social Implications of Neurobiobanking and Stroke Genomic Research in Africa: A Study Protocol of the African Neurobiobank for Precision Stroke Medicine ELSI Project

The ethical, legal, and social implications (ELSI) of emerging neurobiobanks and data resources are unclear in an African scientific landscape with unique cultural, linguistic, and belief systems. The overarching goal of the African Neurobiobank for Precision Stroke Medicine-ELSI Project is to identify, examine, and develop novel approaches to address ELSI issues of biobanking and stroke genomic research in sub-Saharan Africa (SSA). To accomplish the goal we will (1) explore knowledge, attitude, perceptions, barriers, and facilitators influencing ELSI issues related to biobanking and stroke genomic research; (2) use information obtained to craft a community intervention program focused on ELSI issues; and (3) build capacity and careers related to genomics and biobanking for effective client/community engagement while enhancing regulatory, governance, and implementation competences in biobanking science in SSA. A community-based participatory research and mixed-methodological approach, focused on various levels of the social ecological model, will be used to identify and examine relevant ELSI issues. Contextual intervention tools, platforms, and practices will be developed to enhance community understanding and participation in stroke biobanking and genomics research activities while facilitating enduring trust, and equitable and fair utilization of biobanking resources for genetic and trans-omics research. A concurrent capacity building program related to genetic counseling and biobanking will be implemented for early career researchers. The huge potential for neurobiobanking and genomics research in Africa to advance precision medicine applicable to stroke and other neurological disorders requires addressing ELSI challenges while building sustainable research, career, and regulatory capacities in trans-omics and biobanking science.

Rural physical activity interventions in the United States: a systematic review and RE-AIM evaluation

Background

Previous reviews of rural physical activity interventions were focused on intervention effectiveness and had reported overall mixed findings. The purpose of this systematic review was to apply the Reach, Efficacy, Adoption, Implementation and Maintenance (RE-AIM) framework to evaluate the extent to which rural physical activity interventions in the U.S. have reported on dimensions of internal and external validity and to offer suggestions for future physical activity interventions for rural U.S. populations.

Methods

Pubmed, PsychINFO, CINAHL, PAIS, and Web of Science were searched through February 2019 to identify physical activity intervention studies conducted in rural regions in the U.S. with adult populations. Titles, abstracts, and full texts of articles were reviewed against inclusion and exclusion criteria. Data extraction from included articles included a summary of study details, rural classification system used, and the presence or absence of a total 61 RE-AIM indicators, including reach (n = 13), efficacy/effectiveness (n = 10), adoption (n = 21), implementation (n = 9), and maintenance (n = 8).

Results

A total of 40 full-text articles representing 29 unique studies were included. Classifications of rurality included self-statements by authors (n = 19, 65.5%), population/census-based definitions (n = 3, 10.3%), Rural Urban Continuum Codes (n = 3, 10.3%), Rural Urban Commuting Area codes (n = 2, 6.9%), the 2014 Alabama Rural Health Association classification system (n = 1, 3.4%) and the U.S. Office of Management and Budget classification system (n = 1, 3.4%). Individual studies reported between 14.8 to 52.5% of total RE-AIM indicators. Studies reported 15.4 to 84.6% indicators for reach; 20.0 to 70.0% indicators for efficacy/effectiveness; 4.8 to 47.6% indicators for adoption; 11.1 to 88.9% indicators for implementation; and 0 to 25.0% indicators for maintenance.

Conclusions

We found an overall poor reporting of components related to external validity, which hinders the generalizability of intervention findings, and a lack of consistency in the definition of rurality. Future research should focus on balancing factors of internal and external validity, and should aim to develop a greater understanding of how rurality influences health and behavior to provide contextual knowledge needed to advance the translation of physical activity interventions into practice in rural communities and reduce rural health disparities.

Trial registration

The review protocol was registered with PROSPERO: CRD42019116308.

Adapting and Operationalizing the RE-AIM Framework for Implementation Science in Environmental Health: Clean Fuel Cooking Programs in Low Resource Countries

Introduction: The use of models and frameworks to design and evaluate strategies to improve delivery of evidence-based interventions is a foundational element of implementation science. To date, however, evaluative implementation science frameworks such as Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) have not been widely employed to examine environmental health interventions. We take advantage of a unique opportunity to utilize and iteratively adapt the RE-AIM framework to guide NIH-funded case studies of the implementation of clean cooking fuel programs in eleven low- and middle-income countries. Methods: We used existing literature and expert consultation to translate and iteratively adapt the RE-AIM framework across several stages of the NIH Clean Cooking Implementation Science case study project. Checklists and templates to guide investigators were developed at each stage. Results: The RE-AIM framework facilitated identification of important emerging issues across this set of case studies, in particular highlighting the fact that data associated with certain important outcomes related to health and welfare are chronically lacking in clean fuel programs. Monitoring of these outcomes should be prioritized in future implementation efforts. As RE-AIM was not originally designed to evaluate household energy interventions, employing the framework required adaptation. Specific adaptations include the broadening of Effectiveness to encompass indicators of success toward any stated programmatic goal, and expansion of Adoption to include household-level uptake of technology. Conclusions: The RE-AIM implementation science framework proved to be a useful organizing schema for 11 case studies of clean fuel cooking programs, in particular highlighting areas requiring emphasis in future research and evaluation efforts. The iterative approach used here to adapt an implementation science framework to a specific programmatic goal may be of value to other multi-country program efforts, such as those led by international development agencies. The checklists and templates developed for this project are publicly available for others to use and/or further modify.

A single-site pilot implementation of a novel trauma training program for prehospital providers in a resource-limited setting

Background

Prehospital (ambulance) care can reduce morbidity and mortality from trauma. Yet, there is a dearth of effective evidence-based interventions and implementation strategies. Emergency Medical Services Traumatic Shock Care (EMS-TruShoC) is a novel bundle of five core evidence-based trauma care interventions. High-Efficiency EMS Training (HEET) is an innovative training and sensitization program conducted during clinical shifts in ambulances. We assess the feasibility of implementing EMS-TruShoC using the HEET strategy, and feasibility of assessing implementation and clinical outcomes. Findings will inform a main trial.

Methods

We conducted a single-site, prospective cohort, multi-methods pilot implementation study in Western Cape EMS system of South Africa. Of the 120 providers at the study site, 12 were trainers and the remaining were eligible learners. Feasibility of implementation was guided by the RE-AIM (reach, effectiveness, adoption, implementation, and maintenance) framework. Feasibility of assessing clinical outcomes was assessed using shock indices and clinical quality of care scores, collected via abstraction of patients' prehospital trauma charts. Thresholds for progression to a main trial were developed a priori.

Results

The average of all implementation indices was 83% (standard deviation = 10.3). Reach of the HEET program was high, with 84% learners completing at least 75% of training modules. Comparing the proportion of learners attaining perfect scores in post- versus pre-implementation assessments, there was an 8-fold (52% vs. 6%) improvement in knowledge, 3-fold (39% vs. 12%) improvement in skills, and 2-fold (42% vs. 21%) increase in self-efficacy. Clinical outcomes data were successfully calculated-there were clinically significant improvements in shock indices and quality of prehospital trauma care in the post- versus pre-implementation phases. Adoption of HEET was good, evidenced by 83% of facilitator participation in trainings, and 100% of surveyed stakeholders indicating good programmatic fit for their organization. Stakeholders responded that HEET was a sustainable educational solution that aligned well with their organization. Implementation fidelity was very high; 90% of the HEET intervention and 77% of the implementation strategy were delivered as originally planned. Participants provided very positive feedback, and explained that on-the-job timing enhanced their participation. Maintenance was not relevant to assess in this pilot study.

Conclusions

We successfully implemented the EMS-TruShoC educational intervention using the HEET training strategy in a single-site pilot study conducted in a low-resource international setting. All clinical outcomes were successfully calculated. Overall, this pilot study suggests high feasibility of our future, planned experimental trial.

Implementation findings from an effectiveness-implementation trial of tablet-based parent training in pediatric primary care

INTRODUCTION

The use of mobile delivery to deliver parent training can address barriers to access and improve the translation of interventions in existing settings like pediatric primary care. Studying implementation provides critical information to identify and address barriers and facilitators and inform sustainability efforts.

METHOD

This study was a descriptive evaluation using the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework as part of a Hybrid Type I Effectiveness-Implementation trial of the ezParent Program within 4 pediatric primary care clinics. We collected data before, during, and after implementation to evaluate provider implementation and their perspectives on program benefits and barriers to implementation.

RESULTS

On average, 14% of eligible parents were introduced to the study. Of these parents, 78% expressed interest in participating and 37% enrolled in the study. Seventy-eight percent of staff providers (n = 36) implemented the procedures at least once, and among those who implemented 3 or more times (n = 24), 79% maintained implementation for 6 months. Barriers to implementation include limited time, lack of information, and full practice buy-in and engagement.

DISCUSSION

Implementation fidelity may improve with additional education and training of the interdisciplinary team, clear messaging regarding the purpose and content of the ezParent program, defining roles within the care team, identifying practice champions, and use of the electronic health record. Findings from this evaluation, including data from the randomized controlled trial and literature to support intervention effectiveness and implementation, will be used to develop an implementation toolkit to include specific strategies for implementation and ideas for local adaptations. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

A flexible framework for planning and evaluating early-stage health interventions: FRAME-IT

Health interventions exhibit three stages of maturity: early-, mid-, and late-stages. Early-stage interventions have innovative content necessitating evaluation; however existing evaluation frameworks omit constructs and guidelines relevant to this evaluation. Early-stage interventions require planning and evaluation that supports creating, testing, and exploring content to establish general feasibility and enable refinement for further testing, prior to randomised controlled trialling and wider dissemination. Feasibility, Reach-out, Acceptability, Maintenance, Efficacy, Implementation, Tailorability (FRAME-IT) was developed for a mixed methods feasibility study of a novel well-being intervention. FRAME-IT was conceived as a complementary framework to Reach, Efficacy, Adoption, Implementation, Maintenance (RE-AIM; Glasgow et al., 1999) which is better suited for mid- and late- stage interventions. FRAME-IT is proposed to support: (1) early-stage intervention planning and design, by guiding research focus and data sourcing strategy with relevant constructs; (2) comprehensive evaluation, by including constructs appropriate for early-stage interventions, i.e. feasibility, acceptability, and tailorability; (3) future intervention scalability, by including and adapting some of RE-AIM's constructs to encourage a smoother translation of research into practice as interventions are scaled-up.

Understanding cost data collection tools to improve economic evaluations of health interventions

Micro-costing data collection tools often used in literature include standardized comprehensive templates, targeted questionnaires, activity logs, on-site administrative databases, and direct observation. These tools are not mutually exclusive and are often used in combination. Each tool has unique merits and limitations, and some may be more applicable than others under different circumstances. Proper application of micro-costing tools can produce quality cost estimates and enhance the usefulness of economic evaluations to inform resource allocation decisions. A common method to derive both fixed and variable costs of an intervention involves collecting data from the bottom up for each resource consumed (micro-costing). We scanned economic evaluation literature published in 2008-2018 and identified micro-costing data collection tools used. We categorized the identified tools and discuss their practical applications in an example study of health interventions, including their potential strengths and weaknesses. Sound economic evaluations of health interventions provide valuable information for justifying resource allocation decisions, planning for implementation, and enhancing the sustainability of the interventions. However, the quality of intervention cost estimates is seldom addressed in the literature. Reliable cost data forms the foundation of economic evaluations, and without reliable estimates, evaluation results, such as cost-effectiveness measures, could be misleading. In this project, we identified data collection tools often used to obtain reliable data for estimating costs of interventions that prevent and manage chronic conditions and considered practical applications to promote their use.

Behavioral health risk factor profiles in general hospital patients: identifying the need for screening and brief intervention

BACKGROUND

Little is known about the clustering of behavioral health risk factors (HRFs), namely the occurrence of 16 specific combinations of tobacco smoking, at-risk alcohol use, overweight and physical inactivity in general hospital patients. Furthermore, social inequalities in HRFs, health and life expectancy are a major concern in public health. In order to establish the need for screening and intervention in general hospital care, the study aimed to determine the co-occurrence of HRFs in patients in four medical departments, and to investigate differences by gender, age and socio-economic characteristics.

METHODS

Over 17 months, a systematic multiple HRF screening was conducted at one general hospital in northeastern Germany. In total, 6251 18-64 year old patients (92% of eligibles) participated. Proportions and confidence intervals were calculated for all 16 HRF profiles stratified by department, gender, age group, school education, and employment status.

RESULTS

In total, 92.2% of the participants (58.6% male) reported ≥1 HRF, and 65.7% ≥2 HRFs. Men (71.2%), patients aged 35-49 (67.9%) and 50-64 years (69.5%), lower educated (79.0%), and unemployed (77.8%) patients had larger proportions of ≥2 HRFs than their counterparts. In all departments, the most common HRF profiles included overweight. HRF profiles that included alcohol and/ or smoking were more common in ear-nose-throat and trauma surgery than in internal medicine and general surgery patients. Men had higher rates concerning almost all HRF profiles including ≥2 HRFs and alcohol; women concerning profiles that included ≤2 HRFs and inactivity. In older patients, profiles with ≥2 HRFs including overweight; and in younger patients, profiles with smoking and/or alcohol were more common. In lower educated patients, profiles with ≥2 HRFs including inactivity; and in higher educated patients profiles with ≤2 HRFs including alcohol were more common. Compared to others, unemployed patients had higher rates of profiles with ≥3 HRFs including smoking.

CONCLUSIONS

Two in three patients require interventions targeting two or more HRFs. The findings help to develop screening and brief intervention for patients with specific health risk profiles, that can reach most patients, including those most in need and those most hard to reach, with socio-economically disadvantaged people in particular. REGISTRY: clinicaltrials.gov: NCT01291693.

RE-AIM in the Real World: Use of the RE-AIM Framework for Program Planning and Evaluation in Clinical and Community Settings

Background: The RE-AIM framework has been widely used in health research but it is unclear the extent to which this framework is also used for planning and evaluating health-related programs in clinical and community settings. Our objective was to evaluate how RE-AIM is used in the "real-world" and identify opportunities for improving use outside of research contexts. Methods: We used purposive and snowball sampling to identify clinical and community health programs that used RE-AIM for planning and/or evaluation. Recruitment methods included surveys with email follow-up to funders, implementers, and RE-AIM working group members. We identified 17 programs and conducted structured in-depth interviews with key informants (n = 18). Across RE-AIM dimensions, respondents described motivations, uses, and measures; rated understandability and usefulness; discussed benefits and challenges, strategies to overcome challenges, and resources used. We used descriptive statistics for quantitative ratings, and content analysis for qualitative data. Results: Program content areas included chronic disease management and prevention, healthy aging, mental health, or multiple, often behavioral health-related topics. During planning, most programs considered reach (n = 9), adoption (n = 11), and implementation (n = 12) while effectiveness (n = 7) and maintenance (n = 6) were considered less frequently. In contrast, most programs evaluated all RE-AIM dimensions, ranging from 13 programs assessing maintenance to 15 programs assessing implementation and effectiveness. On five-point scales, all RE-AIM dimensions were rated as easy to understand (Overall M = 4.7 ± 0.5), but obtaining data was rated as somewhat challenging (Overall M = 3.4 ± 0.9). Implementation was the most frequently used dimension to inform program design (M = 4.7 ± 0.6) relative to the other dimensions (3.0-3.9). All dimensions were considered similarly important for decision-making (average M = 4.1 ± 1.4), with the exception of maintenance (M = 3.4 ± 1.7). Qualitative corresponded to the quantitative findings in that RE-AIM was reported to be a practical, easy to understand, and well-established implementation science framework. Challenges included understanding differences among RE-AIM dimensions and data acquisition. Valuable resources included the RE-AIM website and collaborating with an expert. Discussion: RE-AIM is an efficient framework for planning and evaluation of clinical and community-based projects. It provides structure to systematically evaluate health program impact. Programs found planning for and assessing maintenance difficult, providing opportunities for further refinement.

Assessing the Integrated Community-Based Health Systems Strengthening initiative in northern Togo: a pragmatic effectiveness-implementation study protocol

BACKGROUND

Over the past decade, prevalence of maternal and child morbidity and mortality in Togo, particularly in the northern regions, has remained high despite global progress. The causes of under-five child mortality in Togo are diseases with effective and low-cost prevention and/or treatment strategies, including malaria, acute lower respiratory infections, and diarrheal diseases. While Togo has a national strategy for implementing the integrated management of childhood illness (IMCI) guidelines, including a policy on integrated community case management (iCCM), challenges in implementation and low public sector health service utilization persist. There are critical gaps to access and quality of community health systems throughout the country. An integrated facility- and community-based initiative, the Integrated Community-Based Health Systems Strengthening (ICBHSS) initiative, seeks to address these gaps while strengthening the public sector health system in northern Togo. This study aims to evaluate the effect and implementation strategy of the ICBHSS initiative over 48 months in the catchment areas of 21 public sector health facilities.

METHODS

The ICBHSS model comprises a bundle of evidence-based interventions targeting children under five, women of reproductive age, and people living with HIV through (1) community engagement and feedback; (2) elimination of point-of-care costs; (3) proactive community-based IMCI using community health workers (CHWs) with additional services including family planning, HIV testing, and referrals; (4) clinical mentoring and enhanced supervision; and (5) improved supply chain management and facility structures. Using a pragmatic type II hybrid effectiveness-implementation study, we will evaluate the ICBHSS initiative with two primary aims: (1) determine effectiveness through changes in under-five mortality rates and (2) assess the implementation strategy through measures of reach, adoption, implementation, and maintenance. We will conduct a mixed-methods assessment using the RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework. This assessment consists of four components: (1) a stepped-wedge cluster randomized control trial using a community-based household survey, (2) annual health facility assessments, (3) key informant interviews, and (4) costing and return-on-investment assessments for each randomized cluster.

DISCUSSION

Our research is expected to contribute to continuous quality improvement initiatives, optimize implementation factors, provide knowledge regarding health service delivery, and accelerate health systems improvements in Togo and more broadly.

TRIAL REGISTRATION

ClinicalTrials.gov , NCT03694366 , registered 3 October 2018.

Effects and acceptability of implementing improved cookstoves and heaters to reduce household air pollution: a FRESH AIR study

The objective was to evaluate the effectiveness and acceptability of locally tailored implementation of improved cookstoves/heaters in low- and middle-income countries. This interventional implementation study among 649 adults and children living in rural communities in Uganda, Vietnam and Kyrgyzstan, was performed after situational analyses and awareness programmes. Outcomes included household air pollution (PM_2.5 and CO), self-reported respiratory symptoms (with CCQ and MRC-breathlessness scale), chest infections, school absence and intervention acceptability. Measurements were conducted at baseline, 2 and 6–12 months after implementing improved cookstoves/heaters. Mean PM_2.5 values decrease by 31% (to 95.1 µg/m³) in Uganda (95%CI 71.5–126.6), by 32% (to 31.1 µg/m³) in Vietnam (95%CI 24.5–39.5) and by 65% (to 32.4 µg/m³) in Kyrgyzstan (95%CI 25.7–40.8), but all remain above the WHO guidelines. CO-levels remain below the WHO guidelines. After intervention, symptoms and infections diminish significantly in Uganda and Kyrgyzstan, and to a smaller extent in Vietnam. Quantitative assessment indicates high acceptance of the new cookstoves/heaters. In conclusion, locally tailored implementation of improved cookstoves/heaters is acceptable and has considerable effects on respiratory symptoms and indoor pollution, yet mean PM_2.5 levels remain above WHO recommendations.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.

Health professional-delivered obesity prevention interventions during the first 1,000 days: A systematic review of external validity reporting

Background: Childhood obesity prevention interventions delivered by health professionals during the first 1,000 days show some evidence of effectiveness, particularly in relation to behavioural outcomes. External validity refers to how generalisable interventions are to populations or settings beyond those in the original study. The degree to which external validity elements are reported in such studies is unclear however. This systematic review aimed to determine the extent to which childhood obesity interventions delivered by health professionals during the first 1,000 days report on elements that can be used to inform generalizability across settings and populations. Methods: Eligible studies meeting study inclusion and exclusion criteria were identified through a systematic review of 11 databases and three trial registers. An assessment tool based on the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used to assess the external validity of included studies. It comprised five dimensions: reach and representativeness of individuals, reach and representativeness of settings, implementation and adaptation, outcomes for decision making maintenance and/or institutionalisation. Two authors independently assessed the external validity of 20% of included studies; discrepancies were resolved, and then one author completed assessments of the remaining studies. Results: In total, 39 trials involving 46 interventions published between 1999 and 2019 were identified. The majority of studies were randomized controlled trials (n=24). Reporting varied within and between dimensions. External validity elements that were poorly described included: representativeness of individuals and settings, treatment receipt, intervention mechanisms and moderators, cost effectiveness, and intervention sustainability and acceptability. Conclusions: Our review suggests that more emphasis is needed on research designs that consider generalisability, and the reporting of external validity elements in early life childhood obesity prevention interventions. Important gaps in external validity reporting were identified that could facilitate decisions around the translation and scale-up of interventions from research to practice. Registration: PROSPERO CRD42016050793 03/11/16.

Designing a program evaluation for a medical-dental service for adults with autism and intellectual disabilities using the RE-AIM framework

INTRODUCTION

Robust evaluation of service models can improve the quality and efficiency of care while articulating the models for potential replication. Even though it is an essential part of learning health systems, evaluations that benchmark and sustain models serving adults with developmental disabilities are lacking, impeding pilot programs from becoming official care pathways. Here, we describe the development of a program evaluation for a specialized medical-dental community clinic serving adults with autism and intellectual disabilities in Montreal, Canada.

METHOD

Using a Participatory Action-oriented approach, researchers and staff co-designed an evaluation for a primary care service for this population. We performed an evaluability assessment to identify the processes and outcomes that were feasible to capture and elicited perspectives at both clinical and health system levels. The RE-AIM framework was used to categorize and select tools to capture data elements that would inform practice at the clinic.

RESULTS

We detail the process of conceptualizing the evaluation framework and operationalizing the domains using a mixed-methods approach. Our experience demonstrated (1) the utility of a comprehensive framework that captures contextual factors in addition to clinical outcomes, (2) the need for validated measures that are not cumbersome for everyday practice, (3) the importance of understanding the functional needs of the organization and building a sustainable data infrastructure that addresses those needs, and (4) the need to commit to an evolving, "living" evaluation in a dynamic health system.

CONCLUSIONS

Evaluation employing rigorous patient-centered and systems-relevant metrics can help organizations effectively implement and continuously improve service models. Using an established framework and a collaborative approach provides an important blueprint for a program evaluation in a learning health system. This work provides insight into the process of integrating care for vulnerable populations with chronic conditions in health care systems and integrated knowledge generation processes between research and health systems.

Can brief alcohol interventions in general hospital inpatients improve mental and general health over 2 years? Results from a randomized controlled trial

BACKGROUND

Little is known about the impact of brief alcohol interventions on mental and general health. The aim was to investigate whether brief interventions for general hospital inpatients with at-risk drinking can improve mental and general health over 2 years; and whether effects are dependent on how they are delivered: in-person or through computer-generated feedback letters (CO).

METHODS

Three-arm randomized controlled trial with 6-, 12-, 18-, and 24-month follow-ups. Data were collected on 13 general hospital wards from four medical departments (internal medicine, surgical medicine, trauma surgery, and ear-nose-throat) of one university hospital in northeastern rural Germany. A consecutive sample of 961 18- to 64-year-old general hospital inpatients with at-risk alcohol use was recruited through systematic screening. Inpatients with particularly severe alcohol problems were excluded. Participants were allocated to: in-person counseling (PE), CO, and assessment only (AO). PE and CO included three contacts: on the ward, 1, and 3 months later. Mental and general health were assessed using the five-item mental health inventory (0-100) and a one-item general health measure (0, poor - 4, excellent).

RESULTS

Latent growth models including all participants revealed: after 24 months and in contrast to AO, mental and general health were improved in PE (change in mean difference, ΔMmental = 5.13, p = 0.002, Cohen's d = 0.51; ΔMgeneral = 0.20, p = 0.005, d = 0.71) and CO (ΔMmental = 6.98, p < 0.001, d = 0.69; ΔMgeneral = 0.24, p = 0.001, d = 0.86). PE and CO did not differ significantly.

CONCLUSIONS

Beyond drinking reduction, PE and CO can improve general hospital inpatients' self-reported mental and general health over 2 years.

Operationalizing the reach, effectiveness, adoption, implementation, maintenance (RE-AIM) framework to evaluate the collective impact of autonomous community programs that promote health and well-being

Background

The RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework is a useful tool for evaluating the impact of programs in community settings. RE-AIM has been applied to evaluate individual programs but seldom used to evaluate the collective impact of community-based, public health programming developed and delivered by multiple autonomous organizations. The purposes of this paper were to (a) demonstrate how RE-AIM can be operationalized and applied to evaluate the collective impact of similar autonomous programs that promote health and well-being and (b) provide preliminary data on the collective impact of Canadian spinal cord injury (SCI) peer mentorship programs on the delivery of peer mentorship services.

Methods

Criteria from all five RE-AIM dimensions were operationalized to evaluate multiple similar community-based programs. For this study, nine provincial organizations that serve people with SCI were recruited from across Canada. Organizations completed a structured self-report questionnaire and participated in a qualitative telephone interview to examine different elements of their peer mentorship program. Data were analyzed using summary statistics.

Results

Having multiple indicators to assess RE-AIM dimensions provided a broad evaluation of the impact of Canadian SCI peer mentorship programs. Peer mentorship programs reached 1.63% of the estimated Canadian SCI population. The majority (67%) of organizations tracked the effectiveness of peer mentorship through testimonials and reports. Setting-level adoption rates were high with 100% of organizations offering peer mentorship in community and hospital settings. On average, organizations allocated 10.4% of their operating budget and 9.8% of their staff to implement peer mentorship and 89% had maintained their programming for over 10 years. Full interpretation of the collective impact of peer mentorship programs was limited as complete data were only collected for 52% of survey questions.

Conclusions

The lack of available organizational data highlights a significant challenge when using RE-AIM to evaluate the collective impact of multiple programs that promote health and well-being. Although researchers are encouraged to use RE-AIM to evaluate the collective impact of programs delivered by different organizations, documenting limitations and providing recommendations should be done to further the understanding of how best to operationalize RE-AIM in these contexts.

Electronic supplementary material

The online version of this article (10.1186/s12889-019-7131-4) contains supplementary material, which is available to authorized users.

Challenges in assessing population reach in a pragmatic trial

Strategies and Opportunities to STOP Colon Cancer in Priority Populations (STOP CRC) was a pragmatic cluster-randomized trial conducted at federally qualified health centers and designed to “Reach” as many unscreened patients as possible by directly mailing them fecal screening tests.

STOP CRC used an electronic health record registry to identify individuals' needing CRC screening and mail interventions to them. The registry was updated daily removing individuals completing CRC screening or those who no longer were clinic patients. Reach, a component RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance), is defined as the absolute number, percent, and representativeness of individuals “willing to participate in” or “exposed to” an initiative. We describe the complexities of measuring Reach in a pragmatic trial.

Overall 21,134 patients were on the registry list for at least one day, with 18,226 remaining after removing patients completing screening before any mailings. Observed Reached (the percent of individuals exposed to the intervention) using each denominator was 30.7% and 35.6% respectively. Reach improved only modestly after accounting for factors that made it impossible for clinics to send mailings. Few differences were observed in demographic and health care utilization factors among individuals Reached versus not Reached, suggesting that health center Implementation was more influential than patients' willingness or ability to participate.

A pragmatic definition of Reach that accounted for dynamic changes the absolute number eligible and the proportion exposed was more useful than traditional definitions of Reach. Actual Reach was dependent on Implementation and not patient level characteristics.

Clinical Trials Registration Number: ClincalTrials.gov (NCT01742065).

Closed-loop electronic referral to SmokefreeTXT for smoking cessation support: a demonstration project in outpatient care

Too few smokers who present for outpatient healthcare receive evidence-based interventions to stop smoking. Referral to nationally available smoking cessation support may enhance tobacco intervention reach during healthcare visits. This study evaluated the feasibility of outpatient electronic health record (EHR)-enabled, closed-loop referral (eReferral) to SmokefreeTXT, a National Cancer Institute text message smoking cessation program. SmokefreeTXT eReferral for adult patients who smoke was implemented in a family medicine clinic and an allergy and asthma clinic in an integrated Midwestern healthcare system. Interoperable, HIPAA-compliant eReferral returned referral outcomes to the EHR. In Phase 1 of implementation, clinicians were responsible for eReferral; in Phase 2 this responsibility shifted to Medical Assistants and/or nurses. EHR data were extracted to compute eReferral rates among adult smokers and compare demographics among those eReferred versus not referred. SmokefreeTXT data were used to compute SmokefreeTXT enrollment rates among those eReferred. Descriptive analyses of clinic staff surveys assessed implementation context and staff attitudes toward and adaptations of eReferral processes. During clinician implementation, 43 of 299 adult smokers (14.4%) were eReferred. During medical assistant (MA) implementation, 36 of 401 adult smokers (9.0%) were eReferred. Overall, among those eReferred, 25.7% completed SmokefreeTXT enrollment (3.1% of patients eligible for eReferral). Staff survey responses indicated that eReferral was efficient and easy. eReferral rates and relevant attitudes varied meaningfully by clinic. Thus, interoperable eReferral via outpatient EHR to SmokefreeTXT is feasible and acceptable to clinic staff and enrolls roughly 3.0% of smokers. Clinic context and implementation approach may influence reach.

Walk This Way: Our Perspective on Challenges and Opportunities for Extension Statewide Walking Promotion Programs

Evidence-based walking programs exist and some have been tailored specifically for the national Cooperative Extension System; however, program outcomes and translational challenges and successes are underreported. This has presented a challenge to scaling the best-fit intervention for walking promotion within this national system. Here, we describe existing open-access walking programs as well as implications for improving the fit and sustainability of this intervention type within the system. Our experience provides suggestions for pragmatic data collection, infrastructure to support pragmatic data collection, and novel ways to disseminate best practices, as well as considerations for de-implementing what is not working.

RE-AIM Planning and Evaluation Framework: Adapting to New Science and Practice With a 20-Year Review

The RE-AIM planning and evaluation framework was conceptualized two decades ago. As one of the most frequently applied implementation frameworks, RE-AIM has now been cited in over 2,800 publications. This paper describes the application and evolution of RE-AIM as well as lessons learned from its use. RE-AIM has been applied most often in public health and health behavior change research, but increasingly in more diverse content areas and within clinical, community, and corporate settings. We discuss challenges of using RE-AIM while encouraging a more pragmatic use of key dimensions rather than comprehensive applications of all elements. Current foci of RE-AIM include increasing the emphasis on cost and adaptations to programs and expanding the use of qualitative methods to understand "how" and "why" results came about. The framework will continue to evolve to focus on contextual and explanatory factors related to RE-AIM outcomes, package RE-AIM for use by non-researchers, and integrate RE-AIM with other pragmatic and reporting frameworks.

Making Health Research Matter: A Call to Increase Attention to External Validity

Most of the clinical research conducted with the goal of improving health is not generalizable to nonresearch settings. In addition, scientists often fail to replicate each other's findings due, in part, to lack of attention to contextual factors accounting for their relative effectiveness or failure. To address these problems, we review the literature on assessment of external validity and summarize approaches to designing for generalizability. When investigators conduct systematic reviews, a critical need is often unmet: to evaluate the pragmatism and context of interventions, as well as their effectiveness. Researchers, editors, and grant reviewers can implement key changes in how they consider and report on external validity issues. For example, the recently published expanded CONSORT figure may aid scientists and potential program adopters in summarizing participation in and representativeness of a program across different settings, staff, and patients. Greater attention to external validity is needed to increase reporting transparency, improve program dissemination, and reduce failures to replicate research.

Online Resources for Dissemination and Implementation Science: Meeting Demand and Lessons Learned

A dramatically increased interest in dissemination and implementation (D&I) science, with relatively few training programs for D&I scientists, highlights the need for innovative ways to deliver educational materials, training, and resources. We described nine interactive, web-based D&I science resources appropriate for trainees and Clinical and Translational Science Awards. We used audience feedback and design thinking to develop resources iteratively. Primary target users are T3–T4 researchers, although T2 researchers can benefit from “designing for dissemination” resources. Workforce development resources were used in D&I science workshops, as stand-alone, self-directed resources, and for consultations and trainings. We assessed resource design (purpose, functionality), usage, user experience and engagement. Educational resources addressed included: D&I science basics, pragmatic trials, getting proposals funded, designing for dissemination, and D&I science theory selection. We reviewed the purpose, functionality, status, and usage of these interactive resources. All resources engaged users; provided interactive feedback for learners; and linked users to additional learning resources. Online resources can be valuable for preparing clinical and translational mentees for research consultations, as follow-up training activities, and as D&I workforce development resources. The resources described are publicly available and we encourage their use, further development, and evaluation by Clinical and Translational Science Awards and other programs.

Beginning With the End in Mind: Contextual Considerations for Scaling-Out a Community-Based Intervention

Introduction: A number of effective physical activity programs for older adults exist, but are not widely delivered within community settings, such as the Cooperative Extension System. The purpose of this paper was to determine if an evidence-based intervention (EBI) developed in one state Extension system could be scaled-out to a new state system. Methods and results: The RE-AIM (reach, effectiveness, adoption, implementation, maintenance) framework was used to guide an iterative evaluation of three translational stages. Stage 1: Before program adoption, Extension health educators were surveyed and interviewed to assess physical activity programming perceptions and factors that may influence their decision to attend training or deliver the program in practice. Results indicated that a virtual, scalable training protocol would be necessary and that training needed to include hands-on instruction and be catered to those who were less confident in physical activity program delivery. Stage 2: Training attendees were surveyed pre- and post-training on factors related to the adoption-decision making process and contacted post-training to assess program delivery status. Training did not influence perceptions of the program, intent to deliver, or confidence in delivering the program. Stage 3: During program implementation, the program was evaluated through the RE-AIM framework by surveying across three key stakeholder groups: (1) program participants, (2) potential delivery personnel, and (3) Extension administrators. Findings indicate that the program has the potential to reach a large and representative proportion of the target audience, especially in rural areas. However, adoption and implementation rates among Extension health educators and community partners were low and data collection for effectiveness, implementation, and maintenance was a challenge. Conclusion: Overall, the results indicate initial struggles to translating and evaluating the program in a large, rural state. Implications for practice include making system-level changes to increase physical activity program adoption rates among Extension health educators and improve data collection and program evaluation through this community-based organization. More work is needed to identify infrastructure support and capacity to scale-out EBIs.

The Swedish version of the Normalization Process Theory Measure S-NoMAD: translation, adaptation, and pilot testing

BACKGROUND

The original British instrument the Normalization Process Theory Measure (NoMAD) is based on the four core constructs of the Normalization Process Theory: Coherence, Cognitive Participation, Collective Action, and Reflexive Monitoring. They represent ways of thinking about implementation and are focused on how interventions can become part of everyday practice.

AIM

To translate and adapt the original NoMAD into the Swedish version S-NoMAD and to evaluate its psychometric properties based on a pilot test in a health care context including in-hospital, primary, and community care contexts.

METHODS

A systematic approach with a four-step process was utilized, including forward and backward translation and expert reviews for the test and improvement of content validity of the S-NoMAD in different stages of development. The final S-NoMAD version was then used for process evaluation in a pilot study aimed at the implementation of a new working method for individualized care planning. The pilot was executed in two hospitals, four health care centres, and two municipalities in a region in northern Sweden. The S-NoMAD pilot results were analysed for validity using confirmatory factor analysis, i.e. a one-factor model fitted for each of the four constructs of the S-NoMAD. Cronbach's alpha was used to ascertain the internal consistency reliability.

RESULTS

In the pilot, S-NoMAD data were collected from 144 individuals who were different health care professionals or managers. The initial factor analysis model showed good fit for two of the constructs (Coherence and Cognitive Participation) and unsatisfactory fit for the remaining two (Collective Action and Reflexive Monitoring) based on three items. Deleting those items from the model yielded a good fit and good internal consistency (alphas between 0.78 and 0.83). However, the estimation of correlations between the factors showed that the factor Reflexive Monitoring was highly correlated (around 0.9) with the factors Coherence and Collective Action.

CONCLUSIONS

The results show initial satisfactory psychometric properties for the translation and first validation of the S-NoMAD. However, development of a highly valid and reliable instrument is an iterative process, requiring more extensive validation in various settings and populations. Thus, in order to establish the validity and reliability of the S-NoMAD, additional psychometric testing is needed.

An analysis of efforts to scale up clean household energy for cooking around the world

Approximately 3 billion people, most of whom live in Asia, Africa, and the Americas, rely on solid fuels (i.e. wood, crop wastes, dung, charcoal) and kerosene for their cooking needs. Exposure to household air pollution from burning these fuels is estimated to account for approximately 3 million premature deaths a year. Cleaner fuels - such as liquefied petroleum gas, biogas, electricity, and certain compressed biomass fuels - have the potential to alleviate much of this significant health burden. A wide variety of clean cooking intervention programs are being implemented around the world, but very few of these efforts have been analyzed to enable global learning. The Clean Cooking Implementation Science Network (ISN), supported by the U.S. National Institutes of Health (NIH) and partners, identified the need to augment the publicly available literature concerning what has worked well and in what context. The ISN has supported the development of a systematic set of case studies, contained in this Special Issue, examining clean cooking program rollouts in a variety of low- and middle-income settings around the world. We used the RE-AIM (reach, effectiveness, adaptation, implementation, maintenance) framework to coordinate and evaluate the case studies. This paper describes the clean cooking case studies project, introduces the individual studies contained herein, and proposes a general conceptual model to support future planning and evaluation of household energy programs.

Expanding the CONSORT Figure: Increasing Transparency in Reporting on External Validity

INTRODUCTION

There are major problems with failure to replicate research findings. Contributing to this problem is a failure to report on factors related to external validity. Frequently, researchers have little knowledge whether findings apply more generally, especially to low-resource settings and underserved populations. The CONSORT flow diagram has improved reporting on variables related to internal validity, but it has very limited detail on issues related to external validity. A recent CONSORT update and other publications have called for more transparent reporting on external validity and context, and information regarding the sustainability of interventions. All of these elements influence the generalizability of findings from outcomes research.

METHODS

Drawing on theory, a prior meeting, and recent recommendations for reporting factors related to external validity, the authors propose an expansion of the basic CONSORT flow diagram for clinical trials to concisely summarize these data that recent CONSORT statements and other guidelines have recommended.

RESULTS

The authors propose the use of an expanded CONSORT figure and illustrate its utility with an example. The expanded CONSORT figure adds data about participation and representativeness at the levels of settings and staff, and about intervention sustainability after project support ends. The authors provide an expanded CONSORT figure reporting template, and demonstrate its use.

CONCLUSIONS

Improving transparent reporting on external validity by using the proposed expanded CONSORT figure would help to address both the scientific replication crisis and health equity concerns. This figure provides a method to efficiently address the representativeness, generalizability, and sustainability of outcomes research.

An Algorithm for Digital Medicine Testing: A NODE.Health Perspective Intended to Help Emerging Technology Companies and Healthcare Systems Navigate the Trial and Testing Period prior to Full-Scale Adoption

Over the last several years, there has been rapid growth of digital technologies attempting to transform healthcare. Unique features of digital medicine technology lead to both challenges and opportunities for testing and validation. Yet little guidance exists to help a health system decide whether to undertake a pilot test of new technology, move right to full-scale adoption or start somewhere in between. To navigate this complexity, this paper proposes an algorithm to help choose the best path toward validation and adoption. Special attention is paid to considering whether the needs of patients with limited digital skills, equipment (e.g., smart-phones) and connectivity (e.g., data plans) have been considered in technology development and deployment. The algorithm reflects the collective experience of 20+ health systems and academic institutions that have established the Network of Digital Evidence for Health, NODE. Health, plus insights from existing clinical research taxonomies, syntheses or frameworks for assessing technology or for reporting clinical trials.

Pragmatic dissemination and implementation research models, methods and measures and their relevance for nursing research

BACKGROUND

Pragmatic dissemination and implementation (D&I) research approaches can benefit patient care because they emphasize real-world settings and populations. Nurse scientists have an opportunity to reduce the gap between science and practice by using pragmatic D&I research and sustainability strategies.

PURPOSE

This article discusses pragmatic models, methods, and measures used in D&I research and their relevance for nursing research and enhancing population health.

METHODS

Summary of pragmatic D&I models and related methods for designing a pragmatic studies. We discuss the RE-AIM framework and the PRECIS-2 planning aid and figure in detail. A case study is provided and application to nursing research is discussed.

DISCUSSION

Successful translation of pragmatic D&I research demands an approach that addresses external validity, and customization at multiple levels including the patient, clinician, and setting. Context is critically important, and it is never too early to design for dissemination.

CONCLUSIONS

Pragmatic D&I approaches are needed to speed research translation, reduce avoidable waste of funding, improve clinical care, and enhance population health. Pragmatic D&I research is an area of tremendous opportunity for the nursing science community.

RE-AIM in Clinical, Community, and Corporate Settings: Perspectives, Strategies, and Recommendations to Enhance Public Health Impact

The RE-AIM Framework is a planning and evaluation model that has been used in a variety of settings to address various programmatic, environmental, and policy innovations for improving population health. In addition to the broad application and diverse use of the framework, there are lessons learned and recommendations for the future use of the framework across clinical, community, and corporate settings. The purposes of this article are to: (A) provide a brief overview of the RE-AIM Framework and its pragmatic use for planning and evaluation; (B) offer recommendations to facilitate the application of RE-AIM in clinical, community, and corporate settings; and (C) share perspectives and lessons learned about employing RE-AIM dimensions in the planning, implementation, and evaluation phases within these different settings. In this article, we demonstrate how the RE-AIM concepts and elements within each dimension can be applied by researchers and practitioners in diverse settings, among diverse populations and for diverse health topics.

Qualitative approaches to use of the RE-AIM framework: rationale and methods

BACKGROUND

There have been over 430 publications using the RE-AIM model for planning and evaluation of health programs and policies, as well as numerous applications of the model in grant proposals and national programs. Full use of the model includes use of qualitative methods to understand why and how results were obtained on different RE-AIM dimensions, however, recent reviews have revealed that qualitative methods have been used infrequently. Having quantitative and qualitative methods and results iteratively inform each other should enhance understanding and lessons learned.

METHODS

Because there have been few published examples of qualitative approaches and methods using RE-AIM for planning or assessment and no guidance on how qualitative approaches can inform these processes, we provide guidance on qualitative methods to address the RE-AIM model and its various dimensions. The intended audience is researchers interested in applying RE-AIM or similar implementation models, but the methods discussed should also be relevant to those in community or clinical settings.

RESULTS

We present directions for, examples of, and guidance on how qualitative methods can be used to address each of the five RE-AIM dimensions. Formative qualitative methods can be helpful in planning interventions and designing for dissemination. Summative qualitative methods are useful when used in an iterative, mixed methods approach for understanding how and why different patterns of results occur.

CONCLUSIONS

In summary, qualitative and mixed methods approaches to RE-AIM help understand complex situations and results, why and how outcomes were obtained, and contextual factors not easily assessed using quantitative measures.