Adapting and Operationalizing the RE-AIM Framework for Implementation Science in Environmental Health: Clean Fuel Cooking Programs in Low Resource Countries

Applying Implementation Science to Advance Electronic Health Record-Driven Learning Health Systems: Case Studies, Challenges, and Recommendations

With the widespread implementation of electronic health records (EHRs), there has been significant progress in developing learning health systems (LHSs) aimed at improving health and health care delivery through rapid and continuous knowledge generation and translation. To support LHSs in achieving these goals, implementation science (IS) and its frameworks are increasingly being leveraged to ensure that LHSs are feasible, rapid, iterative, reliable, reproducible, equitable, and sustainable. However, 6 key challenges limit the application of IS to EHR-driven LHSs: barriers to team science, limited IS experience, data and technology limitations, time and resource constraints, the appropriateness of certain IS approaches, and equity considerations. Using 3 case studies from diverse health settings and 1 IS framework, we illustrate these challenges faced by LHSs and offer solutions to overcome the bottlenecks in applying IS and utilizing EHRs, which often stymie LHS progress. We discuss the lessons learned and provide recommendations for future research and practice, including the need for more guidance on the practical application of IS methods and a renewed emphasis on generating and accessing inclusive data.

Leveraging Implementation Science to Advance Environmental Justice Research and Achieve Health Equity through Neighborhood and Policy Interventions

Environmental justice research is increasingly focused on community-engaged, participatory investigations that test interventions to improve health. Such research is primed for the use of implementation science-informed approaches to optimize the uptake and use of interventions proven to be effective. This review identifies synergies between implementation science and environmental justice with the goal of advancing both disciplines. Specifically, the article synthesizes the literature on neighborhood-, community-, and policy-level interventions in environmental health that address underlying structural determinants (e.g., structural racism) and social determinants of health. Opportunities to facilitate and scale the equitable implementation of evidence-based environmental health interventions are highlighted, using urban greening as an illustrative example. An environmental justice-focused version of the implementation science subway is provided, which highlights these principles: Remember and Reflect, Restore and Reclaim, and Reinvest. The review concludes with existing gaps and future directions to advance the science of implementation to promote environmental justice.

Interventions connecting young people living in Africa to healthcare; a systematic review using the RE-AIM framework

Introduction

Africa's young people are among the least focused groups in healthcare linkage. The disproportionally high burden of youth-related health problems is a burden, especially in developing regions like Africa, which have a high population of young people. More information is needed about factors that impact linkages in healthcare and the sustainability of health interventions among young people in Africa.

Methods

A systematic literature search was performed from October 2020 to May 2022 in PubMed, CINAHL, Scopus, Global Health, and the Web of Science. Studies included in the review were conducted among young people aged 10-24 living in Africa, written in English, and published between 2011 and 2021. Results were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Data was analyzed using narrative synthesis, synthesizing the details of the RE-AIM reporting component. Interventions were systematically compared using the Cochrane Collaboration risk-of-bias tool to evaluate the rigor of each intervention.

Results

A total of 2,383 potentially relevant citations were obtained after an initial database search. Retained in the final group were seventeen articles from electronic data searches; among these articles, 16 interventions were identified. Out of the seventeen studies, nine (53%) were randomized controlled trials, three (18%) were quasi-experimental designs, and five (29%) were observational studies. At the same time, the included interventions were reported on 20 (76.92%) of the 26 components of the RE-AIM dimensions. In eastern Africa, twelve (80%) interventions were conducted, and all the interventions addressed linkage to care for young people in preventing and treating HIV. The least reported RE-AIM dimensions were implementing and maintaining interventions connecting young people to care.

Discussion

Timely care remains critical to treating and preventing ailments. This review indicates that interventions created to link young people to care, especially HIV care, can help link them to health care and strengthen the programs. It is also clear that further research with more extended follow-up periods is needed to examine connections to care in all other aspects of health and to bridge the gap between research and practice in the care of young people in Africa.

Systematic Review Registration

PROSPERO [CRD42022288227].

Applying RE-AIM to evaluations of Veterans Health Administration Enterprise-Wide Initiatives: lessons learned

Introduction

The United States Veterans Health Administration (VHA) Office of Rural Health funds Enterprise-Wide Initiatives (system-wide initiatives) to spread promising practices to rural Veterans. The Office requires that evaluations of Enterprise-Wide Initiatives use the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. This presents a unique opportunity to understand the experience of using RE-AIM across a series of evaluations. The authors conducted a study to document the benefits and pitfalls of using RE-AIM, capture the variety of ways that the team captured the elements of RE-AIM, and develop recommendations for the future use of RE-AIM in evaluation.

Materials and methods

The authors first conducted a document review to capture pre-existing information about how RE-AIM was used. They subsequently facilitated two focus groups to gather more detailed information from team members who had used RE-AIM. Finally, they used member-checking throughout the writing process to ensure accurate data representation and interpretation and to gather additional feedback.

Results

Four themes emerged from the document review, focus groups, and member checking. RE-AIM: provides parameters and controls the evaluation scope, "buckets" are logical, plays well with other frameworks, and can foster collaboration or silo within a team. Challenges and attributes for each RE-AIM dimension were also described.

Discussion

Overall, participants reported both strengths and challenges to using RE-AIM as an evaluation framework. The overarching theme around the challenges with RE-AIM dimensions was the importance of context. Many of these benefits and challenges of using RE-AIM may not be unique to RE-AIM and would likely occur when using any prescribed framework. The participants reported on the RE-AIM domains in a variety of ways in their evaluation reports and were not always able capture data as originally planned. Recommendations included: start with an evaluation framework (or frameworks) and revisit it throughout the evaluation, consider applying RE-AIM PRISM (Practical Robust Implementation Framework) to gain a broader perspective, and intentionally integrate quantitative and qualitative team members, regardless of the framework used.

Implementation of an evidence-based tobacco control intervention for school teachers in India: Evaluating the effects of a capacity-building strategy

Background

Tobacco-Free Teachers, Tobacco-Free Society (TFT-TFS) is an evidence-based intervention that promotes tobacco use cessation among teachers and tobacco control policies among schools in India. This study tested an implementation model to build Bihar Department of Education (DOE) capacity to support and deliver TFT-TFS within schools, leveraging DOE training infrastructure.

Method

We used a training-of-trainers (TOT) "cascade" implementation strategy to embed the TFT-TFS program into the Bihar DOE infrastructure. We trained 46 Cluster Coordinators to train and support Headmasters to implement TFT-TFS in their schools over one academic year. We selected three school districts, representing approximately 46 clusters and 219 schools. We used the RE-AIM framework to assess program adoption (Headmaster participation in at least one of six TFT-TFS trainings), implementation (of four core program components), and reach (teachers' participation in three or more group discussions). Using a non-inferiority design, we hypothesized that program adoption, implementation, and reach would not be inferior to the high standards demonstrated when TFT-TFS was originally tested in the Bihar School Teachers Study. We used self-reported checklists to measure outcomes and SPSS Version 25 to analyze data.

Results

For adoption, 94% of Headmasters attended the first training, although participation declined by the sixth training. Among the 112 schools out of 219 with complete Headmaster checklist data, all met our minimum criteria for implementing TFT-TFS. Over 99% of schools posted a school tobacco control policy and distributed quit booklets. However, only 69% of schools met our criteria for program reach.

Conclusions

This study outlines the processes for taking a tobacco control intervention to scale and implementing it through the Bihar DOE infrastructure. These findings provide a foundation for other Indian states and low- and middle-income countries to implement tobacco control and other health programs for schoolteachers.

Trial registration

NCT05346991.

Plain Language Summary

Each year in India, more than 1.2 million people die from tobacco-related causes, and India has the world's highest oral cancer burden. The world needs more evidence on how to bring cost-effective tobacco control interventions to scale, especially in low- and middle-income countries (LMICs). To address this gap, from 2017 to 2021, we examined the process of scaling up Tobacco-Free Teachers, Tobacco-Free Society (TFT-TFS), an evidence-based intervention promoting tobacco use cessation among teachers and tobacco control policies in schools. Our study tested an implementation model aimed at building the Bihar State Department of Education (DOE) capacity to support and deliver TFT-TFS. We used a training-of-trainers model to embed TFT-TFS into Bihar DOE infrastructure, training 46 Cluster Coordinators to in turn train and support Headmasters to implement TFT-TFS over one academic year. We hypothesized that program adoption, implementation, and reach would not be inferior to the high standards demonstrated when we originally tested TFT-TFS through the Bihar School Teachers Study (2013-2017). For adoption, 94% of Headmasters attended the first training, although participation declined by the sixth training. Of 112 schools (out of 219 with complete Headmaster checklist data), all met our minimum criteria for implementing TFT-TFS. Over 99% of schools posted a school tobacco control policy and distributed quit booklets. However, only 69% of schools met our criteria for program reach. Study findings offer other Indian states and LMICs lessons to implement tobacco control and other health programs for schoolteachers within educational systems.

Needs Assessment and Best Practices for Digital Trainings for Health Professionals in Ethiopia Using the RE-AIM Framework: COVID-19, Case Study

OBJECTIVES

This study is aimed to assess the implementation science outcomes of the coronavirus disease (COVID-19) e-health educational intervention in Ethiopia targeting health care workers via the RE-AIM (Reach, Effectiveness, Adaption, Implementation, Maintenance) framework.

METHODS

A series of three 1-hour medical seminars focused on COVID-19 prevention and treatment education were conducted between May and August 2020. Educational content was built from medical sites previously impacted by COVID-19. Post-seminar evaluation information was collected from physician and other participants by a survey instrument. Cross-sectional evaluation results are reported here by RE-AIM constructs.

RESULTS

The medical seminars reached 324 participants. Key success metrics include that 90% reporting the information delivered in a culturally sensitive/tailored manner (effectiveness), 80% reporting that they planned to share the information presented with someone else (adoption and implementation), and 64% reporting using information presented in their daily clinical responsibilities 6 months after the first medical seminars (maintenance).

CONCLUSION

Grounded in a theoretical framework and following evidence-based best practices, this intervention advances the field of dissemination and implementation science by demonstrating how to transition health care training and delivery from an in-person to digital medium in low-resource settings like Ethiopia.

Diversifying Implementation Science: A Global Perspective

We present a joint global perspective about the urgent need to diversify the loci of knowledge creation and sharing in global implementation science. We underscore the imperative of addressing implementation research questions relevant to practitioners, policy makers, and researchers from low- and middle-income countries.

Implementation Evaluation of HUGS/Abrazos During the COVID-19 Pandemic: A Program to Foster Resiliency in Pregnancy and Early Childhood

Early life adversity can significantly impact child development and health outcomes throughout the life course. With the COVID-19 pandemic exacerbating preexisting and introducing new sources of toxic stress, social programs that foster resilience are more necessary now than ever. The Helping Us Grow Stronger (HUGS/Abrazos) program fills a crucial need for protective buffers during the COVID-19 pandemic, which has escalated toxic stressors affecting pregnant women and families with young children. HUGS/Abrazos combines patient navigation, behavioral health support, and innovative tools to ameliorate these heightened toxic stressors. We used a mixed-methods approach, guided by the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework, to evaluate the implementation of the HUGS/Abrazos program at Massachusetts General Hospital from 6/30/2020-8/31/2021. Results of the quality improvement evaluation revealed that the program was widely adopted across the hospital and 392 unique families were referred to the program. The referred patients were representative of the communities in Massachusetts disproportionately affected by the COVID-19 pandemic. Furthermore, 79% of referred patients followed up with the initial referral, with sustained high participation rates throughout the program course; and they were provided with an average of four community resource referrals. Adoption and implementation of the key components in HUGS/Abrazos were found to be appropriate and acceptable. Furthermore, the implemented program remained consistent to the original design. Overall, HUGS/Abrazos was well adopted as an emergency relief program with strong post-COVID-19 applicability to ameliorate continuing toxic stressors while decreasing burden on the health system.

Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda

Background: We describe the development and testing of a hybrid implementation research (IR) framework to understand the pathways, successes, and challenges in addressing amenable under-5 mortality (U5M) - deaths preventable through health system-delivered evidence-based interventions (EBIs) - in low- and middle-income countries (LMICs). Methods: We reviewed existing IR frameworks to develop a hybrid framework designed to better understand U5M reduction in LMICs from identification of leading causes of amenable U5M, to EBI choice, identification, and testing of strategies, work to achieve sustainability at scale, and key contextual factors. We then conducted a mixed-methods case study of Rwanda using the framework to explore its utility in understanding the steps the country took in EBI-related decision-making and implementation between 2000-2015, key contextual factors which hindered or facilitated success, and to extract actionable knowledge for other countries working to reduce U5M. Results: While relevant frameworks were identified, none individually covered the scope needed to understand Rwanda's actions and success. Building on these frameworks, we combined and adapted relevant frameworks to capture exploration, planning, implementation, contextual factors in LMICs such as Rwanda, and outcomes beyond effectiveness and coverage. Utilizing our hybrid framework in Rwanda, we studied multiple EBIs and identified a common pathway and cross-cutting strategies and contextual factors that supported the country's success in reducing U5M through the health system EBIs. Using these findings, we identified transferable lessons for other countries working to accelerate reduction in U5M. Conclusions: We found that a hybrid framework building on and adapting existing frameworks was successful in guiding data collection and interpretation of results, emerging new insights into how and why Rwanda achieved equitable introduction and implementation of health system EBIs that contributed to the decline in U5M, and generated lessons for countries working to drop U5M.

Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda

Background: We describe the development and testing of a hybrid implementation research (IR) framework to understand the pathways, successes, and challenges in addressing amenable under-5 mortality (U5M) - deaths preventable through health system-delivered evidence-based interventions (EBIs) - in low- and middle-income countries (LMICs). Methods: We reviewed existing IR frameworks to develop a hybrid framework designed to better understand U5M reduction in LMICs from identification of leading causes of amenable U5M, to EBI choice, identification, and testing of strategies, work to achieve sustainability at scale, and key contextual factors. We then conducted a mixed-methods case study of Rwanda using the framework to explore its utility in understanding the steps the country took in EBI-related decision-making and implementation between 2000-2015, key contextual factors which hindered or facilitated success, and to extract actionable knowledge for other countries working to reduce U5M. Results: While relevant frameworks were identified, none individually covered the scope needed to understand Rwanda's actions and success. Building on these frameworks, we combined and adapted relevant frameworks to capture exploration, planning, implementation, contextual factors in LMICs such as Rwanda, and outcomes beyond effectiveness and coverage. Utilizing our hybrid framework in Rwanda, we studied multiple EBIs and identified a common pathway and cross-cutting strategies and contextual factors that supported the country's success in reducing U5M through the health system EBIs. Using these findings, we identified transferable lessons for other countries working to accelerate reduction in U5M. Conclusions: We found that a hybrid framework building on and adapting existing frameworks was successful in guiding data collection and interpretation of results, emerging new insights into how and why Rwanda achieved equitable introduction and implementation of health system EBIs that contributed to the decline in U5M, and generated lessons for countries working to drop U5M.

Development and application of a hybrid implementation research framework to understand success in reducing under-5 mortality in Rwanda

Background: We describe the development and testing of a hybrid implementation research (IR) framework to understand the pathways, successes, and challenges in addressing amenable under-5 mortality (U5M) - deaths preventable through health system-delivered evidence-based interventions (EBIs) - in low- and middle-income countries (LMICs). Methods: We reviewed existing IR frameworks to develop a hybrid framework designed to better understand U5M reduction in LMICs from identification of leading causes of amenable U5M, to EBI choice, identification and testing of strategies, work to achieve sustainability at scale and key contextual factors. We then conducted a mixed-methods case study of Rwanda using the framework to explore its utility in understanding the steps the country took in EBI-related decision-making and implementation between 2000-2015, key contextual factors which hindered or facilitated success, and extract actionable knowledge for other countries working to reduce U5M. Results: While relevant frameworks were identified, none individually covered the scope needed to understand Rwanda's actions and success. Building on these frameworks, we combined and adapted relevant frameworks to capture exploration, planning, implementation, contextual factors in LMICs such as Rwanda, and outcomes beyond effectiveness and coverage. Utilizing our hybrid framework in Rwanda, we studied multiple EBIs and identified a common pathway and cross-cutting strategies and contextual factors that supported the country's success in reducing U5M through the health system EBIs. Using these findings, we identified transferable lessons for other countries working to accelerate reduction in U5M. Conclusions: We found that a hybrid framework building on and adapting existing frameworks was successful in guiding data collection and interpretation of results, emerging new insights into how and why Rwanda achieved equitable introduction and implementation of health system EBIs that contributed to the decline in U5M, and generated lessons for countries working to drop U5M.

Everybody Stacks: Lessons from household energy case studies to inform design principles for clean energy transitions

Stove stacking (concurrent use of multiple stoves and/or fuels) is a poorly quantified practice in regions where efforts to transition household energy to cleaner stoves/or fuels are on-going. Using biomass-burning stoves alongside clean stoves undermines health and environmental goals. This review synthesizes stove stacking data gathered from eleven case studies of clean cooking programs in low/middle-income country settings. Analyzed data are from ministry and program records, research studies, and informant interviews. Thematic analysis identify key drivers of stove stacking behavior in each setting. Significant (28%-100%) stacking with traditional cooking methods was observed in all cases. Reason for traditional fuel use includes: costs of clean fuel; mismatches between cooking technologies and household needs; and unreliable fuel supply. National household surveys often focus on 'primary' cookstoves and miss stove stacking data. Thus more attention should be paid to discontinuation of traditional stove use, not solely adoption of cleaner stoves/fuels. Future energy policies and programs should acknowledge the realities of stacking and incorporate strategies at the design stage to transition away from polluting stoves/fuels. Seven principles for clean cooking system program design and policy are presented, focused on a shift toward "cleaner stacking" that could yield household air pollution reductions approaching WHO targets.

Making Implementation Science More Rapid: Use of the RE-AIM Framework for Mid-Course Adaptations Across Five Health Services Research Projects in the Veterans Health Administration

Introduction: Implementation science frameworks have helped advance translation of research to practice. They have been widely used for planning and post-hoc evaluation, but seldom to inform and guide mid-course adjustments to intervention and implementation strategies.

Materials and Methods: This study developed an innovative methodology using the RE-AIM framework and related tools to guide mid-course assessments and adaptations across five diverse health services improvement projects in the Veterans Health Administration (VA). Using a semi-structured guide, project team members were asked to assess the importance of and progress on each RE-AIM dimension (i.e., reach, effectiveness, adoption, implementation, maintenance) at the current phase of their project. Based on these ratings, each team identified one or two RE-AIM dimensions for focused attention. Teams developed proximal goals and implementation strategies to improve progress on their selected dimension(s). A follow-up meeting with each team occurred approximately 6 weeks after the goal setting meeting to evaluate the usefulness of the iterative process. Results were evaluated using both descriptive quantitative analyses and qualitative assessments from interviews and meeting notes.

Results: A median of seven team members participated in the two meetings. Qualitative and descriptive data revealed that the process was feasible, understandable and useful to teams in adjusting their interventions and implementation strategies. The RE-AIM dimensions identified as most important were adoption and effectiveness, and the dimension that had the largest gap between importance and rated progress was reach. The dimensions most frequently selected for improvement were reach and adoption. Examples of action plans were summarizing stakeholder interviews for leadership, revising exclusion criteria, and conducting in-service trainings. Follow-up meetings indicated that teams found the process very useful and were able to implement the action plans they set.

Discussion: The iterative use of RE-AIM to support adjustments during project implementation proved feasible and useful across diverse projects in the VA setting. Building on this and related examples, future research should replicate these findings and further develop the methodology, as well as explore the optimal frequency and timing for these iterative applications of RE-AIM. More generally, greater focus on more rapid and iterative use of implementation science frameworks is encouraged to facilitate successful translation of research to practice.