National Working Group on the RE-AIM Planning and Evaluation Framework: Goals, Resources, and Future Directions

Comments, suggestions, and criticisms of the Pragmatic Explanatory Continuum Indicator Summary-2 design tool: a citation analysis

INTRODUCTION

The pragmatic explanatory continuum indicator summary (PRECIS) tool, initially published in 2009 and revised in 2015, was created to assist trialists to align their design choices with the intended purpose of their randomised controlled trial (RCT): either to guide real-world decisions between alternative interventions (pragmatic) or to test hypotheses about intervention mechanisms by minimising sources of variation (explanatory). There have been many comments, suggestions, and criticisms of PRECIS-2. This summary will be used to facilitate the development of to the next revision, which is PRECIS-3.

METHODS

We used Web of Science to identify all publication types citing PRECIS-2, published between May 2015 and July 2023. Citations were eligible if they contained 'substantive' suggestions, comments, or criticism of the PRECIS-2 tool. We defined 'substantive' as comments explicitly referencing at least one PRECIS-2 domain or a concept directly linked to an existing or newly proposed domain. Two reviewers independently extracted comments, suggestions, and criticisms, noting their implications for the update. These were discussed among authors to achieve consensus on the interpretation of each comment and its implications for PRECIS-3.

RESULTS

The search yielded 885 publications, and after full-text review, 89 articles met the inclusion criteria. Comments pertained to new domains, changes in existing domains, or were relevant across several or all domains. Proposed new domains included assessment of the comparator arm and a domain to describe blinding. There were concerns about scoring eligibility and recruitment domains for cluster trials. Suggested areas for improvement across domains included the need for more scoring guidance for explanatory design choices.

DISCUSSION

Published comments recognise PRECIS-2's success in aiding trialists with pragmatic or explanatory design choices. Enhancing its implementation and widespread use will involve adding new domains, refining domain definitions, and addressing overall tool issues. This citation review offers valuable user feedback, pivotal for shaping the upcoming version of the PRECIS tool, PRECIS-3.

Evaluating adoption and reach in a pragmatic randomized trial of community paramedicine for intermediate acuity patient care

Introduction

Pragmatic trials aim to speed translation to practice by integrating study procedures in routine care settings. This study evaluated implementation outcomes related to clinician and patient recruitment and participation in a trial of community paramedicine (CP) and presents successes and challenges of maintaining pragmatic study features.

Methods

Adults in the pre-hospital setting, emergency department (ED), or hospital being considered for referral to the ED/hospital or continued hospitalization for intermediate-level care were randomized 1:1 to CP care or usual care. Referral and enrollment data were tracked administratively, and patient characteristics were abstracted from the electronic health record (EHR). Enrolled patients completed baseline surveys, and a subset of intervention patients were interviewed. All CPs and a sample of clinicians and administrators were invited to complete a survey and interview.

Results

Between January 2022 and February 2023, 240 enrolled patients (42% rural) completed surveys, and 22 completed an interview; 63 staff completed surveys and 20 completed an interview. Ninety-three clinicians in 27 departments made at least one referral. Factors related to referrals included program awareness and understanding the CP practice scope. Most patients were enrolled in the hospital, but characteristics were similar to the primary care population and included older and medically complex patients. Challenges to achieving representativeness included limited EHR infrastructure, constraints related to patient consenting, and clinician concerns about patient randomization disrupting preferred care.

Conclusion

Future pragmatic trials in busy clinical settings may benefit from regulatory policies and EHR capabilities that allow for real-world study conduct and representative participation. Trial registration: NCT05232799.

Occupational Therapy for Adults With Acute Hand Conditions: A Mixed-Methods Feasibility Study

Hand injuries account for up to one third of acute hospital presentations. Current guidelines and standards of care recommend patients with hand trauma are seen by hand therapists, typically occupational therapists. This study explored the feasibility of implementing an occupational therapy-led hand therapy service for adults presenting acutely with a hand condition, using the RE-AIM framework. Purposeful sampling was used to recruit adult participants (n = 13). Outcome measures assessing function, health-related quality of life, pain, and occupational value and competence were completed. Focus groups were completed with patients and health care professionals. Data analysis revealed improvements in outcome measures post intervention and at 6-month follow-up. Participants were receptive of therapy-led services, and barriers and facilitators to service integration were identified using the RE-AIM framework. Occupational therapy-led hand therapy clinics show potential in treating people with acute hand conditions at emergency department settings, based on RE-AIM evaluation.

RE-AIM for rural health innovations: perceptions of (mis) alignment between the RE-AIM framework and evaluation reporting in the Department of Veterans Affairs Enterprise-Wide Initiatives program

Background

The Department of Veterans Affairs (VA) Office of Rural Health (ORH) supports national VA program offices' efforts to expand health care to rural Veterans through its Enterprise-Wide Initiatives (EWIs) program. In 2017, ORH selected Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM), an implementation science framework, to structure the EWI evaluation and reporting process. As part of its mandate to improve EWI program evaluation, the Center for the Evaluation of Enterprise-Wide Initiatives conducted a qualitative evaluation to better understand EWI team' perceptions of, and barriers and facilitators to, the EWI evaluation process.

Methods

We conducted 43 semi-structured interviews with 48 team members (e.g., evaluators, program office leads, and field-based leads) representing 21 EWIs from April-December 2020. Questions focused on participants' experiences using strategies targeting each RE-AIM dimension. Interviews were inductively analyzed in MAXQDA. We also systematically reviewed 51 FY19-FY20 EWI annual reports to identify trends in misapplications of RE-AIM.

Results

Participants had differing levels of experience with RE-AIM. While participants understood ORH's rationale for selecting a common framework to structure evaluations, the perceived misalignment between RE-AIM and EWIs' work emerged as an important theme. Concerns centered around 3 sub-themes: (1) (Mis)Alignment with RE-AIM Dimensions, (2) (Mis)Alignment between RE-AIM and the EWI, and (3) (Mis)Alignment with RE-AIM vs. other Theories, Models, or Frameworks. Participants described challenges differentiating between and operationalizing dimensions in unique contexts. Participants also had misconceptions about RE-AIM and its relevance to their work, e.g., that it was meant for established programs and did not capture aspects of initiative planning, adaptations, or sustainability. Less commonly, participants shared alternative models or frameworks to RE-AIM. Despite criticisms, many participants found RE-AIM useful, cited training as important to understanding its application, and identified additional training as a future need.

Discussion

The selection of a shared implementation science framework can be beneficial, but also challenging when applied to diverse initiatives or contexts. Our findings suggest that establishing a common understanding, operationalizing framework dimensions for specific programs, and assessing training needs may better equip partners to integrate a shared framework into their evaluations.

Mixed-method approach to informing a lifestyle intervention to improve the survivorship of patients with endometrial cancer

OBJECTIVES

To identify endometrial cancer survivors' (ECS) barriers and facilitators for participation in lifestyle interventions to improve their dietary and exercise behaviors. Our secondary objective is to determine baseline information: physical activity level, quality of life (QoL), and impact of COVID-19 on exercise, diet, and mental health.

METHODS

Obese, early-stage ECS participated in 2-part mixed-methods data collection; Part 1: survey gathering sample characteristics, QoL, exercise, and basic endometrial cancer- related knowledge. Part 2: virtual focus group or individual interviews using a brainwriting premortem protocol. Statistical analysis was performed using SAS (version 8.3). Qualitative data were analyzed using deductive thematic coding guided by the RE-AIM framework.

RESULTS

Twenty percent (70/358) of ECS from a survivorship database and clinic recruitment completed the survey; 16 ECS provided qualitative feedback. Common barriers to intervention participation included time and resource costs, meeting frequency, and pessimism about weight loss maintenance. Facilitators included an opportunity to connect with other survivors and a focus on health rather than weight loss. Most ECS could not identify exercise guidelines (60%) and 83% were not meeting these guidelines. Higher BMI was correlated with a lower confidence in completing in moderate physical activity (p-value = 0.0206). Post-COVID-19 pandemic, physical activity, nutritional decisions, and/or mental health worsened for 67% of ECS.

CONCLUSION

ECS are a disparate population, with worsening behaviors and mental health following the pandemic. The identified ECS-specific barriers and facilitators to behavioral intervention participation are being used to simultaneously improve the reach of and adherence to a lifestyle intervention aimed at improving their health and QoL.

Application of the ConNECT Framework to achieve digital health equity

BACKGROUND

The emphasis on digital technology and informatics in health care (digital health) has introduced innovative ways to deliver health care and engage populations in health research. However, inadequate attention to the development and implementation of digital health interventions can exacerbate health disparities.

PURPOSE

We applied the transdisciplinary ConNECT Framework principles within the context of digital health, with an aim to describe strategies to achieve digital health equity.

METHODS

We described the five ConNECT principles of (a) integrating context, (b) fostering a norm of inclusion, (c) ensuring equitable diffusion of innovations, (d) harnessing communication technology, and (e) prioritizing specialized training within the framework of achieving digital health equity.

FINDINGS AND DISCUSSION

We describe proactive, actionable strategies for the systematic application of the ConNECT Framework principles to address digital health equity. Recommendations to reduce the digital health divide in nursing research and practice are also described.

Translation, cross-cultural adaptation and measurement properties of three implementation measures into Brazilian-Portuguese

BACKGROUND

To translate and cross-culturally adapt into Brazilian-Portuguese, and to test the measurement properties of the following items of implementation outcome measures: Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM) and Feasibility of Intervention Measure (FIM).

METHODS

This was a measurement properties study in accordance with the Consensus-based Standards for the selection of health status Measurement Instruments (COSMIN). We conducted a translation and cross-cultural adaptation of three implementation measures according to guidelines for translation and cross-cultural adaptation, then we collected information from patients who had participated in remotely delivered physical therapy treatment for musculoskeletal condition. The patients answered the translated versions of the implementation outcome measures. The measurement properties of the three implementation outcome measures were collected in a test-retest assessment, with an interval of 7 to 14 days.. The measurement properties evaluated in this study were interpretability, measured using Ceiling and Floor Effects, reliability in test-retest evaluation, measured using Cronbach's Alpha Coefficient, internal consistency, measured using Intraclass Correlation Coefficient and construct validity, measured using Pearson Correlation.

RESULTS

We included 104 participants (76 female). The average age of the sample was 56.8 (SD 14.8) years old. The items of implementation outcome measures (AIM, IAM, and FIM) showed 66.39%, 63.11%, and 63.93% of ceiling effects. The items of implementation outcome measures showed adequate internal consistency measured using Cronbach's Alpha Coefficient (AIM: 0.89, IAM: 0.91, FIM: 0.93) and values of Standard Error of Measurement between 5 and 10%, showing good measurement error. The results of AIM and IAM was classified as moderate reliability and the FIM as substantial reliability. In a total 96 correlations, > 75% of correlations met our prior hypothesis.

CONCLUSION

The three Brazilian-Portuguese versions of items of implementation outcome measures had adequate internal consistency, measurement error and construct validity. The three implementation outcome measures showed moderate to substantial reliability values. The Ceiling Effect was observed in the three measures, showing maximum values ​​in more than 15% of the evaluations.

Planning for Implementation Science in Clinical Practice Research: An Augmentative and Alternative Communication Example

PURPOSE

Most evidence-based clinical research findings in communication sciences and disorders never achieve full implementation by practicing clinicians. To address this almost universal problem, this tutorial focuses on the initial planning steps of including implementation methodologies as part of clinical practice research programs. Integrating the principles and methodologies of implementation science can shift the current reality, leading to better uptake of evidence-based interventions.

METHOD

A step-by-step approach to integrating implementation science into new or existing research projects is provided, including the construction of a logic model, the selection of an implementation framework, and the selection of research methodologies. A detailed example of an augmentative and alternative communication research program is provided to illustrate the process.

CONCLUSIONS

Readily available online tools can help researchers start the process of integrating implementation science into clinical practice research, and existing frameworks can assist with developing and guiding research programs to ensure maximal impact. If the ultimate goal is to improve the daily lives of individuals with communication disorders through the use of evidence-based practices, clinical practice researchers must integrate these approaches into their research programs.

Exploring perspectives on changing opioid prescribing practices: A qualitative study of community stakeholders in the HEALing Communities Study

BACKGROUND

Community-based perspectives are needed to more broadly inform policy-makers, public health practitioners, prescribers, and pharmacists about community-led and broader efforts to reduce opioid overprescribing, and ultimately reduce prescription opioid use disorder, overdoses and fatalities. The aim of this study is to explore community-based perspectives on efforts to change opioid prescribing practices in their communities.

METHODS

Semi-structured interviews were conducted with 388 community stakeholders across four states (Kentucky, Massachusetts, New York, Ohio) from November 2019 to January 2020 about community approaches and goals of community-led responses to the opioid crisis. Data analysis combined deductive and inductive approaches to identify themes and sub-themes related to improving opioid prescribing practices.

RESULTS

Three major themes and different subthemes were characterized: (1) acknowledging progress (i.e., healthcare providers being part of the solution, provider education, and prescription drug monitoring programs); (2) emergent challenges (i.e., physician nonadherence with safer opioid prescribing guidelines, difficulty identifying appropriate use of opioids, and concerns about accelerating the progression from opioid misuse to drug abuse); and (3) opportunities for change (i.e., educating patients about safer use and proper disposal of opioids, expanding prescriber and pharmacist education, changing unrealistic expectations around eliminating pain, expanding and increasing insurance coverage for alternative treatment options).

CONCLUSIONS

Community stakeholders appeared to support specific opportunities to reduce prescription opioid misuse and improve safer prescribing. The opportunities included culture change around pain expectations, awareness of safe disposal, additional provider education, and increased coverage and acceptability of non-opioid treatments.

Understanding and applying the RE-AIM framework: Clarifications and resources

INTRODUCTION

Understanding, categorizing, and using implementation science theories, models, and frameworks is a complex undertaking. The issues involved are even more challenging given the large number of frameworks and that some of them evolve significantly over time. As a consequence, researchers and practitioners may be unintentionally mischaracterizing frameworks or basing actions and conclusions on outdated versions of a framework.

METHODS

This paper addresses how the RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance) framework has been described, summarizes how the model has evolved over time, and identifies and corrects several misconceptions.

RESULTS

We address 13 specific areas where misconceptions have been noted concerning the use of RE-AIM and summarize current guidance on these issues. We also discuss key changes to RE-AIM over the past 20 years, including the evolution to Pragmatic Robust Implementation and Sustainability Model, and provide resources for potential users to guide application of the framework.

CONCLUSIONS

RE-AIM and many other theories and frameworks have evolved, been misunderstood, and sometimes been misapplied. To some degree, this is inevitable, but we conclude by suggesting some actions that reviewers, framework developers, and those selecting or applying frameworks can do to prevent or alleviate these problems.

Shared Decision-Making for Left Ventricular Assist Devices: Rationale and Design of a Nationwide Dissemination and Implementation Project

Background The left ventricular assist device (LVAD) has become a common medical option for patients with end-stage heart failure. Although patients' chances of survival may increase with an LVAD compared with medical therapy, the LVAD poses many risks and requires major lifestyle changes, thus making it a complex medical decision. Our prior work found that a decision aid for LVADs significantly increased decision quality for both patients and caregivers and was successfully implemented at 6 LVAD programs. Methods In follow-up, we are conducting a nationwide dissemination and implementation project, with the goal of implementing the decision aid at as many of the 176 LVAD programs in the United States as possible. Guided by the Theory of Diffusion of Innovations, the project consists of 4 phases: (1) building a network; (2) promoting adoption; (3) supporting implementation; and (4) encouraging maintenance. Developing an LVAD network of contacts occurs by using a national baseline survey of LVAD clinicians, existing professional relationships, and an internet-based strategy. A suite of resources targeted to promote adoption and support implementation of the decision aid into standard LVAD education processes are provided to the network. Evaluation is guided by the Reach, Effectiveness, Adoption, Implementation, Maintenance framework, where clinician and patient surveys and qualitative interviews determine the reach, effectiveness, adoption, implementation, and maintenance achieved. Conclusions This project is a true dissemination study in that it targets the entire population of LVAD programs in the United States and is unique in its use of social marketing principles to promote adoption and implementation. The implementation plan is intended to serve as a test case and model for dissemination and implementation of other evidence-based decision support aids and strategies.

Model and approach for assessing implementation context and fidelity in the HEALing Communities Study

BACKGROUND

In response to the U.S. opioid epidemic, the HEALing (Helping to End Addiction Long-term^SM) Communities Study (HCS) is a multisite, wait-listed, community-level cluster-randomized trial that aims to test the novel Communities That HEAL (CTH) intervention, in 67 communities. CTH will expand an integrated set of evidence-based practices (EBPs) across health care, behavioral health, justice, and other community-based settings to reduce opioid overdose deaths. We present the rationale for and adaptation of the RE-AIM/PRISM framework and methodological approach used to capture the CTH implementation context and to evaluate implementation fidelity.

METHODS

HCS measures key domains of the internal and external CTH implementation context with repeated annual surveys and qualitative interviews with community coalition members and key stakeholders. Core constructs of fidelity include dosage, adherence, quality, and program differentiation-the adaptation of the CTH intervention to fit each community's needs. Fidelity measures include a monthly CTH checklist, collation of artifacts produced during CTH activities, coalition and workgroup attendance, and coalition meeting minutes. Training and technical assistance delivered by the research sites to the communities are tracked monthly.

DISCUSSION

To help attenuate the nation's opioid epidemic, the adoption of EBPs must be increased in communities. The HCS represents one of the largest and most complex implementation research experiments yet conducted. Our systematic examination of implementation context and fidelity will significantly advance understanding of how to best evaluate community-level implementation of EBPs and assess relations among implementation context, fidelity, and intervention impact.

A systematic review of the use and reporting of evaluation frameworks within evaluations of physical activity interventions

BACKGROUND

Evaluation of physical activity interventions is vital to inform, and justify, evidence-based policy and practice to support population-wide changes in physical activity. Several evaluation frameworks and guidance documents have been developed to facilitate the evaluation and reporting of evaluation studies in public health. However, there is a lack of evidence about whether frameworks are being used to guide evaluation. There continues to be claims of poor and inconsistent reporting in evaluation studies. The aim of this review was to assess the use of evaluation frameworks and the quality of reporting of how they were applied within evaluation studies of physical activity interventions.

OBJECTIVES

1. To identify whether evaluation frameworks are reported to have been used within evaluation studies of physical activity interventions, and which frameworks have been used. 2. To appraise the quality of reporting with regards to how evaluation frameworks have been used.

METHOD

We developed a checklist of indicators to enable a critical appraisal of the use and reporting of different evaluation frameworks in evaluation studies. We conducted a systematic search and review of evaluation studies published between 2015 and the date of the search to appraise the use and reporting of evaluation frameworks. A narrative synthesis is provided.

RESULTS

The review identified 292 evaluation studies of physical activity interventions, only 69 (23%) of these mentioned using an evaluation framework, and only 16 different frameworks were referred to. There was variation in the quality of reporting of framework use. 51 (74%) studies were identified as being explicitly based on the stated framework, however only 26 (38%) provided detailed descriptions consistently across all the checklist indicators. Details of adaptations and limitations in how frameworks were applied were less frequently reported. The review also highlighted variability in the reporting of intervention components. More consistent and precise reporting of framework and intervention components is needed.

CONCLUSION

Evaluation frameworks can facilitate a more systematic evaluation report and we argue their limited use suggests missed opportunities to apply frameworks to guide evaluation and reporting in evaluation studies. Variability in the quality of reporting of framework use limits the comparability and transferability of evidence. Where a framework has been used, the checklist of indicators can be employed to facilitate the reporting of an evaluation study and to review the quality of an evaluation report.