Applying the PRECIS criteria to describe three effectiveness trials of weight loss in obese patients with comorbid conditions

Identification of weight loss interventions for translation among endometrial cancer survivors: A RE-AIM analysis

Interventions for obesity-related cancers that combine nutrition and physical activity for weight loss exist; however, their application to survivors of endometrial cancer is unknown. Furthermore, little is known about pre-implementation perceptions of existing programs from a variety of interested persons (physicians, researchers) who may be part of the implementation team. Adapting an existing intervention rather than developing a new intervention may speed the translational lag time as long as intervention characteristics and fit within the delivery system are considered during the planning phase. To describe the process of determining the core elements of obesity-related interventions for cancer survivors and determine which one might be best delivered by an urban healthcare system that predominantly serves individuals who live in rural areas of Virginia and West Virginia. A pragmatic review of the literature was conducted via PubMed and Google Scholar with broad search terms of cancer survivor AND weight loss AND health intervention. Identified interventions were scored related to the Practical, Robust Implementation and Sustainability Model-which is an extension of RE-AIM framework to guide the understanding of who, what, where, when, and how the intervention was conducted. Intervention characteristics are reported. In addition, ratings from three independent reviewers on the validated 5-point Likert scale of an intervention's acceptability, appropriateness, and feasibility in the intended delivery system were collected and summarized. Twelve interventions were identified with an average sample size of 241(±195) and a range of 48-683 participants. Target populations included survivors of colorectal, breast, and endometrial cancers as well as general cancer survivors and included both men and women or only women. Most participants (74%) identified as white/Caucasian and average age ranged from 47.1 to 65.9 years. Program duration ranged from 4 weeks to 18 months, with an average duration of 32 weeks. Intervention dosage ranged from three times a week to once a month. Intervention acceptability, appropriateness, and feasibility had average and standard deviation ratings of 3.52(±0.46), 3.41(±0.45), and 3.21(±0.46), respectively, out of 5. The four interventions with the highest combined acceptable, appropriate, and feasible scores are being considered for potential use as an obesity-related intervention for survivors of endometrial cancer. Future work is needed to determine relevant adaptations and efficacy among survivors of endometrial cancer with obesity. Our approach may be beneficial for other interventionists aiming to speed intervention development and implementation.

Using PRECIS-2 in Chinese herbal medicine randomized controlled trials for irritable bowel syndrome: A methodological exploration based on literature

Background

The pragmatism levels of randomized controlled trials (RCTs) mean how similar the interventions delivered in the trial setting match those in the setting where the results will be applied. We aimed to investigate the association between the consistency of pragmatism among the characteristics of RCT design and its effect size of results in Chinese herbal medicine (CHM) for irritable bowel syndrome (IBS).

Methods

Eight English and Chinese language databases were searched for RCTs on CHM for IBS. Six reviewers independently assessed the pragmatism of trials using the pragmatic-explanatory continuum indicator summary 2 (PRECIS-2) tool. The consistency of pragmatism levels among the characteristics of RCT design was calculated using the coefficient of variation. Linear regression models were adopted to explore influence factors of the pragmatism of RCTs.

Results

78 RCTs were included. The level of consistency in the pragmatism for RCT's design was significantly correlated with the effect size of the results (binary outcome, r = -0.413; P = 0.005; continuous outcome, r = -0.779, P < 0.001). PRECIS-2 score was higher in trials with individualized interventions than fixed interventions (3.29 [0.32] vs 2.90 [0.32]; Cohen's d relative effect size, 0.52; P < 0.001) and in standard or usual-treatment-controlled trials than placebo-controlled (3.05 [0.37] vs 2.83 [0.28]; Cohen's d relative effect size, 0.32; P = 0.048).

Conclusion

The consistency of pragmatism level across the 9 domains of the PRECIS-2 tool in CHM IBS RCTs was positively correlated with the effect size of the results.

Bridging the gap: leveraging implementation science to advance pediatric behavioral sleep interventions

STUDY OBJECTIVES

This review synthesizes the pediatric behavioral sleep intervention (BSI) evidence base, summarizes challenges in translating BSIs from research to practice, and provides recommendations for integrating implementation science methods to advance pediatric BSI research.

METHODS

We briefly review the common behavioral sleep disturbances among youth, discuss the pediatric BSIs with well-established evidence, and identify gaps in pediatric BSI research. We then identify contributors to the dearth of research evaluating pediatric BSIs in accessible settings and present a model for applying implementation science strategies to address identified gaps across the continuum of translational research.

RESULTS

Relatively few BSI trials include older children and adolescents. Similarly, there is limited research evaluating BSIs among racially and ethnically minoritized children and families and/or those of lower socioeconomic status backgrounds. Access to scalable and easily disseminable tools to treat pediatric sleep disturbances early in their development is crucial for promoting positive child outcomes. To address these gaps, researchers should apply implementation science theories, models, and frameworks to design new interventions for implementation, adapt existing interventions with end users and settings in mind, conduct hybrid effectiveness-implementation trials, and test implementation strategies.

CONCLUSIONS

Given the prevalence and consequences of poor sleep across developmental periods, pediatric BSIs must be effective as well as adaptable, scalable, and easily disseminable. Implementation science theories, models, and frameworks can enhance access to, engagement in, and the implementation and dissemination of scalable BSIs across diverse pediatric care settings and heterogeneous populations.

CITATION

Levenson JC, Williamson AA. Bridging the gap: leveraging implementation science to advance pediatric behavioral sleep interventions. J Clin Sleep Med. 2023;19(7):1321-1336.

Integrating pragmatic and implementation science randomized clinical trial approaches: a PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) analysis

BACKGROUND

Over the past two decades, pragmatic and implementation science clinical trial research methods have advanced substantially. Pragmatic and implementation studies have natural areas of overlap, particularly relating to the goal of using clinical trial data to leverage health care system policy changes. Few investigations have addressed pragmatic and implementation science randomized trial methods development while also considering policy impact.

METHODS

The investigation used the PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) and PRECIS-2-Provider Strategies (PRECIS-2-PS) tools to evaluate the design of two multisite randomized clinical trials that targeted patient-level effectiveness outcomes, provider-level practice changes and health care system policy. Seven raters received PRECIS-2 training and applied the tools in the coding of the two trials. Descriptive statistics were produced for both trials, and PRECIS-2 wheel diagrams were constructed. Interrater agreement was assessed with the Intraclass Correlation (ICC) and Kappa statistics. The Rapid Assessment Procedure Informed Clinical Ethnography (RAPICE) qualitative approach was applied to understanding integrative themes derived from the PRECIS-2 ratings and an end-of-study policy summit.

RESULTS

The ICCs for the composite ratings across the patient and provider-focused PRECIS-2 domains ranged from 0.77 to 0.87, and the Kappa values ranged from 0.25 to 0.37, reflecting overall fair-to-good interrater agreement for both trials. All four PRECIS-2 wheels were rated more pragmatic than explanatory, with composite mean and median scores ≥ 4. Across trials, the primary intent-to-treat analysis domain was consistently rated most pragmatic (mean = 5.0, SD = 0), while the follow-up/data collection domain was rated most explanatory (mean range = 3.14-3.43, SD range = 0.49-0.69). RAPICE field notes identified themes related to potential PRECIS-2 training improvements, as well as policy themes related to using trial data to inform US trauma care system practice change; the policy themes were not captured by the PRECIS-2 ratings.

CONCLUSIONS

The investigation documents that the PRECIS-2 and PRECIS-2-PS can be simultaneously used to feasibly and reliably characterize clinical trials with patient and provider-level targets. The integration of pragmatic and implementation science clinical trial research methods can be furthered by using common metrics such as the PRECIS-2 and PRECIS-2-PS. Future study could focus on clinical trial policy research methods development.

TRIAL REGISTRATION

DO-SBIS ClinicalTrials.gov NCT00607620. registered on January 29, 2008. TSOS ClinicalTrials.gov NCT02655354, registered on July 27, 2015.

Comparison of explanatory and pragmatic design choices in a cluster-randomized hypertension trial: effects on enrollment, participant characteristics, and adherence

Background

Explanatory trials are designed to assess intervention efficacy under ideal conditions, while pragmatic trials are designed to assess whether research-proven interventions are effective in “real-world” settings without substantial research support.

Methods

We compared two trials (Hyperlink 1 and 3) that tested a pharmacist-led telehealth intervention in adults with uncontrolled hypertension. We applied PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) scores to describe differences in the way these studies were designed and enrolled study-eligible participants, and the effect of these differences on participant characteristics and adherence to study interventions.

Results

PRECIS-2 scores demonstrated that Hyperlink 1 was more explanatory and Hyperlink 3 more pragmatic. Recruitment for Hyperlink 1 was conducted by study staff, and 2.9% of potentially eligible patients enrolled. Enrollees were older, and more likely to be male and White than non-enrollees. Study staff scheduled the initial pharmacist visit and adherence to attending this visit was 98%. Conversely for Hyperlink 3, recruitment was conducted by clinic staff at routine encounters and 81% of eligible patients enrolled. Enrollees were younger, and less likely to be male and White than non-enrollees. Study staff did not assist with scheduling the initial pharmacist visit and adherence to attending this visit was only 27%. Compared to Hyperlink 1, patients in Hyperlink 3 were more likely to be female, and Asian or Black, had lower socioeconomic indicators, and were more likely to have comorbidities. Owing to a lower BP for eligibility in Hyperlink 1 (>140/90 mm Hg) than in Hyperlink 3 (>150/95 mm Hg), mean baseline BP was 148/85 mm Hg in Hyperlink 1 and 158/92 mm Hg in Hyperlink 3.

Conclusion

The pragmatic design features of Hyperlink 3 substantially increased enrollment of study-eligible patients and of those traditionally under-represented in clinical trials (women, minorities, and patients with less education and lower income), and demonstrated that identification and enrollment of a high proportion of study-eligible subjects could be done by usual primary care clinic staff. However, the trade-off was much lower adherence to the telehealth intervention than in Hyperlink 1, which is likely to reflect uptake under real-word conditions and substantially dilute intervention effect on BP.

Trial registration

The Hyperlink 1 study (NCT00781365) and the Hyperlink 3 study (NCT02996565) are registered at ClinicalTrials.gov.

Feasibility of Implementing Cancer-Specific Community-Based Exercise Programming: A Multi-Centre Randomized Trial

BACKGROUND

There is growing recognition of the importance of reporting preliminary work on the feasibility of a trial. The present study aimed to assess the feasibility of (1) a proposed fitness testing battery, and (2) processes related to the implementation of cancer-specific exercise programming in a community setting.

METHODS/DESIGN

A randomized controlled implementation feasibility trial was performed in advance of a large-scale implementation study. Eligible participants within 18 months of a cancer diagnosis were randomized to immediate or delayed community-based exercise at YMCA locations in Calgary and Edmonton, Canada for an 8-week period. The primary outcome for the trial was the feasibility of the physical fitness testing battery, defined as a 70% or greater completion rate across the 24-week study period. The Reach, Effectiveness, Adoption, Implementation and Maintenance (RE-AIM) framework was used to evaluate processes related to implementation of the exercise program across the two sites.

RESULTS

Eighty participants were recruited, 73 (91%) completed the 8-week trial, and 68 (85%) completed the 16- and 24-week follow-ups. Sixty participants (75%) completed the full physical fitness test battery at each time point, and 59 (74%) completed the patient-reported outcome measures. Statistically significant between-group differences were found in favor of the exercise group for functional aerobic capacity, upper and lower extremity strength, and symptoms. Differences were found between the sites, however, in completion rates and processes related to program implementation.

DISCUSSION

Findings suggest the need for minor adaptations to the physical fitness battery and outcome measures to better fit the community context. While findings support feasibility, context-specific challenges related to implementation processes were identified.

Pragmatic trials of pain therapies: a systematic review of methods

ABSTRACT

Pragmatic randomised clinical trials aim to directly inform clinical or health policy decision making. Here, we systematically review methods and design of pragmatic trials of pain therapies to examine methods, identify common challenges, and areas for improvement. Seven databases were searched for pragmatic randomised controlled clinical trials that assessed pain treatment in a clinical population of adults reporting pain. All screening steps and data extractions were performed twice. Data were synthesised descriptively, and correlation analyses between prespecified trial features and PRECIS-2 (PRagmatic-Explanatory Continuum Indicator Summary 2) ratings and attrition were performed. Protocol registration: PROSPERO-ID CRD42020178954. Of 57 included trials, only 21% assessed pharmacological interventions, the remainder physical, surgical, psychological, or self-management pain therapies. Three-quarters of the trials were comparative effectiveness designs, often conducted in multiple centres (median: 5; Q1/3: 1, 9.25) and with a median sample size of 234 patients at randomization (Q1/3: 135.5; 363.5). Although most trials recruited patients with chronic pain, reporting of pain duration was poor and not well described. Reporting was comprehensive for most general items, while often deficient for specific pragmatic aspects. Average ratings for pragmatism were highest for treatment adherence flexibility and clinical relevance of outcome measures. They were lowest for patient recruitment methods and extent of follow-up measurements and appointments. Current practice in pragmatic trials of pain treatments can be improved in areas such as patient recruitment and reporting of methods, analysis, and interpretation of data. These improvements will facilitate translatability to other real-world settings-the purpose of pragmatic trials.

Patient-provided e-support in reduced intensity obesity treatment: The INSPIRE randomized controlled trial

OBJECTIVE

There are two types of patient supporters, peers (two individuals initiating health behavior change who support one another) and mentors (a previously successful patient who supports incoming patients). Social comparison theory suggests that peers and mentors may elicit social comparison processes (patients may compare their progress to that of their peer/mentor), and these social comparisons could impact treatment outcomes. This randomized controlled trial is the first to examine the differential impact of peers and mentors on obesity treatment outcomes and social comparison processes when added to reduced intensity treatment.

METHOD

Participants (N = 278) were randomly assigned to reduced intensity behavioral weight loss treatment alone (rBWL), rBWL plus peer e-support (rBWL + Peer), or rBWL plus mentor e-support (rBWL + Mentor). rBWL involved periodic group sessions that decreased over time; when group sessions decreased, intensity of peer/mentor e-support increased. Weight and social comparison processes were assessed throughout the 12-month intervention.

RESULTS

There was a significant treatment effect; when group sessions became less frequent and peer/mentor e-support became more frequent, rBWL + Peer had significantly greater weight loss than rBWL alone, and rBWL + Mentor was not significantly different from the other two. Social comparison processes differed by treatment arm; rBWL + Peer participants tended to report more lateral social comparisons ("my weight loss progress is 'similar' to my peer's"), whereas rBWL + Mentor participants reported more upward comparisons ("my weight loss progress is 'worse than' my mentor's"). Upward comparisons were associated with poorer weight loss outcomes.

CONCLUSIONS

Peer e-support may be an effective, low-cost, sustainable method for improving longer-term weight loss outcomes in reduced intensity obesity treatment. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Real-world experiences with generating real-world evidence: Case Studies from PCORI's pragmatic clinical Studies program

BACKGROUND

Over the last decade, randomized studies evaluating outcomes of health care interventions conducted in real-world settings-often termed "pragmatic trials"-have come to be seen as an important means of obtaining relevant, actionable evidence to guide health care decisions. Despite extensive writing on methodological considerations in pragmatic trial design, limited information exists regarding the practical and logistical challenges encountered in carrying out rigorous randomized evaluations in highly representative, real-world contexts.

METHODS

The Patient Centered Outcomes Research Institute (PCORI) convened an expert panel in 2017 to examine common tradeoffs in study design and implementation through 3 "case studies" of in-progress, PCORI-funded pragmatic trials. This paper summarizes the findings of this panel, using the 3 examples to illustrate common implementation challenges encountered in pragmatic trials.

RESULTS

Investigators aimed to generate highly generalizable findings that could address real-world clinical decisions; however, practical considerations required that each study incorporate traditionally "explanatory" elements to achieve a "fit-for-purpose" approach to design and implementation. Within individual studies, efforts to balance pragmatic versus explanatory perspectives often involved multiple, diverse aspects of trial design and implementation, and the aspects of design and implementation where investigators reported encountering such tradeoffs varied across the three cases we examined.

CONCLUSIONS

Efforts to generate rigorous evidence that is generalizable to "real-world" practice require continuous and iterative efforts to balance "pragmatic" and "explanatory" perspectives. In each study examined, these tradeoffs were guided both by an overriding effort to maintain pragmatism and practical considerations that varied depending on the research question and study context.

Parents lose less weight than nonparents in an intensive lifestyle intervention

OBJECTIVE

Understand whether parents lose less weight than nonparents in behavioural weight interventions.

METHODS

The Look AHEAD (Action for Health in Diabetes) trial randomized adults with Type 2 diabetes and overweight to an intensive lifestyle intervention (ILI) or control (diabetes support and education [DSE]). Participants who reported living with a child under age 18 were designated as 'parents' for this analysis. Intention to treat analysis was performed of the effect of the ILI on change in weight at 1 year by parental status. Adherence to attending intervention visits was compared between parents and nonparents. Subgroup analyses were done based on previous subgroup findings in the Look AHEAD study.

RESULTS

Among 4,547 participants, 15% were parents. Parents were younger and more likely to have self-identified as African American or Hispanic/Latino. Comparing ILI with DSE, parents lost less weight than nonparents (-7.1% vs. -8.3%, p = 0.021). African American female parents lost 4% body weight compared with 7% in African American female nonparents (p = 0.01).

CONCLUSIONS

In a randomized trial, parents lost less weight than nonparents, and this difference was largest for African American women. These findings suggest parents face unique challenges achieving weight loss; more research is needed to understand and optimize interventions for parents.

Trends in the Explanatory or Pragmatic Nature of Cardiovascular Clinical Trials Over 2 Decades

Importance

Pragmatic trials test interventions using designs that produce results that may be more applicable to the population in which the intervention will be eventually applied.

Objective

To investigate how pragmatic or explanatory cardiovascular (CV) randomized clinical trials (RCT) are, and if this has changed over time.

Data Source

Six major medical and CV journals, including New England Journal of Medicine, Lancet, JAMA, Circulation, European Heart Journal, and Journal of the American College of Cardiology.

Study Selection

All CV-related RCTs published during 2000, 2005, 2010, and 2015 were identified and included.

Data Extraction and Synthesis

Included RCTs were assessed by 2 independent adjudicators with expertise in RCT and CV medicine.

Main Outcomes and Measures

The outcome measure was the level of pragmatism evaluated using the Pragmatic Explanatory Continuum Index Summary (PRECIS)-2 tool, which uses a 5-point ordinal scale (ranging from very pragmatic to very explanatory) across 9 domains of trial design, including eligibility, recruitment, setting, organization, intervention delivery, intervention adherence, follow-up, primary outcome, and analysis.

Results

Of 616 RCTs, the mean (SD) PRECIS-2 score was 3.26 (0.70). The level of pragmatism increased over time from a mean (SD) score of 3.07 (0.74) in 2000 to 3.46 (0.67) in 2015 (P < .001 for trend; Cohen d relative effect size, 0.56). The increase occurred mainly in the domains of eligibility, setting, intervention delivery, and primary end point. PRECIS-2 score was higher for neutral trials than those with positive results (P < .001) and in phase III/IV trials compared with phase I/II trials (P < .001) but similar between different sources of funding (public, industry, or both; P = .38). More pragmatic trials had more sites, larger sample sizes, longer follow-ups, and mortality as the primary end point.

Conclusions and Relevance

The level of pragmatism increased moderately over 2 decades of CV trials. Understanding the domains of current and future clinical trials will aid in the design and delivery of CV trials with broader application.

The importance of decision intent within descriptions of pragmatic trials

OBJECTIVE

It is now more than 50 years since the concepts of explanatory and pragmatic attitudes toward trials were first discussed by Schwartz and Lellouch in their influential 1967 paper. Since then, there has been increasing focus on design aspects that may be consistent with more pragmatic attitudes within clinical trials, and a number of tools developed to assist investigators prospectively think about their trial design. Researchers have subsequently expressed interest in using these tools retrospectively to characterize trials as pragmatic or explanatory.

RESULTS

We suggest that recent attempts to retrospectively dichotomize trials solely on the basis of quantitative scoring of trial design features are flawed. Instead, we argue that there is a need to consider both the intent and design when assessing the degree of pragmatism within a trial.

CONCLUSION

The practical implication of our suggestion for trial reporting is that investigators should explicitly state the intent of the trial through a clear articulation of the decision that they hope will be informed by the trial results. This should be coupled with a completed PRagmatic-Explanatory Continuum Indicator Summary 2 assessment (or similar) with an explanation of study design choices to appropriately assess whether the study design is consistent with the study intent. We believe this will assist reviewers and knowledge users in making assessments of trials.

Recruitment and reach in a pragmatic behavioral weight loss randomized controlled trial: implications for real-world primary care practice

BACKGROUND

Obesity is a major risk factor behind some of the most common problems encountered in primary care. Although effective models for obesity treatment have been developed, the 'reach' of these interventions is poor and only a small fraction of primary care patients receive evidence-based treatment. The purpose of this study is to identify factors that impact the uptake (reach) of an evidence-based obesity treatment program within the context of a pragmatic cluster randomized controlled trial comparing three models of care delivery.

METHODS

Recruitment and reach were evaluated by the following measures: 1) mailing response rates, 2) referral sources among participants contacting the study team, 3) eligibility rates, 4) participation rates, and 5) representativeness based on demographics, co-morbid conditions, and healthcare utilization of 1432 enrolled participants compared to > 17,000 non-participants from the clinic-based patient populations. Referral sources and participation rates were compared across study arms and level of clinic engagement.

RESULTS

The response rate to clinic-based mailings was 13.2% and accounted for 66% of overall program recruitment. An additional 22% of recruitment came from direct clinic referrals and 11% from media, family, or friends. Of those screened, 87% were eligible; among those eligible, 86% enrolled in the trial. Participation rates did not vary across the three care delivery arms, but were higher at clinics with high compared to low provider involvement. In addition, clinics with high provider involvement had a higher rate of in clinic referrals (33% versus 16%) and a more representative sample with regards to BMI, rurality, and months since last clinic visit. However, across clinics, enrolled participants compared to non-participants were older, more likely to be female, more likely to have had a joint replacement but less likely to have CVD or smoke, and had fewer hospitalizations.

CONCLUSIONS

A combination of direct patient mailings and in-clinic referrals may enhance the reach of primary care behavioral weight loss interventions, although more proactive outreach is likely necessary for men, younger patients, and those at greater medial risk. Strategies are needed to enhance provider engagement in referring patients to behavioral weight loss programs.

TRIAL REGISTRATION

clnicialtrials.gov NCT02456636. Registered May 28, 2015, https://www.clinicaltrials.gov/ct2/results?cond=&term=RE-POWER&cntry=&state=&city=&dist=.

Feasibility and implementation of a healthy lifestyles program in a community setting in Ontario, Canada: protocol for a pragmatic mixed methods pilot study

INTRODUCTION

Rates of chronic conditions, such as diabetes, cardiovascular disease and obesity are increasing in Canada and internationally. There are effective lifestyle interventions that are known to improve chronic conditions. However, there is often a gap in 'how to' make lifestyle changes. Mental health and other determinants of health play a role in the development and progression of chronic conditions. Changing habits takes time and requires the use of multiple techniques, including mental health and behavioural change strategies, based on a person's needs. A new, multidisciplinary, person-centred and evidence-based and practice-based programme has been created to address these needs. This proposal aims to evaluate the feasibility and implementation of this programme and to determine changes in participant-directed and clinical outcomes through a pilot study.

METHODS AND ANALYSIS

A pragmatic mixed methods design will be used to study multiple dimensions of the year-long healthy lifestyles programme. The pilot study includes a randomised controlled trial, with 30 participants randomised to either the programme or to a comparator arm, and qualitative components to determine the feasibility of the programme, including recruitment and retention, data missing rates and resources needed to run this programme. Changes in participant-directed and clinical outcomes will be measured. Descriptive statistics, t-tests and repeated measures analysis of variance (ANOVA) for within group comparisons and generalised estimating equations for between group analyses will be used. Qualitative interviews of programme staff and healthcare providers and family focus groups will be used to further enhance the findings and improve the programme.

ETHICS AND DISSEMINATION

Approval from the Hamilton Integrated Research Ethics Board (HiREB) has been obtained. Informed consent will be obtained prior to enrolling any participant into the study. Participant IDs will be used during data collection and entry. Peer-reviewed publications and presentations will target researchers, health professionals and stakeholders.

TRIAL REGISTRATION NUMBER

ClinicalTrials.gov Identifier: NCT03258138.HiREB project number: 3793.

Recommendations and Alerting for Delirium Alleviation in Real-Time (RADAR): Protocol for a pilot randomized controlled trial

Background: Delirium is a common and serious complication of major surgery for older adults. Postoperative social and behavioral support (e.g., early mobilization, mealtime assistance) may reduce the incidence and impact of delirium, and these efforts are possible with proactive patient-care programs. This pilot trial tests the hypothesis that a multicomponent decision support system, which sends automated alerts and recommendations to patient-care programs and family members for high-risk patients, will improve the postoperative environment for neurocognitive and clinical recovery. Methods: This will be a randomized, controlled, factorial pilot trial at a large academic medical center. High-risk, non-cardiac surgery patients (≥70 years old) will be recruited. Patients will be allocated to a usual care group (n=15), Hospital Elder Life Program (HELP)-based paging system (n=15), family-based paging system (n=15), or combined HELP- and family-based system (n=15). The primary outcome will be the presence of delirium, defined by positive long-form Confusion Assessment Method screening. Secondary outcomes will include additional HELP- and family-based performance metrics along with various neurocognitive and clinical recovery measures. Exploratory outcomes include the incidence of positive family-based delirium assessments post-discharge, 36-item Short Form Survey, PROMIS Cognitive Function Abilities Subset 4a, and 30-day readmission rates. Ethics and dissemination: This trial has received approval by the University of Michigan Medical Institutional Review Board (IRBMED). Dissemination plans include presentation at scientific conferences, publication in medical journals, and distribution via educational and news media. Registration: ClinicalTrials.gov Identifier NCT04007523, registered on 7/3/2019.

Recommendations and Alerting for Delirium Alleviation in Real-Time (RADAR): Protocol for a pilot randomized controlled trial

Background: Delirium is a common and serious complication of major surgery for older adults. Postoperative social and behavioral support (e.g., early mobilization, mealtime assistance) may reduce the incidence and impact of delirium, and these efforts are possible with proactive patient-care programs. This pilot trial tests the hypothesis that a multicomponent decision support system, which sends automated alerts and recommendations to patient-care programs and family members for high-risk patients, will improve the postoperative environment for neurocognitive and clinical recovery. Methods: This will be a randomized, controlled, factorial pilot trial at a large academic medical center. High-risk, non-cardiac surgery patients (≥70 years old) will be recruited. Patients will be allocated to a usual care group (n=15), Hospital Elder Life Program (HELP)-based paging system (n=15), family-based paging system (n=15), or combined HELP- and family-based system (n=15). The primary outcome will be the presence of delirium, defined by positive long-form Confusion Assessment Method screening. Secondary outcomes will include additional HELP- and family-based performance metrics along with various neurocognitive and clinical recovery measures. Exploratory outcomes include the incidence of positive family-based delirium assessments post-discharge, 36-item Short Form Survey, PROMIS Cognitive Function Abilities Subset 4a, and 30-day readmission rates. Ethics and dissemination: This trial has received approval by the University of Michigan Medical Institutional Review Board (IRBMED). Dissemination plans include presentation at scientific conferences, publication in medical journals, and distribution via educational and news media. Registration: ClinicalTrials.gov Identifier NCT04007523, registered on 7/3/2019.

Making Health Research Matter: A Call to Increase Attention to External Validity

Most of the clinical research conducted with the goal of improving health is not generalizable to nonresearch settings. In addition, scientists often fail to replicate each other's findings due, in part, to lack of attention to contextual factors accounting for their relative effectiveness or failure. To address these problems, we review the literature on assessment of external validity and summarize approaches to designing for generalizability. When investigators conduct systematic reviews, a critical need is often unmet: to evaluate the pragmatism and context of interventions, as well as their effectiveness. Researchers, editors, and grant reviewers can implement key changes in how they consider and report on external validity issues. For example, the recently published expanded CONSORT figure may aid scientists and potential program adopters in summarizing participation in and representativeness of a program across different settings, staff, and patients. Greater attention to external validity is needed to increase reporting transparency, improve program dissemination, and reduce failures to replicate research.

Development of a scalable weight loss intervention for low-income workers through adaptation of interactive obesity treatment approach (iOTA)

BACKGROUND

Describing how and why an evidence-based intervention is adapted for a new population and setting using a formal evaluation and an adaptation framework can inform others seeking to modify evidence-based weight management interventions for different populations or settings. The Working for You intervention was adapted, to fit a workplace environment, from Be Fit Be Well, an evidence-based intervention that targets weight-control and hypertension in patients at an outpatient clinic. Workplace-based efforts that promote diet and activity behavior change among low-income employees have potential to address the obesity epidemic. This paper aims to explicitly describe how Be Fit Be Well was adapted for this new setting and population.

METHODS

To describe and understand the worksite culture, environment, and policies that support or constrain healthy eating and activity in the target population, we used qualitative and quantitative methods including key informant interviews, focus groups, and a worker survey; these data informed intervention adaptation. We organized the adaptations made to Be Fit Be Well using an adaptation framework from implementation science.

RESULTS

The adapted intervention, Working for You, maintains the theoretical premise and evidence-base underpinning Be Fit Be Well. However, it was modified in terms of the means of delivery (i.e., rather than using interactive voice response, Working for You employs automated SMS text messaging), defined as a modification to context by the adaptation framework. The adaptation framework also includes modifications to content; in this case the behavioral goals were modified for the target population based on updated science related to weight loss and to target a workplace population (e.g., a goal to avoiding free food at work).

CONCLUSIONS

If effective, this scalable and relatively inexpensive intervention can be translated to other work settings to reduce obesity and diabetes risk among low-SES workers, a group with a higher prevalence of these conditions. Using a formal evaluation and framework to guide and organize how and why an evidence-based intervention is adapted for a new population and setting can push the field of intervention research forward.

TRIAL REGISTRATION

ClinicalTrials.gov: NCT02934113 ; Received: October 12, 2016; Updated: November 7, 2017.

Comparative effectiveness trials in asthma - how will I recognize one?

PURPOSE OF REVIEW

Comparative efficacy trials are designed to evaluate the harms and benefits of health care in a research environment. There is increasing interest in the results of comparative effectiveness trials, which are intended to fill gaps in evidence to inform decision-making in real-life clinical environments. The objective of this report is to review various tools to classify trials along the efficacy to effectiveness continuum.

RECENT FINDINGS

Three tools [Pragmatic-Explanatory Continuum Indicator Summary (PRECIS), PRECIS-2, and PRAgmatic Clinical Trial Assessment Scale (PRACTAS)] are available that use a star diagram to illustrate where each element of a clinical trial design falls along the efficacy to effectiveness continuum (e.g., selectivity of eligibility criteria, supports to promote participant adherence). The number and type of design element to be classified varies (e.g., 10 elements for PRECIS and PRACTAS vs. nine elements for PRECIS-2; only the PRACTAS tool includes stakeholder engagement). There is substantial interrater reliability when using all three tools and interrater reliability varies across the different design elements (intraclass correlation of coefficient 0.4-0.8).

SUMMARY

The PRECIS, PRECIS-2, and PRACTAS tools are options when classifying trials along the efficacy to effectiveness continuum. Researchers and decision-making stakeholders are likely to disagree about the extent to which clinical trials employ efficacy or effectiveness designs.

A dynamic application of PRECIS-2 to evaluate implementation in a pragmatic, cluster randomized clinical trial in two nursing home systems

Background

PRECIS-2 (PRagmatic Explanatory Continuum Indicator Summary-2) can assess how clinical trial design decisions (along the explanatory-pragmatic continuum) influence the applicability of trial results to intended stakeholders. The tool has been used to assess features of trials during the trial design phase and also upon completion. The ongoing PRagmatic trial Of Video Education in Nursing homes (PROVEN), which is evaluating the effectiveness of a suite of videos to improve advance care planning, is one of the first large pragmatic, cluster randomized trials within nursing home health care systems. While certain features of pragmatic trials remain static once designed (e.g., recruitment, outcomes), successful implementation of a system-wide program requires on-going evaluation and adaptation. This report’s objectives were to apply PRECIS-2 in a novel manner during the actual conduct of the PROVEN trial to assess how dynamic adaptations shifted implementation to either a more explanatory or a more pragmatic approach.

Methods

We assessed PROVEN’s protocol as initially designed according to the three PRECIS-2 domains pertinent to implementation: (1) Organization, (2) Flexibility-Delivery, and (3) Flexibility-Adherence. We then applied this framework to conduct a formative evaluation of decisions made while the trial was ongoing to adapt the implementation approach along the pragmatic versus the explanatory continuum in response to emergent challenges.

Results

Based on the PRECIS-2 rubric, the initial design of the PROVEN implementation approach reflected a hybrid of pragmatic and explanatory features. Most notably, within the Flexibility-Delivery, the trial had a relatively pragmatic approach to protocol delivery by front-line nursing home providers, balanced with a more explanatory approach to protocol monitoring enabled by the analytic capabilities of the research team. This more intensive monitoring proved critical in revealing implementation problems once the study began. Dynamic adaptations made in response to these challenges generally reflected shifts to more explanatory approaches within the Flexibility-Delivery and Flexibility-Adherence domains including ever more intensive compliance monitoring, as well as detailed coaching of front-line providers delivering the intervention by the research team.

Conclusions

Pragmatic trials conducted in the nursing home setting may benefit from a more dynamic approach to implementation. Allowing fluidity between pragmatic and explanatory features may still preserve the trial's applicability to intended stakeholders’ needs. PRECIS-2 provides a useful formative evaluation tool to assess these adaptations in “real-time.”

Trial registration

US National Library of Medicine, ClinicalTrials.gov, ID: NCT02612688. Registered on 19 November 2015

A comparison of meta-analytic methods for synthesizing evidence from explanatory and pragmatic trials

BACKGROUND

The pragmatic-explanatory continuum indicator summary version 2 (PRECIS-2) tool has recently been developed to classify randomized clinical trials (RCTs) as pragmatic or explanatory based on their design characteristics. Given that treatment effects in explanatory trials may be greater than those obtained in pragmatic trials, conventional meta-analytic approaches may not accurately account for the heterogeneity among the studies and may result in biased treatment effect estimates. This study investigates if the incorporation of PRECIS-2 classification of published trials can improve the estimation of overall intervention effects in meta-analysis.

METHODS

Using data from 31 published trials of intervention aimed at reducing obesity in children, we evaluated the utility of incorporating PRECIS-2 ratings of published trials into meta-analysis of intervention effects in clinical trials. Specifically, we compared random-effects meta-analysis, stratified meta-analysis, random-effects meta-regression, and mixture random-effects meta-regression methods for estimating overall pooled intervention effects.

RESULTS

Our analyses revealed that mixture meta-regression models that incorporate PRECIS-2 classification as covariate resulted in a larger pooled effect size (ES) estimate (ES = - 1.01, 95%CI = [- 1.52, - 0.43]) than conventional random-effects meta-analysis (ES = - 0.15, 95%CI = [- 0.23, - 0.08]).

CONCLUSIONS

In addition to the original intent of PRECIS-2 tool of aiding researchers in their choice of trial design, PRECIS-2 tool is useful for explaining between study variations in systematic review and meta-analysis of published trials. We recommend that researchers adopt mixture meta-regression methods when synthesizing evidence from explanatory and pragmatic trials.

Understanding the applicability of results from primary care trials: lessons learned from applying PRECIS-2

OBJECTIVE

To compare two approaches for trial teams to apply PRECIS-2 to pragmatic trials: independent scoring and scoring following a group discussion.

STUDY DESIGN AND SETTING

We recruited multidisciplinary teams who were conducting or had conducted trials in primary care in collaboration with the Pragmatic Clinical Trials Unit, Queen Mary University of London. Each team carried out two rounds of scoring on the nine PRECIS-2 domains: first independently using an online version of PRECIS-2 and second following a discussion.

RESULTS

Seven teams took part in the study. Before the discussion, within-team agreement in scores was generally poor and not all raters were able to score all domains; agreement improved after the discussion. The PRECIS-2 wheels suggested that the trials were pragmatic, although some domains were more pragmatic than others.

CONCLUSION

PRECIS-2 can facilitate information exchange within trial teams. To apply PRECIS-2 successfully, we recommend a discussion between those with detailed understanding of what usual care is for the intervention, the trial's design including operational and technical aspects, and the PRECIS-2 domains. For some cluster-randomized trials, greater insight may be gained by plotting two PRECIS-2 wheels, one at the individual participant level and another at the cluster level.

The PRECIS-2 tool has good interrater reliability and modest discriminant validity

OBJECTIVES

PRagmatic Explanatory Continuum Indicator Summary (PRECIS)-2 is a tool that could improve design insight for trialists. Our aim was to validate the PRECIS-2 tool, unlike its predecessor, testing the discriminant validity and interrater reliability.

STUDY DESIGN AND SETTING

Over 80 international trialists, methodologists, clinicians, and policymakers created PRECIS-2 helping to ensure face validity and content validity. The interrater reliability of PRECIS-2 was measured using 19 experienced trialists who used PRECIS-2 to score a diverse sample of 15 randomized controlled trial protocols. Discriminant validity was tested with two raters to independently determine if the trial protocols were more pragmatic or more explanatory, with scores from the 19 raters for the 15 trials as predictors of pragmatism.

RESULTS

Interrater reliability was generally good, with seven of nine domains having an intraclass correlation coefficient over 0.65. Flexibility (adherence) and recruitment had wide confidence intervals, but raters found these difficult to rate and wanted more information. Each of the nine PRECIS-2 domains could be used to differentiate between trials taking more pragmatic or more explanatory approaches with better than chance discrimination for all domains.

CONCLUSION

We have assessed the validity and reliability of PRECIS-2. An elaboration study and web site provide guidance to help future users of the tool which is continuing to be tested by trial teams, systematic reviewers, and funders.

Hypertension with unsatisfactory sleep health (HUSH): study protocol for a randomized controlled trial

BACKGROUND

Insomnia is common in primary care medical practices. Although behavioral treatments for insomnia are safe, efficacious, and recommended in practice guidelines, they are not widely-available, and their effects on comorbid medical conditions remain uncertain. We are conducting a pragmatic clinical trial to test the efficacy of two cognitive behavioral treatments for insomnia (Brief Behavioral Treatment for Insomnia (BBTI) and Sleep Healthy Using the Internet (SHUTi)) versus an enhanced usual care condition (EUC).

METHODS/DESIGN

The study is a three-arm, parallel group, randomized controlled trial. Participants include 625 adults with hypertension and insomnia, recruited via electronic health records from primary care practices affiliated with a large academic medical center. After screening and baseline assessments, participants are randomized to treatment. BBTI is delivered individually with a live therapist via web-interface/telehealth sessions, while SHUTi is a self-guided, automated, interactive, web-based form of cognitive behavioral therapy for insomnia. Participants in EUC receive an individualized sleep report, educational resources, and an online educational video. Treatment outcomes are measured at 9 weeks, 6 months, and 12 months. The primary outcome is patient-reported sleep disturbances. Secondary outcomes include other self-reported sleep measures, home blood pressure, body mass index, quality of life, health functioning, healthcare utilization, and side effects.

DISCUSSION

This randomized clinical trial compares two efficacious insomnia interventions to EUC, and provides a cost-effective and efficient examination of their similarities and differences. The pragmatic orientation of this trial may impact sleep treatment delivery in real world clinical settings and advance the dissemination and implementation of behavioral sleep interventions.

TRIAL REGISTRATION

ClinicalTrials.gov (Identifier: NCT02508129 ; Date Registered: July 21, 2015).

Rating of Included Trials on the Efficacy-Effectiveness Spectrum: development of a new tool for systematic reviews

BACKGROUND AND OBJECTIVE

Randomized trials may be designed to provide evidence more strongly related to efficacy or effectiveness of an intervention. When systematic reviews are used to inform clinical or policy decisions, it is important to know the efficacy-effectiveness nature of the included trials. The objective of this study was to develop a tool to characterize randomized trials included in a systematic review on an efficacy-effectiveness continuum.

METHODS

We extracted rating domains and descriptors from existing tools and used a modified Delphi procedure to condense the domains and develop a new tool. The feasibility and interrater reliability of the tool was tested on trials from four systematic reviews.

RESULTS

The Rating of Included Trials on the Efficacy-Effectiveness Spectrum (RITES) tool rates clinical trials on a five-point Likert scale in four domains: (1) participant characteristics, (2) trial setting, (3) flexibility of interventions, and (4) clinical relevance of interventions. When RITES was piloted on trials from three reviews by unaffiliated raters, ratings were variable (intraclass correlation coefficient [ICC] 0.25-0.66 for the four domains); but, when RITES was used on one review by the review authors with expertise on the topic, the ratings were consistent (ICCs > 0.80.

CONCLUSION

RITES may help to characterize the efficacy-effectiveness nature of trials included in systematic reviews.

CHARTing a Path to Pragmatic Tobacco Treatment Research

INTRODUCTION

It is important to consider the degree to which studies are explanatory versus pragmatic to understand the implications of their findings for patients, healthcare professionals, and policymakers. Pragmatic trials test the effectiveness of interventions in real-world conditions; explanatory trials test for efficacy under ideal conditions. The Consortium of Hospitals Advancing Research on Tobacco (CHART) is a network of seven NIH-funded trials designed to identify effective programs that can be widely implemented in routine clinical practice.

METHODS

A cross-sectional analysis of CHART trial study designs was conducted to place each study on the pragmatic-explanatory continuum. After reliability training, six raters independently scored each CHART study according to ten PRagmatic Explanatory Continuum Indicator Summary (PRECIS) dimensions, which covered participant eligibility criteria, intervention flexibility, practitioner expertise, follow-up procedures, participant compliance, practitioner adherence, and outcome analyses. Means and SDs were calculated for each dimension of each study, with lower scores representing more pragmatic elements. Results were plotted on "spoke and wheel" diagrams. The rating process and analyses were performed in October 2014 to September 2015.

RESULTS

All seven CHART trials tended toward the pragmatic end of the spectrum, although there was a range from 0.76 (SD=0.23) to 1.85 (SD=0.58). Most studies included some explanatory design elements.

CONCLUSIONS

CHART findings should be relatively applicable to clinical practice. Funders and reviewers could integrate PRECIS criteria into their guidelines to better facilitate pragmatic research. CHART study protocols, coupled with scores reported here, may help readers improve the design of their own pragmatic trials.

Protocol for the Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes (ENGAGES) study: a pragmatic, randomised clinical trial

INTRODUCTION

Postoperative delirium, arbitrarily defined as occurring within 5 days of surgery, affects up to 50% of patients older than 60 after a major operation. This geriatric syndrome is associated with longer intensive care unit and hospital stay, readmission, persistent cognitive deterioration and mortality. No effective preventive methods have been identified, but preliminary evidence suggests that EEG monitoring during general anaesthesia, by facilitating reduced anaesthetic exposure and EEG suppression, might decrease incident postoperative delirium. This study hypothesises that EEG-guidance of anaesthetic administration prevents postoperative delirium and downstream sequelae, including falls and decreased quality of life.

METHODS AND ANALYSIS

This is a 1232 patient, block-randomised, double-blinded, comparative effectiveness trial. Patients older than 60, undergoing volatile agent-based general anaesthesia for major surgery, are eligible. Patients are randomised to 1 of 2 anaesthetic approaches. One group receives general anaesthesia with clinicians blinded to EEG monitoring. The other group receives EEG-guidance of anaesthetic agent administration. The outcomes of postoperative delirium (≤5 days), falls at 1 and 12 months and health-related quality of life at 1 and 12 months will be compared between groups. Postoperative delirium is assessed with the confusion assessment method, falls with ProFaNE consensus questions and quality of life with the Veteran's RAND 12-item Health Survey. The intention-to-treat principle will be followed for all analyses. Differences between groups will be presented with 95% CIs and will be considered statistically significant at a two-sided p<0.05.

ETHICS AND DISSEMINATION

Electroencephalography Guidance of Anesthesia to Alleviate Geriatric Syndromes (ENGAGES) is approved by the ethics board at Washington University. Recruitment began in January 2015. Dissemination plans include presentations at scientific conferences, scientific publications, internet-based educational materials and mass media.

TRIAL REGISTRATION NUMBER

NCT02241655; Pre-results.

Fast versus slow weight loss: development process and rationale behind the dietary interventions for the TEMPO Diet Trial

OBJECTIVE AND METHODS

Finding effective solutions to curb the obesity epidemic is a great global public health challenge. The need for long-term follow-up necessitates weight loss trials conducted in real-world settings, outside the confines of tightly controlled laboratory or clinic conditions. Given the complexity of eating behaviour and the food supply, this makes the process of designing a practical dietary intervention that stands up to scientific rigor difficult. Detailed information about the dietary intervention itself, as well as the process of developing the final intervention and its underlying rationale, is rarely reported in scientific weight management publications but is valuable and essential for translating research into practice. Thus, this paper describes the design process and underlying rationale behind the dietary interventions in an exemplar weight loss trial - the TEMPO Diet Trial (Type of Energy Manipulation for Promoting optimal metabolic health and body composition in Obesity). This trial assesses the long-term effects of fast versus slow weight loss on adiposity, fat free mass, muscle strength and bone density in women with obesity (body mass index 30-40 kg m-2) that are 45-65 years of age, postmenopausal and sedentary.

RESULTS AND CONCLUSIONS

This paper is intended as a resource for researchers and/or clinicians to illustrate how theoretical values based on a hypothesis can be translated into a dietary weight loss intervention to be used in free-living women of varying sizes.

Effectiveness of centre-based childcare interventions in increasing child physical activity: a systematic review and meta-analysis for policymakers and practitioners

CONTEXT

The review describes the effectiveness of physical activity interventions implemented in centre-based childcare services and (i) examines characteristics of interventions that may influence intervention effects; (ii) describes the effects of pragmatic interventions and non-pragmatic interventions; (iii) assesses adverse effects; and (iv) describes cost-effectiveness of interventions

METHODS

Data sources were Cochrane Central Register of Controlled trials, MEDLINE, EMBASE, PsycINFO, ERIC, CINAHL, SCOPUS and SPORTDISCUS. Studies selected included randomized controlled trials conducted in centre-based childcare including an intervention to increase objectively measured physical activity in children aged less than 6 years. Data were converted into standardized mean difference (SMD) and analysed using a random effects model.

RESULTS

Overall interventions significantly improved child physical activity (SMD 0.44; 95% confidence interval [CI]: 0.12-0.76). Significant effects were found for interventions that included structured activity (SMD 0.53; 95% CI: 0.12-0.94), delivery by experts (SMD 1.26; 95% CI: 0.20-2.32) and used theory (SMD 0.76; 95% CI: 0.08-1.44). Non-pragmatic (SMD 0.80; 95% CI: 0.12-1.48) but not pragmatic interventions (SMD 0.10; 95% CI:-0.13-0.33) improved child physical activity. One trial reported adverse events, and no trials reported cost data.

CONCLUSIONS

Intervention effectiveness varied according to intervention and trial design characteristics. Pragmatic trials were not effective, and information on cost and adverse effects was lacking. Evidence gaps remain for policymakers and practitioners regarding the effectiveness and feasibility of childcare-based physical activity interventions.

Use of PRECIS ratings in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory

BACKGROUND

The National Institutes of Health (NIH) Health Care Systems Research Collaboratory (NIH Collaboratory) seeks to produce generalizable knowledge about the conduct of pragmatic research in health systems. This analysis applied the PRECIS-2 pragmatic trial criteria to five NIH Collaboratory pragmatic trials to better understand 1) the pragmatic aspects of the design and implementation of treatments delivered in real world settings and 2) the usability of the PRECIS-2 criteria for assessing pragmatic features across studies and across time.

METHODS/DESIGN

Using the PRECIS-2 criteria, five pragmatic trials were each rated by eight raters. For each trial, we reviewed the original grant application and a required progress report written at the end of a 1-year planning period that included changes to the protocol or implementation approach. We calculated median scores and interrater reliability for each PRECIS domain and for the overall trial at both time points, as well as the differences in scores between the two time points. We also reviewed the rater comments associated with the scores.

RESULTS

All five trials were rated to be more pragmatic than explanatory, with comments indicating that raters generally perceived them to closely mirror routine clinical care across multiple domains. The PRECIS-2 domains for which the trials were, on average, rated as most pragmatic on the 1 to 5 scale at the conclusion of the planning period included primary analysis (mean = 4.7 (range = 4.5 to 4.9)), recruitment (4.3 (3.6 to 4.8)), eligibility (4.1 (3.4 to 4.8)), setting (4.1 (4.0 to 4.4)), follow-up (4.1 (3.4 to 4.9)), and primary outcome (4.1 (3.5 to 4.9)). On average, the less pragmatic domains were organization (3.3 (2.6 to 4.4)), flexibility of intervention delivery (3.5 (2.1-4.5)), and flexibility of intervention adherence (3.8 (2.8-4.5)). Interrater agreement was modest but statistically significant for four trials (Gwet's AC1 statistic range 0.23 to 0.40) and the intraclass correlation coefficient ranged from 0.05 to 0.31. Rating challenges included assigning a single score for domains that may relate to both patients and care settings (that is, eligibility or recruitment) and determining to what extent aspects of complex research interventions differ from usual care.

CONCLUSIONS

These five trials in diverse healthcare settings were rated as highly pragmatic using the PRECIS-2 criteria. Applying the tool generated insightful discussion about real-world design decisions but also highlighted challenges using the tool. PRECIS-2 raters would benefit from additional guidance about how to rate the interwoven patient and practice-level considerations that arise in pragmatic trials.

TRIAL REGISTRATIONS

Clinicaltrials.gov trial registrations: NCT02019225 , NCT01742065 , NCT02015455 , NCT02113592 , NCT02063867 .

Differences between the real and the desired worlds in the results of clinical trials

OBJECTIVE

We refer to the effectiveness (known as pragmatic or real world) and efficacy (known as explanatory or desired or ideal world) of interventions. However, these terms seem to be randomly chosen by investigators who design clinical trials and do not always reflect the true purpose of the study. A pragmatic-explanatory continuum indicator summary tool was thus developed with the aim of identifying the characteristics of clinical trials that distinguish between effectiveness and efficacy issues. We verified whether clinical trials used the criteria proposed by the indicator summary tool, and we categorized these clinical trials according to a new classification.

METHOD

A systematic survey of randomized clinical trials was performed. We added a score ranging from 0 (more efficacious) to 10 (more effective) to each domain of the indicator summary tool and proposed the following classifications: high efficacy (<25), moderate efficacy (25-50), moderate effectiveness (51-75), and high effectiveness (<75).

RESULTS

A total of 844 randomized trials were analyzed. No analyzed trials used the criteria proposed by the indicator summary tool. Approximately 44% of the trials were classified as having moderate effectiveness, and 43.82% were classified as having moderate efficacy.

CONCLUSIONS

Most clinical trials used the term "efficacy" to illustrate the application of results in clinical practice, but the majority of those were classified as having moderate effectiveness according to our proposed score. The classification based on the 0-100 score is still highly subjective and can be easily misunderstood in all domains based on each investigator's own experiences and knowledge.

Technology-assisted weight loss interventions in primary care: a systematic review

BACKGROUND

The US Preventive Services Task Force recommends screening for and treating obesity. However, there are many barriers to successfully treating obesity in primary care (PC). Technology-assisted weight loss interventions offer novel ways of improving treatment, but trials are overwhelmingly conducted outside of PC and may not translate well into this setting. We conducted a systematic review of technology-assisted weight loss interventions specifically tested in PC settings.

METHODS

We searched the literature from January 2000 to March 2014.

INCLUSION CRITERIA

(1) Randomized controlled trial; (2) trials that utilized the Internet, personal computer, and/or mobile device; and (3) occurred in an ambulatory PC setting. We applied the Cochrane Effective Practice and Organization of Care (EPOC) and Delphi criteria to assess bias and the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) criteria to assess pragmatism (whether trials occurred in the real world versus under ideal circumstances). Given heterogeneity, results were not pooled quantitatively.

RESULTS

Sixteen trials met inclusion criteria. Twelve (75 %) interventions achieved weight loss (range: 0.08 kg - 5.4 kg) compared to controls, while 5-45 % of patients lost at least 5 % of baseline weight. Trial duration and attrition ranged from 3-36 months and 6-80 %, respectively. Ten (63 %) studies reported results after at least 1 year of follow-up. Interventions used various forms of personnel, technology modalities, and behavior change elements; trials most frequently utilized medical doctors (MDs) (44 %), web-based applications (63 %), and self-monitoring (81 %), respectively. Interventions that included clinician-guiding software or feedback from personnel appeared to promote more weight loss than fully automated interventions. Only two (13 %) studies used publically available technologies. Many studies had fair pragmatism scores (mean: 2.8/4), despite occurring in primary care.

DISCUSSION

Compared to usual care, technology-assisted interventions in the PC setting help patients achieve weight loss, offering evidence-based options to PC providers. However, best practices remain undetermined. Despite occurring in PC, studies often fall short in utilizing pragmatic methodology and rarely provide publically available technology. Longitudinal, pragmatic, interdisciplinary, and open-source interventions are needed.

Effectiveness of app-based relaxation for patients with chronic low back pain (Relaxback) and chronic neck pain (Relaxneck): study protocol for two randomized pragmatic trials

BACKGROUND

Chronic low back pain (LBP) and neck pain (NP) are highly prevalent conditions resulting in high economic costs. Treatment guidelines recommend relaxation techniques, such as progressive muscle relaxation, as adjuvant therapies. Self-care interventions could have the potential to reduce costs in the health care system, but their effectiveness, especially in a usual care setting, is unclear. The aim of these two pragmatic randomized studies is to evaluate whether an additional app-delivered relaxation is more effective in the reduction of chronic LBP or NP than usual care alone.

METHODS/DESIGN

Each pragmatic randomized two-armed study aims to include a total of 220 patients aged 18 to 65 years with chronic (>12 weeks) LBP or NP and an average pain intensity of ≥ 4 on a numeric rating scale (NRS) in the 7 days before recruitment. The participants will be randomized into an intervention and a usual care group. The intervention group will be instructed to practice one of these 3 relaxation techniques on at least 5 days/week for 15 minutes/day over a period of 6 months starting on the day of randomization: autogenic training, mindfulness meditation, or guided imagery. Instructions and exercises will be provided using a smartphone app, baseline information will be collected using paper and pencil. Follow-up information (daily, weekly, and after 3 and 6 months) will be collected using electronic diaries and questionnaires included in the app. The primary outcome measure will be the mean LBP or NP intensity during the first 3 months of intervention based on daily pain intensity measurements on a NRS (0 = no pain, 10 = worst possible pain). The secondary outcome parameters will include the mean pain intensity during the first 6 months after randomization based on daily measurements, the mean pain intensity measured weekly as the average pain intensity of the previous 7 days over 3 and 6 months, pain acceptance, 'LBP- and NP-related' stress, sick leave days, pain medication intake, adherence, suspected adverse reaction, and serious adverse events.

DISCUSSION

The designed studies reflect a usual self-care setting and will provide evidence on a pragmatic self-care intervention that is easy to combine with care provided by medical professionals.

TRIAL REGISTRATION

ClinicalTrials.gov identifier Relaxback NCT02019498, Relaxneck NCT02019134 registered on 18 December 2013.

How pragmatic is it? Lessons learned using PRECIS and RE-AIM for determining pragmatic characteristics of research

Background

The need for high-quality evidence that is applicable in real-world, routine settings continues to increase. Pragmatic trials are designed to evaluate the effectiveness of interventions in real-world settings, whereas explanatory trials aim to test whether an intervention works under optimal situations. There is a continuum between explanatory and pragmatic trials. Most trials have aspects of both, making it challenging to label and categorize a trial and to evaluate its potential for translation into practice.

Methods

We summarize our experience applying the Pragmatic-Explanatory Continuum Indicator Summary (PRECIS) combined with external validity items based on the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework to three studies to provide a more robust and comprehensive assessment of trial characteristics related to translation of research. We summarize lessons learned using domains from the combined frameworks for use in study planning, evaluating specific studies, and reviewing the literature and make recommendations for future use.

Results

A variety of coders can be trained to use the PRECIS and RE-AIM domains. These domains can also be used for diverse purposes, content areas, and study types, but are not without challenges. Both PRECIS and RE-AIM domains required modification in two of the three studies to evaluate and rate domains specific to study type. Lessons learned involved: dedicating enough time for training activities related to the domains; use of reviewers with a range of familiarity with specific study protocols; how to best adapt ratings that reflect complex study designs; and differences of opinion regarding the value of creating a composite score for these criteria.

Conclusions

Combining both frameworks can specifically help identify where and how a study is and is not pragmatic. Using both PRECIS and RE-AIM allows for standard reporting of key study characteristics related to pragmatism and translation. Such measures should be used more consistently to help plan more pragmatic studies, evaluate progress, increase transparency of reporting, and integrate literature to facilitate translation of research into practice and policy.

Electronic supplementary material

The online version of this article (doi:10.1186/s13012-014-0096-x) contains supplementary material, which is available to authorized users.

What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation

BACKGROUND

One of the reasons for the slow and uncertain translation of research into practice is likely due to the emphasis in science on explanatory models and efficacy designs rather than more pragmatic approaches.

METHODS

Following a brief definition of what constitutes a pragmatic approach, I provide examples of pragmatic methods, measures, and models and how they have been applied.

RESULTS

Descriptions are provided of pragmatic trials and related designs, practical measures including patient-reported items for the electronic health record, and the Evidence Integration Triangle and RE-AIM practical models, each of which can help increase the relevance of research to policy makers, practitioners, and patients/consumers.

CONCLUSIONS

By focusing on the perspective of stakeholders and the context for application of scientific findings, pragmatic approaches can accelerate the integration of research, policy, and practice. Progress has been made, especially in pragmatic trials but even more opportunities remain.

Economic analyses of the Be Fit Be Well program: a weight loss program for community health centers

BACKGROUND

The U.S. Preventive Services Task Force has released new guidelines on obesity, urging primary care physicians to provide obese patients with intensive, multi-component behavioral interventions. However, there are few studies of weight loss in real world nonacademic primary care, and even fewer in largely racial/ethnic minority, low-income samples.

OBJECTIVE

To evaluate the recruitment, intervention and replications costs of a 2-year, moderate intensity weight loss and blood pressure control intervention.

DESIGN

A comprehensive cost analysis was conducted, associated with a weight loss and hypertension management program delivered in three community health centers as part of a pragmatic randomized trial.

PARTICIPANTS

Three hundred and sixty-five high risk, low-income, inner city, minority (71 % were Black/African American and 13 % were Hispanic) patients who were both hypertensive and obese.

MAIN MEASURES

Measures included total recruitment costs and intervention costs, cost per participant, and incremental costs per unit reduction in weight and blood pressure.

KEY RESULTS

Recruitment and intervention costs were estimated $2,359 per participant for the 2-year program. Compared to the control intervention, the cost per additional kilogram lost was $2,204 /kg, and for blood pressure, $621 /mmHg. Sensitivity analyses suggest that if the program was offered to a larger sample and minor modifications were made, the cost per participant could be reduced to the levels of many commercially available products.

CONCLUSIONS

The costs associated with the Be Fit Be Well program were found to be significantly more expensive than many commercially available products, and much higher than the amount that the Centers for Medicare and Medicaid reimburse physicians for obesity counseling. However, given the serious and costly health consequences associated with obesity in high risk, multimorbid and socioeconomically disadvantaged patients, the resources needed to provide interventions like those described here may still prove to be cost-effective with respect to producing long-term behavior change.

A systematic review of eHealth cancer prevention and control interventions: new technology, same methods and designs?

There has been a recent surge of eHealth programs in cancer and other content areas, but few reviews have focused on the methodologies and designs employed in these studies. We conducted a systematic review of studies on eHealth interventions on cancer prevention and control published between 2001 and 2010 applying the Pragmatic Explanatory Continuum Indicator Summary (PRECIS) criteria and external validity components from the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) framework. We identified 113 studies that focused on cancer prevention and control of eHealth interventions. Most studies fell midway along the explanatory/pragmatic trial continuum, but few reported on various practical feasibility criteria for translation. Despite vast interest in cancer eHealth and the applied nature of this field, few studies considered key external validity issues. There is a need for use of alternative pragmatic study designs and transparent reporting of external validity components to produce more rapid and generalizable results.

Implementation science approaches for integrating eHealth research into practice and policy

PURPOSE

To summarize key issues in the eHealth field from an implementation science perspective and to highlight illustrative processes, examples and key directions to help more rapidly integrate research, policy and practice.

METHODS

We present background on implementation science models and emerging principles; discuss implications for eHealth research; provide examples of practical designs, measures and exemplar studies that address key implementation science issues; and make recommendations for ways to more rapidly develop and test eHealth interventions as well as future research, policy and practice.

RESULTS

The pace of eHealth research has generally not kept up with technological advances, and many of our designs, methods and funding mechanisms are incapable of providing the types of rapid and relevant information needed. Although there has been substantial eHealth research conducted with positive short-term results, several key implementation and dissemination issues such as representativeness, cost, unintended consequences, impact on health inequities, and sustainability have not been addressed or reported. Examples of studies in several of these areas are summarized to demonstrate this is possible.

CONCLUSIONS

eHealth research that is intended to translate into policy and practice should be more contextual, report more on setting factors, employ more responsive and pragmatic designs and report results more transparently on issues important to potential adopting patients, clinicians and organizational decision makers. We outline an alternative development and assessment model, summarize implementation science findings that can help focus attention, and call for different types of more rapid and relevant research and funding mechanisms.

Designing a valid randomized pragmatic primary care implementation trial: the my own health report (MOHR) project

BACKGROUND

There is a pressing need for greater attention to patient-centered health behavior and psychosocial issues in primary care, and for practical tools, study designs and results of clinical and policy relevance. Our goal is to design a scientifically rigorous and valid pragmatic trial to test whether primary care practices can systematically implement the collection of patient-reported information and provide patients needed advice, goal setting, and counseling in response.

METHODS

This manuscript reports on the iterative design of the My Own Health Report (MOHR) study, a cluster randomized delayed intervention trial. Nine pairs of diverse primary care practices will be randomized to early or delayed intervention four months later. The intervention consists of fielding the MOHR assessment--addresses 10 domains of health behaviors and psychosocial issues--and subsequent provision of needed counseling and support for patients presenting for wellness or chronic care. As a pragmatic participatory trial, stakeholder groups including practice partners and patients have been engaged throughout the study design to account for local resources and characteristics. Participatory tasks include identifying MOHR assessment content, refining the study design, providing input on outcomes measures, and designing the implementation workflow. Study outcomes include the intervention reach (percent of patients offered and completing the MOHR assessment), effectiveness (patients reporting being asked about topics, setting change goals, and receiving assistance in early versus delayed intervention practices), contextual factors influencing outcomes, and intervention costs.

DISCUSSION

The MOHR study shows how a participatory design can be used to promote the consistent collection and use of patient-reported health behavior and psychosocial assessments in a broad range of primary care settings. While pragmatic in nature, the study design will allow valid comparisons to answer the posed research question, and findings will be broadly generalizable to a range of primary care settings. Per the pragmatic explanatory continuum indicator summary (PRECIS) framework, the study design is substantially more pragmatic than other published trials. The methods and findings should be of interest to researchers, practitioners, and policy makers attempting to make healthcare more patient-centered and relevant.

TRIAL REGISTRATION

Clinicaltrials.gov: NCT01825746.

Generalizing the findings from group dynamics-based physical activity research to practice settings: what do we know?

The general purpose of this secondary analysis of a prior systematic review was to determine the degree to which group dynamics-based physical activity interventions align with research processes and outcomes that are more likely to facilitate the translation of research into practice. To accomplish this, a systematic search was conducted within Science Citation Index Expanded, Social Sciences Citation Index, Arts and Humanities Citation Index, and MEDLINE databases to identify articles published prior to 2010 that used at least one group dynamics-based strategy (e.g., group interaction, group goal setting) in physical activity promotion. These 17 intervention studies were identified and coded based on the reach, effectiveness, adoption, implementation, maintenance framework and Pragmatic-Explanatory Continuum Indicator Summary (for trial design characteristics). Reporting was infrequent for external validity factors (i.e., representativeness, adoption, cost, maintenance) but more frequent for internal validity factors (e.g., inclusion criteria). Intervention costs were not reported. Studies were more likely to be pragmatic (i.e., designed to determine the effects of an intervention under the usual conditions in which it will be applied) in areas of participant compliance and practitioner adherence and explanatory (i.e., designed to determine the effects of an intervention under ideal conditions) and in areas of practitioner expertise and flexibility of intervention protocol. While a number of these interventions were tested in more pragmatic settings, external validity factors were still underreported.

Making clinical trials more relevant: improving and validating the PRECIS tool for matching trial design decisions to trial purpose

Background

If you want to know which of two or more healthcare interventions is most effective, the randomised controlled trial is the design of choice. Randomisation, however, does not itself promote the applicability of the results to situations other than the one in which the trial was done. A tool published in 2009, PRECIS (PRagmatic Explanatory Continuum Indicator Summaries) aimed to help trialists design trials that produced results matched to the aim of the trial, be that supporting clinical decision-making, or increasing knowledge of how an intervention works. Though generally positive, groups evaluating the tool have also found weaknesses, mainly that its inter-rater reliability is not clear, that it needs a scoring system and that some new domains might be needed. The aim of the study is to: Produce an improved and validated version of the PRECIS tool. Use this tool to compare the internal validity of, and effect estimates from, a set of explanatory and pragmatic trials matched by intervention.

Methods

The study has four phases. Phase 1 involves brainstorming and a two-round Delphi survey of authors who cited PRECIS. In Phase 2, the Delphi results will then be discussed and alternative versions of PRECIS-2 developed and user-tested by experienced trialists. Phase 3 will evaluate the validity and reliability of the most promising PRECIS-2 candidate using a sample of 15 to 20 trials rated by 15 international trialists. We will assess inter-rater reliability, and raters’ subjective global ratings of pragmatism compared to PRECIS-2 to assess convergent and face validity. Phase 4, to determine if pragmatic trials sacrifice internal validity in order to achieve applicability, will compare the internal validity and effect estimates of matched explanatory and pragmatic trials of the same intervention, condition and participants. Effect sizes for the trials will then be compared in a meta-regression. The Cochrane Risk of Bias scores will be compared with the PRECIS-2 scores of pragmatism.

Discussion

We have concrete suggestions for improving PRECIS and a growing list of enthusiastic individuals interested in contributing to this work. By early 2014 we expect to have a validated PRECIS-2.

Effectiveness of additional self-care acupressure for women with menstrual pain compared to usual care alone: using stakeholder engagement to design a pragmatic randomized trial and study protocol

Background

Self-care acupressure might be successful in treating menstrual pain, which is common among young women. There is a need for comparative effectiveness research with stakeholder engagement in all phases seeking to address the needs of decision-makers. Our aim was to design a study on the effectiveness of additional self-care acupressure for menstrual pain comparing usual care alone using different methods of stakeholder engagement.

Methods

The study was designed using multiple mixed methods for stakeholder engagement. Based on the results of a survey and focus group discussion, a stakeholder advisory group developed the study design.

Results

Stakeholder engagement resulted in a two-arm pragmatic randomized trial. Two hundred and twenty women aged 18 to 25 years with menstrual pain will be included in the study. Outcome measurement will be done using electronic questionnaires provided by a study specific mobile application (App). Primary outcome will be the mean pain intensity at the days of pain during the third menstruation after therapy start.

Conclusion

Stakeholder engagement helped to develop a study design that better serves the needs of decision makers, including an App as a modern tool for both intervention and data collection in a young target group.

Trial registration

Clinicaltrials.gov identifier http://NCT01582724

Recruitment and retention of participants in a pragmatic randomized intervention trial at three community health clinics: results and lessons learned

Background

Obesity and hypertension and their associated health complications disproportionately affect communities of color and people of lower socioeconomic status. Recruitment and retention of these populations in research trials, and retention in weight loss trials has been an ongoing challenge.

Methods

Be Fit, Be Well was a pragmatic randomized weight loss and hypertension management trial of patients attending one of three community health centers in Boston, Massachusetts. Participants were asked to complete follow-up assessments every 6-months for two years. We describe challenges encountered and strategies implemented to recruit and retain trial participants over the 24-month intervention. We also identify baseline participant characteristics associated with retention status. Retention strategies included financial incentives, contact between assessment visits, building relationships with health center primary care providers (PCPs) and staff, and putting participant convenience first.

Results

Active refusal rates were low with 130 of 2,631 patients refusing participation (4.9%). Of 474 eligible persons completing telephone screening, 365 (77.0%) completed their baseline visit and were randomized into the study. The study population was predominantly non-Hispanic Black (71.2%), female (68.5%) and reported annual household income of less than $35,000 (70.1%). Recruitment strategies included use of passive approval of potential participants by PCPs, use of part-time staff, and outsourcing calls to a call center. A total of 314 (86.0%) people completed the 24-month visit. Retention levels varied across study visits and intervention condition. Most participants completed three or more visits (69.6%), with 205 (56.2%) completing all four. At 24-months, lower retention was observed for males and the intervention condition. Retention strategies included building strong relationships with clinic staff, flexibility in overcoming participant barriers through use of taxi vouchers, night and weekend appointments, and keeping participants engaged via newsletters and social gatherings.

Conclusion

We were able to retain 86.0% of participants at 24-months. Recruitment and retention of high percentages of racial/ethnic minorities and lower income samples is possible with planning, coordination with a trusted community setting and staff (e.g. community health centers and RAs), adaptability and building strong relationships.

Trial registration

Clinicaltrials.gov Identifier: NCT00661817

Use of RE-AIM to Address Health Inequities: Application in a low-income community health center based weight loss and hypertension self-management program

BACKGROUND

While health inequities are well documented, and there are helpful frameworks to understand health disparities, implementation frameworks are also needed to focus the design, evaluation and reporting on interventions targeting populations at increased risk.

PURPOSE

Describe how the RE-AIM framework (Reach, Effectiveness, Adoption, Implementation, and Maintenance) can be used for these purposes and illustrate its application in the context of a randomized, pragmatic weight-loss and hypertension self-management intervention.

METHODS

RE-AIM was used to both plan and evaluate the Be Fit Be Well program for urban community health center patients.

RESULTS

The RE-AIM framework helped to focus attention on and produce high rates of adoption and reach. Implementation rates varied across components. Weight losses were statistically significant, but not clinically significant. They were robust across a variety of patient characteristics, and the program was relatively low cost. Individual weight losses and blood pressure reductions were maintained throughout the 24-month period, but the program was not sustained at any of the three settings.

CONCLUSION

Implementation frameworks such as RE-AIM can help design pragmatic interventions that focus on both the context for disparities reduction and the ultimate goal of public health impact.

Implementation science and its application to population health

Implementation science studies the use of strategies to adapt and use evidence-based interventions in targeted settings (e.g., schools, workplaces, health care facilities, public health departments) to sustain improvements to population health. This nascent field of research is in the early stages of developing theories of implementation and evaluating the properties of measures. Stakeholder engagement, effectiveness studies, research synthesis, and mathematical modeling are some of the methods used by implementation scientists to identify strategies to embed evidence-based interventions in clinical and public health programs. However, for implementation science to reach its full potential to improve population health the existing paradigm for how scientists create evidence, prioritize publications, and synthesize research needs to shift toward greater stakeholder input and improved reporting on external validity. This shift will improve the relevance of the research that is produced and provide information that will help guide decision makers in their selection of research-tested interventions.

How pragmatic or explanatory is the randomized, controlled trial? The application and enhancement of the PRECIS tool to the evaluation of a smoking cessation trial

BACKGROUND

Numerous explanatory randomized trials support the efficacy of chronic disease interventions, including smoking cessation treatments. However, there is often inadequate adoption of these interventions for various reasons, one being the limitation of generalizability of the explanatory studies in real-world settings. Randomized controlled trials can be rated as more explanatory versus pragmatic along 10 dimensions. Pragmatic randomized clinical trials generate more realistic estimates of effectiveness with greater relevance to clinical practice and for health resource allocation decisions. However, there is no clear method to scale each dimension during the trial design phase to ensure that the design matches the intended purpose of the study.

METHODS

We designed a pragmatic, randomized, controlled study to maximize external validity by addressing several barriers to smoking cessation therapy in ambulatory care. We analyzed our design and methods using the recently published 'Pragmatic-Explanatory Continuum Indicatory Summary (PRECIS)' tool, a qualitative method to assess trial design across 10 domains. We added a 20-point numerical rating scale and a modified Delphi process to improve consensus in rating these domains.

RESULTS

After two rounds of review, there was consensus on all 10 domains of study design. No single domain was scored as either fully pragmatic or fully explanatory; but overall, the study scored high on pragmatism.

CONCLUSIONS

This addition to the PRECIS tool may assist other trial designers working with interdisciplinary co-investigators to rate their study design while building consensus.