A comparison of meta-analytic methods for synthesizing evidence from explanatory and pragmatic trials

Methods for pragmatic randomized clinical trials of pain therapies: IMMPACT statement

ABSTRACT

Pragmatic, randomized, controlled trials hold the potential to directly inform clinical decision making and health policy regarding the treatment of people experiencing pain. Pragmatic trials are designed to replicate or are embedded within routine clinical care and are increasingly valued to bridge the gap between trial research and clinical practice, especially in multidimensional conditions, such as pain and in nonpharmacological intervention research. To maximize the potential of pragmatic trials in pain research, the careful consideration of each methodological decision is required. Trials aligned with routine practice pose several challenges, such as determining and enrolling appropriate study participants, deciding on the appropriate level of flexibility in treatment delivery, integrating information on concomitant treatments and adherence, and choosing comparator conditions and outcome measures. Ensuring data quality in real-world clinical settings is another challenging goal. Furthermore, current trials in the field would benefit from analysis methods that allow for a differentiated understanding of effects across patient subgroups and improved reporting of methods and context, which is required to assess the generalizability of findings. At the same time, a range of novel methodological approaches provide opportunities for enhanced efficiency and relevance of pragmatic trials to stakeholders and clinical decision making. In this study, best-practice considerations for these and other concerns in pragmatic trials of pain treatments are offered and a number of promising solutions discussed. The basis of these recommendations was an Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) meeting organized by the Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks.

The need for more pragmatic trials in glaucoma research

AIM

There have been a number of clinical trials in glaucoma research published in the past two decades. Most of these trials were designed to evaluate very specific issues in selected populations placing them in the explanatory end of the pragmatic-explanatory continuum. The purpose of this study was to assess the level of pragmatism of published randomized controlled trials in glaucoma.

METHODS

A PubMed search using 'glaucoma' from 1995 to 2022 and randomized controlled trial (RCT) article type was done. Each study was assessed by three independent examiners using the Pragmatic-Explanatory Continuum Indicator Summary version 2 (PRECIS-2) toolkit. Scores were calculated for each study to determine the level of pragmatism. A summed score ≥36 was indicative of a very pragmatic study.

RESULTS

Thirty-two different articles were included in the analysis. These papers represented 13 different landmark trials. The median PRECIS-2 score was 32 (range, 25 for the Early Manifest Glaucoma Trial (EMGT) to 34 to the Collaborative Normal Tension Glaucoma Study (CNTGS) and the Ocular Hypertension Treatment Study). The Treatment of Advanced Glaucoma Study (TAGS), was considered very pragmatic and scored 33 points.

CONCLUSION

Despite the number of RCTs in glaucoma, there is still a need for more pragmatic studies.

Meta-research on pragmatism of randomized trials: rationale and design of the PragMeta database

BACKGROUND

Pragmatic trials provide decision-oriented, real-world evidence that is highly applicable and generalizable. The interest in real-world evidence is fueled by the assumption that effects in the "real-world" are different to effects obtained under artificial, controlled, research conditions as often used for traditional explanatory trials. However, it is unknown which features of pragmatism, generalizability, and applicability would be responsible for such differences. There is a need to provide empirical evidence and promote meta-research to answer these fundamental questions on the pragmatism of randomized trials and real-world evidence. Here, we describe the rationale and design of the PragMeta database which pursues this goal ( www.PragMeta.org ).

METHODS

PragMeta is a non-commercial, open data platform and infrastructure to facilitate research on pragmatic trials. It collects and shares data from published randomized trials that either have a specific design feature or other characteristic related to pragmatism or they form clusters of trials addressing the same research question but having different aspects of pragmatism. This lays the foundation to determine the relationship of various features of pragmatism, generalizability, and applicability with intervention effects or other trial characteristics. The database contains trial data actively collected for PragMeta but also allows to import and link existing datasets of trials collected for other purposes, forming a large-scale meta-database. PragMeta captures data on (1) trial and design characteristics (e.g., sample size, population, intervention/comparison, outcome, longitudinal structure, blinding), (2) effects estimates, and (3) various determinants of pragmatism (e.g., the use of routinely collected data) and ratings from established tools used to determine pragmatism (e.g., the PRagmatic-Explanatory Continuum Indicator Summary 2; PRECIS-2). PragMeta is continuously provided online, inviting the meta-research community to collaborate, contribute, and/or use the database. As of April 2023, PragMeta contains data from > 700 trials, mostly with assessments on pragmatism.

CONCLUSIONS

PragMeta will inform a better understanding of pragmatism and the generation and interpretation of real-world evidence.

The benefits of data sharing and ensuring open sources of systematic review data

AIMS

The benefits of increasing public access to data from clinical trials are widely accepted. Such benefits extend to the sharing of data from high-quality systematic reviews, given the time and cost involved with undertaking reviews. We describe the application of open sources of review data, outline potential challenges and highlight efforts made to address these challenges, with the intent of encouraging publishers, funders and authors to consider sharing review data more broadly.

RESULTS

We describe the application of systematic review data in: (i) advancing understanding of clinical trials and systematic review methods, (ii) repurposing of data to answer public health policy and practice relevant questions, (iii) identification of research gaps and (iv) accelerating the conduct of rapid reviews to inform decision making. While access, logistical, motivational and legal challenges exist, there has been progress made by systematic review, academic and funding agencies to incentivise data sharing and create infrastructure to support greater access to systematic review data.

CONCLUSION

There is opportunity to maximize the benefits of research investment in undertaking systematic reviews by ensuring open sources of systematic review data. Efforts to create such systems should draw on learnings and principles outlined for sharing clinical trial data.

Implementation of e-Mental Health Interventions for Informal Caregivers of Adults With Chronic Diseases: Mixed Methods Systematic Review With a Qualitative Comparative Analysis and Thematic Synthesis

BACKGROUND

Informal caregivers commonly experience mental health difficulties related to their caregiving role. e-Mental health interventions provide mental health support in a format that may be more accessible to informal caregivers. However, e-mental health interventions are seldom implemented in real-world practice.

OBJECTIVE

This mixed methods systematic review aimed to examine factors associated with the effectiveness and implementation of e-mental health interventions for informal caregivers of adults with chronic diseases. To achieve this aim, two approaches were adopted: combinations of implementation and intervention characteristics sufficient for intervention effectiveness were explored using qualitative comparative analysis, and barriers to and facilitators of implementation of e-mental health interventions for informal caregivers were explored using thematic synthesis.

METHODS

We identified relevant studies published from January 1, 2007, to July 6, 2022, by systematically searching 6 electronic databases and various secondary search strategies. Included studies reported on the effectiveness or implementation of e-mental health interventions for informal caregivers of adults with cancer, chronic obstructive pulmonary disease, dementia, diabetes, heart disease, or stroke. Randomized controlled trials reporting on caregivers' mental health outcomes were included in a crisp-set qualitative comparative analysis. We assessed randomized controlled trials for bias using the Risk of Bias 2.0 tool, and we assessed how pragmatic or explanatory their trial design was using the Pragmatic Explanatory Continuum Indicator Summary 2 tool. Studies of any design reporting on implementation were included in a thematic synthesis using the Consolidated Framework for Implementation Research to identify barriers to and facilitators of implementation.

RESULTS

Overall, 53 reports, representing 29 interventions, were included in the review. Most interventions (27/29, 93%) focused on informal cancer or dementia caregivers. In total, 14 reports were included in the qualitative comparative analysis, exploring conditions including the presence of peer or professional support and key persuasive design features. Low consistency and coverage prevented the determination of condition sets sufficient for intervention effectiveness. Overall, 44 reports were included in the thematic synthesis, and 152 barriers and facilitators were identified, with the majority related to the intervention and individual characteristic domains of the Consolidated Framework for Implementation Research. Implementation barriers and facilitators in the inner setting (eg, organizational culture) and outer setting (eg, external policies and resources) domains were largely unexplored.

CONCLUSIONS

e-Mental health interventions for informal caregivers tend to be well-designed, with several barriers to and facilitators of implementation identified related to the intervention and individual user characteristics. Future work should focus on exploring the views of stakeholders involved in implementation to determine barriers to and facilitators of implementing e-mental health interventions for informal caregivers, focusing on inner and outer setting barriers and facilitators.

TRIAL REGISTRATION

PROSPERO (International Prospective Register of Systematic Reviews) CRD42020155727; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020155727.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.1136/bmjopen-2019-035406.

Meta-analysis of Pragmatic and Explanatory Trials

Pooling studies with different clinical and methodological features may lead to statistical heterogeneity, particularly if heterogeneity remains unexplained. One potential source of heterogeneity may be how much the included trials tend toward a pragmatic or explanatory design. Many tools have been developed to aid researchers in quantifying pragmatism in clinical trials, at both the design and appraisal stages.In this chapter we review these tools, illustrate examples of their use, and discuss methods of including pragmatism in meta-analysis as a way of exploring heterogeneity.We suggest a stepwise approach to incorporating evidence from pragmatic and explanatory trials which includes planning to assess pragmatism at the protocol stage, collecting data on pragmatism, extracting data on treatment effects, incorporating pragmatism in meta-analysis using subgroup analysis or meta-regression techniques, and interpreting and reporting the findings transparently.

No inexplicable disagreements between real-world data-based nonrandomized controlled studies and randomized controlled trials were found

OBJECTIVES

We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements.

RESULTS

We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement.

CONCLUSION

Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Designing provider-focused implementation trials with purpose and intent: introducing the PRECIS-2-PS tool

Background

First articulated by Schwartz and Lellouch (1967), randomized controlled trials (RCTs) can be conceptualized along a continuum from more explanatory to more pragmatic. The purpose and intent of the former is to test interventions under ideal contexts, and the purpose and intent of the latter is to test interventions in real-world contexts. The PRagmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) is a validated tool that helps researchers make decisions about the elements of the trial to match the overall purpose and intent of the trial along the continuum. The PRECIS-2 tool has guided the design of hundreds of RCTs. However, a few aspects of the tool would benefit from greater clarity, including its application to provider-focused implementation trials rather than patient-focused intervention trials.

Main text

We describe the newly developed PRECIS-2-Provider Strategies (PRECIS-2-PS) tool, an extension of the PRECIS-2 tool, which has been adapted for trials testing provider-focused strategies. We elaborate on nine domains that can make a provider-focused trial more explanatory or more pragmatic, including eligibility, recruitment, setting, implementation resources, flexibility of provider strategies, flexibility of intervention, data collection, primary outcome, and primary analysis. We detail the complementary roles that researchers and stakeholders play in the trial design phase, with implications for generalizability of trial results to the contexts in which they are intended to be applied.

Conclusions

The PRECIS-2-PS tool is designed to help research and practice teams plan for provider-focused trials that reflect the overall intent and purpose of the trial. The tool has potential to help advance the science of provider-focused strategies across a range of trials, with the ultimate goal of facilitating the adoption, integration, and sustainability of provider-focused strategies outside the context of trials.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13012-020-01075-y.

Design characteristics of comparative effectiveness trials for the relief of symptomatic dyspepsia: A systematic review

Background

Dyspepsia represents a symptom domain rather than a diagnostic condition and covers a wide range of complex, underlying pathophysiologies that are not well understood. The review explores comparative effectiveness interventions for the treatment of symptomatic dyspepsia along a pragmatic-explanatory continuum. The aim is to identify relevant design characteristics applicable to future upper gastrointestinal comparative effectiveness research employing integrative medicine.

Methods

Medline, CINAHL, Scopus, Cochrane Central Register of Controlled Trials (CENTRAL) and WHO Clinical Trials were systematically searched until January 2019. Included articles were original research with two or more comparative intervention arms for the primary outcome; relief of symptomatic dyspepsia. Evaluation of the studies was conducted using the pragmatic-explanatory continuum indicator summary (PRECIS-2) tool.

Results

Thirty-six articles were included in the review. A total of 68 Patient Reported Outcome Measurements (PROMs), utilizing 50 different formats were deployed across the studies. The appraisal process revealed eligibility, flexibility in adherence, flexibility in delivery and organization domains further aligned towards an explanatory design.

Conclusion

This review identified three design characteristics relevant for future comparative effectiveness research for the treatment of upper gastrointestinal disorders in a community setting. Extensive exclusion eligibility criteria limited the generalization of comparative effectiveness study results by removing sub-groups of the target populations more at risk of dyspeptic symptoms. The requirement for entry endoscopy was found to be common and not always pragmatically justifiable. Development of validated PROMs appropriate for a generic application to upper gastrointestinal disorders would be advantageous for future comparative effectiveness research within integrative medicine.

Effect of pragmatic versus explanatory interventions on medication adherence in people with cardiometabolic conditions: a systematic review and meta-analysis

OBJECTIVE

To synthesise findings from randomised controlled trials (RCTs) of interventions aimed at increasing medication adherence in individuals with type 2 diabetes (T2DM) and/or cardiovascular disease (CVD). And, in a novel approach, to compare the intervention effect of studies which were categorised as being more pragmatic or more explanatory using the Pragmatic-Explanatory Continuum Indicator Summary-2 (PRECIS-2) tool, to identify whether study design affects outcomes. As explanatory trials are typically held under controlled conditions, findings from such trials may not be relatable to real-world clinical practice. In comparison, pragmatic trials are designed to replicate real-world conditions and therefore findings are more likely to represent those found if the intervention were to be implemented in routine care.

DESIGN

Systematic review and meta-analysis.

DATA SOURCES

Ovid Medline, Ovid Embase, Web of Science and CINAHL from 1 January 2013 to 31 December 2018.

ELIGIBILITY CRITERIA FOR SELECTING STUDIES

RCTs lasting ≥3 months (90 days), involving ≥200 patients in the analysis, with either established CVD and/or T2DM and which measured medication adherence. From 4403 citations, 103 proceeded to full text review. Studies published in any language other than English and conference abstracts were excluded.

MAIN OUTCOME MEASURE

Change in medication adherence.

RESULTS

Of 4403 records identified, 34 studies were considered eligible, of which 28, including 30 861 participants, contained comparable outcome data for inclusion in the meta-analysis. Overall interventions were associated with an increase in medication adherence (OR 1.57 (95% CI: 1.33 to 1.84), p<0.001; standardised mean difference 0.24 (95% CI: -0.10 to 0.59) p=0.101). The effectiveness of interventions did not differ significantly between studies considered pragmatic versus explanatory (p=0.598), but did differ by intervention type, with studies that included a multifaceted rather than a single-faceted intervention having a more significant effect (p=0.010). The analysis used random effect models and used the revised Cochrane Risk of Bias Tool to assess study quality.

CONCLUSIONS

In this meta-analysis, interventions were associated with a significant increase in medication adherence. Overall multifaceted interventions which included an element of education alongside regular patient contact or follow-up showed the most promise. Effectiveness of interventions between pragmatic and explanatory trials was comparable, suggesting that findings can be transferred from idealised to real-word conditions.

PROSPERO REGISTRATION NUMBER

CRD42017059460.

Implementation of e-mental health interventions for informal caregivers of adults with chronic diseases: a protocol for a mixed-methods systematic review with a qualitative comparative analysis

INTRODUCTION

Informal caregivers provide the majority of care to individuals with chronic health conditions, benefiting the care recipient and reducing use of formal care services. However, providing informal care negatively impacts the mental health of many caregivers. E-mental health interventions have emerged as a way to provide accessible mental healthcare to caregivers. Much attention has been given to reviewing the effectiveness and efficacy of such interventions, however, factors related to implementation have received less consideration. Therefore, this mixed-methods systematic review will aim to examine factors associated with the effectiveness and implementation of e-mental health interventions for caregivers.

METHODS AND ANALYSIS

Eligible studies published since 1 January 2007 will be searched for in several electronic databases (CINAHL Plus with Full Text, the Cochrane Library, EMBASE, PsycINFO, PubMed and Web of Science), clinical trial registries and OpenGrey, with all screening steps conducted by two independent reviewers. Studies will be included if they focus on the implementation or effectiveness of e-mental health interventions designed for informal adult caregivers of adults with cancer, heart disease, stroke, diabetes, dementia or chronic obstructive pulmonary disease. Pragmatic randomised controlled trials quantitatively reporting on caregiver anxiety, depression, psychological distress or stress will be used for a qualitative comparative analysis to identify combinations of conditions that result in effective interventions. Qualitative and quantitative data on implementation of e-mental health interventions for caregivers will be integrated in a thematic synthesis to identify barriers and facilitators to implementation. These results will inform future development and implementation planning of e-mental health interventions for caregivers.

ETHICS AND DISSEMINATION

Ethical approval is not required for this study as no primary data will be collected. Results will be disseminated in the form of a scientific publication and presentations at academic conferences and plain language summaries for various stakeholders.

PROSPERO REGISTRATION NUMBER

CRD42020155727.

The importance of decision intent within descriptions of pragmatic trials

OBJECTIVE

It is now more than 50 years since the concepts of explanatory and pragmatic attitudes toward trials were first discussed by Schwartz and Lellouch in their influential 1967 paper. Since then, there has been increasing focus on design aspects that may be consistent with more pragmatic attitudes within clinical trials, and a number of tools developed to assist investigators prospectively think about their trial design. Researchers have subsequently expressed interest in using these tools retrospectively to characterize trials as pragmatic or explanatory.

RESULTS

We suggest that recent attempts to retrospectively dichotomize trials solely on the basis of quantitative scoring of trial design features are flawed. Instead, we argue that there is a need to consider both the intent and design when assessing the degree of pragmatism within a trial.

CONCLUSION

The practical implication of our suggestion for trial reporting is that investigators should explicitly state the intent of the trial through a clear articulation of the decision that they hope will be informed by the trial results. This should be coupled with a completed PRagmatic-Explanatory Continuum Indicator Summary 2 assessment (or similar) with an explanation of study design choices to appropriately assess whether the study design is consistent with the study intent. We believe this will assist reviewers and knowledge users in making assessments of trials.

Pragmatic Trials in Genomic Medicine: The Integrating Pharmacogenetics In Clinical Care (I-PICC) Study

Pragmatic clinical trials (PCTs) have an established presence in clinical research and yet have only recently garnered attention within the landscape of genomic medicine. Using the PRagmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2) as a framework, this paper illustrates the application of PCT principles to The Integrating Pharmacogenetics In Clinical Care (I-PICC) Study, a trial of pharmacogenetic testing prior to statin initiation for cardiovascular disease prevention in primary care. The trial achieved high engagement with providers (85% enrolled of those approached) and enrolled a representative sample of participants for which statin therapy would be recommended. The I-PICC Study has a high level of pragmatism, which should enhance the generalizability of its findings. The PRECIS-2 may be useful in the design and evaluation of PCTs of genomic medicine interventions, contributing to the generation of evidence that can bridge the gap between genomics innovation and clinical adoption.