Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Phenobarbital treatment of alcohol withdrawal in the emergency department: A systematic review and meta-analysis

OBJECTIVE

Despite frequent treatment of alcohol withdrawal syndrome (AWS) in the emergency department (ED), evidence for phenobarbital (PB) as an ED alternative therapy is mixed. We conducted a systematic review and meta-analysis comparing safety and efficacy of PB to benzodiazepines (BZDs) for treatment of AWS in the ED.

METHODS

We searched articles and references published in English in PubMed, Web of Science, and Embase from inception through May 2022. We included randomized trials and cohort studies comparing treatment with PB to BZD controls and excluded studies focused on non-AWS conditions. Review was conducted by two blinded investigators and a third author; eight of 59 (13.6%) abstracts met inclusion criteria for review and meta-analysis using a random-effects model. Treatment superiority was evaluated through utilization, pharmacologic, and clinical outcomes. Primary outcomes for meta-analysis were the proportion of patients (1) admitted to the intensive care unit (ICU), (2) admitted to the hospital, (3) readmitted to the ED after discharge, and (4) who experienced adverse events.

RESULTS

Eight studies (two randomized controlled trials, six retrospective cohorts) comprised data from 1507 patients in 2012 treatment encounters for AWS. All studies were included in meta-analysis for adverse events, seven for hospital admission, five for ICU admission, and three for readmission to the ED after discharge. Overall methodological quality was low-moderate, risk of bias moderate-high, and statistical heterogeneity moderate. Pooled relative risk of ICU admission for those treated with PB versus BZD was 0.92 (95% confidence interval [CI] 0.54-1.55). Risk for admission to the hospital was 0.98 (95% CI 0.89-1.07) and for any adverse event was 1.1 (95% CI 0.78-1.57); heterogeneity prevented meta-analysis for ED readmission.

CONCLUSIONS

The current literature base does not show that treatment with PB significantly reduces ICU admissions, hospital admissions, ED readmissions, or adverse events in ED patients with AWS compared with BZDs alone.

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

Are e-Health Interventions Effective in Reducing Diabetes-Related Distress and Depression in Patients with Type 2 Diabetes? A Systematic Review with Meta-Analysis

Background: e-Health refers to any health care service delivered through the internet or related technologies, to improve quality of life. Despite the increasing use of e-health interventions to manage type 2 diabetes (T2D), there is a lack of evidence about the effectiveness on diabetes distress and depression, which are common issues in those living with T2D. Purpose: To synthesize and determine the effects of e-health interventions on diabetes distress and depression among patients with T2D. Methods: We systematically searched PubMed, Scopus, Cochrane CENTRAL, and Web of Science for randomized controlled trials (RCTs), non-RCTs and observational cohort studies for the effects of e-health interventions on diabetes distress and depression in patients with T2D up to September 14, 2022. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses 2020 recommendations were followed. The risk of bias was assessed according to the Risk-of-Bias 2 tool (RCTs), the Risk Of Bias In Non-randomised Studies-of Interventions (ROBINS-I) (non-RCTs) and the National Institute of Health tool (observational). The standardized mean difference (SMD) and its related 95% confidence intervals (CIs) were estimated with the DerSimonian-Laird method through random-effect models. A pooled raw mean difference (MD) meta-analysis was conducted for RCTs comparing the effects of e-health versus control on diabetes distress screening to display the clinical impact. Results: A total of 41 studies (24 RCTs, 14 non-RCTs, and 3 observational) involving 8,667 individuals were included. The pooled SMD for the effect of e-health versus the control group on diabetes distress was -0.14 (95% CI = -0.24 to -0.04; I2 = 23.9%; n = 10 studies), being -0.06 (95% CI = -0.15 to 0.02; I2 = 7.8%; n = 16 studies) for depression. The pooled raw MD on diabetes distress screening showed a reduction of -0.54 points (95% CI = -0.81 to -0.27; I2 = 85.1%; n = 7 studies). Conclusion: e-Health interventions are effective in diminishing diabetes distress among adults with T2D, inducing clinically meaningful reductions.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

BACKGROUND

Researchers and decision-makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up-to-date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question.

OBJECTIVES

To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types.

SEARCH METHODS

We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews.

SELECTION CRITERIA

We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case-control and cross-sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation.

DATA COLLECTION AND ANALYSIS

Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach.

MAIN RESULTS

We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty-four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs - for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case-control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty-three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: - pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); - RCTs and observational studies with substantial or high heterogeneity; that is, I2 ≥ 50% (ratio of ratios 1.11, 95% CI 1.04 to 1.18); - no use (ratio of ratios 1.07, 95% CI 1.03 to 1.11) or unclear use (ratio of ratios 1.13, 95% CI 1.03 to 1.25) of propensity score adjustment in observational studies; and - observational studies without further specification of the study design (ratio of ratios 1.06, 95% CI 0.96 to 1.18). We detected no clear difference in other subgroup analyses.

AUTHORS' CONCLUSIONS

We found no difference or a very small difference between effect estimates from RCTs and observational studies. These findings are largely consistent with findings from recently published research. Factors other than study design need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies, such as differences in the population, intervention, comparator, and outcomes investigated in the respective studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding is needed of how these factors might yield estimates reflective of true effectiveness.

Reporting, handling, and interpretation of time-varying drug treatments in observational studies using routinely collected healthcare data

BACKGROUND

Time-varying drug treatments are common in studies using routinely collected health data (RCD) for assessing treatment effects. This study aimed to examine how these studies reported, handled, and interpreted time-varying drug treatments.

METHODS

A systematic search was conducted on PubMed from 2018 to 2020. Eligible studies were those used RCD to explore drug treatment effects. We summarized the reporting characteristics and methods employed for handling time-varying treatments. Logistic regressions were performed to investigate the association between study characteristics and the reporting of time-varying treatments.

RESULTS

Two hundred and fifty-six studies were included, and 225 (87.9%) studies involved time-varying treatments. Of these, 24 (10.7%) reported the proportion of time-varying treatments and 105 (46.7%) reported methods used to handle time-varying treatments. Multivariable logistic regression showed that medical studies, prespecified protocol, and involvement of methodologists were associated with a higher likelihood of reporting the methods applied to handle time-varying treatments. Among the 105 studies that reported methods, as-treated analyses were the most commonly used analysis sets, which were employed in 73.9%, 75.3% and 88.2% of studies that reported approaches for treatment discontinuation, treatment switching and treatment add-on. Among the 225 studies involved time-varying treatments, 27 (12.0%) acknowledged the potential bias introduced by treatment change, of which 14 (51.9%) suggested that potential biases may impact acceptance or rejection of the null hypothesis.

CONCLUSIONS

Among observational studies using RCD, the underreporting about the presence and methods for handling time-varying treatments was largely common. The potential biases due to time-varying treatments have frequently been disregarded. Collaborative endeavors are strongly needed to enhance the prevailing practices.

Evaluating the impact of including non-randomised studies of interventions in meta-analysis of randomised controlled trials: a protocol for a meta-epidemiological study

INTRODUCTION

Although interest in including non-randomised studies of interventions (NRSIs) in meta-analysis of randomised controlled trials (RCTs) is growing, estimates of effectiveness obtained from NRSIs are vulnerable to greater bias than RCTs. The objectives of this study are to: (1) explore how NRSIs can be integrated into a meta-analysis of RCTs; (2) assess concordance of the evidence from non-randomised and randomised trials and explore factors associated with agreement; and (3) investigate the impact on estimates of pooled bodies of evidence when NRSIs are included.

METHODS AND ANALYSIS

We will conduct a systematic survey of 210 systematic reviews that include both RCTs and NRSIs, published from 2017 to 2022. We will randomly select reviews, stratified in a 1:1 ratio by Core vs non-Core clinical journals, as defined by the National Library of Medicine. Teams of paired reviewers will independently determine eligibility and abstract data using standardised, pilot-tested forms. The concordance of the evidence will be assessed by exploring agreement in the relative effect reported by NRSIs and RCT addressing the same clinical question, defined as similarity of the population, intervention/exposure, control and outcomes. We will conduct univariable and multivariable logistic regression analyses to examine the association of prespecified study characteristics with agreement in the estimates between NRSIs and RCTs. We will calculate the ratio of the relative effect estimate from NRSIs over that from RCTs, along with the corresponding 95% CI. We will use a bias-corrected meta-analysis model to investigate the influence on pooled estimates when NRSIs are included in the evidence synthesis.

ETHICS AND DISSEMINATION

Ethics approval is not required. The findings of this study will be disseminated through peer-reviewed publications, conference presentations and condensed summaries for clinicians, health policymakers and guideline developers regarding the design, conduct, analysis, and interpretation of meta-analysis that integrate RCTs and NRSIs.

Editorial Commentary: Autologous and Synthetic Bone Fillers for Medial Open-Wedge High Tibial Osteotomy Have No Effect on Clinical Outcomes But Autologous Graft Promotes Complete Bony Union

In contrast to closed-wedge high tibial osteotomy, medial based open-wedge high tibial osteotomy produces gaps of various sizes. Synthetic bone void fillers are an attractive option to close these gaps, potentially increase bone union, decrease time to union, and improve clinical outcomes. Autologous bone grafts are the accepted standard and result in reliable and reproducible outcomes. However, harvesting of autologous bone requires an additional procedure and is associated with potential complications. The use of synthetic bone void fillers could theoretically avoid these issues and reduce operating times. The current evidence suggests that autologous bone grafting has higher union rates but is not associated with better clinical and functional outcomes. Unfortunately, the certainty of evidence to support the use of bone void fillers is low, and the question of whether bone grafting of the gap should be performed in medial based open-wedge high tibial osteotomies cannot be answered with confidence.

Inclusion of nonrandomized studies of interventions in systematic reviews of interventions: updated guidance from the Agency for Health Care Research and Quality Effective Health Care program

OBJECTIVES

We developed guidance to inform decisions regarding the inclusion of nonrandomized studies of interventions (NRSIs) in systematic reviews (SRs) of the effects of interventions.

STUDY DESIGN AND SETTING

The guidance workgroup comprised SR experts and used an informal consensus generation method.

RESULTS

Instead of recommending NRSI inclusion only if randomized controlled trials (RCTs) are insufficient to address the SR key question, different topics may require different decisions regarding NRSI inclusion. We identified important considerations to inform such decisions from topic refinement through protocol development. During topic scoping and refinement, considerations were related to the clinical decisional dilemma, adequacy of RCTs to address the key questions, risk of bias in NRSIs, and the extent to which NRSIs are likely to complement RCTs. When NRSIs are included, during SR team formation, familiarity with topic-specific data sources and advanced analytic methods for NRSIs should be considered. During protocol development, the decision regarding NRSI inclusion or exclusion should be justified, and potential implications explained. When NRSIs are included, the protocol should describe the processes for synthesizing evidence from RCTs and NRSIs and determining the overall strength of evidence.

CONCLUSION

We identified specific considerations for decisions regarding NRSI inclusion in SRs and highlight the importance of flexibility and transparency.

An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research

BACKGROUND

Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review.

METHODS

BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs.

RESULTS

Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I² was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%).

CONCLUSIONS

Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.

Mediterranean Diet and Cardiovascular Prevention: Why Analytical Observational Designs Do Support Causality and Not Only Associations

Causal reductions in cardiovascular disease (CVD) with the Mediterranean diet (MedDiet) are supported by randomized trials, but the ability of nonrandomized studies to provide causal inferences in nutritional epidemiology is questioned. The "Seguimiento Universidad de Navarra" (SUN) project, conducted during 1999-2019 with 18,419 participants, was used to try to refute non-causal explanations for the inverse association found between adherence to the MedDiet and reduced CVD risk. A framework of different analytical strategies is proposed: alternative definitions of the exposure, exploration of residual confounding, resampling methods, depiction of absolute risks across the follow-up period, trial emulation, and negative controls. Additionally, we calculated the rate advancement period (RAP). We found that one standard deviation increase in the most frequently used MedDiet score was associated with a 29% relative reduction in CVD risk (95% Confidence Interval [CI] 14-41%), which is almost identical to that found in 2 randomized trials. The RAP of CVD would be postponed by an average of 7.9 years (95% CI: 1.6 to 14.2 years) by switching from low (MDS = 0 to2) to high (MDS = 7 to 9) adherence to the MedDiet in the fully adjusted model. Sensitivity analyses, graphical representations of absolute risks, trial emulation, and negative controls also supported causality. In conclusion, a framework of analytical approaches supported the causal effect of the MedDiet on CVD prevention using observational data. Similar methodology could be applied for causal inferences regarding other hypotheses.