Evaluating Concordance of Bodies of Evidence from Randomized Controlled Trials, Dietary Intake, and Biomarkers of Intake in Cohort Studies: A Meta-Epidemiological Study

Rapid review: A review of methods and recommendations based on current evidence

Rapid review (RR) could accelerate the traditional systematic review (SR) process by simplifying or omitting steps using various shortcuts. With the increasing popularity of RR, numerous shortcuts had emerged, but there was no consensus on how to choose the most appropriate ones. This study conducted a literature search in PubMed from inception to December 21, 2023, using terms such as "rapid review" "rapid assessment" "rapid systematic review" and "rapid evaluation". We also scanned the reference lists and performed citation tracking of included impact studies to obtain more included studies. We conducted a narrative synthesis of all RR approaches, shortcuts and studies assessing their effectiveness at each stage of RRs. Based on the current evidence, we provided recommendations on utilizing certain shortcuts in RRs. Ultimately, we identified 185 studies focusing on summarizing RR approaches and shortcuts, or evaluating their impact. There was relatively sufficient evidence to support the use of the following shortcuts in RRs: limiting studies to those published in English-language; conducting abbreviated database searches (e.g., only searching PubMed/MEDLINE, Embase, and CENTRAL); omitting retrieval of grey literature; restricting the search timeframe to the recent 20 years for medical intervention and the recent 15 years for reviewing diagnostic test accuracy; conducting a single screening by an experienced screener. To some extent, the above shortcuts were also applicable to SRs. This study provided a reference for future RR researchers in selecting shortcuts, and it also presented a potential research topic for methodologists.

Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

Synthesis methods used to combine observational studies and randomised trials in published meta-analyses

BACKGROUND

This study examined the synthesis methods used in meta-analyses pooling data from observational studies (OSs) and randomised controlled trials (RCTs) from various medical disciplines.

METHODS

We searched Medline via PubMed to identify reports of systematic reviews of interventions, including and pooling data from RCTs and OSs published in 110 high-impact factor general and specialised journals between 2015 and 2019. Screening and data extraction were performed in duplicate. To describe the synthesis methods used in the meta-analyses, we considered the first meta-analysis presented in each article.

RESULTS

Overall, 132 reports were identified with a median number of included studies of 14 [9-26]. The median number of OSs was 6.5 [3-12] and that of RCTs was 3 [1-6]. The effect estimates recorded from OSs (i.e., adjusted or unadjusted) were not specified in 82% (n = 108) of the meta-analyses. An inverse-variance common-effect model was used in 2% (n = 3) of the meta-analyses, a random-effects model was used in 55% (n = 73), and both models were used in 40% (n = 53). A Poisson regression model was used in 1 meta-analysis, and 2 meta-analyses did not report the model they used. The mean total weight of OSs in the studied meta-analyses was 57.3% (standard deviation, ± 30.3%). Only 44 (33%) meta-analyses reported results stratified by study design. Of them, the results between OSs and RCTs had a consistent direction of effect in 70% (n = 31). Study design was explored as a potential source of heterogeneity in 79% of the meta-analyses, and confounding factors were investigated in only 10% (n = 13). Publication bias was assessed in 70% (n = 92) of the meta-analyses. Tau-square was reported in 32 meta-analyses with a median of 0.07 [0-0.30].

CONCLUSION

The inclusion of OSs in a meta-analysis on interventions could provide useful information. However, considerations of several methodological and conceptual aspects of OSs, that are required to avoid misleading findings, were often absent or insufficiently reported in our sample.

Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study

BACKGROUND

Researchers and decision-makers often use evidence from randomised controlled trials (RCTs) to determine the efficacy or effectiveness of a treatment or intervention. Studies with observational designs are often used to measure the effectiveness of an intervention in 'real world' scenarios. Numerous study designs and their modifications (including both randomised and observational designs) are used for comparative effectiveness research in an attempt to give an unbiased estimate of whether one treatment is more effective or safer than another for a particular population. An up-to-date systematic analysis is needed to identify differences in effect estimates from RCTs and observational studies. This updated review summarises the results of methodological reviews that compared the effect estimates of observational studies with RCTs from evidence syntheses that addressed the same health research question.

OBJECTIVES

To assess and compare synthesised effect estimates by study type, contrasting RCTs with observational studies. To explore factors that might explain differences in synthesised effect estimates from RCTs versus observational studies (e.g. heterogeneity, type of observational study design, type of intervention, and use of propensity score adjustment). To identify gaps in the existing research comparing effect estimates across different study types.

SEARCH METHODS

We searched MEDLINE, the Cochrane Database of Systematic Reviews, Web of Science databases, and Epistemonikos to May 2022. We checked references, conducted citation searches, and contacted review authors to identify additional reviews.

SELECTION CRITERIA

We included systematic methodological reviews that compared quantitative effect estimates measuring the efficacy or effectiveness of interventions tested in RCTs versus in observational studies. The included reviews compared RCTs to observational studies (including retrospective and prospective cohort, case-control and cross-sectional designs). Reviews were not eligible if they compared RCTs with studies that had used some form of concurrent allocation.

DATA COLLECTION AND ANALYSIS

Using results from observational studies as the reference group, we examined the relative summary effect estimates (risk ratios (RRs), odds ratios (ORs), hazard ratios (HRs), mean differences (MDs), and standardised mean differences (SMDs)) to evaluate whether there was a relatively larger or smaller effect in the ratio of odds ratios (ROR) or ratio of risk ratios (RRR), ratio of hazard ratios (RHR), and difference in (standardised) mean differences (D(S)MD). If an included review did not provide an estimate comparing results from RCTs with observational studies, we generated one by pooling the estimates for observational studies and RCTs, respectively. Across all reviews, we synthesised these ratios to produce a pooled ratio of ratios comparing effect estimates from RCTs with those from observational studies. In overviews of reviews, we estimated the ROR or RRR for each overview using observational studies as the reference category. We appraised the risk of bias in the included reviews (using nine criteria in total). To receive an overall low risk of bias rating, an included review needed: explicit criteria for study selection, a complete sample of studies, and to have controlled for study methodological differences and study heterogeneity. We assessed reviews/overviews not meeting these four criteria as having an overall high risk of bias. We assessed the certainty of the evidence, consisting of multiple evidence syntheses, with the GRADE approach.

MAIN RESULTS

We included 39 systematic reviews and eight overviews of reviews, for a total of 47. Thirty-four of these contributed data to our primary analysis. Based on the available data, we found that the reviews/overviews included 2869 RCTs involving 3,882,115 participants, and 3924 observational studies with 19,499,970 participants. We rated 11 reviews/overviews as having an overall low risk of bias, and 36 as having an unclear or high risk of bias. Our main concerns with the included reviews/overviews were that some did not assess the quality of their included studies, and some failed to account appropriately for differences between study designs - for example, they conducted aggregate analyses of all observational studies rather than separate analyses of cohort and case-control studies. When pooling RORs and RRRs, the ratio of ratios indicated no difference or a very small difference between the effect estimates from RCTs versus from observational studies (ratio of ratios 1.08, 95% confidence interval (CI) 1.01 to 1.15). We rated the certainty of the evidence as low. Twenty-three of 34 reviews reported effect estimates of RCTs and observational studies that were on average in agreement. In a number of subgroup analyses, small differences in the effect estimates were detected: - pharmaceutical interventions only (ratio of ratios 1.12, 95% CI 1.04 to 1.21); - RCTs and observational studies with substantial or high heterogeneity; that is, I2 ≥ 50% (ratio of ratios 1.11, 95% CI 1.04 to 1.18); - no use (ratio of ratios 1.07, 95% CI 1.03 to 1.11) or unclear use (ratio of ratios 1.13, 95% CI 1.03 to 1.25) of propensity score adjustment in observational studies; and - observational studies without further specification of the study design (ratio of ratios 1.06, 95% CI 0.96 to 1.18). We detected no clear difference in other subgroup analyses.

AUTHORS' CONCLUSIONS

We found no difference or a very small difference between effect estimates from RCTs and observational studies. These findings are largely consistent with findings from recently published research. Factors other than study design need to be considered when exploring reasons for a lack of agreement between results of RCTs and observational studies, such as differences in the population, intervention, comparator, and outcomes investigated in the respective studies. Our results underscore that it is important for review authors to consider not only study design, but the level of heterogeneity in meta-analyses of RCTs or observational studies. A better understanding is needed of how these factors might yield estimates reflective of true effectiveness.

Novel Lines of Research on the Environmental and Human Health Impacts of Nut Consumption

Nuts have formed part of human diets throughout the ages. In recent decades, research has shown they are key foods in dietary patterns associated with lower chronic disease risk. The current state of climate change, however, has introduced an imperative to review the impact of dietary patterns on the environment with a shift to plant-based diets. Nuts emerge as a significant source of protein in plant-based diets and are a minimally processed and sustainable food. Research in this area is evolving to drive better production methods in varying climate conditions. Nevertheless, nut consumption remains an important contributor to human health. The mechanisms of action can be explained in terms of the nutrients they deliver. Studies of nut consumption have linked components such as monounsaturated fatty acids, plant omega-3 fatty acids, antioxidants, and plant sterols to improved lipoprotein profiles, lower blood pressure, and reduced cardiovascular disease risk. Preliminary research also indicates possible beneficial effects of nut consumption on reproductive health. In any case, the ultimate effects of foods on health are the results of multiple interactive factors, so where nuts fit within dietary patterns is a significant consideration for research translation. This has implications for research methodologies, including categorization within food groups and inclusion in Healthy Dietary Indices. The aim of this narrative review is to outline new focal points for investigation that examine the environmental and some novel human health impacts of nut consumption and discuss future directions for research.

The Relative Merits of Observational and Experimental Research: Four Key Principles for Optimising Observational Research Designs

The main barrier to the publication of observational research is a perceived inferiority to randomised designs with regard to the reliability of their conclusions. This commentary addresses this issue and makes a set of recommendations. It analyses the issue of research reliability in detail and fully describes the three sources of research unreliability (certainty, risk and uncertainty). Two of these (certainty and uncertainty) are not adequately addressed in most research texts. It establishes that randomised designs are vulnerable as observation studies to these two sources of unreliability, and are therefore not automatically superior to observational research in all research situations. Two key principles for reducing research unreliability are taken from R.A. Fisher’s early work on agricultural research. These principles and their application are described in detail. The principles are then developed into four key principles that observational researchers should follow when they are designing observational research exercises in nutrition. It notes that there is an optimal sample size for any particular research exercise that should not be exceeded. It concludes that best practice in observational research is to replicate this optimal sized observational exercise multiple times in order to establish reliability and credibility.

Red and processed meat intakes and cardiovascular disease and type 2 diabetes mellitus: An umbrella systematic review and assessment of causal relations using Bradford Hill's criteria

Observational research suggests higher red and processed meat intakes predict greater risks of developing or dying from cardiovascular diseases (CVD) and type 2 diabetes mellitus (T2DM), but this research limits causal inference. This systematic review of reviews utilizes both observational and experimental research findings to infer causality of these relations. Reviews from four databases were screened by two researchers. Reviews included unprocessed red meat (URM), processed meat (PM), or mixed URM + PM intake, and reported CVD or T2DM outcomes. Twenty-nine reviews were included, and causality was inferred using Bradford Hill's Criteria. Observational assessments of CVD outcomes and all meat types consistently reported weak associations while, T2DM outcomes and PM and Mixed URM + PM assessments consistently reported strong associations. Experimental assessments of Mixed URM + PM on CVD and T2DM risk factors were predominately not significant which lacked coherence with observational findings. For all meat types and outcomes, temporality and plausible mechanisms were established, but specificity and analogous relationships do not support causality. Evidence was insufficient for URM and T2DM. More experimental research is needed to strengthen these inferences. These results suggest that red and processed meat intakes are not likely causally related to CVD but there is potential for a causal relationship with T2DM.

Dietary fibre in hypertension and cardiovascular disease management: systematic review and meta-analyses

BACKGROUND

Higher dietary fibre intakes are associated with a reduced risk of developing cardiovascular disease (CVD), and increasing intake has been shown to reduce blood pressure and other cardiometabolic risk factors. The extent to which dietary fibre can further reduce risk for those with CVD and treated with cardioprotective drugs has not been clearly established. We have examined the evidence for dietary fibre as adjunct therapy in those with CVD or hypertension.

METHODS

Ovid MEDLINE, Embase, PubMed, and CENTRAL were searched to June 2021. Prospective observational studies reporting on fibre intakes and mortality in those with pre-existing CVD and controlled trials of increasing fibre intakes on cardiometabolic risk factors in those with CVD or hypertension were eligible. Outcomes were mortality (studies) and cardiometabolic risk factors (trials). Data synthesis was with random effects and dose response. Certainty of evidence was assessed using GRADE.

RESULTS

Three prospective studies including 7469 adults with CVD, and 12 trials of 878 adults with CVD or hypertension were identified. Moderate certainty evidence indicates reduced all-cause mortality (relative risk, RR0.75 (95% confidence interval, CI 0.58-0.97)) when comparing higher with lower fibre intakes. Low certainty evidence from trials of adults with cardiovascular disease indicates increasing fibre intakes reduced total (mean difference, MD - 0.42 mmol/L (95%CI - 0.78 to - 0.05) and low-density lipoprotein (LDL) cholesterol (MD - 0.47mmol/L (95%CI - 0.85 to - 0.10)). High certainty evidence from trials of adults with hypertension indicates increasing fibre intakes reduces systolic (MD 4.3 mmHg (95% CI 2.2 to 5.8)) and diastolic blood pressure (MD 3.1 mmHg (95% CI 1.7 to 4.4)). Moderate and low certainty evidence indicated improvements in fasting blood glucose (MD 0.48 mmol/L (- 0.91 to - 0.05)) and LDL cholesterol (MD 0.29 mmol/L (95% CI 0.17 to 0.40)). Benefits were observed irrespective of cardioprotective drug use.

CONCLUSIONS

These findings emphasise the likely benefits of promoting greater dietary fibre intakes for patients with CVD and hypertension. Further trials and cohort analyses in this area would increase confidence in these results.

An Empirical Evaluation of the Impact Scenario of Pooling Bodies of Evidence from Randomized Controlled Trials and Cohort Studies in Nutrition Research

Only very few Cochrane nutrition reviews include cohort studies (CSs), but most evidence in nutrition research comes from CSs. We aimed to pool bodies of evidence (BoE) from randomized controlled trials (RCTs) derived from Cochrane reviews with matched BoE from CSs. The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs. BoE from RCTs were pooled together with BoE from CSs using random-effects and common-effect models. Heterogeneity, 95% prediction intervals, contributed weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from CSs modified the conclusion from BoE of RCTs were evaluated. Overall, 80 diet-disease outcome pairs based on 773 RCTs and 720 CSs were pooled. By pooling BoE from RCTs and CSs with a random-effects model, for 45 (56%) out of 80 diet-disease associations the 95% CI excluded no effect and showed mainly a reduced risk/inverse association. By pooling BoE from RCTs and CSs, median I2 = 46% and the median contributed weight of RCTs to the pooled estimates was 34%. The direction of effect between BoE from RCTs and pooled effect estimates was rarely opposite (n = 17; 21%). The integration of BoE from CSs modified the result (by examining the 95% CI) from BoE of RCTs in 35 (44%) of the 80 diet-disease associations. Our pooling scenario showed that the integration of BoE from CSs modified the conclusion from BoE of RCTs in nearly 50% of the associations, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. Our findings provide insights for the potential impact of pooling both BoE in Cochrane nutrition reviews. CSs should be considered for inclusion in future Cochrane nutrition reviews, and we recommend analyzing RCTs and CSs in separate meta-analyses, or, if combined together, with a subgroup analysis.