Evaluating agreement between evidence from randomised controlled trials and cohort studies in nutrition: a meta-research replication study

This meta-research study aims to evaluate the agreement of effect estimates between bodies of evidence (BoE) from RCTs and cohort studies included in the same nutrition evidence synthesis, to identify factors associated with disagreement, and to replicate the findings of a previous study. We searched Medline, Epistemonikos and the Cochrane Database of Systematic Reviews for nutrition systematic reviews that included both RCTs and cohort studies for the same patient-relevant outcome or intermediate-disease marker. We rated similarity of PI/ECO (population, intervention/exposure, comparison, outcome) between BoE from RCTs and cohort studies. Agreement of effect estimates across BoE was analysed by pooling ratio of risk ratios (RRR) for binary outcomes and difference of standardised mean differences (DSMD) for continuous outcomes. We performed subgroup and sensitivity analyses to explore determinants associated with disagreements. We included 82 BoE-pairs from 51 systematic reviews. For binary outcomes, the RRR was 1.04 (95% confidence interval (CI) 0.99 to 1.10, I2 = 59%, τ2 = 0.02, prediction interval (PI) 0.77 to 1.41). For continuous outcomes, the pooled DSMD was  - 0.09 (95% CI  - 0.26 to 0.09, PI  - 0.55 to 0.38). Subgroup analyses yielded that differences in type of intake/exposure were drivers towards disagreement. We replicated the findings of a previous study, where on average RCTs and cohort studies had similar effect estimates. Disagreement and wide prediction intervals were mainly driven by PI/ECO-dissimilarities. More research is needed to explore other potentially influencing factors (e.g. risk of bias) on the disagreement between effect estimates of both BoE.Trial registration: CRD42021278908.

An empirical evaluation of the impact scenario of pooling bodies of evidence from randomized controlled trials and cohort studies in medical research

BACKGROUND

Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess treatment effects of medical interventions. We aimed to hypothetically pool bodies of evidence (BoE) from RCTs with matched BoE from cohort studies included in the same systematic review.

METHODS

BoE derived from systematic reviews of RCTs and cohort studies published in the 13 medical journals with the highest impact factor were considered. We re-analyzed effect estimates of the included systematic reviews by pooling BoE from RCTs with BoE from cohort studies using random and common effects models. We evaluated statistical heterogeneity, 95% prediction intervals, weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from cohort studies modified the conclusion from BoE of RCTs.

RESULTS

Overall, 118 BoE-pairs based on 653 RCTs and 804 cohort studies were pooled. By pooling BoE from RCTs and cohort studies with a random effects model, for 61 (51.7%) out of 118 BoE-pairs, the 95% confidence interval (CI) excludes no effect. By pooling BoE from RCTs and cohort studies, the median I² was 48%, and the median contributed percentage weight of RCTs to the pooled estimates was 40%. The direction of effect between BoE from RCTs and pooled effect estimates was mainly concordant (79.7%). The integration of BoE from cohort studies modified the conclusion (by examining the 95% CI) from BoE of RCTs in 32 (27%) of the 118 BoE-pairs, but the direction of effect was mainly concordant (88%).

CONCLUSIONS

Our findings provide insights for the potential impact of pooling both BoE in systematic reviews. In medical research, it is often important to rely on both evidence of RCTs and cohort studies to get a whole picture of an investigated intervention-disease association. A decision for or against pooling different study designs should also always take into account, for example, PI/ECO similarity, risk of bias, coherence of effect estimates, and also the trustworthiness of the evidence. Overall, there is a need for more research on the influence of those issues on potential pooling.

Evaluating agreement between bodies of evidence from randomized controlled trials and cohort studies in medical research: a meta-epidemiological study

BACKGROUND

Randomized controlled trials (RCTs) and cohort studies are the most common study design types used to assess the treatment effects of medical interventions. To evaluate the agreement of effect estimates between bodies of evidence (BoE) from randomized controlled trials (RCTs) and cohort studies and to identify factors associated with disagreement.

METHODS

Systematic reviews were published in the 13 medical journals with the highest impact factor identified through a MEDLINE search. BoE-pairs from RCTs and cohort studies with the same medical research question were included. We rated the similarity of PI/ECO (Population, Intervention/Exposure, Comparison, Outcome) between BoE from RCTs and cohort studies. The agreement of effect estimates across BoE was analyzed by pooling ratio of ratios (RoR) for binary outcomes and difference of mean differences for continuous outcomes. We performed subgroup analyses to explore factors associated with disagreements.

RESULTS

One hundred twenty-nine BoE pairs from 64 systematic reviews were included. PI/ECO-similarity degree was moderate: two BoE pairs were rated as "more or less identical"; 90 were rated as "similar but not identical" and 37 as only "broadly similar". For binary outcomes, the pooled RoR was 1.04 (95% CI 0.97-1.11) with considerable statistical heterogeneity. For continuous outcomes, differences were small. In subgroup analyses, degree of PI/ECO-similarity, type of intervention, and type of outcome, the pooled RoR indicated that on average, differences between both BoE were small. Subgroup analysis by degree of PI/ECO-similarity revealed high statistical heterogeneity and wide prediction intervals across PI/ECO-dissimilar BoE pairs.

CONCLUSIONS

On average, the pooled effect estimates between RCTs and cohort studies did not differ. Statistical heterogeneity and wide prediction intervals were mainly driven by PI/ECO-dissimilarities (i.e., clinical heterogeneity) and cohort studies. The potential influence of risk of bias and certainty of the evidence on differences of effect estimates between RCTs and cohort studies needs to be explored in upcoming meta-epidemiological studies.

An Empirical Evaluation of the Impact Scenario of Pooling Bodies of Evidence from Randomized Controlled Trials and Cohort Studies in Nutrition Research

Only very few Cochrane nutrition reviews include cohort studies (CSs), but most evidence in nutrition research comes from CSs. We aimed to pool bodies of evidence (BoE) from randomized controlled trials (RCTs) derived from Cochrane reviews with matched BoE from CSs. The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs. BoE from RCTs were pooled together with BoE from CSs using random-effects and common-effect models. Heterogeneity, 95% prediction intervals, contributed weight of BoE from RCTs to the pooled estimate, and whether integration of BoE from CSs modified the conclusion from BoE of RCTs were evaluated. Overall, 80 diet-disease outcome pairs based on 773 RCTs and 720 CSs were pooled. By pooling BoE from RCTs and CSs with a random-effects model, for 45 (56%) out of 80 diet-disease associations the 95% CI excluded no effect and showed mainly a reduced risk/inverse association. By pooling BoE from RCTs and CSs, median I2 = 46% and the median contributed weight of RCTs to the pooled estimates was 34%. The direction of effect between BoE from RCTs and pooled effect estimates was rarely opposite (n = 17; 21%). The integration of BoE from CSs modified the result (by examining the 95% CI) from BoE of RCTs in 35 (44%) of the 80 diet-disease associations. Our pooling scenario showed that the integration of BoE from CSs modified the conclusion from BoE of RCTs in nearly 50% of the associations, although the direction of effect was mainly concordant between BoE of RCTs and pooled estimates. Our findings provide insights for the potential impact of pooling both BoE in Cochrane nutrition reviews. CSs should be considered for inclusion in future Cochrane nutrition reviews, and we recommend analyzing RCTs and CSs in separate meta-analyses, or, if combined together, with a subgroup analysis.

Evaluating agreement between bodies of evidence from randomised controlled trials and cohort studies in nutrition research: meta-epidemiological study

OBJECTIVE

To evaluate the agreement between diet-disease effect estimates of bodies of evidence from randomised controlled trials and those from cohort studies in nutrition research, and to investigate potential factors for disagreement.

DESIGN

Meta-epidemiological study.

DATA SOURCES

Cochrane Database of Systematic Reviews, and Medline.

REVIEW METHODS

Population, intervention or exposure, comparator, outcome (PI/ECO) elements from a body of evidence from cohort studies (BoE(CS)) were matched with corresponding elements of a body of evidence from randomised controlled trials (BoE(RCT)). Pooled ratio of risk ratios or difference of mean differences across all diet-disease outcome pairs were calculated. Subgroup analyses were conducted to explore factors for disagreement. Heterogeneity was assessed through I2 and τ2. Prediction intervals were calculated to assess the range of possible values for the difference in the results between evidence from randomised controlled trials and evidence from cohort studies in future comparisons.

RESULTS

97 diet-disease outcome pairs (that is, matched BoE(RCT) and BoE(CS)) were identified overall. For binary outcomes, the pooled ratio of risk ratios comparing estimates from BoE(RCT) with BoE(CS) was 1.09 (95% confidence interval 1.04 to 1.14; I2=68%; τ2=0.021; 95% prediction interval 0.81 to 1.46). The prediction interval indicated that the difference could be much more substantial, in either direction. We further explored heterogeneity and found that PI/ECO dissimilarities, especially for the comparisons of dietary supplements in randomised controlled trials and nutrient status in cohort studies, explained most of the differences. When the type of intake or exposure between both types of evidence was identical, the estimates were similar. For continuous outcomes, small differences were observed between randomised controlled trials and cohort studies.

CONCLUSION

On average, the difference in pooled results between estimates from BoE(RCT) and BoE(CS) was small. But wide prediction intervals and some substantial statistical heterogeneity in cohort studies indicate that important differences or potential bias in individual comparisons or studies cannot be excluded. Observed differences were mainly driven by dissimilarities in population, intervention or exposure, comparator, and outcome. These findings could help researchers further understand the integration of such evidence into prospective nutrition evidence syntheses and improve evidence based dietary guidelines.

Evaluating Concordance of Bodies of Evidence from Randomized Controlled Trials, Dietary Intake, and Biomarkers of Intake in Cohort Studies: A Meta-Epidemiological Study

We aimed to identify and compare empirical data to determine the concordance of diet-disease effect estimates of bodies of evidence (BoE) from randomized controlled trials (RCTs), dietary intake, and biomarkers of dietary intake in cohort studies (CSs). The Cochrane Database of Systematic Reviews and MEDLINE were searched for systematic reviews (SRs) of RCTs and SRs of CSs that investigated both dietary intake and biomarkers of intake published between 1 January 2010 and 31 December 2019. For matched diet-disease associations, the concordance between results from the 3 different BoE was analyzed using 2 definitions: qualitative (e.g., 95% CI within a predefined range) and quantitative (test hypothesis on the z score). Moreover, the differences in the results coming from BoERCTs, BoECSs dietary intake, and BoECSs biomarkers were synthesized to get a pooled ratio of risk ratio (RRR) across all eligible diet-disease associations, so as to compare the 3 BoE. Overall, 49 diet-disease associations derived from 41 SRs were identified and included in the analysis. Twenty-four percent, 10%, and 39% of the diet-disease associations were qualitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively; 88%, 69%, and 90% of the diet-disease associations were quantitatively concordant comparing BoERCTs with BoECSs dietary intake, BoERCTs with BoECSs biomarkers, and comparing both BoE from CSs, respectively. The pooled RRRs comparing effects from BoERCTs with effects from BoECSs dietary intake were 1.09 (95% CI: 1.06, 1.13) and 1.18 (95% CI: 1.10, 1.25) compared with BoECSs biomarkers. Comparing both BoE from CSs, the difference in the results was also small (RRR: 0.92; 95% CI: 0.88, 0.96). Our findings suggest that BoE from RCTs and CSs are often quantitatively concordant. Prospective SRs in nutrition research should include, whenever possible, BoE from RCTs and CSs on dietary intake and biomarkers of intake to provide the whole picture for an investigated diet-disease association.

A Scoping Review of Current Guidelines on Dietary Fat and Fat Quality

INTRODUCTION

We conducted a scoping review of dietary guidelines with the intent of developing a position paper by the "IUNS Task force on Dietary Fat Quality" tasked to summarize the available evidence and provide the basis for dietary recommendations.

METHODS

We systematically searched several databases and Web sites for relevant documents published between 2015 and 2019.

RESULTS

Twenty documents were included. Quantitative range intake recommendations for daily total fat intake included boundaries from 20 to 35% of total energy intake (TEI), for monounsaturated fat (MUFA) 10-25%, for polyunsaturated fat (PUFA) 6-11%, for saturated-fat (SFA) ≤11-≤7%, for industrial trans-fat (TFA) ≤2-0%, and <300-<200 mg/d for dietary cholesterol. The methodological approaches to grade the strength of recommendations were heterogeneous, and varied highly between the included guidelines. Only the World Health Organization applied the GRADE approach and graded the following recommendation as "strong": to reduce SFA to below 10%, and TFA to below 1% and replace both with PUFA if SFA intake is greater than 10% of TEI.

CONCLUSION

Although the methodological approaches of the dietary guidelines were heterogeneous, most of them recommend total fat intakes of 30-≤35% of TEI, replacement of SFA with PUFA and MUFA, and avoidance of industrial TFA.

Total Dietary Fat Intake, Fat Quality, and Health Outcomes: A Scoping Review of Systematic Reviews of Prospective Studies

INTRODUCTION

We conducted a scoping review of systematic reviews (SRs) on dietary fat intake and health outcomes in human adults within the context of a position paper by the "International Union of Nutritional Sciences Task force on Dietary Fat Quality" tasked to summarize the available evidence and provide dietary recommendations.

METHODS

We systematically searched several databases for relevant SRs of randomized controlled trials (RCTs) and/or prospective cohort studies published between 2015 and 2019 assessing the association between dietary fat and health outcomes.

RESULTS

Fifty-nine SRs were included. The findings from SRs of prospective cohort studies, which frequently compare the highest versus lowest intake categories, found mainly no association of total fat, monounsaturated fatty acid (MUFA), polyunsaturated fatty acid (PUFA), and saturated fatty acid (SFA) with risk of chronic diseases. SRs of RCTs applying substitution analyses indicate that SFA replacement with PUFA and/or MUFA improves blood lipids and glycemic control, with the effect of PUFA being more pronounced. A higher intake of total trans-fatty acid (TFA), but not ruminant TFA, was probably associated with an increased risk of mortality and cardiovascular disease based on existing SRs.

CONCLUSION

Overall, the available published evidence deems it reasonable to recommend replacement of SFA with MUFA and PUFA and avoidance of consumption of industrial TFA.