Characteristics and quality of systematic reviews and meta-analyses of observational nutritional epidemiology: a cross-sectional study

Identification of application and interpretation errors that can occur in pairwise meta-analyses in systematic reviews of interventions: a systematic review

OBJECTIVES

To generate a bank of items describing application and interpretation errors that can arise in pairwise meta-analyses in systematic reviews of interventions.

STUDY DESIGN AND SETTING

MEDLINE, Embase, and Scopus were searched to identify studies describing types of errors in meta-analyses. Descriptions of errors and supporting quotes were extracted by multiple authors. Errors were reviewed at team meetings to determine if they should be excluded, reworded, or combined with other errors, and were categorized into broad categories of errors and subcategories within.

RESULTS

Fifty articles met our inclusion criteria, leading to the identification of 139 errors. We identified 25 errors covering data extraction/manipulation, 74 covering statistical analyses, and 40 covering interpretation. Many of the statistical analysis errors related to the meta-analysis model (eg, using a two-stage strategy to determine whether to select a fixed or random-effects model) and statistical heterogeneity (eg, not undertaking an assessment for statistical heterogeneity).

CONCLUSION

We generated a comprehensive bank of possible errors that can arise in the application and interpretation of meta-analyses in systematic reviews of interventions. This item bank of errors provides the foundation for developing a checklist to help peer reviewers detect statistical errors.

Grilling the data: application of specification curve analysis to red meat and all-cause mortality

OBJECTIVES

To present an application of specification curve analysis-a novel analytic method that involves defining and implementing all plausible and valid analytic approaches for addressing a research question-to nutritional epidemiology.

STUDY DESIGN AND SETTING

We reviewed all observational studies addressing the effect of red meat on all-cause mortality, sourced from a published systematic review, and documented variations in analytic methods (eg, choice of model, covariates, etc.). We enumerated all defensible combinations of analytic choices to produce a comprehensive list of all the ways in which the data may reasonably be analyzed. We applied specification curve analysis to data from National Health and Nutrition Examination Survey 2007 to 2014 to investigate the effect of unprocessed red meat on all-cause mortality. The specification curve analysis used a random sample of all reasonable analytic specifications we sourced from primary studies.

RESULTS

Among 15 publications reporting on 24 cohorts included in the systematic review on red meat and all-cause mortality, we identified 70 unique analytic methods, each including different analytic models, covariates, and operationalizations of red meat (eg, continuous vs quantiles). We applied specification curve analysis to National Health and Nutrition Examination Survey, including 10,661 participants. Our specification curve analysis included 1208 unique analytic specifications, of which 435 (36.0%) yielded a hazard ratio equal to or more than 1 for the effect of red meat on all-cause mortality and 773 (64.0%) less than 1. The specification curve analysis yielded a median hazard ratio of 0.94 (interquartile range: 0.83-1.05). Forty-eight specifications (3.97%) were statistically significant, 40 of which indicated unprocessed red meat to reduce all-cause mortality and eight of which indicated red meat to increase mortality.

CONCLUSION

We show that the application of specification curve analysis to nutritional epidemiology is feasible and presents an innovative solution to analytic flexibility.

Appraisal methods and outcomes of AMSTAR 2 assessments in overviews of systematic reviews of interventions in the cardiovascular field: A methodological study

This study aimed to assess the methods and outcomes of The Measurement Tool to Assess systematic Reviews (AMSTAR) 2 appraisals in overviews of reviews (overviews) of interventions in the cardiovascular field and identify factors that are associated with these outcomes. MEDLINE, Scopus, and the Cochrane Database of Systematic Reviews were searched until November 2022. Eligible were overviews of cardiovascular interventions, analyzing systematic reviews (SRs) of randomized controlled trials (RCTs). Extracted data included characteristics of overviews and SRs and AMSTAR 2 appraisal methods and outcomes. Data were synthesized using descriptive statistics and logistic regression to explore potential associations between the characteristics of SRs and extracted AMSTAR 2 overall ratings ("High-Moderate" vs. "Low-Critically low"). The original results on individual AMSTAR 2 items were entered into the official AMSTAR 2 online tool and the recalculated overall confidence ratings were compared to those provided in overviews. All 34 overviews identified were published between 2019 and 2022. Rating of overall confidence following the algorithm suggested by AMSTAR 2 developers was noted in 74% of overviews. The 679 unique included SRs were mainly of "Critically low" (53%) or "Low" (18.7%) confidence and underperformed in items 2 (Protocol, no = 65.2%) and 7 (List of excluded studies, no = 84%). The following characteristics of SRs were significantly associated with higher overall ratings: Cochrane origin, pharmacological interventions, including exclusively RCTs, citation of methodological and reporting guidelines, protocol, absence of funding and publication after AMSTAR 2 release. Generally, overviews' authors tended to deviate from the original rating scheme and ascribe higher ratings to SRs compared to the official AMSTAR 2 online tool. Most SRs included in overviews of cardiovascular interventions have critically low or low confidence in their results. Overviews' authors should be more transparent about the methods used to derive the overall confidence in SRs.

Reporting quality and risk of bias of systematic reviews of ultra-processed foods: a methodological study

A dramatic shift in the global food system is occurring with the rapid growth of ultra-processed foods (UPFs) consumption, which poses potentially serious health risks. Systematic review (SR) method has been used to summarise the association between UPF consumption and multiple health outcomes; however, a suboptimal-quality SR may mislead the decision-making in clinical practices and health policies. Therefore, a methodological review was conducted to identify the areas that can be improved regarding the risk of bias and reporting quality of relevant SRs. Systematic searches to collect SRs with meta-analyses of UPFs were performed using four databases from their inception to April 14, 2023. The risk of bias and reporting quality were evaluated using ROBIS and PRISMA 2020, respectively. The key characteristics of the included SRs were summarised descriptively. Excel 2019 and R 4.2.3 were used to analyse the data and draw graphs. Finally, 16 relevant SRs written in English and published between 2020 and 2023 in 12 academic journals were included. Only one SR was rated as low risk of bias, and the others were rated as higher risk of bias mainly because the risk of bias in the original studies was not explicitly addressed when synthesising the evidence. The reporting was required to be advanced significantly, involving amendments of registration and protocol, data and analytic code statement, and lists of excluded studies with justifications. The reviews' results could improve the quality, strengthen future relevant SRs' robustness, and further underpin the evidence base for supporting clinical decisions and health policies.

The effects of inulin-type fructans on cardiovascular disease risk factors: systematic review and meta-analysis of randomized controlled trials

BACKGROUND

Inulin-type fructans (ITF) are the leading prebiotics in the market. Available evidence provides conflicting results regarding the beneficial effects of ITF on cardiovascular disease risk factors.

OBJECTIVES

This study aimed to evaluate the effects of ITF supplementation on cardiovascular disease risk factors in adults.

METHODS

We searched MEDLINE, EMBASE, Emcare, AMED, CINAHL, and the Cochrane Library databases from inception through May 15, 2022. Eligible randomized controlled trials (RCTs) administered ITF or placebo (for example, control, foods, diets) to adults for ≥2 weeks and reported one or more of the following: low, very-low, or high-density lipoprotein cholesterol (LDL-C, VLDL-C, HDL-C); total cholesterol; apolipoprotein A1 or B; triglycerides; fasting blood glucose; body mass index; body weight; waist circumference; waist-to-hip ratio; systolic or diastolic blood pressure; or hemoglobin A1c. Two reviewers independently and in duplicate screened studies, extracted data, and assessed risk of bias. We pooled data using random-effects model, and assessed the certainty of evidence (CoE) using the Grading of Recommendations, Assessment, Development and Evaluation approach.

RESULTS

We identified 1767 studies and included 55 RCTs with 2518 participants in meta-analyses. The pooled estimate showed that ITF supplementation reduced LDL-C [mean difference (MD) -0.14 mmol/L, 95% confidence interval (95% CI: -0.24, -0.05), 38 RCTs, 1879 participants, very low CoE], triglycerides (MD -0.06 mmol/L, 95% CI: -0.12, -0.01, 40 RCTs, 1732 participants, low CoE), and body weight (MD -0.97 kg, 95% CI: -1.28, -0.66, 36 RCTs, 1672 participants, low CoE) but little to no significant effect on other cardiovascular disease risk factors. The effects were larger when study duration was ≥6 weeks and in pre-obese and obese participants.

CONCLUSION

ITF may reduce low-density lipoprotein, triglycerides, and body weight. However, due to low to very low CoE, further well-designed and executed trials are needed to confirm these effects.

PROSPERO REGISTRATION NUMBER

CRD42019136745.

The Role of Diet in the Prevention of Hypertension and Management of Blood Pressure: An Umbrella Review of Meta-Analyses of Interventional and Observational Studies

High blood pressure (BP) is a major pathological risk factor for the development of several cardiovascular diseases. Diet is a key modifier of BP, but the underlying relationships are not clearly demonstrated. This is an umbrella review of published meta-analyses to critically evaluate the wide range of dietary evidence from bioactive compounds to dietary patterns on BP and risk of hypertension. PubMed, Embase, Web of Science, and Cochrane Central Register of Controlled Trials were searched from inception until October 31, 2021, for relevant meta-analyses of randomized controlled trials or meta-analyses of observational studies. A total of 175 publications reporting 341 meta-analyses of randomized controlled trials (145 publications) and 70 meta-analyses of observational studies (30 publications) were included in the review. The methodological quality of the included publications was assessed using Assessment of Multiple Systematic Reviews 2 and the evidence quality of each selected meta-analysis was assessed using NutriGrade. This umbrella review supports recommended public health guidelines for prevention and control of hypertension. Dietary patterns including the Dietary Approaches to Stop Hypertension and the Mediterranean-type diets that further restrict sodium, and moderate alcohol intake are advised. To produce high-quality evidence and substantiate strong recommendations, future research should address areas where the low quality of evidence was observed (for example, intake of dietary fiber, fish, egg, meat, dairy products, fruit juice, and nuts) and emphasize focus on dietary factors not yet conclusively investigated.

Epidemiology and Reporting Characteristics of Systematic Reviews in Orthopedic Journals: A Meta-Epidemiological Study

Systematic reviews (SRs) with complete reporting or rigorous methods can lead to less biased recommendations and decisions. A comprehensive analysis of the epidemiological and reporting characteristics of SRs in orthopedics is lacking. We evaluated 360 SRs, including 165 and 195 published in orthopedic journals in 2012 and 2022. According to the established reporting guidelines, we examined these SRs for key epidemiological characteristics, including focus areas, type of meta-analysis (MA), and reporting characteristics. Most SRs (71%) were therapy-related, with a significant proportion originating from authors in the USA, UK, and China. Pairwise MA was performed on half of the SRs. The proportion of protocol registrations improved by 2022 but remained low (33%). Despite a formal declaration of adherence to the reporting guidelines (68%), they were often not used and reported enough. Only 10% of the studies used full search strategies, including trial registries. Publication bias assessments, subgroup analyses, and sensitivity analyses were not even planned. The risk of bias assessment improved in 2022; however, the certainty of the evidence remained largely unassessed (8%). The use and reporting of standard methods in orthopedic SRs have remained suboptimal. Thus, authors, peer reviewers, journal editors, and readers should criticize the results more.

Saturated fat, the estimated absolute risk and certainty of risk for mortality and major cancer and cardiometabolic outcomes: an overview of systematic reviews

OBJECTIVE

To assess the impact of reducing saturated fat or fatty foods, or replacing saturated fat with unsaturated fat, carbohydrate or protein, on the risk of mortality and major cancer and cardiometabolic outcomes in adults.

METHODS

We searched MEDLINE, EMBASE, CINAHL, and references of included studies for systematic reviews and meta-analyses (SRMAs) of randomized controlled trials (RCTs) and observational studies in adults published in the past 10 years. Eligible reviews investigated reducing saturated fat or fatty foods or replacing saturated fat with unsaturated fat, carbohydrate or protein, on the risk of cancer and cardiometabolic outcomes and assessed the certainty of evidence for each outcome using, for example, the GRADE (Grading of Recommendations, Assessment, Development, and Evaluations) approach. We assessed the quality of SRMAs using a modified version of AMSTAR-2. Results were summarized as absolute estimates of effect together with the certainty of effects using a narrative synthesis approach.

RESULTS

We included 17 SRMAs (13 reviews of observational studies with follow-up 1 to 34 years; 4 reviews of RCTs with follow-up 1 to 17 years). The quality of two-thirds of the SRMAs was critically low to moderate; the main limitations included deficient reporting of study selection, absolute effect estimates, sources of funding, and a priori subgroups to explore heterogeneity. Our included reviews reported > 100 estimates of effect across 11 critically important cancer and cardiometabolic outcomes. High quality SRMAs consistently and predominantly reported low to very low certainty evidence that reducing or replacing saturated fat was associated with a very small risk reduction in cancer and cardiometabolic endpoints. The risk reductions where approximately divided, some being statistically significant and some being not statistically significant. However, based on 2 moderate to high quality reviews, we found moderate certainty evidence for a small but important effect that was statistically significant for two outcomes (total mortality events [20 fewer events per 1000 followed] and combined cardiovascular events [16 fewer per 1000 followed]). Conversely, 4 moderate to high quality reviews showed very small effects on total mortality, with 3 of these reviews showing non-statistically significant mortality effects.

CONCLUSION

Systematic reviews investigating the impact of SFA on mortality and major cancer and cardiometabolic outcomes almost universally suggest very small absolute changes in risk, and the data is based primarily on low and very low certainty evidence.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO CRD42020172141.

Perspectives on the application of CONSORT guidelines to randomised controlled trials in nutrition

PURPOSE

Reporting guidelines facilitate quality and completeness in research reporting. The CONsolidated Standards Of Reporting Trials (CONSORT) statement is widely applied to dietary and nutrition trials but has no extension specific to nutrition. Evidence suggests poor reporting in nutrition research. The Federation of European Nutrition Societies led an initiative to make recommendations for a nutrition extension to the CONSORT statement towards a more robust reporting of the evidence base.

METHODS

An international working group was formed of nutrition researchers from 14 institutions in 12 different countries and on five continents. Using meetings over a period of one year, we interrogated the CONSORT statement specifically for its application to report nutrition trials.

RESULTS

We provide a total of 28 new nutrition-specific recommendations or emphasised recommendations for the reporting of the introduction (three), methods (twelve), results (five) and discussion (eight). We also added two additional recommendations that were not allocated under the standard CONSORT headings.

CONCLUSION

We identify a need to provide guidance in addition to CONSORT to improve the quality and consistency of the reporting and propose key considerations for further development of formal guidelines for the reporting of nutrition trials. Readers are encouraged to engage in this process, provide comments and conduct specific studies to inform further work on the development of reporting guidelines for nutrition trials.

A Cross-Sectional Study Based on Forty Systematic Reviews of Foods with Function Claims (FFC) in Japan: Quality Assessment Using AMSTAR 2

BACKGROUND

The Foods with Function Claims (FFC) was introduced in Japan in April 2015 to make more products available that are labeled with health functions. The products' functionality of function claims must be explained by scientific evidence presented in systematic reviews (SRs), but the quality of recent SRs is unclear. This study assessed the quality of SRs in the FFC registered on the Consumer Affairs Agency (CAA) website in Japan.

METHODS

We searched the database from 1 April to 31 October 2022. Confidence in the methodological quality of each SR was evaluated by the AMSTAR 2 checklist.

RESULTS

Forty SRs were randomly extracted on the basis of the eligibility criteria and recruitment procedures. Overall confidence was rated as "high" (N = 0, 0%), "moderate" (N = 0, 0%), "low" (N = 2, 5%), or "critically low" (N = 38, 95%). The mean AMSTAR 2 score was 51.1% (SD 12.1%; range 19-73%). Among the 40 SRs, the number of critical domain deficiencies was 4 in 7.5% of SRs, 3 in 52.5% of SRs, 2 in 35% of SRs, and 1 in 5% of SRs. Registering the review's protocol and comprehensive search strategies were particularly common deficiencies. Additionally, the risk of bias (RoB) was insufficiently considered.

CONCLUSION

Overall, the methodological quality of the SRs based on the FFC, introduced eight years earlier, was very poor. This was especially true in the interpretation and discussion of critical domains, which had many deficiencies in terms of protocol registration, a comprehensive literature search strategy, and accounting for the RoB.

Characteristics of the sources, evaluation, and grading of the certainty of evidence in systematic reviews in public health: A methodological study

Objectives

To systematically explore how the sources of evidence, types of primary studies, and tools used to assess the quality of the primary studies vary across systematic reviews (SRs) in public health.

Methods

We conducted a methodological survey of SRs in public health by searching the of literature in selected journals from electronic bibliographic databases. We selected a 10% random sample of the SRs that met the explicit inclusion criteria. Two researchers independently extracted data for analysis.

Results

We selected 301 SRs for analysis: 94 (31.2%) of these were pre-registered, and 211 (70.1%) declared to have followed published reporting standard. All SRs searched for evidence in electronic bibliographic databases, and more than half (n = 180, 60.0%) searched also the references of the included studies. The common types of primary studies included in the SRs were primarily cross-sectional studies (n = 132, 43.8%), cohort studies (n = 126, 41.9%), randomized controlled trials (RCTs, n = 89, 29.6%), quasi-experimental studies (n = 83, 27.6%), case-control studies (n = 58, 19.3%) qualitative studies (n = 38, 12.6%) and mixed-methods studies (n = 32, 10.6%). The most frequently used quality assessment tools were the Newcastle-Ottawa Scale (used for 50.0% of cohort studies and 55.6% of case-control studies), Cochrane Collaboration's Risk of Bias tool (50.7% of RCTs) and Critical Appraisal Skills Program (38.5% of qualitative studies). Only 20 (6.6%) of the SRs assessed the certainty of the body of evidence, of which 19 (95.0%) used the GRADE approach. More than 65% of the evidence in the SRs using GRADE was of low or very low certainty.

Conclusions

SRs should always assess the quality both at the individual study level and the body of evidence for outcomes, which will benefit patients, health care practitioners, and policymakers.

Saturated fat and human health: a protocol for a methodologically innovative systematic review and meta-analysis to inform public health nutrition guidelines

BACKGROUND

The health effects of dietary fats are a controversial issue on which experts and authoritative organizations have often disagreed. Care providers, guideline developers, policy-makers, and researchers use systematic reviews to advise patients and members of the public on optimal dietary habits, and to formulate public health recommendations and policies. Existing reviews, however, have serious limitations that impede optimal dietary fat recommendations, such as a lack of focus on outcomes important to people, substantial risk of bias (RoB) issues, ignoring absolute estimates of effects together with comprehensive assessments of the certainty of the estimates for all outcomes.

OBJECTIVE

We therefore propose a methodologically innovative systematic review using direct and indirect evidence on diet and food-based fats (i.e., reduction or replacement of saturated fat with monounsaturated or polyunsaturated fat, or carbohydrates or protein) and the risk of important health outcomes.

METHODS

We will collaborate with an experienced research librarian to search MEDLINE, EMBASE, CINAHL, and the Cochrane Database of Systematic Reviews (CDSR) for randomized clinical trials (RCTs) addressing saturated fat and our health outcomes of interest. In duplicate, we will screen, extract results from primary studies, assess their RoB, conduct de novo meta-analyses and/or network meta-analysis, assess the impact of missing outcome data on meta-analyses, present absolute effect estimates, and assess the certainty of evidence for each outcome using the GRADE contextualized approach. Our work will inform recommendations on saturated fat based on international standards for reporting systematic reviews and guidelines.

CONCLUSION

Our systematic review and meta-analysis will provide the most comprehensive and rigorous summary of the evidence addressing the relationship between saturated fat modification for people-important health outcomes. The evidence from this review will be used to inform public health nutrition guidelines.

TRIAL REGISTRATION

PROSPERO Registration: CRD42023387377 .

Nuts and seeds consumption and risk of cardiovascular disease, type 2 diabetes and their risk factors: a systematic review and meta-analysis

Objectives

We aimed to systematically review studies and evaluate the strength of the evidence on nuts/seeds consumption and cardiometabolic diseases and their risk factors among adults.

Methods

A protocol was pre-registered in PROSPERO (CRD42021270554). We searched MEDLINE, Embase, Cochrane Central Register of Controlled Trials and Scopus up to September 20, 2021 for prospective cohort studies and ≥12-week randomized controlled trials (RCTs). Main outcomes were cardiovascular disease (CVD), coronary heart disease (CHD), stroke and type 2 diabetes (T2D), secondary total-/low density lipoprotein (LDL)-cholesterol, blood pressure and glycaemic markers. Data extraction and risk of bias (RoB) assessments (using RoB 2.0 and RoB-NObS) were performed in duplicate. Effect sizes were pooled using random-effects meta-analyses and expressed as relative risk (RR) or weighted mean differences with 95% confidence intervals (CI); heterogeneity quantified as I ². One-stage dose-response analyses assessed the linear and non-linear associations with CVD, CHD, stroke and T2D. The strength of evidence was classified per the World Cancer Research Fund criteria.

Results

After screening 23,244 references, we included 42 papers from cohort studies (28 unique cohorts, 1,890,573 participants) and 18 RCTs (2,266 participants). In the cohorts, mainly populations with low consumption, high versus low total nuts/seeds consumption was inversely associated with total CVD (RR 0.81; 95% CI 0.75, 0.86; I ² = 67%), CVD mortality (0.77; 0.72, 0.82; I ² = 59.3%), CHD (0.82; 0.76, 0.89; I ² = 64%), CHD mortality (0.75; 0.65, 0.87; I ² = 66.9%) and non-fatal CHD (0.85; 0.75, 0.96; I ² = 62.2%). According to the non-linear dose-response analyses, consumption of 30 g/day of total nuts/seeds was associated with RRs of similar magnitude. For stroke and T2D the summary RR for high versus low intake was 0.91 (95% CI 0.85, 0.97; I ² = 24.8%) and 0.95 (0.75, 1.21; I ² = 82.2%). Intake of nuts (median ~50 g/day) lowered total (-0.15 mmol/L; -0.22, -0.08; I ² = 31.2%) and LDL-cholesterol (-0.13 mmol/L; -0.21, -0.05; I ² = 68.6%), but not blood pressure. Findings on fasting glucose, HbA1c and insulin resistance were conflicting. The results were robust to sensitivity and subgroup analyses. We rated the associations between nuts/seeds and both CVD and CHD as probable. There was limited but suggestive evidence for no association with stroke. No conclusion could be made for T2D.

Conclusion

There is a probable relationship between consumption of nuts/seeds and lower risk of CVD, mostly driven by CHD, possibly in part through effects on blood lipids. More research on stroke and T2D may affect the conclusions. The evidence of specific nuts should be further investigated.

Reporting and interpretation of effects in non-randomized nutritional and environmental epidemiology: a methods study

PURPOSE

The presentation of absolute effects, in addition to relative effects, is critical to the optimal interpretation of effect estimates. Failure to present and interpret absolute effects may obscure the magnitude of the effect of an intervention or exposure and mislead evidence users.

OBJECTIVE

In this study, we estimate the proportion of systematic reviews and meta-analyses (SRMAs) addressing the health effects of nutritional and environmental exposures that report absolute effects.

METHODS

We searched MEDLINE and EMBASE from 2019 through 2021 for SRMAs addressing the health effects of nutritional and environmental exposures and patient-important health outcomes. We included a sample of 200 SRMAs. Pairs of reviewers, working independently and in duplicate, reviewed search records for eligibility and collected data from SRMAs.

RESULTS

More than two-thirds (153/200; 76.5%) of eligible systematic reviews reported on one or more dichotomous outcomes that could be translated to absolute effects. Only eight (8/153, 5.2%)5.2%), however, reported absolute effects. A similar proportion of reviews published in high-impact journals and in other journals reported absolute effects (4/131; 3.1% vs. 4/69; 5.9%). Among reviews that reported absolute effects, six reviews (6/8; 75%) reported absolute risk differences as fractions (e.g., 2 fewer cases per 1000 people) and two reviews (2/8; 25%) presented the number of cases prevented by modifying the exposure (e.g., 2000 cases prevented in United States annually).

CONCLUSIONS

Reviews addressing the effects of nutritional and environmental exposures on health outcomes rarely report absolute effects, which precludes effective interpretation of magnitudes of effects and their importance. We present guidance for review authors, editors, peer reviewers, and evidence users to calculate and interpret absolute effects.

Estimating the impact of nutrition and physical activity policies with quasi-experimental methods and simulation modelling: an integrative review of methods, challenges and synergies

BACKGROUND

The promotion of healthy lifestyles has high priority on the global public health agenda. Evidence on the real-world (cost-)effectiveness of policies addressing nutrition and physical activity is needed. To estimate short-term policy impacts, quasi-experimental methods using observational data are useful, while simulation models can estimate long-term impacts. We review the methods, challenges and potential synergies of both approaches for the evaluation of nutrition and physical activity policies.

METHODS

We performed an integrative review applying purposive literature sampling techniques to synthesize original articles, systematic reviews and lessons learned from public international workshops conducted within the European Union Policy Evaluation Network.

RESULTS

We highlight data requirements for policy evaluations, discuss the distinct assumptions of instrumental variable, difference-in-difference, and regression discontinuity designs and describe the necessary robustness and falsification analyses to test them. Further, we summarize the specific assumptions of comparative risk assessment and Markov state-transition simulation models, including their extension to microsimulation. We describe the advantages and limitations of these modelling approaches and discuss future directions, such as the adequate consideration of heterogeneous policy responses. Finally, we highlight how quasi-experimental and simulation modelling methods can be integrated into an evidence cycle for policy evaluation.

CONCLUSIONS

Assumptions of quasi-experimental and simulation modelling methods in policy evaluations should be credible, rigorously tested and transparently communicated. Both approaches can be applied synergistically within a coherent framework to compare policy implementation scenarios and improve the estimation of nutrition and physical activity policy impacts, including their distribution across population sub-groups.

Mediterranean dietary pattern and the risk of type 2 diabetes: a systematic review and dose-response meta-analysis of prospective cohort studies

PURPOSE

Previous meta-analyses assessed the association of adherence to the Mediterranean dietary pattern (MedDiet) with the risk of type 2 diabetes (T2D). Since then, new large-scale cohort studies have been published. In addition, dose-response relation was not previously investigated and the certainty of evidence was not assessed. We aimed to explore the dose-response relationship between adherence to the MedDiet and the risk of T2D.

METHODS

We did a systematic search using PubMed, Scopus, and ISI Web of Science upto April 2021 for prospective cohort studies of the relationship between adherence to the MedDiet and the risk of T2D in the general population. The summary relative risks (RR) and 95%CI were estimated by applying a random-effects model.

RESULTS

Fourteen prospective cohort studies (410,303 participants and 41,466 cases) were included. There was an inverse association for the highest versus lowest category of adherence to the MedDiet (RR: 0.79, 95%CI 0.72, 0.88; I² = 82%, n = 14; Risk difference:  - 21 per 1000 person, 95%CI  - 28,  - 12; GRADE = moderate certainty), and for a 2-point increment in the MedDiet adherence score (RR: 0.86, 95%CI 0.82, 0.91; n = 13). The RR remained significant after controlling for important confounders and in almost all subgroups, especially subgroups defined by geographical region. We observed an inverse linear association between MedDiet adherence score and T2D incidence.

CONCLUSION

Adherence to the MedDiet was inversely related to T2D risk in a dose-response manner. Adherence to a Mediterranean-style diet may be a good advice for the primary prevention of T2D.

REGISTRY AND REGISTRY NUMBER

PROSPERO (CRD42021246589).

Predictors of Higher Quality of Systematic Reviews Addressing Nutrition and Cancer Prevention

Systematic reviews/meta-analyses (SR/MAs) are considered a reliable source of information in healthcare. We aimed to explore the association of several characteristics of SR/MAs addressing nutrition in cancer prevention and their quality/risk of bias (using assessments from AMSTAR-2 and ROBIS tools). The analysis included 101 SR/MAs identified in a systematic survey. Associations of each specified characteristic (e.g., information about the protocol, publication year, reported use of GRADE, or other methods for assessing overall certainty of evidence) with the number of AMSTAR-2 not met (‘No’ responses) and the number of ROBIS items met (‘Probably Yes’ or “Yes’ responses) were examined. Poisson regression was used to identify predictors of the number of ‘No’ answers (indicating lower quality) for all AMSTAR-2 items and the number of ‘Yes’ or ‘Probably Yes’ answers (indicating higher quality/lower concern for bias) for all ROBIS items. Logistic regression was used to identify variables associated with at least one domain assessed as ‘low concern for bias’ in the ROBIS tool. In multivariable analysis, SR/MAs not reporting use of any quality/risk of bias assessment instrument for primary studies were associated with a higher number of ‘No’ answers for all AMSTAR-2 items (incidence rate ratio (IRR) 1.26, 95% confidence interval (CI) 1.09–1.45), and a lower number of ‘Yes’ or ‘Probably Yes’ answers for all ROBIS items (IRR 0.76, 95% CI 0.66–0.87). Providing information about the protocol and search for unpublished studies was associated with a lower number of ‘No’ answers (IRR 0.73, 95% CI 0.56–0.97 and IRR 0.75, 95% CI 0.59–0.95, respectively) and a higher number of ‘Yes’ or ‘Probably Yes’ answers (IRR 1.43, 95% CI 1.17–1.74 and IRR 1.28, 95% CI 1.07–1.52, respectively). Not using at least one quality/risk of bias assessment tool for primary studies within an SR/MA was associated with lower odds that a study would be assessed as ‘low concern for bias’ in at least one ROBIS domain (odds ratio 0.061, 95% CI 0.007–0.527). Adherence to methodological standards in the development of SR/MAs was associated with a higher overall quality of SR/MAs addressing nutrition for cancer prevention.

Are systematic reviews addressing nutrition for cancer prevention trustworthy? A systematic survey of quality and risk of bias

Context

The last 30 years have yielded a vast number of systematic reviews and/or meta-analyses addressing the link between nutrition and cancer risk.

Objective

The aim of this survey was to assess overall quality and potential for risk of bias in systematic reviews and meta-analyses (SRMAs) that examined the role of nutrition in cancer prevention.

Data Sources

MEDLINE, Embase, and the Cochrane Library databases were searched (last search performed November 2018).

Study Selection

Studies identified as SRMAs that investigated a nutritional or dietary intervention or exposure for cancer prevention in the general population or in people at risk of cancer and in which primary studies had a comparison group were eligible for inclusion. Screening, data extraction, and quality assessment were conducted independently by 2 reviewers.

Data Extraction

Altogether, 101 studies were randomly selected for analysis. The methodological quality and risk of bias were evaluated using the AMSTAR-2 and ROBIS tools, respectively.

Results

Most SRMAs included observational studies. Less than 10% of SRMAs reported a study protocol, and only 51% of SRMAs assessed the risk of bias in primary studies. Most studies conducted subgroup analyses, but only a few reported tests of interaction or specified subgroups of interest a priori. Overall, according to AMSTAR-2, only 1% of SRMAs were of high quality, while 97% were of critically low quality. Only 3% had a low risk of bias, according to ROBIS.

Conclusions

This systematic survey revealed substantial limitations with respect to quality and risk of bias of SRMAs. SRMAs examining nutrition and cancer prevention cannot be considered trustworthy, and results should be interpreted with caution. Peer reviewers as well as users of SRMAs should be advised to use the AMSTAR-2 and/or ROBIS instruments to help to determine the overall quality and risk of bias of SRMAs.

Systematic Review Registration

PROSPERO registration number CRD42019121116.

Assessments of risk of bias in systematic reviews of observational nutritional epidemiologic studies are often not appropriate or comprehensive: a methodological study

BACKGROUND

An essential component of systematic reviews is the assessment of risk of bias. To date, there has been no investigation of how reviews of non-randomised studies of nutritional exposures (called 'nutritional epidemiologic studies') assess risk of bias.

OBJECTIVE

To describe methods for the assessment of risk of bias in reviews of nutritional epidemiologic studies.

METHODS

We searched MEDLINE, EMBASE and the Cochrane Database of Systematic Reviews (Jan 2018-Aug 2019) and sampled 150 systematic reviews of nutritional epidemiologic studies.

RESULTS

Most reviews (n=131/150; 87.3%) attempted to assess risk of bias. Commonly used tools neglected to address all important sources of bias, such as selective reporting (n=25/28; 89.3%), and frequently included constructs unrelated to risk of bias, such as reporting (n=14/28; 50.0%). Most reviews (n=66/101; 65.3%) did not incorporate risk of bias in the synthesis. While more than half of reviews considered biases due to confounding and misclassification of the exposure in their interpretation of findings, other biases, such as selective reporting, were rarely considered (n=1/150; 0.7%).

CONCLUSION

Reviews of nutritional epidemiologic studies have important limitations in their assessment of risk of bias.

Similarities, reliability and gaps in assessing the quality of conduct of systematic reviews using AMSTAR-2 and ROBIS: systematic survey of nutrition reviews

BACKGROUND

AMSTAR-2 ('A Measurement Tool to Assess Systematic Reviews, version 2') and ROBIS ('Risk of Bias in Systematic Reviews') are independent instruments used to assess the quality of conduct of systematic reviews/meta-analyses (SR/MAs). The degree of overlap in methodological constructs together with the reliability and any methodological gaps have not been systematically assessed and summarized in the field of nutrition.

METHODS

We performed a systematic survey of MEDLINE, EMBASE, and the Cochrane Library for SR/MAs published between January 2010 and November 2018 that examined the effects of any nutritional intervention/exposure for cancer prevention. We followed a systematic review approach including two independent reviewers at each step of the process. For AMSTAR-2 (16 items) and ROBIS (21 items), we assessed the similarities, the inter-rater reliability (IRR) and any methodological limitations of the instruments. Our protocol for the survey was registered in PROSPERO (CRD42019121116).

RESULTS

We found 4 similar domain constructs based on 11 comparisons from a total of 12 AMSTAR-2 and 14 ROBIS items. Ten comparisons were considered fully overlapping. Based on Gwet's agreement coefficients, six comparisons provided almost perfect (> 0.8), three substantial (> 0.6), and one a moderate level of agreement (> 0.4). While there is considerable overlap in constructs, AMSTAR-2 uniquely addresses explaining the selection of study designs for inclusion, reporting on excluded studies with justification, sources of funding of primary studies, and reviewers' conflict of interest. By contrast, ROBIS uniquely addresses appropriateness and restrictions within eligibility criteria, reducing risk of error in risk of bias (RoB) assessments, completeness of data extracted for analyses, the inclusion of all necessary studies for analyses, and adherence to predefined analysis plan.

CONCLUSIONS

Among the questions on AMSTAR-2 and ROBIS, 70.3% (26/37 items) address the same or similar methodological constructs. While the IRR of these constructs was moderate to perfect, there are unique methodological constructs that each instrument independently addresses. Notably, both instruments do not address the reporting of absolute estimates of effect or the overall certainty of the evidence, items that are crucial for users' wishing to interpret the importance of SR/MA results.

Methods used to select results to include in meta-analyses of nutrition research: a meta-research study

OBJECTIVES

To investigate how often review authors encounter multiple results from included studies that are eligible for inclusion in a particular meta-analysis, and how often methods to select results are specified.

METHODS

MEDLINE and Epistemonikos were searched (January 2018 - June 2019) to identify systematic reviews with meta-analysis of the association between food/diet and health-related outcomes. A random sample of these reviews was selected, and for the first presented ('index') meta-analysis, rules used to select effect estimates to include in this meta-analysis were extracted from the reviews and their protocols. All effect estimates from the primary studies that were eligible for inclusion in the index meta-analyses were extracted (e.g. when a study report presented effect estimates for blood pressure at 3 weeks and 6 weeks, both unadjusted and adjusted for covariates, and all were eligible for inclusion in a meta-analysis of the effect of red meat consumption on blood pressure, we extracted all estimates and classified the study as having "multiplicity of results").

RESULTS

Forty-two systematic reviews with 325 studies (104 randomized, 221 non-randomized) were included; 14 reviews had a protocol. In 29% of review protocols and 69% of reviews, authors specified at least one decision rule to select effect estimates when multiple were available. In 68% of studies included in the index meta-analyses, there was at least one type of multiplicity of results.

CONCLUSIONS

Authors of systematic reviews of nutrition studies should anticipate encountering multiplicity of results in the included primary studies. Specification of methods to handle multiplicity when designing reviews is therefore recommended.

NUQUEST-NUtrition QUality Evaluation Strengthening Tools: development of tools for the evaluation of risk of bias in nutrition studies

BACKGROUND

Dietary exposure assessments are a critical issue in evaluating human nutrition studies; however, nutrition-specific criteria are not consistently included in existing bias assessment tools.

OBJECTIVES

Our objective was to develop a set of risk of bias (RoB) tools that integrated nutrition-specific criteria into validated generic assessment tools to address RoB issues, including those specific to dietary exposure assessment.

METHODS

The Nutrition QUality Evaluation Strengthening Tools (NUQUEST) development and validation process included 8 steps. The first steps identified 1) a development strategy; 2) generic assessment tools with demonstrated validity; and 3) nutrition-specific appraisal issues. This was followed by 4) generation of nutrition-specific items and 5) development of guidance to aid users of NUQUEST. The final steps used established ratings of selected studies and feedback from independent raters to 6) assess reliability and validity; 7) assess formatting and usability; and 8) finalize NUQUEST.

RESULTS

NUQUEST is based on the Scottish Intercollegiate Guidelines Network checklists for randomized controlled trials, cohort studies, and case-control studies. Using a purposive sample of 45 studies representing the 3 study designs, interrater reliability was high (Cohen's κ: 0.73; 95% CI: 0.52, 0.93) across all tools and at least moderate for individual tools (range: 0.57-1.00). The use of a worksheet improved usability and consistency of overall interrater agreement across all study designs (40% without worksheet, 80%-100% with worksheet). When compared to published ratings, NUQUEST ratings for evaluated studies demonstrated high concurrent validity (93% perfect or near-perfect agreement). Where there was disagreement, the nutrition-specific component was a contributing factor in discerning exposure methodological issues.

CONCLUSIONS

NUQUEST integrates nutrition-specific criteria with generic criteria from assessment tools with demonstrated reliability and validity. NUQUEST represents a consistent and transparent approach for evaluating RoB issues related to dietary exposure assessment commonly encountered in human nutrition studies.

Dairy Product Consumption and Cardiovascular Health: a Systematic Review and Meta-Analysis of Prospective Cohort Studies

The association between dairy product consumption and cardiovascular health remains highly debated. We quantitatively synthesized prospective cohort evidence on the associations between dairy consumption and risk of hypertension (HTN), coronary heart disease (CHD) and stroke. We systematically searched PubMed, Embase, and Web of Science through August 1st, 2020 to retrieve prospective cohort studies that reported on dairy consumption and risk of HTN, CHD or stroke. We used random-effects models to calculate the pooled relative risk (RR) and 95% confidence interval (CI) for the highest vs the lowest category of intake and for 1 serving/day increase in consumption. We rated the quality of evidence using NutriGrade. Fifty-five studies were included. Total dairy consumption was associated with a lower risk of HTN (RR for highest vs lowest level of intake: 0.91, 95% CI: 0.86-0.95, I2 = 73.5%; RR for 1 serving/day increase: 0.96, 95% CI: 0.94-0.97, I2 = 66.5%), CHD (highest vs lowest level of intake: 0.96, 95% CI: 0.92-1.00, I2 = 46.6%; 1 serving/day increase: 0.98, 95% CI: 0.95-1.00, I2 = 56.7%), and stroke (highest vs lowest level of intake: 0.90, 95% CI: 0.85-0.96, I2 = 60.8%; 1 serving/day increase: 0.96, 95% CI: 0.93-0.99, I2 = 74.7%). Despite moderate to considerable heterogeneity, these associations remained consistent across multiple subgroups. Evidence on the relationship between total dairy and risk of HTN and CHD were of moderate quality and of low quality for stroke. Low-fat dairy consumption was associated with lower risk of HTN and stroke, and high-fat dairy with a lower risk of stroke. Milk, cheese, or yogurt consumption showed inconsistent associations with the cardiovascular outcomes in high vs. low intake and dose-response meta-analyses. Total dairy consumption was associated with a modestly lower risk of hypertension, CHD and stroke. Moderate to considerable heterogeneity was observed in the estimates and the overall quality of the evidence was low to moderate.