1
|
Kanukula R, McKenzie JE, Bero L, Dai Z, McDonald S, Kroeger CM, Korevaar E, Forbes A, Page MJ. Investigation of bias due to selective inclusion of study effect estimates in meta-analyses of nutrition research. Res Synth Methods 2024; 15:524-542. [PMID: 38316613 DOI: 10.1002/jrsm.1706] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2022] [Revised: 12/14/2023] [Accepted: 01/04/2024] [Indexed: 02/07/2024]
Abstract
We aimed to explore, in a sample of systematic reviews (SRs) with meta-analyses of the association between food/diet and health-related outcomes, whether systematic reviewers selectively included study effect estimates in meta-analyses when multiple effect estimates were available. We randomly selected SRs of food/diet and health-related outcomes published between January 2018 and June 2019. We selected the first presented meta-analysis in each review (index meta-analysis), and extracted from study reports all study effect estimates that were eligible for inclusion in the meta-analysis. We calculated the Potential Bias Index (PBI) to quantify and test for evidence of selective inclusion. The PBI ranges from 0 to 1; values above or below 0.5 suggest selective inclusion of effect estimates more or less favourable to the intervention, respectively. We also compared the index meta-analytic estimate to the median of a randomly constructed distribution of meta-analytic estimates (i.e., the estimate expected when there is no selective inclusion). Thirty-nine SRs with 312 studies were included. The estimated PBI was 0.49 (95% CI 0.42-0.55), suggesting that the selection of study effect estimates from those reported was consistent with a process of random selection. In addition, the index meta-analytic effect estimates were similar, on average, to what we would expect to see in meta-analyses generated when there was no selective inclusion. Despite this, we recommend that systematic reviewers report the methods used to select effect estimates to include in meta-analyses, which can help readers understand the risk of selective inclusion bias in the SRs.
Collapse
Affiliation(s)
- Raju Kanukula
- Methods in Evidence Synthesis Unit, School of Public Health and Preventative Medicine, Monash University, Melbourne, Victoria, Australia
| | - Joanne E McKenzie
- Methods in Evidence Synthesis Unit, School of Public Health and Preventative Medicine, Monash University, Melbourne, Victoria, Australia
| | - Lisa Bero
- Center for Bioethics and Humanities, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
| | - Zhaoli Dai
- Charles Perkins Centre, School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Camperdown, New South Wales, Australia
| | - Sally McDonald
- Charles Perkins Centre, School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Camperdown, New South Wales, Australia
| | - Cynthia M Kroeger
- Charles Perkins Centre, Central Clinical School, Faculty of Medicine and Health, The University of Sydney, Camperdown, New South Wales, Australia
| | - Elizabeth Korevaar
- Methods in Evidence Synthesis Unit, School of Public Health and Preventative Medicine, Monash University, Melbourne, Victoria, Australia
| | - Andrew Forbes
- Methods in Evidence Synthesis Unit, School of Public Health and Preventative Medicine, Monash University, Melbourne, Victoria, Australia
| | - Matthew J Page
- Methods in Evidence Synthesis Unit, School of Public Health and Preventative Medicine, Monash University, Melbourne, Victoria, Australia
| |
Collapse
|
2
|
Kanukula R, McKenzie JE, Cashin AG, Korevaar E, McDonald S, Mello AT, Nguyen PY, Saldanha IJ, Wewege MA, Page MJ. Variation observed in consensus judgments between pairs of reviewers when assessing the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research. J Clin Epidemiol 2024; 166:111244. [PMID: 38142761 DOI: 10.1016/j.jclinepi.2023.111244] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 11/18/2023] [Accepted: 12/19/2023] [Indexed: 12/26/2023]
Abstract
OBJECTIVES To evaluate the risk of bias due to missing evidence in a sample of published meta-analyses of nutrition research using the Risk Of Bias due to Missing Evidence (ROB-ME) tool and determine inter-rater agreement in assessments. STUDY DESIGN AND SETTING We assembled a random sample of 42 meta-analyses of nutrition research. Eight assessors were randomly assigned to one of four pairs. Each pair assessed 21 randomly assigned meta-analyses, and each meta-analysis was assessed by two pairs. We calculated raw percentage agreement and chance corrected agreement using Gwet's Agreement Coefficient (AC) in consensus judgments between pairs. RESULTS Across the eight signaling questions in the ROB-ME tool, raw percentage agreement ranged from 52% to 100%, and Gwet's AC ranged from 0.39 to 0.76. For the risk-of-bias judgment, the raw percentage agreement was 76% (95% confidence interval 60% to 92%) and Gwet's AC was 0.47 (95% confidence interval 0.14 to 0.80). In seven (17%) meta-analyses, either one or both pairs judged the risk of bias due to missing evidence as "low risk". CONCLUSION Our findings indicated substantial variation in assessments in consensus judgments between pairs for the signaling questions and overall risk-of-bias judgments. More tutorials and training are needed to help researchers apply the ROB-ME tool more consistently.
Collapse
Affiliation(s)
- Raju Kanukula
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Joanne E McKenzie
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Aidan G Cashin
- Centre for Pain IMPACT, Neuroscience Research Australia, Sydney, NSW, Australia; School of Health Sciences, Faculty of Medicine & Health, University of New South Wales, Sydney, NSW, Australia
| | - Elizabeth Korevaar
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Sally McDonald
- Charles Perkins Centre, School of Pharmacy, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia
| | - Arthur T Mello
- Post-Graduate Program in Nutrition, Federal University of Santa Catarina, Florianopolis, Santa Catarina, Brazil
| | - Phi-Yen Nguyen
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia
| | - Ian J Saldanha
- Center for Clinical Trials and Evidence Synthesis, Johns Hopkins University Bloomberg School of Public Health, Baltimore, MD, USA
| | - Michael A Wewege
- Centre for Pain IMPACT, Neuroscience Research Australia, Sydney, NSW, Australia; School of Health Sciences, Faculty of Medicine & Health, University of New South Wales, Sydney, NSW, Australia
| | - Matthew J Page
- Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia.
| |
Collapse
|
3
|
Gkiouras K, Choleva ME, Verrou A, Goulis DG, Bogdanos DP, Grammatikopoulou MG. A Meta-Epidemiological Study of Positive Results in Clinical Nutrition Research: The Good, the Bad and the Ugly of Statistically Significant Findings. Nutrients 2022; 14:nu14235164. [PMID: 36501193 PMCID: PMC9738552 DOI: 10.3390/nu14235164] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2022] [Revised: 12/01/2022] [Accepted: 12/02/2022] [Indexed: 12/12/2022] Open
Abstract
Positive (statistically significant) findings are easily produced in nutrition research when specific aspects of the research design and analysis are not accounted for. To address this issue, recently, a pledge was made to reform nutrition research and improve scientific trust on the science, encompass research transparency and achieve reproducibility. The aim of the present meta-epidemiological study was to evaluate the statistical significance status of research items published in three academic journals, all with a focus on clinical nutrition science and assessing certain methodological/transparency issues. All research items were published between the years 2015 and 2019. Study design, primary and secondary findings, sample size and age group, funding sources, positivist findings, the existence of a published research protocol and the adjustment of nutrients/dietary indexes to the energy intake (EI) of participants, were extracted for each study. Out of 2127 studies in total, those with positive findings consisted of the majority, in all three journals. Most studies had a published research protocol, however, this was mainly due to the randomized controlled trials and not to the evidence-synthesis studies. No differences were found in the distribution of positive findings according to the existence/inexistence of a published research protocol. In the pooled sample of studies, positive findings differed according to study design and more significant findings were reported by researchers failing to report any funding source. The majority of items published in the three journals (65.9%) failed to account for the EI of participants. The present results indicate that there is still room for the improvement of nutrition research in terms of design, analyses and reporting.
Collapse
Affiliation(s)
- Konstantinos Gkiouras
- Department of Rheumatology and Clinical Immunology, Faculty of Medicine, School of Health Sciences, University of Thessaly, Biopolis, 41110 Larissa, Greece
| | - Maria-Eleftheria Choleva
- Department of Nutritional Sciences and Dietetics, Faculty of Health Sciences, International Hellenic University, Alexander Campus, 57400 Thessaloniki, Greece
| | - Aikaterini Verrou
- Department of Nutritional Sciences and Dietetics, Faculty of Health Sciences, International Hellenic University, Alexander Campus, 57400 Thessaloniki, Greece
| | - Dimitrios G. Goulis
- Unit of Reproductive Endocrinology, 1st Department of Obstetrics and Gynecology, Medical School, Faculty of Health Sciences, Aristotle University of Thessaloniki, 56403 Thessaloniki, Greece
| | - Dimitrios P. Bogdanos
- Department of Rheumatology and Clinical Immunology, Faculty of Medicine, School of Health Sciences, University of Thessaly, Biopolis, 41110 Larissa, Greece
| | - Maria G. Grammatikopoulou
- Department of Rheumatology and Clinical Immunology, Faculty of Medicine, School of Health Sciences, University of Thessaly, Biopolis, 41110 Larissa, Greece
- Correspondence:
| |
Collapse
|