1
|
Wang Y, Parpia S, Couban R, Wang Q, Armijo-Olivo S, Bassler D, Briel M, Brignardello-Petersen R, Gluud LL, Keitz SA, Letelier LM, Ravaud P, Schulz KF, Siemieniuk RAC, Zeraatkar D, Guyatt GH. Compelling evidence from meta-epidemiological studies demonstrates overestimation of effects in randomized trials that fail to optimize randomization and blind patients and outcome assessors. J Clin Epidemiol 2024; 165:111211. [PMID: 37939743 DOI: 10.1016/j.jclinepi.2023.11.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2023] [Revised: 10/29/2023] [Accepted: 11/01/2023] [Indexed: 11/10/2023]
Abstract
OBJECTIVES To investigate the impact of potential risk of bias elements on effect estimates in randomized trials. STUDY DESIGN AND SETTING We conducted a systematic survey of meta-epidemiological studies examining the influence of potential risk of bias elements on effect estimates in randomized trials. We included only meta-epidemiological studies that either preserved the clustering of trials within meta-analyses (compared effect estimates between trials with and without the potential risk of bias element within each meta-analysis, then combined across meta-analyses; between-trial comparisons), or preserved the clustering of substudies within trials (compared effect estimates between substudies with and without the element, then combined across trials; within-trial comparisons). Separately for studies based on between- and within-trial comparisons, we extracted ratios of odds ratios (RORs) from each study and combined them using a random-effects model. We made overall inferences and assessed certainty of evidence based on Grading of Recommendations, Assessment, development, and Evaluation and Instrument to assess the Credibility of Effect Modification Analyses. RESULTS Forty-one meta-epidemiological studies (34 of between-, 7 of within-trial comparisons) proved eligible. Inadequate random sequence generation (ROR 0.94, 95% confidence interval [CI] 0.90-0.97) and allocation concealment (ROR 0.92, 95% CI 0.88-0.97) probably lead to effect overestimation (moderate certainty). Lack of patients blinding probably overestimates effects for patient-reported outcomes (ROR 0.36, 95% CI 0.28-0.48; moderate certainty). Lack of blinding of outcome assessors results in effect overestimation for subjective outcomes (ROR 0.69, 95% CI 0.51-0.93; high certainty). The impact of patients or outcome assessors blinding on other outcomes, and the impact of blinding of health-care providers, data collectors, or data analysts, remain uncertain. Trials stopped early for benefit probably overestimate effects (moderate certainty). Trials with imbalanced cointerventions may overestimate effects, while trials with missing outcome data may underestimate effects (low certainty). Influence of baseline imbalance, compliance, selective reporting, and intention-to-treat analysis remain uncertain. CONCLUSION Failure to ensure random sequence generation or adequate allocation concealment probably results in modest overestimates of effects. Lack of patients blinding probably leads to substantial overestimates of effects for patient-reported outcomes. Lack of blinding of outcome assessors results in substantial effect overestimation for subjective outcomes. For other elements, though evidence for consistent systematic overestimate of effect remains limited, failure to implement these safeguards may still introduce important bias.
Collapse
Affiliation(s)
- Ying Wang
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada.
| | - Sameer Parpia
- Department of Oncology, McMaster University, Hamilton, Ontario, Canada
| | - Rachel Couban
- Department of Anesthesia, McMaster University, Hamilton, Ontario, Canada
| | - Qi Wang
- School of Public Health, Capital Medical University, Beijing, China
| | - Susan Armijo-Olivo
- University of Applied Sciences, Faculty of Business and Social Sciences, Osnabrück, Germany; Department of Physical Therapy, Faculty of Rehabilitation Medicine, University of Alberta, Edmonton Canada
| | - Dirk Bassler
- Department of Neonatology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Matthias Briel
- Department of Clinical Research, Meta-Research Centre Basel, University Hospital Basel, Basel, Switzerland
| | | | - Lise Lotte Gluud
- Gastro Unit, Copenhagen University Hospital Hvidovre, Copenhagen, Denmark
| | - Sheri A Keitz
- Department of Medicine, Lahey Hospital & Medical Center, Burlington, MA, USA
| | - Luz M Letelier
- Department of Internal Medicine, Escuela de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile
| | - Philippe Ravaud
- Epidemiology and Statistics Sorbonne Paris Cité Research Center (CRESS), INSERM, Université Paris Descartes, Paris, France
| | - Kenneth F Schulz
- School of Medicine, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
| | - Reed A C Siemieniuk
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Dena Zeraatkar
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| | - Gordon H Guyatt
- Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, Ontario, Canada
| |
Collapse
|
2
|
Marshall IJ, Marshall R, Wallace BC, Brassey J, Thomas J. Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study. J Clin Epidemiol 2019; 109:30-41. [PMID: 30590190 PMCID: PMC6524137 DOI: 10.5281/zenodo.1447087] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2018] [Revised: 12/07/2018] [Accepted: 12/19/2018] [Indexed: 08/31/2023]
Abstract
OBJECTIVE To simulate possible changes in systematic review results if rapid review methods were used. STUDY DESIGN AND SETTING We recalculated meta-analyses for binary primary outcomes in Cochrane systematic reviews, simulating rapid review methods. We simulated searching only PubMed, excluding older articles (5, 7, 10, 15, and 20 years before the search date), excluding smaller trials (<50, <100, and <200 participants), and using the largest trial only. We examined percentage changes in pooled odds ratios (ORs) (classed as no important change [<5%], small [<20%], moderate [<30%], or large [≥30%]), statistical significance, and biases observed using rapid methods. RESULTS Two thousand five hundred and twelve systematic reviews (16,088 studies) were included. Rapid methods resulted in the loss of all data in 3.7-44.7% of meta-analyses. Searching only PubMed had the smallest risk of changed ORs (19% [477/2,512] were small changes or greater; 10% [260/2,512] were moderate or greater). Changes in ORs varied substantially with each rapid review method; 8.4-21.3% were small, 1.9-8.8% were moderate, and 4.7-34.1% were large. Changes in statistical significance occurred in 6.5-38.6% of meta-analyses. Changes from significant to nonsignificant were most common (2.1-13.7% meta-analyses). We found no evidence of bias with any rapid review method. CONCLUSION Searching PubMed only might be considered where a ∼10% risk of the primary outcome OR changing by >20% could be tolerated. This could be the case in scoping reviews, resource limitation, or where syntheses are needed urgently. Other situations, such as clinical guidelines and regulatory decisions, favor more comprehensive systematic review methods.
Collapse
Affiliation(s)
- Iain J Marshall
- School of Population Health and Environmental Sciences, King's College London, London, UK.
| | | | - Byron C Wallace
- College of Computer and Information Science, Northeastern University, Boston, MA, USA
| | | | - James Thomas
- UCL Institute of Education, University College London, London, UK
| |
Collapse
|
3
|
Marshall IJ, Marshall R, Wallace BC, Brassey J, Thomas J. Rapid reviews may produce different results to systematic reviews: a meta-epidemiological study. J Clin Epidemiol 2018; 109:30-41. [PMID: 30590190 PMCID: PMC6524137 DOI: 10.1016/j.jclinepi.2018.12.015] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2018] [Revised: 12/07/2018] [Accepted: 12/19/2018] [Indexed: 11/17/2022]
Abstract
Objective To simulate possible changes in systematic review results if rapid review methods were used. Study Design and Setting We recalculated meta-analyses for binary primary outcomes in Cochrane systematic reviews, simulating rapid review methods. We simulated searching only PubMed, excluding older articles (5, 7, 10, 15, and 20 years before the search date), excluding smaller trials (<50, <100, and <200 participants), and using the largest trial only. We examined percentage changes in pooled odds ratios (ORs) (classed as no important change [<5%], small [<20%], moderate [<30%], or large [≥30%]), statistical significance, and biases observed using rapid methods. Results Two thousand five hundred and twelve systematic reviews (16,088 studies) were included. Rapid methods resulted in the loss of all data in 3.7–44.7% of meta-analyses. Searching only PubMed had the smallest risk of changed ORs (19% [477/2,512] were small changes or greater; 10% [260/2,512] were moderate or greater). Changes in ORs varied substantially with each rapid review method; 8.4–21.3% were small, 1.9–8.8% were moderate, and 4.7–34.1% were large. Changes in statistical significance occurred in 6.5–38.6% of meta-analyses. Changes from significant to nonsignificant were most common (2.1–13.7% meta-analyses). We found no evidence of bias with any rapid review method. Conclusion Searching PubMed only might be considered where a ∼10% risk of the primary outcome OR changing by >20% could be tolerated. This could be the case in scoping reviews, resource limitation, or where syntheses are needed urgently. Other situations, such as clinical guidelines and regulatory decisions, favor more comprehensive systematic review methods.
Collapse
Affiliation(s)
- Iain J Marshall
- School of Population Health and Environmental Sciences, King's College London, London, UK.
| | | | - Byron C Wallace
- College of Computer and Information Science, Northeastern University, Boston, MA, USA
| | | | - James Thomas
- UCL Institute of Education, University College London, London, UK
| |
Collapse
|