1
|
Huang Q, Trinquart L. Relative likelihood ratios for neutral comparisons of statistical tests in simulation studies. Biom J 2024; 66:e2200102. [PMID: 36642800 DOI: 10.1002/bimj.202200102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2022] [Revised: 11/11/2022] [Accepted: 11/15/2022] [Indexed: 01/17/2023]
Abstract
When comparing the performance of two or more competing tests, simulation studies commonly focus on statistical power. However, if the size of the tests being compared are either different from one another or from the nominal size, comparing tests based on power alone may be misleading. By analogy with diagnostic accuracy studies, we introduce relative positive and negative likelihood ratios to factor in both power and size in the comparison of multiple tests. We derive sample size formulas for a comparative simulation study. As an example, we compared the performance of six statistical tests for small-study effects in meta-analyses of randomized controlled trials: Begg's rank correlation, Egger's regression, Schwarzer's method for sparse data, the trim-and-fill method, the arcsine-Thompson test, and Lin and Chu's combined test. We illustrate that comparing power alone, or power adjusted or penalized for size, can be misleading, and how the proposed likelihood ratio approach enables accurate comparison of the trade-off between power and size between competing tests.
Collapse
Affiliation(s)
- Qiuxi Huang
- Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts, USA
| | - Ludovic Trinquart
- Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts, USA
- Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, Massachusetts, USA
- Tufts Clinical and Translational Science Institute, Tufts University, Boston, Massachusetts, USA
| |
Collapse
|
2
|
Tang X, Trinquart L. Bayesian multivariate network meta-analysis model for the difference in restricted mean survival times. Stat Med 2021; 41:595-611. [PMID: 34883534 DOI: 10.1002/sim.9276] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Revised: 10/15/2021] [Accepted: 10/23/2021] [Indexed: 11/08/2022]
Abstract
Network meta-analysis (NMA) is essential for clinical decision-making. NMA enables inference for all pair-wise comparisons between interventions available for the same indication, by using both direct evidence and indirect evidence. In randomized trials with time-to event outcome data, such as lung cancer data, conventional NMA methods rely on the hazard ratio and the proportional hazards assumption, and ignore the varying follow-up durations across trials. We introduce a novel multivariate NMA model for the difference in restricted mean survival times (RMST). Our model synthesizes all the available evidence from multiple time points simultaneously and borrows information across time points through within-study covariance and between-study covariance for the differences in RMST. We propose an estimator of the within-study covariance and we then assume it to be known. We estimate the model under the Bayesian framework. We evaluated our model by conducting a simulation study. Our multiple-time-point model yields lower mean squared error over the conventional single-time-point model at all time points, especially when the availability of evidence decreases. We illustrated the model on a network of randomized trials of second-line treatments of advanced non-small-cell lung cancer. Our multiple-time-point model yielded increased precision and detected evidence of benefit at earlier time points as compared to the single-time-point model. Our model has the advantage of providing clinically interpretable measures of treatment effects.
Collapse
Affiliation(s)
- Xiaoyu Tang
- Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts, USA
| | - Ludovic Trinquart
- Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts, USA.,Institute for Clinical Research and Health Policy Studies, Tufts Medical Center, Boston, Massachusetts, USA.,Tufts Clinical and Translational Science Institute, Tufts University, Boston, Massachusetts, USA
| |
Collapse
|
3
|
Stanley TD, Doucouliagos H, Ioannidis JPA, Carter EC. Detecting publication selection bias through excess statistical significance. Res Synth Methods 2021; 12:776-795. [PMID: 34196473 DOI: 10.1002/jrsm.1512] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2021] [Revised: 05/08/2021] [Accepted: 06/09/2021] [Indexed: 11/11/2022]
Abstract
We introduce and evaluate three tests for publication selection bias based on excess statistical significance (ESS). The proposed tests incorporate heterogeneity explicitly in the formulas for expected and ESS. We calculate the expected proportion of statistically significant findings in the absence of selective reporting or publication bias based on each study's SE and meta-analysis estimates of the mean and variance of the true-effect distribution. A simple proportion of statistical significance test (PSST) compares the expected to the observed proportion of statistically significant findings. Alternatively, we propose a direct test of excess statistical significance (TESS). We also combine these two tests of excess statistical significance (TESSPSST). Simulations show that these ESS tests often outperform the conventional Egger test for publication selection bias and the three-parameter selection model (3PSM).
Collapse
Affiliation(s)
- T D Stanley
- School of Business and Law, Deakin University, Burwood, Victoria, Australia
| | | | - John P A Ioannidis
- Epidemiology and Population Health, and (by courtesy) of Biomedical Data Science, and of Statistics, and Co-Director, METRICS, Stanford University, Stanford, California, USA.,METRIC B, Berlin Institute of Health, Berlin, Germany
| | - Evan C Carter
- Human Research and Engineering Directorate, United States Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, Maryland, USA
| |
Collapse
|
4
|
Weir IR. Multivariate meta-analysis model for the difference in restricted mean survival times. Biostatistics 2021; 22:82-96. [PMID: 31175828 PMCID: PMC7846118 DOI: 10.1093/biostatistics/kxz018] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/22/2018] [Revised: 04/26/2019] [Accepted: 04/28/2019] [Indexed: 01/01/2023] Open
Abstract
In randomized controlled trials (RCTs) with time-to-event outcomes, the difference in restricted mean survival times (RMSTD) offers an absolute measure of the treatment effect on the time scale. Computation of the RMSTD relies on the choice of a time horizon, $\tau$. In a meta-analysis, varying follow-up durations may lead to the exclusion of RCTs with follow-up shorter than $\tau$. We introduce an individual patient data multivariate meta-analysis model for RMSTD estimated at multiple time horizons. We derived the within-trial covariance for the RMSTD enabling the synthesis of all data by borrowing strength from multiple time points. In a simulation study covering 60 scenarios, we compared the statistical performance of the proposed method to that of two univariate meta-analysis models, based on available data at each time point and based on predictions from flexible parametric models. Our multivariate model yields smaller mean squared error over univariate methods at all time points. We illustrate the method with a meta-analysis of five RCTs comparing transcatheter aortic valve replacement (TAVR) with surgical replacement in patients with aortic stenosis. Over 12, 24, and 36 months of follow-up, those treated by TAVR live 0.28 [95% confidence interval (CI) 0.01 to 0.56], 0.46 (95% CI $-$0.08 to 1.01), and 0.79 (95% CI $-$0.43 to 2.02) months longer on average compared to those treated by surgery, respectively.
Collapse
Affiliation(s)
- Isabelle R Weir
- Department of Biostatistics, Boston University School of Public Health, Boston, 801 Massachusetts Avenue, MA, USA
| |
Collapse
|
5
|
Cavus M, Yazici B, Sezer A. Penalized power approach to compare the power of the tests when Type I error probabilities are different. COMMUN STAT-SIMUL C 2019. [DOI: 10.1080/03610918.2019.1588310] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Affiliation(s)
- Mustafa Cavus
- Department of Statistics, Eskisehir Technical University, Eskisehir, Turkey
| | - Berna Yazici
- Department of Statistics, Eskisehir Technical University, Eskisehir, Turkey
| | - Ahmet Sezer
- Department of Statistics, Eskisehir Technical University, Eskisehir, Turkey
| |
Collapse
|
6
|
Lin L, Xing A, Kofler MJ, Murad MH. Borrowing of strength from indirect evidence in 40 network meta-analyses. J Clin Epidemiol 2018; 106:41-49. [PMID: 30342086 DOI: 10.1016/j.jclinepi.2018.10.007] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2018] [Revised: 09/02/2018] [Accepted: 10/09/2018] [Indexed: 12/13/2022]
Abstract
OBJECTIVES Network meta-analysis (NMA) is increasingly being used to synthesize direct and indirect evidence and help decision makers simultaneously compare multiple treatments. We empirically evaluate the incremental gain in precision achieved by incorporating indirect evidence in NMAs. STUDY DESIGN AND SETTING We performed both network and pairwise meta-analyses on 40 published data sets of multiple-treatment comparisons. Their results were compared using the recently proposed borrowing of strength (BoS) statistic, which quantifies the percentage reduction in the uncertainty of the effect estimate when adding indirect evidence to an NMA. RESULTS We analyzed 915 possible treatment comparisons, from which 484 (53%) had no direct evidence (BoS = 100%). In 181 comparisons with only one study contributing direct evidence, NMAs resulted in reduced precision (BoS < 0) and no appreciable improvements in precision (0 < BoS < 30%) for 104 (57.5%) and 23 (12.7%) comparisons, respectively. In 250 comparisons with at least two studies contributing direct evidence, NMAs provided increased precision with BoS ≥ 30% for 166 (66.4%) comparisons. CONCLUSION Although NMAs have the potential to provide more precise results than those only based on direct evidence, the incremental gain may reliably occur only when at least two head-to-head studies are available and treatments are well connected. Researchers should routinely report and compare the results from both network and pairwise meta-analyses.
Collapse
Affiliation(s)
- Lifeng Lin
- Department of Statistics, Florida State University, Tallahassee, FL 32306, USA.
| | - Aiwen Xing
- Department of Statistics, Florida State University, Tallahassee, FL 32306, USA
| | - Michael J Kofler
- Department of Psychology, Florida State University, Tallahassee, FL 32306, USA
| | | |
Collapse
|
7
|
Hwang H, DeSantis SM. Multivariate network meta-analysis to mitigate the effects of outcome reporting bias. Stat Med 2018; 37:3254-3266. [PMID: 29882392 PMCID: PMC7259375 DOI: 10.1002/sim.7815] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2017] [Revised: 02/09/2018] [Accepted: 04/22/2018] [Indexed: 02/04/2023]
Abstract
Outcome reporting bias (ORB) is recognized as a threat to the validity of both pairwise and network meta-analysis (NMA). In recent years, multivariate meta-analytic methods have been proposed to reduce the impact of ORB in the pairwise setting. These methods have shown that multivariate meta-analysis can reduce bias and increase efficiency of pooled effect sizes. However, it is unknown whether multivariate NMA (MNMA) can similarly reduce the impact of ORB. Additionally, it is quite challenging to implement MNMA due to the fact that correlation between treatments and outcomes must be modeled; thus, the dimension of the covariance matrix and number of components to estimate grows quickly with the number of treatments and number of outcomes. To determine whether MNMA can reduce the effects of ORB on pooled treatment effect sizes, we present an extensive simulation study of Bayesian MNMA. Via simulation studies, we show that MNMA reduces the bias of pooled effect sizes under a variety of outcome missingness scenarios, including missing at random and missing not at random. Further, MNMA improves the precision of estimates, producing narrower credible intervals. We demonstrate the applicability of the approach via application of MNMA to a multi-treatment systematic review of randomized controlled trials of anti-depressants for the treatment of depression in older adults.
Collapse
Affiliation(s)
- Hyunsoo Hwang
- Department of Biostatistics and Data Science, The University of Texas School of Public Health, TX, 77030, USA
| | - Stacia M DeSantis
- Department of Biostatistics and Data Science, The University of Texas School of Public Health, TX, 77030, USA
| |
Collapse
|
8
|
Martinez V, Beloeil H, Marret E, Fletcher D, Ravaud P, Trinquart L. Non-opioid analgesics in adults after major surgery: systematic review with network meta-analysis of randomized trials. Br J Anaesth 2017; 118:22-31. [DOI: 10.1093/bja/aew391] [Citation(s) in RCA: 124] [Impact Index Per Article: 17.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/17/2023] Open
|
9
|
Response to letter: Whole scalp EEG power change is not a prerequisite for further EEG processing. Hear Res 2016; 339:217-8. [DOI: 10.1016/j.heares.2016.07.002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/07/2016] [Accepted: 07/09/2016] [Indexed: 11/18/2022]
|
10
|
Mueller KF, Meerpohl JJ, Briel M, Antes G, von Elm E, Lang B, Motschall E, Schwarzer G, Bassler D. Methods for detecting, quantifying, and adjusting for dissemination bias in meta-analysis are described. J Clin Epidemiol 2016; 80:25-33. [PMID: 27502970 DOI: 10.1016/j.jclinepi.2016.04.015] [Citation(s) in RCA: 26] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2015] [Revised: 04/08/2016] [Accepted: 04/25/2016] [Indexed: 12/20/2022]
Abstract
OBJECTIVE To systematically review methodological articles which focus on nonpublication of studies and to describe methods of detecting and/or quantifying and/or adjusting for dissemination in meta-analyses. To evaluate whether the methods have been applied to an empirical data set for which one can be reasonably confident that all studies conducted have been included. STUDY DESIGN AND SETTING We systematically searched Medline, the Cochrane Library, and Web of Science, for methodological articles that describe at least one method of detecting and/or quantifying and/or adjusting for dissemination bias in meta-analyses. RESULTS The literature search retrieved 2,224 records, of which we finally included 150 full-text articles. A great variety of methods to detect, quantify, or adjust for dissemination bias were described. Methods included graphical methods mainly based on funnel plot approaches, statistical methods, such as regression tests, selection models, sensitivity analyses, and a great number of more recent statistical approaches. Only few methods have been validated in empirical evaluations using unpublished studies obtained from regulators (Food and Drug Administration, European Medicines Agency). CONCLUSION We present an overview of existing methods to detect, quantify, or adjust for dissemination bias. It remains difficult to advise which method should be used as they are all limited and their validity has rarely been assessed. Therefore, a thorough literature search remains crucial in systematic reviews, and further steps to increase the availability of all research results need to be taken.
Collapse
Affiliation(s)
- Katharina Felicitas Mueller
- Center for Pediatric Clinical Studies, University Children's Hospital Tuebingen, Frondsbergstraße 23, 72070 Tuebingen, Germany
| | - Joerg J Meerpohl
- Cochrane Germany, Medical Center-University of Freiburg, Berliner Allee 29, 79110 Freiburg im Breisgau, Germany
| | - Matthias Briel
- Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Hebelstrasse 10, 4031 Basel, Switzerland
| | - Gerd Antes
- Cochrane Germany, Medical Center-University of Freiburg, Berliner Allee 29, 79110 Freiburg im Breisgau, Germany
| | - Erik von Elm
- Cochrane Switzerland, IUMSP, University Hospital Lausanne, Route de la Corniche 10, 1010 Lausanne, Switzerland
| | - Britta Lang
- Cochrane Germany, Medical Center-University of Freiburg, Berliner Allee 29, 79110 Freiburg im Breisgau, Germany
| | - Edith Motschall
- Center for Medical Biometry and Medical Informatics, Medical Center-University of Freiburg, Freiburg, Germany
| | - Guido Schwarzer
- Center for Medical Biometry and Medical Informatics, Medical Center-University of Freiburg, Freiburg, Germany
| | - Dirk Bassler
- Department of Neonatology, University Hospital Zurich, University of Zurich, Frauenklinikstrasse 10, 8091 Zurich, Switzerland.
| |
Collapse
|
11
|
Sa Cunha A, Carrere N, Meunier B, Fabre JM, Sauvanet A, Pessaux P, Ortega-Deballon P, Fingerhut A, Lacaine F. Stump closure reinforcement with absorbable fibrin collagen sealant sponge (TachoSil) does not prevent pancreatic fistula after distal pancreatectomy: the FIABLE multicenter controlled randomized study. Am J Surg 2015; 210:739-48. [PMID: 26160763 DOI: 10.1016/j.amjsurg.2015.04.015] [Citation(s) in RCA: 50] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2014] [Revised: 03/29/2015] [Accepted: 04/30/2015] [Indexed: 01/07/2023]
Abstract
BACKGROUND The aim of this study is to evaluate the effectiveness of TachoSil sponge on distal pancreatectomy remnant stump in reducing the rate and severity of postoperative pancreatic fistula (POPF). METHODS All consecutive patients requiring distal pancreatectomy were randomized in 45 centers. The principal end point was onset of "clinically relevant" POPF. Univariate and multivariate analyses were searched for predictive factors. RESULTS Of the 270 patients randomized (134 with TachoSil; 136 without), 150 (55.6%) patients sustained a POPF [74 clinically relevant and 76 clinically silent (27.4% and 28.1%), respectively]: no statistically significant difference was found between patients sustaining clinically relevant POPF [41 (30.6%) with vs 33 (24.3%) without TachoSil (P = .276)], or overall POPF [73 (54.5%) with vs 77 (56.6%) without TachoSil, (P = .807)], but there were more clinically relevant POPF after hand-sewn (32.3%) versus mechanical closure (19.8%) (P = .025) and, in case of splenic preservation, after splenic vessel ligation (15/32, 46.9%) versus vascular preservation (17/72, 23.6%) (P = .024). Hand-sewn pancreatic remnant closure (P = .023) and splenic vessel ligation in splenic preservation (P = .035) were independent predictive factors for the onset of clinically relevant POPF. CONCLUSION TachoSil sponge reinforcement of the proximal remnant after distal pancreatectomy reduced neither the rate nor the severity of POPF.
Collapse
Affiliation(s)
- Antonio Sa Cunha
- Service de Chirurgie Hépato-biliare, Centre Hépato-Biliaire, Hôpital Paul Brousse, Villejuif, France
| | - Nicolas Carrere
- Service de Chirurgie Générale et Digestive, Hôpital Purpan, Toulouse, France
| | - Bernard Meunier
- Service de Chirurgie Hépato-Biliaire et Digestive, Hôpital Pontchaillou, Rennes, France
| | - Jean-Michel Fabre
- Service de Chirurgie Digestive A, Hôpital St Eloi, Montpellier, France
| | - Alain Sauvanet
- Service de Chirurgie Hépato-Bilio-Pancréatique, Hôpital Beaujon, Clichy, France
| | - Patrick Pessaux
- Service de Chirurgie Hépato-Bilio-Pancréatique, Hôpital Hautepierre, Strasbourg, France
| | | | - Abe Fingerhut
- Section for Surgical Research, Department of Surgery, Medical University of Graz, Graz, Austria.
| | - François Lacaine
- Service de Chirurgie Digestive et Viscérale, Hôpital Tenon, Paris, France
| |
Collapse
|
12
|
Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, Ioannidis JPA, Straus S, Thorlund K, Jansen JP, Mulrow C, Catalá-López F, Gøtzsche PC, Dickersin K, Boutron I, Altman DG, Moher D. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med 2015; 162:777-84. [PMID: 26030634 DOI: 10.7326/m14-2385] [Citation(s) in RCA: 4443] [Impact Index Per Article: 493.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 02/06/2023] Open
Abstract
The PRISMA statement is a reporting guideline designed to improve the completeness of reporting of systematic reviews and meta-analyses. Authors have used this guideline worldwide to prepare their reviews for publication. In the past, these reports typically compared 2 treatment alternatives. With the evolution of systematic reviews that compare multiple treatments, some of them only indirectly, authors face novel challenges for conducting and reporting their reviews. This extension of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) statement was developed specifically to improve the reporting of systematic reviews incorporating network meta-analyses. A group of experts participated in a systematic review, Delphi survey, and face-to-face discussion and consensus meeting to establish new checklist items for this extension statement. Current PRISMA items were also clarified. A modified, 32-item PRISMA extension checklist was developed to address what the group considered to be immediately relevant to the reporting of network meta-analyses. This document presents the extension and provides examples of good reporting, as well as elaborations regarding the rationale for new checklist items and the modification of previously existing items from the PRISMA statement. It also highlights educational information related to key considerations in the practice of network meta-analysis. The target audience includes authors and readers of network meta-analyses, as well as journal editors and peer reviewers.
Collapse
|