1
|
Schimmack U, Bartoš F. Estimating the false discovery risk of (randomized) clinical trials in medical journals based on published p-values. PLoS One 2023; 18:e0290084. [PMID: 37647247 PMCID: PMC10468063 DOI: 10.1371/journal.pone.0290084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Accepted: 08/01/2023] [Indexed: 09/01/2023] Open
Abstract
The influential claim that most published results are false raised concerns about the trustworthiness and integrity of science. Since then, there have been numerous attempts to examine the rate of false-positive results that have failed to settle this question empirically. Here we propose a new way to estimate the false positive risk and apply the method to the results of (randomized) clinical trials in top medical journals. Contrary to claims that most published results are false, we find that the traditional significance criterion of α = .05 produces a false positive risk of 13%. Adjusting α to.01 lowers the false positive risk to less than 5%. However, our method does provide clear evidence of publication bias that leads to inflated effect size estimates. These results provide a solid empirical foundation for evaluations of the trustworthiness of medical research.
Collapse
Affiliation(s)
- Ulrich Schimmack
- Department of Psychology, University of Toronto Mississauga, Mississauga, Canada
| | - František Bartoš
- Department of Psychological Methods, University of Amsterdam, Amsterdam, The Netherlands
- Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic
| |
Collapse
|
2
|
|
3
|
Pauli F. A Statistical Model to Investigate the Reproducibility Rate Based on Replication Experiments. Int Stat Rev 2018. [DOI: 10.1111/insr.12273] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Affiliation(s)
- Francesco Pauli
- DEAMSUniversity of Trieste P.le Europa 1 Trieste 34127 Italy
| |
Collapse
|
4
|
Abstract
False positive findings in science are inevitable, but are they particularly common in psychology and psychiatry? The evidence that we review suggests that while not restricted to our field, the problem is acute. We describe the concept of researcher 'degrees-of-freedom' to explain how many false-positive findings arise, and how the various strategies of registration, pre-specification, and reporting standards that are being adopted both reduce and make these visible. We review possible benefits and harms of proposed statistical solutions, from tougher requirements for significance, to Bayesian and machine learning approaches to analysis. Finally we consider the organisation and methods for replication and systematic review in psychology and psychiatry.
Collapse
Affiliation(s)
- D Stahl
- Department of Biostatistics and Health Informatics,Institute of Psychiatry,Psychology and Neuroscience,King's College London,London,UK
| | - A Pickles
- Department of Biostatistics and Health Informatics,Institute of Psychiatry,Psychology and Neuroscience,King's College London,London,UK
| |
Collapse
|
5
|
Domínguez Islas C, Rice KM. Addressing the estimation of standard errors in fixed effects meta-analysis. Stat Med 2018; 37:1788-1809. [PMID: 29574827 PMCID: PMC6001579 DOI: 10.1002/sim.7625] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2017] [Revised: 12/27/2017] [Accepted: 01/14/2018] [Indexed: 01/20/2023]
Abstract
Standard methods for fixed effects meta-analysis assume that standard errors for study-specific estimates are known, not estimated. While the impact of this simplifying assumption has been shown in a few special cases, its general impact is not well understood, nor are general-purpose tools available for inference under more realistic assumptions. In this paper, we aim to elucidate the impact of using estimated standard errors in fixed effects meta-analysis, showing why it does not go away in large samples and quantifying how badly miscalibrated standard inference will be if it is ignored. We also show the important role of a particular measure of heterogeneity in this miscalibration. These developments lead to confidence intervals for fixed effects meta-analysis with improved performance for both location and scale parameters.
Collapse
Affiliation(s)
- Clara Domínguez Islas
- Fred Hutchinson Cancer Research Center, Seattle, WA, USA.,MRC Biostatistics Unit, School of Clinical Medicine, University of Cambridge, Cambridge, UK
| | - Kenneth M Rice
- Department of Biostatistics, University of Washington, Seattle, WA, USA
| |
Collapse
|
6
|
Karabatsos G. A Bayesian nonparametric test of significance chasing biases. Res Synth Methods 2017; 9:51-61. [PMID: 28985020 DOI: 10.1002/jrsm.1269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2017] [Revised: 08/21/2017] [Accepted: 09/18/2017] [Indexed: 11/07/2022]
Abstract
There is a growing concern that much of the published research literature is distorted by the pursuit of statistically significant results. In a seminal article, Ioannidis and Trikalinos (2007, Clinical Trials) proposed an omnibus (I&T) test for significance chasing (SC) biases. This test compares the observed number of studies that report statistically significant results, against their expected number based on study power, assuming a common effect size across studies. The current article extends this approach by developing a Bayesian nonparametric (BNP) meta-regression model and test of SC bias, which can diagnose bias at the individual study level. This new BNP test is based on a flexible model of the predictive distribution of study power, conditionally on study-level covariates which account for study diversity, including diversity due to heterogeneous effect sizes across studies. A test of SC bias proceeds by comparing each study's significant outcome report indicator against its estimated posterior predictive distribution of study power, conditionally on the study's covariates. The BNP model and SC bias test are illustrated through the analyses of 3 meta-analytic data sets and through a simulation study. Software code for the BNP model and test, and the data sets, are provided as Supporting Information.
Collapse
Affiliation(s)
- George Karabatsos
- Departments of Educational Psychology, and Mathematics, Statistics, and by courtesy, Computer Science, University of Illinois at Chicago, Chicago, IL 60607, USA
| |
Collapse
|
7
|
Perneger TV, Combescure C. The distribution of P -values in medical research articles suggested selective reporting associated with statistical significance. J Clin Epidemiol 2017; 87:70-77. [DOI: 10.1016/j.jclinepi.2017.04.003] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2016] [Revised: 03/27/2017] [Accepted: 04/04/2017] [Indexed: 10/19/2022]
|
8
|
Forstmeier W, Wagenmakers E, Parker TH. Detecting and avoiding likely false‐positive findings – a practical guide. Biol Rev Camb Philos Soc 2016; 92:1941-1968. [DOI: 10.1111/brv.12315] [Citation(s) in RCA: 212] [Impact Index Per Article: 26.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2016] [Revised: 10/17/2016] [Accepted: 10/19/2016] [Indexed: 12/17/2022]
Affiliation(s)
- Wolfgang Forstmeier
- Department of Behavioural Ecology and Evolutionary Genetics Max Planck Institute for Ornithology 82319 Seewiesen Germany
| | - Eric‐Jan Wagenmakers
- Department of Psychology University of Amsterdam PO Box 15906 1001 NK Amsterdam The Netherlands
| | | |
Collapse
|
9
|
Transparency in Ecology and Evolution: Real Problems, Real Solutions. Trends Ecol Evol 2016; 31:711-719. [PMID: 27461041 DOI: 10.1016/j.tree.2016.07.002] [Citation(s) in RCA: 105] [Impact Index Per Article: 13.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2016] [Revised: 07/05/2016] [Accepted: 07/06/2016] [Indexed: 01/09/2023]
Abstract
To make progress scientists need to know what other researchers have found and how they found it. However, transparency is often insufficient across much of ecology and evolution. Researchers often fail to report results and methods in detail sufficient to permit interpretation and meta-analysis, and many results go entirely unreported. Further, these unreported results are often a biased subset. Thus the conclusions we can draw from the published literature are themselves often biased and sometimes might be entirely incorrect. Fortunately there is a movement across empirical disciplines, and now within ecology and evolution, to shape editorial policies to better promote transparency. This can be done by either requiring more disclosure by scientists or by developing incentives to encourage disclosure.
Collapse
|
10
|
Hartgerink CH, van Aert RC, Nuijten MB, Wicherts JM, van Assen MA. Distributions of p-values smaller than .05 in psychology: what is going on? PeerJ 2016; 4:e1935. [PMID: 27077017 PMCID: PMC4830257 DOI: 10.7717/peerj.1935] [Citation(s) in RCA: 35] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2016] [Accepted: 03/23/2016] [Indexed: 11/20/2022] Open
Abstract
Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP "incorrect rounding of p-value" (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.
Collapse
Affiliation(s)
- Chris H.J. Hartgerink
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Robbie C.M. van Aert
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Michèle B. Nuijten
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Jelte M. Wicherts
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
| | - Marcel A.L.M. van Assen
- Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands
- Department of Sociology, Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
11
|
Bishop DV, Thompson PA. Problems in using p-curve analysis and text-mining to detect rate of p-hacking and evidential value. PeerJ 2016; 4:e1715. [PMID: 26925335 PMCID: PMC4768688 DOI: 10.7717/peerj.1715] [Citation(s) in RCA: 40] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2015] [Accepted: 01/29/2016] [Indexed: 11/20/2022] Open
Abstract
Background. The p-curve is a plot of the distribution of p-values reported in a set of scientific studies. Comparisons between ranges of p-values have been used to evaluate fields of research in terms of the extent to which studies have genuine evidential value, and the extent to which they suffer from bias in the selection of variables and analyses for publication, p-hacking. Methods. p-hacking can take various forms. Here we used R code to simulate the use of ghost variables, where an experimenter gathers data on several dependent variables but reports only those with statistically significant effects. We also examined a text-mined dataset used by Head et al. (2015) and assessed its suitability for investigating p-hacking. Results. We show that when there is ghost p-hacking, the shape of the p-curve depends on whether dependent variables are intercorrelated. For uncorrelated variables, simulated p-hacked data do not give the "p-hacking bump" just below .05 that is regarded as evidence of p-hacking, though there is a negative skew when simulated variables are inter-correlated. The way p-curves vary according to features of underlying data poses problems when automated text mining is used to detect p-values in heterogeneous sets of published papers. Conclusions. The absence of a bump in the p-curve is not indicative of lack of p-hacking. Furthermore, while studies with evidential value will usually generate a right-skewed p-curve, we cannot treat a right-skewed p-curve as an indicator of the extent of evidential value, unless we have a model specific to the type of p-values entered into the analysis. We conclude that it is not feasible to use the p-curve to estimate the extent of p-hacking and evidential value unless there is considerable control over the type of data entered into the analysis. In particular, p-hacking with ghost variables is likely to be missed.
Collapse
Affiliation(s)
- Dorothy V.M. Bishop
- Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
| | - Paul A. Thompson
- Department of Experimental Psychology, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
12
|
Bruns SB, Ioannidis JPA. p-Curve and p-Hacking in Observational Research. PLoS One 2016; 11:e0149144. [PMID: 26886098 PMCID: PMC4757561 DOI: 10.1371/journal.pone.0149144] [Citation(s) in RCA: 72] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2015] [Accepted: 01/27/2016] [Indexed: 11/30/2022] Open
Abstract
The p-curve, the distribution of statistically significant p-values of published studies, has been used to make inferences on the proportion of true effects and on the presence of p-hacking in the published literature. We analyze the p-curve for observational research in the presence of p-hacking. We show by means of simulations that even with minimal omitted-variable bias (e.g., unaccounted confounding) p-curves based on true effects and p-curves based on null-effects with p-hacking cannot be reliably distinguished. We also demonstrate this problem using as practical example the evaluation of the effect of malaria prevalence on economic growth between 1960 and 1996. These findings call recent studies into question that use the p-curve to infer that most published research findings are based on true effects in the medical literature and in a wide range of disciplines. p-values in observational research may need to be empirically calibrated to be interpretable with respect to the commonly used significance threshold of 0.05. Violations of randomization in experimental studies may also result in situations where the use of p-curves is similarly unreliable.
Collapse
Affiliation(s)
- Stephan B. Bruns
- Meta-Research in Economics Group, University of Kassel, Kassel, Germany
| | - John P. A. Ioannidis
- Departments of Medicine, Health Research and Policy, and Statistics, and Meta-Research Innovation Center at Stanford, Stanford University, Stanford, United States of America
| |
Collapse
|
13
|
Affiliation(s)
- Jeffery T Leek
- Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD 21205, USA.
| | - Roger D Peng
- Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD 21205, USA
| |
Collapse
|
14
|
Samsa GP. Has It Really Been Demonstrated That Most Genomic Research Findings Are False? AM STAT 2015. [DOI: 10.1080/00031305.2014.951127] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
|