1
|
Fitzpatrick BG, Gorman DM, Trombatore C. Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model. PLoS One 2024; 19:e0303262. [PMID: 38753677 PMCID: PMC11098386 DOI: 10.1371/journal.pone.0303262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2023] [Accepted: 04/23/2024] [Indexed: 05/18/2024] Open
Abstract
In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.
Collapse
Affiliation(s)
- Ben G. Fitzpatrick
- Department of Mathematics, Loyola Marymount University, Los Angeles, California, United States of America
- Tempest Technologies, Los Angeles, California, United States of America
| | - Dennis M. Gorman
- Department of Epidemiology & Biostatistics, School of Public Health, Texas A&M University, College Station, Texas, United States of America
| | - Caitlin Trombatore
- Department of Mathematics, Loyola Marymount University, Los Angeles, California, United States of America
| |
Collapse
|
2
|
Waters P, Rucker B, Love M, Vassar M. Lowering the statistical significance threshold of randomized controlled trials in three major general anesthesiology journals. Can J Anaesth 2023; 70:1441-1448. [PMID: 37561351 DOI: 10.1007/s12630-023-02529-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2022] [Revised: 01/09/2023] [Accepted: 01/10/2023] [Indexed: 08/11/2023] Open
Abstract
PURPOSE The primary objective of our study was to determine how lowering a P value threshold from 0.05 to 0.005 would affect the statistical significance of previously published randomized controlled trials (RCTs) in major anesthesiology journals. METHODS We searched the PubMed database for studies electronically published in 2020 within three major general anesthesiology journals as indexed by both Google Metrics and Scimago Journal & Country Rank. Studies included were RCTs published in 2020 in Anesthesiology, Anesthesia & Analgesia, and the British Journal of Anaesthesia; had a primary endpoint, and used a P value threshold to determine the effect of the intervention. We performed screening and data extraction in a masked duplicate fashion. RESULTS Ninety-one RCTs met inclusion criteria. The most frequently studied type of intervention was drugs (44/91, 48%). From the 91 trials, 99 primary endpoints, and thus P values, were obtained. Fifty-eight (59%) endpoints had a P value < 0.05 and 41 (41%) had a P value ≥ 0.05. Of the 58 primary endpoints previously considered statistically significant, 21 (36%) P values would maintain statistical significance at P < 0.005, and 37 (64%) would be reclassified as "suggestive." CONCLUSIONS Lowering a P value threshold of 0.05 to 0.005 would have altered one third of significance interpretations of RCTs in the surveyed anesthesiology literature. Thus, it is important for readers to consider post hoc probabilities when evaluating clinical trial results. Although the present study focused on the anesthesiology literature, we suggest that our results warrant further research within other fields of medicine to help avoid clinical misinterpretation of RCT findings and improve quality of care.
Collapse
Affiliation(s)
- Philo Waters
- Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA.
| | - Brayden Rucker
- Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
| | - Mitchell Love
- Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
| | - Matt Vassar
- Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
- Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK, USA
| |
Collapse
|
3
|
Chuang Z, Martin J, Shapiro J, Nguyen D, Neocleous P, Jones PM. Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study. Br J Anaesth 2023; 130:412-420. [PMID: 36503825 DOI: 10.1016/j.bja.2022.11.001] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2022] [Revised: 11/01/2022] [Accepted: 11/03/2022] [Indexed: 12/14/2022] Open
Abstract
BACKGROUND Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P-value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals. METHODS We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P-values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR50). The primary outcomes were the percentage of RCTs maintaining statistical significance at P<0.005 and minFPR50. RESULTS We included 318 RCTs. P-values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P<0.005 threshold. The mean (standard deviation) minFPR50 was 22% (20). At P=0.005, the minFPR50 was approximately 5%. CONCLUSIONS These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P-value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.
Collapse
Affiliation(s)
- Zachary Chuang
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada
| | - Janet Martin
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada; Department of Anesthesia & Perioperative Medicine, University of Western Ontario, London, ON, Canada; Department of Epidemiology & Biostatistics, University of Western Ontario, London, ON, Canada
| | - Jordan Shapiro
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada
| | - Derek Nguyen
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada
| | - Penelope Neocleous
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada
| | - Philip M Jones
- Schulich School of Medicine & Dentistry, University of Western Ontario, London, ON, Canada; Department of Anesthesia & Perioperative Medicine, University of Western Ontario, London, ON, Canada; Department of Epidemiology & Biostatistics, University of Western Ontario, London, ON, Canada.
| |
Collapse
|
4
|
Reddy AK, Scott JT, Joshua Stephens B, Patel A, Checketts JX, Stotler WM, Hawkins BJ, Vassar M. Evaluation of Proposed Protocol Changing Statistical Significance From 0.05 to 0.005 in Foot and Ankle Randomized Controlled Trials. J Foot Ankle Surg 2022; 61:925-926. [PMID: 35367112 DOI: 10.1053/j.jfas.2022.03.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 03/17/2022] [Indexed: 02/03/2023]
Affiliation(s)
- Arjun K Reddy
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK; Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK.
| | - Jared T Scott
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - B Joshua Stephens
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - Ashini Patel
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - Jake X Checketts
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - Wesley M Stotler
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - Bryan J Hawkins
- Department of Orthopedic Surgery, Oklahoma State University Medical Center, Tulsa, OK
| | - Matt Vassar
- Office of Medical Student Research, Oklahoma State University Center for Health Sciences, Tulsa, OK; Department of Psychiatry and Behavioral Sciences, Oklahoma State University Center for Health Sciences, Tulsa, OK
| |
Collapse
|
5
|
Potential effects of lowering the threshold of statistical significance in the field of chronic rhinosinusitis - A meta-research on published randomized controlled trials over last decade. Braz J Otorhinolaryngol 2021; 88 Suppl 5:S83-S89. [PMID: 35331655 PMCID: PMC9801036 DOI: 10.1016/j.bjorl.2021.11.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2021] [Revised: 10/01/2021] [Accepted: 11/10/2021] [Indexed: 01/03/2023] Open
Abstract
OBJECTIVE To evaluate the impact of change in p-value threshold from 0.05 to 0.005, on published Randomized Controlled Trials (RCTs) over the last ten years in the field of chronic rhinosinusitis. METHODS A search of the PubMed database from 1st January 2011 to 31st December 2020 was conducted to include all RCTs that used p-value to determine the effects of an intervention. Data extracted included p-values, type of intervention, publishing journal with indexing, registration, funding, and multi- or single center status. The proportion of primary endpoints having p < 0.005 was determined first, followed by those with p > 0.005 but <0.05. Logistic regression analysis was used to determine if any trial characteristic was associated with reporting of significant p-values. RESULTS In total, 168 primary endpoints were identified from 123 RCTs. On analysis, 80 had a p-value <0.05, i.e., statistically significant on conventional parameter. Out of these, 53.75% had a p-value <0.005, which would retain significance under the proposed threshold, and the remaining 46.25% to be reclassified as "suggestive". None of the trial characteristics were contributing to reporting of conventional or proposed p-values on logistic regression analysis. CONCLUSION Lowering the p-value threshold would render 46.25% of a decade of published RCTs results (in the field of CRS) to be reclassified as merely "suggestive" and not significant. Trial characteristics were not found contributing to reporting of p-value <0.005 or even <0.05. LEVEL OF EVIDENCE II.
Collapse
|
6
|
Bruno AM, Shea AE, Einerson BD, Metz TD, Allshouse AA, Scott JR, Blue NR. Impact of the p-Value Threshold on Interpretation of Trial Outcomes in Obstetrics and Gynecology. Am J Perinatol 2021; 38:1223-1230. [PMID: 34169500 PMCID: PMC8460605 DOI: 10.1055/s-0041-1731345] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
OBJECTIVE Randomized controlled trials (RCTs) are considered the highest level of evidence to inform clinical practice. However, the reproducibility crisis has raised concerns about the scientific rigor of published RCT findings. Some advocate for a lower p-value threshold. We aimed to review published OB/Gyn topical RCTs in three representative OB/Gyn journals and three high impact non-OB/Gyn journals to determine if their interpretations would change with adoption of a p-value threshold for significance of 0.005. Secondarily, we evaluated if there were differences in methodologic characteristics between those that did and did not lose significance. STUDY DESIGN A manual search was performed to identify all OB/Gyn RCTs published in the selected journals between July 2017 and June 2019. Data were collected on primary outcome(s), methodology, and p-values. We determined the proportion of primary outcomes that would remain statistically significant with adoption of a p-value significance threshold of 0.005 versus be reinterpreted as "suggestive" (defined as p-value between 0.005 and 0.05). Chi-square or Fisher's exact test were used to compare study characteristics. RESULTS Overall, 202 RCTs met inclusion criteria; 52% in obstetrics and 48% in gynecology. Of 90 studies considered significant with p <0.05 at the time of publication, 54.4% (n = 49) would maintain significant (p < 0.005), while 45.6% (n = 41) would become suggestive using the lower threshold. Most RCTs utilized a single (90.1%) versus composite (8.9%) primary outcome type, used an intent-to-treat analysis (73.3%), and studied a drug intervention (46.5%). Methodologically, 23.7% did not prespecify analysis type, 28.2% did not meet the pre-determined sample size, and 9.4% did not report an a priori sample size calculation. Studies maintaining significance were more likely to be international and report a funding source. CONCLUSION Adopting a p-value significance threshold of 0.005 would require reinterpretation of almost half of RCT results in the OB/Gyn literature. Highly variable methodological quality was identified. KEY POINTS · New p-value threshold results in reinterpretation of nearly half of RCT results in OB/Gyn literature.. · Highly variable methodological quality was identified.. · Reduced use of binary interpretations of significance is necessary..
Collapse
Affiliation(s)
- Ann M. Bruno
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT,Department of Obstetrics & Gynecology, Intermountain Healthcare, Salt Lake City, UT
| | - Ashley E. Shea
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT
| | - Brett D. Einerson
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT,Department of Obstetrics & Gynecology, Intermountain Healthcare, Salt Lake City, UT
| | - Torri D. Metz
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT,Department of Obstetrics & Gynecology, Intermountain Healthcare, Salt Lake City, UT
| | - Amanda A. Allshouse
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT
| | - James R. Scott
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT
| | - Nathan R. Blue
- Department of Obstetrics & Gynecology, University of Utah Health, Salt Lake City, UT,Department of Obstetrics & Gynecology, Intermountain Healthcare, Salt Lake City, UT
| |
Collapse
|
7
|
Evans S, Anderson JM, Johnson AL, Checketts JX, Scott J, Middlemist K, Fishbeck K, Vassar M. The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine. Arthroscopy 2021; 37:1068-1074. [PMID: 33253798 DOI: 10.1016/j.arthro.2020.11.041] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/12/2020] [Revised: 11/14/2020] [Accepted: 11/14/2020] [Indexed: 02/02/2023]
Abstract
PURPOSE To determine how changing the P value threshold of statistical significance from .05 to .005 could affect the statistical significance of findings in previously published orthopaedic sports medicine randomized controlled trials (RCTs). METHODS The authors searched PubMed from January 1, 2016, to December 31, 2017, for RCTs published in the American Journal of Sports Medicine, Arthroscopy, and Knee Surgery, Sports Traumatology, Arthroscopy. Data were extracted blinded and in duplicate fashion by 2 of us. The authors then extracted P value data for primary end points, since RCTs are most often powered for these end points. Discrepancies were resolved by consensus. Google Forms were used for data extraction and STATA 15.1 for the data analysis. RESULTS In total, 275 primary end points were identified from 132 trials. Analysis of primary end points found 45.8% (126/275) had a P value less than .05 and were classified as statistically significant under the current threshold, whereas 54.2% (149/275) had a P value greater than .05 and were not classified as suggestive. Of those end points that were previously considered statistically significant, 38.9% (49/126) were less than .005, whereas 61.1% (77/126) were between .005 and .05 and thereby would be reclassified as suggestive rather than statistically significant under the proposed threshold. Overall, when analyzing the 275 primary end points, we found only 49 (17.8%) of the end points were less than .005 and would hold statistical significance with the proposed threshold. CONCLUSIONS The results suggest that if the threshold of statistical significance were to change to .005, the significance of orthopaedic sports medicine RCTs would be heavily altered. The authors also acknowledge the many issues research faces in regard to P value reliability and therefore interpretation of study results. Because P values from RCTs can often influence the ways physicians choose interventions, it is important to implement methodology that decreases incidence of bias and misrepresentation of these results. However, the authors also understand that lowering the P value could increase the needed sample size and by consequence increase study costs as well, while not directly correlating to clinical significance. Thus, the authors recommend that this proposed threshold should be further evaluated and cautiously interpreted. CLINICAL RELEVANCE If the statistical significance threshold is changed, clinical practice guideline recommendations also may be affected.
Collapse
Affiliation(s)
- Sheridan Evans
- Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, U.S.A..
| | - J Michael Anderson
- Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, U.S.A
| | - Austin L Johnson
- Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, U.S.A
| | - Jake X Checketts
- Department of Orthopaedic Surgery, Oklahoma State University Medical Center, Tulsa, Oklahoma, U.S.A
| | - Jared Scott
- Department of Orthopaedic Surgery, Oklahoma State University Medical Center, Tulsa, Oklahoma, U.S.A
| | - Kevin Middlemist
- Department of Orthopaedic Surgery, Oklahoma State University Medical Center, Tulsa, Oklahoma, U.S.A
| | - Keith Fishbeck
- Department of Orthopaedic Surgery, Oklahoma State University Medical Center, Tulsa, Oklahoma, U.S.A
| | - Matt Vassar
- Oklahoma State University Center for Health Sciences, Tulsa, Oklahoma, U.S.A
| |
Collapse
|
8
|
Reito A. Consequences of altering statistical significance threshold in orthopaedic science. Injury 2020; 51:579. [PMID: 31735478 DOI: 10.1016/j.injury.2019.11.021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/09/2019] [Accepted: 11/11/2019] [Indexed: 02/02/2023]
Affiliation(s)
- Aleksi Reito
- Tampere University Hospital, Department of Orthopaedics, Faculty of Medicine and Health, Technology and Tampere University, Teiskontie 35, 33521 Tampere, Finland.
| |
Collapse
|