Effects of a proposal to alter the statistical significance threshold on previously published orthopaedic trauma randomized controlled trials

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote "statistical significance" and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

Lowering the statistical significance threshold of randomized controlled trials in three major general anesthesiology journals

PURPOSE

The primary objective of our study was to determine how lowering a P value threshold from 0.05 to 0.005 would affect the statistical significance of previously published randomized controlled trials (RCTs) in major anesthesiology journals.

METHODS

We searched the PubMed database for studies electronically published in 2020 within three major general anesthesiology journals as indexed by both Google Metrics and Scimago Journal & Country Rank. Studies included were RCTs published in 2020 in Anesthesiology, Anesthesia & Analgesia, and the British Journal of Anaesthesia; had a primary endpoint, and used a P value threshold to determine the effect of the intervention. We performed screening and data extraction in a masked duplicate fashion.

RESULTS

Ninety-one RCTs met inclusion criteria. The most frequently studied type of intervention was drugs (44/91, 48%). From the 91 trials, 99 primary endpoints, and thus P values, were obtained. Fifty-eight (59%) endpoints had a P value < 0.05 and 41 (41%) had a P value ≥ 0.05. Of the 58 primary endpoints previously considered statistically significant, 21 (36%) P values would maintain statistical significance at P < 0.005, and 37 (64%) would be reclassified as "suggestive."

CONCLUSIONS

Lowering a P value threshold of 0.05 to 0.005 would have altered one third of significance interpretations of RCTs in the surveyed anesthesiology literature. Thus, it is important for readers to consider post hoc probabilities when evaluating clinical trial results. Although the present study focused on the anesthesiology literature, we suggest that our results warrant further research within other fields of medicine to help avoid clinical misinterpretation of RCT findings and improve quality of care.

Minimum false-positive risk of primary outcomes and impact of reducing nominal P-value threshold from 0.05 to 0.005 in anaesthesiology randomised clinical trials: a cross-sectional study

BACKGROUND

Reproducibility of research is poor; this may be because many articles report statistically significant findings that are false positives. Two potential solutions are to lower the P-value for statistical significance testing from 0.05 to 0.005 and to report the minimum false-positive risk (minFPR). This study determined these metrics for randomised controlled trials (RCTs) in general anaesthesiology journals.

METHODS

We identified superiority RCTs published between January 1, 2019 and March 15, 2021 from seven leading anaesthesia journals. P-values for primary outcomes were collected, and minFPRs for these outcomes were calculated using a formula assuming a 50% prior probability of an intervention being effective (minFPR₅₀). The primary outcomes were the percentage of RCTs maintaining statistical significance at P<0.005 and minFPR₅₀.

RESULTS

We included 318 RCTs. P-values below 0.05 were reported in 205/318 (64%) of RCTs. Of these 205 RCTs, 119/205 (58%) maintained statistical significance at the P<0.005 threshold. The mean (standard deviation) minFPR₅₀ was 22% (20). At P=0.005, the minFPR₅₀ was approximately 5%.

CONCLUSIONS

These proposed metrics aimed at mitigating reproducibility concerns would call a significant portion of the anaesthesiology literature into question. We found a minFPR of 22% and determined that 42% of primary outcomes would not maintain statistical significance if the P-value threshold changed from 0.05 to 0.005. These findings could partially explain the lack of reproducibility of research findings.

Potential effects of lowering the threshold of statistical significance in the field of chronic rhinosinusitis - A meta-research on published randomized controlled trials over last decade

OBJECTIVE

To evaluate the impact of change in p-value threshold from 0.05 to 0.005, on published Randomized Controlled Trials (RCTs) over the last ten years in the field of chronic rhinosinusitis.

METHODS

A search of the PubMed database from 1st January 2011 to 31st December 2020 was conducted to include all RCTs that used p-value to determine the effects of an intervention. Data extracted included p-values, type of intervention, publishing journal with indexing, registration, funding, and multi- or single center status. The proportion of primary endpoints having p < 0.005 was determined first, followed by those with p > 0.005 but <0.05. Logistic regression analysis was used to determine if any trial characteristic was associated with reporting of significant p-values.

RESULTS

In total, 168 primary endpoints were identified from 123 RCTs. On analysis, 80 had a p-value <0.05, i.e., statistically significant on conventional parameter. Out of these, 53.75% had a p-value <0.005, which would retain significance under the proposed threshold, and the remaining 46.25% to be reclassified as "suggestive". None of the trial characteristics were contributing to reporting of conventional or proposed p-values on logistic regression analysis.

CONCLUSION

Lowering the p-value threshold would render 46.25% of a decade of published RCTs results (in the field of CRS) to be reclassified as merely "suggestive" and not significant. Trial characteristics were not found contributing to reporting of p-value <0.005 or even <0.05.

LEVEL OF EVIDENCE

II.

Impact of the p-Value Threshold on Interpretation of Trial Outcomes in Obstetrics and Gynecology

OBJECTIVE

Randomized controlled trials (RCTs) are considered the highest level of evidence to inform clinical practice. However, the reproducibility crisis has raised concerns about the scientific rigor of published RCT findings. Some advocate for a lower p-value threshold. We aimed to review published OB/Gyn topical RCTs in three representative OB/Gyn journals and three high impact non-OB/Gyn journals to determine if their interpretations would change with adoption of a p-value threshold for significance of 0.005. Secondarily, we evaluated if there were differences in methodologic characteristics between those that did and did not lose significance.

STUDY DESIGN

A manual search was performed to identify all OB/Gyn RCTs published in the selected journals between July 2017 and June 2019. Data were collected on primary outcome(s), methodology, and p-values. We determined the proportion of primary outcomes that would remain statistically significant with adoption of a p-value significance threshold of 0.005 versus be reinterpreted as "suggestive" (defined as p-value between 0.005 and 0.05). Chi-square or Fisher's exact test were used to compare study characteristics.

RESULTS

Overall, 202 RCTs met inclusion criteria; 52% in obstetrics and 48% in gynecology. Of 90 studies considered significant with p <0.05 at the time of publication, 54.4% (n = 49) would maintain significant (p < 0.005), while 45.6% (n = 41) would become suggestive using the lower threshold. Most RCTs utilized a single (90.1%) versus composite (8.9%) primary outcome type, used an intent-to-treat analysis (73.3%), and studied a drug intervention (46.5%). Methodologically, 23.7% did not prespecify analysis type, 28.2% did not meet the pre-determined sample size, and 9.4% did not report an a priori sample size calculation. Studies maintaining significance were more likely to be international and report a funding source.

CONCLUSION

Adopting a p-value significance threshold of 0.005 would require reinterpretation of almost half of RCT results in the OB/Gyn literature. Highly variable methodological quality was identified.

KEY POINTS

· New p-value threshold results in reinterpretation of nearly half of RCT results in OB/Gyn literature.. · Highly variable methodological quality was identified.. · Reduced use of binary interpretations of significance is necessary..

The Potential Effect of Lowering the Threshold of Statistical Significance From P < .05 to P < .005 in Orthopaedic Sports Medicine

PURPOSE

To determine how changing the P value threshold of statistical significance from .05 to .005 could affect the statistical significance of findings in previously published orthopaedic sports medicine randomized controlled trials (RCTs).

METHODS

The authors searched PubMed from January 1, 2016, to December 31, 2017, for RCTs published in the American Journal of Sports Medicine, Arthroscopy, and Knee Surgery, Sports Traumatology, Arthroscopy. Data were extracted blinded and in duplicate fashion by 2 of us. The authors then extracted P value data for primary end points, since RCTs are most often powered for these end points. Discrepancies were resolved by consensus. Google Forms were used for data extraction and STATA 15.1 for the data analysis.

RESULTS

In total, 275 primary end points were identified from 132 trials. Analysis of primary end points found 45.8% (126/275) had a P value less than .05 and were classified as statistically significant under the current threshold, whereas 54.2% (149/275) had a P value greater than .05 and were not classified as suggestive. Of those end points that were previously considered statistically significant, 38.9% (49/126) were less than .005, whereas 61.1% (77/126) were between .005 and .05 and thereby would be reclassified as suggestive rather than statistically significant under the proposed threshold. Overall, when analyzing the 275 primary end points, we found only 49 (17.8%) of the end points were less than .005 and would hold statistical significance with the proposed threshold.

CONCLUSIONS

The results suggest that if the threshold of statistical significance were to change to .005, the significance of orthopaedic sports medicine RCTs would be heavily altered. The authors also acknowledge the many issues research faces in regard to P value reliability and therefore interpretation of study results. Because P values from RCTs can often influence the ways physicians choose interventions, it is important to implement methodology that decreases incidence of bias and misrepresentation of these results. However, the authors also understand that lowering the P value could increase the needed sample size and by consequence increase study costs as well, while not directly correlating to clinical significance. Thus, the authors recommend that this proposed threshold should be further evaluated and cautiously interpreted.

CLINICAL RELEVANCE

If the statistical significance threshold is changed, clinical practice guideline recommendations also may be affected.