1
|
Hodges CB, Stone BM, Johnson PK, Carter JH, Sawyers CK, Roby PR, Lindsey HM. Researcher degrees of freedom in statistical software contribute to unreliable results: A comparison of nonparametric analyses conducted in SPSS, SAS, Stata, and R. Behav Res Methods 2023; 55:2813-2837. [PMID: 35953660 DOI: 10.3758/s13428-022-01932-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/11/2022] [Indexed: 11/08/2022]
Abstract
Researcher degrees of freedom can affect the results of hypothesis tests and consequently, the conclusions drawn from the data. Previous research has documented variability in accuracy, speed, and documentation of output across various statistical software packages. In the current investigation, we conducted Pearson's chi-square test of independence, Spearman's rank-ordered correlation, Kruskal-Wallis one-way analysis of variance, Wilcoxon Mann-Whitney U rank-sum tests, and Wilcoxon signed-rank tests, along with estimates of skewness and kurtosis, on large, medium, and small samples of real and simulated data in SPSS, SAS, Stata, and R and compared the results with those obtained through hand calculation using the raw computational formulas. Multiple inconsistencies were found in the results produced between statistical packages due to algorithmic variation, computational error, and statistical output. The most notable inconsistencies were due to algorithmic variations in the computation of Pearson's chi-square test conducted on 2 × 2 tables, where differences in p-values reported by different software packages ranged from .005 to .162, largely as a function of sample size. We discuss how such inconsistencies may influence the conclusions drawn from the results of statistical analyses depending on the statistical software used, and we urge researchers to analyze their data across multiple packages to check for inconsistencies and report details regarding the statistical procedure used for data analysis.
Collapse
Affiliation(s)
- Cooper B Hodges
- Department of Neurology, University of Utah School of Medicine, Salt Lake City, UT, USA
- Department of Psychology, Brigham Young University, Provo, UT, USA
- Department of Physical Medicine and Rehabilitation, Virginia Commonwealth University, Richmond, VA, USA
- School of Social and Behavioral Sciences, Andrews University, Berrien Springs, MI, USA
| | - Bryant M Stone
- Department of Psychology, Southern Illinois University, Carbondale, 1125 Lincoln Drive, Carbondale, IL, 62901, USA.
| | - Paula K Johnson
- Department of Neurology, University of Utah School of Medicine, Salt Lake City, UT, USA
- Neuroscience Center, Brigham Young University, Provo, UT, USA
| | - James H Carter
- Department of Psychology, Stanford University, Stanford, CA, USA
| | - Chelsea K Sawyers
- Department of Psychiatry, Virginia Commonwealth University, Richmond, VA, USA
| | - Patricia R Roby
- Center for Injury Research and Prevention, Children's Hospital of Philadelphia, Philadelphia, PA, USA
| | - Hannah M Lindsey
- Department of Neurology, University of Utah School of Medicine, Salt Lake City, UT, USA
- Department of Psychology, Brigham Young University, Provo, UT, USA
| |
Collapse
|