1
|
Curran-Everett D. Primer on reporting statistics: kayaks and walking trees once more. ADVANCES IN PHYSIOLOGY EDUCATION 2024; 48:1-2. [PMID: 37969076 DOI: 10.1152/advan.00231.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Revised: 11/10/2023] [Accepted: 11/10/2023] [Indexed: 11/17/2023]
Affiliation(s)
- Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health, Denver, Colorado, United States
| |
Collapse
|
2
|
Ekkekakis P, Swinton P, Tiller NB. Extraordinary Claims in the Literature on High-Intensity Interval Training (HIIT): I. Bonafide Scientific Revolution or a Looming Crisis of Replication and Credibility? Sports Med 2023; 53:1865-1890. [PMID: 37561389 DOI: 10.1007/s40279-023-01880-7] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/15/2023] [Indexed: 08/11/2023]
Abstract
The literature on high-intensity interval training (HIIT) contains claims that, if true, could revolutionize the science and practice of exercise. This critical analysis examines two varieties of claims: (i) HIIT is effective in improving various indices of fitness and health, and (ii) HIIT is as effective as more time-consuming moderate-intensity continuous exercise. Using data from two recent systematic reviews as working examples, we show that studies in both categories exhibit considerable weaknesses when judged through the prism of fundamental statistical principles. Predominantly, small-to-medium effects are investigated in severely underpowered studies, thus greatly increasing the risk of both type I and type II errors of statistical inference. Studies in the first category combine the volatility of estimates associated with small samples with numerous dependent variables analyzed without consideration of the inflation of the type I error rate. Studies in the second category inappropriately use the p > 0.05 criterion from small studies to support claims of 'similar' or 'comparable' effects. It is concluded that the situation in the HIIT literature is reminiscent of the research climate that led to the replication crisis in psychology. As in psychology, this could be an opportunity to reform statistical practices in exercise science.
Collapse
Affiliation(s)
- Panteleimon Ekkekakis
- Department of Kinesiology, Michigan State University, 308 W Circle Dr #134, East Lansing, MI, 48824, USA.
| | - Paul Swinton
- School of Health Sciences, Robert Gordon University, Aberdeen, Scotland, UK
| | - Nicholas B Tiller
- The Lundquist Institute for Biomedical Innovation at Harbor-UCLA Medical Center, Torrance, CA, USA
| |
Collapse
|
3
|
Gandevia S, Cumming G, Amrhein V, Butler A. Replication: Do not trust your p-value, be it small or large. J Physiol 2021; 599:2989-2990. [PMID: 33963767 DOI: 10.1113/jp281614] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022] Open
Affiliation(s)
- Simon Gandevia
- Neuroscience Research Australia, Sydney, NSW, 2031, Australia
| | - Geoff Cumming
- School of Psychology and Public Health, La Trobe University, Melbourne, Victoria, 3086, Australia
| | - Valentin Amrhein
- Department of Environmental Sciences, Zoology, University of Basel, Basel, Switzerland
| | - Annie Butler
- Neuroscience Research Australia, Sydney, NSW, 2031, Australia
| |
Collapse
|
4
|
Smalheiser NR, Graetz EE, Yu Z, Wang J. Effect size, sample size and power of forced swim test assays in mice: Guidelines for investigators to optimize reproducibility. PLoS One 2021; 16:e0243668. [PMID: 33626103 PMCID: PMC7904226 DOI: 10.1371/journal.pone.0243668] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Accepted: 02/10/2021] [Indexed: 11/19/2022] Open
Abstract
A recent flood of publications has documented serious problems in scientific reproducibility, power, and reporting of biomedical articles, yet scientists persist in their usual practices. Why? We examined a popular and important preclinical assay, the Forced Swim Test (FST) in mice used to test putative antidepressants. Whether the mice were assayed in a naïve state vs. in a model of depression or stress, and whether the mice were given test agents vs. known antidepressants regarded as positive controls, the mean effect sizes seen in the experiments were indeed extremely large (1.5-2.5 in Cohen's d units); most of the experiments utilized 7-10 animals per group which did have adequate power to reliably detect effects of this magnitude. We propose that this may at least partially explain why investigators using the FST do not perceive intuitively that their experimental designs fall short-even though proper prospective design would require ~21-26 animals per group to detect, at a minimum, large effects (0.8 in Cohen's d units) when the true effect of a test agent is unknown. Our data provide explicit parameters and guidance for investigators seeking to carry out prospective power estimation for the FST. More generally, altering the real-life behavior of scientists in planning their experiments may require developing educational tools that allow them to actively visualize the inter-relationships among effect size, sample size, statistical power, and replicability in a direct and intuitive manner.
Collapse
Affiliation(s)
- Neil R. Smalheiser
- Department of Psychiatry, University of Illinois School of Medicine, Chicago, Illinois, United States of America
| | - Elena E. Graetz
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| | - Zhou Yu
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| | - Jing Wang
- Department of Mathematics, Statistics, and Computer Science, University of Illinois at Chicago, Chicago, Illinois, United States of America
| |
Collapse
|
5
|
Gandevia S. Publications, replication and statistics in physiology plus two neglected curves. J Physiol 2021; 599:1719-1721. [DOI: 10.1113/jp281360] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022] Open
Affiliation(s)
- Simon Gandevia
- Neuroscience Research Australia Sydney New South Wales Australia
- University of New South Wales Sydney Australia
| |
Collapse
|
6
|
Curran-Everett D. Evolution in statistics: P values, statistical significance, kayaks, and walking trees. ADVANCES IN PHYSIOLOGY EDUCATION 2020; 44:221-224. [PMID: 32412384 DOI: 10.1152/advan.00054.2020] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Affiliation(s)
- Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health, Denver, Colorado
- Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Denver, Denver, Colorado
| |
Collapse
|
7
|
McLoughlin P, Drummond G. Publishing replication studies to support excellence in physiological research. Exp Physiol 2018; 102:1041-1043. [PMID: 28856808 DOI: 10.1113/ep086344] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
Affiliation(s)
- Paul McLoughlin
- School of Medicine and Conway Institute, University College, Dublin, Ireland
| | - Gordon Drummond
- Department of Anaesthesia, Critical Care and Pain Medicine, University of Edinburgh, UK
| |
Collapse
|
8
|
Lindsey ML, Gray GA, Wood SK, Curran-Everett D. Statistical considerations in reporting cardiovascular research. Am J Physiol Heart Circ Physiol 2018; 315:H303-H313. [PMID: 30028200 PMCID: PMC6139626 DOI: 10.1152/ajpheart.00309.2018] [Citation(s) in RCA: 56] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
The problem of inadequate statistical reporting is long standing and widespread in the biomedical literature, including in cardiovascular physiology. Although guidelines for reporting statistics have been available in clinical medicine for some time, there are currently no guidelines specific to cardiovascular physiology. To assess the need for guidelines, we determined the type and frequency of statistical tests and procedures currently used in the American Journal of Physiology-Heart and Circulatory Physiology. A PubMed search for articles published in the American Journal of Physiology-Heart and Circulatory Physiology between January 1, 2017, and October 6, 2017, provided a final sample of 146 articles evaluated for methods used and 38 articles for indepth analysis. The t-test and ANOVA accounted for 71% (212 of 300 articles) of the statistical tests performed. Of six categories of post hoc tests, Bonferroni and Tukey tests were used in 63% (62 of 98 articles). There was an overall lack in details provided by authors publishing in the American Journal of Physiology-Heart and Circulatory Physiology, and we compiled a list of recommended minimum reporting guidelines to aid authors in preparing manuscripts. Following these guidelines could substantially improve the quality of statistical reports and enhance data rigor and reproducibility.
Collapse
Affiliation(s)
- Merry L Lindsey
- Mississippi Center for Heart Research, Department of Physiology and Biophysics, University of Mississippi Medical Center , Jackson, Mississippi.,Research Service, G. V. (Sonny) Montgomery Veterans Affairs Medical Center , Jackson, Mississippi
| | - Gillian A Gray
- British Heart Foundation/University Centre for Cardiovascular Science, Edinburgh Medical School, University of Edinburgh , Edinburgh , United Kingdom
| | - Susan K Wood
- Department of Pharmacology, Physiology and Neuroscience, University of South Carolina School of Medicine , Columbia, South Carolina
| | - Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health , Denver, Colorado.,Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Denver , Denver, Colorado
| |
Collapse
|
9
|
Curran-Everett D. Explorations in statistics: the log transformation. ADVANCES IN PHYSIOLOGY EDUCATION 2018; 42:343-347. [PMID: 29761718 DOI: 10.1152/advan.00018.2018] [Citation(s) in RCA: 54] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
Learning about statistics is a lot like learning about science: the learning is more meaningful if you can actively explore. This thirteenth installment of Explorations in Statistics explores the log transformation, an established technique that rescales the actual observations from an experiment so that the assumptions of some statistical analysis are better met. A general assumption in statistics is that the variability of some response Y is homogeneous across groups or across some predictor variable X. If the variability-the standard deviation-varies in rough proportion to the mean value of Y, a log transformation can equalize the standard deviations. Moreover, if the actual observations from an experiment conform to a skewed distribution, then a log transformation can make the theoretical distribution of the sample mean more consistent with a normal distribution. This is important: the results of a one-sample t test are meaningful only if the theoretical distribution of the sample mean is roughly normal. If we log-transform our observations, then we want to confirm the transformation was useful. We can do this if we use the Box-Cox method, if we bootstrap the sample mean and the statistic t itself, and if we assess the residual plots from the statistical model of the actual and transformed sample observations.
Collapse
Affiliation(s)
- Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health , Denver, Colorado
- Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Denver , Denver, Colorado
| |
Collapse
|
10
|
Curran-Everett D. Explorations in statistics: the assumption of normality. ADVANCES IN PHYSIOLOGY EDUCATION 2017; 41:449-453. [PMID: 28743689 DOI: 10.1152/advan.00064.2017] [Citation(s) in RCA: 18] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/08/2017] [Revised: 06/26/2017] [Accepted: 06/26/2017] [Indexed: 06/07/2023]
Abstract
Learning about statistics is a lot like learning about science: the learning is more meaningful if you can actively explore. This twelfth installment of Explorations in Statistics explores the assumption of normality, an assumption essential to the meaningful interpretation of a t test. Although the data themselves can be consistent with a normal distribution, they need not be. Instead, it is the theoretical distribution of the sample mean or the theoretical distribution of the difference between sample means that must be roughly normal. The most versatile approach to assess normality is to bootstrap the sample mean, the difference between sample means, or t itself. We can then assess whether the distributions of these bootstrap statistics are consistent with a normal distribution by studying their normal quantile plots. If we suspect that an inference we make from a t test may not be justified-if we suspect that the theoretical distribution of the sample mean or the theoretical distribution of the difference between sample means is not normal-then we can use a permutation method to analyze our data.
Collapse
Affiliation(s)
- Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health, Denver, Colorado; and Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Denver, Denver, Colorado
| |
Collapse
|
11
|
Curran-Everett D. CORP: Minimizing the chances of false positives and false negatives. J Appl Physiol (1985) 2017; 122:91-95. [DOI: 10.1152/japplphysiol.00937.2016] [Citation(s) in RCA: 25] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2016] [Revised: 11/15/2016] [Accepted: 11/23/2016] [Indexed: 11/22/2022] Open
Abstract
Statistics is essential to the process of scientific discovery. An inescapable tenet of statistics, however, is the notion of uncertainty which has reared its head within the arena of reproducibility of research. The Journal of Applied Physiology’s recent initiative, “Cores of Reproducibility in Physiology,” is designed to improve the reproducibility of research: each article is designed to elucidate the principles and nuances of using some piece of scientific equipment or some experimental technique so that other researchers can obtain reproducible results. But other researchers can use some piece of equipment or some technique with expert skill and still fail to replicate an experimental result if they neglect to consider the fundamental concepts of statistics of hypothesis testing and estimation and their inescapable connection to the reproducibility of research. If we want to improve the reproducibility of our research, then we want to minimize the chance that we get a false positive and—at the same time—we want to minimize the chance that we get a false negative. In this review I outline strategies to accomplish each of these things. These strategies are related intimately to fundamental concepts of statistics and the inherent uncertainty embedded in them.
Collapse
Affiliation(s)
- Douglas Curran-Everett
- Division of Biostatistics and Bioinformatics, National Jewish Health, Denver, Colorado; and Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Denver, Denver, Colorado
| |
Collapse
|