1
|
Kroeger CM, Hannon BA, Halliday TM, Ejima K, Teran-Garcia M, Brown AW. Evidence of misuse of nonparametric tests in the presence of heteroscedasticity within obesity research. F1000Res 2021; 10:391. [PMID: 35136571 PMCID: PMC8792877 DOI: 10.12688/f1000research.52693.1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 05/06/2021] [Indexed: 02/02/2023] Open
Abstract
BACKGROUND Classic nonparametric tests (cNPTs), like Kruskal-Wallis or Mann-Whitney U, are sometimes used to detect differences in central tendency ( i.e., means or medians). However, when the tests' assumptions are violated, such as in the presence of unequal variance and other forms of heteroscedasticity, they are no longer valid for testing differences in central tendency. Yet, sometimes researchers erroneously use cNPTs to account for heteroscedasticity. OBJECTIVE To document the appropriateness of cNPT use in obesity literature, characterize studies that use cNPTs, and evaluate the citation and public sharing patterns of these articles. METHODS We reviewed obesity studies published in 2017 to determine whether the authors used cNPTs: (1) to correct for heteroscedasticity (invalid); (2) when heteroscedasticity was clearly not present (correct); or (3) when it was unclear whether heteroscedasticity was present (unclear). Open science R packages were used to transparently search literature and extract data on how often papers with errors have been cited in academic literature, read in Mendeley, and disseminated in the media. RESULTS We identified nine studies that used a cNPT in the presence of heteroscedasticity (some because of the mistaken rationale that the test corrected for heteroscedasticity), 25 articles that did not explicitly state whether heteroscedasticity was present when a cNPT was used, and only four articles that appropriately reported that heteroscedasticity was not present when a cNPT was used. Errors were found in observational and interventional studies, in human and rodent studies, and only when studies were unregistered. Studies with errors have been cited 113 times, read in Mendeley 123 times, and disseminated in the media 41 times, by the public, scientists, science communicators, and doctors. CONCLUSIONS Examples of inappropriate use of cNPTs exist in the obesity literature, and those articles perpetuate the errors via various audiences and dissemination platforms.
Collapse
Affiliation(s)
- Cynthia M Kroeger
- Charles Perkins Centre, Central Clinical School, The University of Sydney, Sydney, NSW, Australia
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health Bloomington, Bloomington, Indiana, USA
| | - Bridget A Hannon
- Division of Nutritional Sciences, University of Illinois at Urbana-Champaign, Urbana, IL, USA
- Abbott Nutrition, Columbus, OH, USA
| | - Tanya M Halliday
- Department of Health and Kinesiology, University of Utah, Salt Lake City, UT, USA
| | - Keisuke Ejima
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health Bloomington, Bloomington, Indiana, USA
| | - Margarita Teran-Garcia
- Division of Nutritional Sciences, University of Illinois at Urbana-Champaign, Urbana, IL, USA
- University of Illinois Extension, Carle Illinois College of Medicine, University of Illinois at Urbana-Champaign, Urbana, IL, USA
| | - Andrew W Brown
- Department of Applied Health Science, Indiana University School of Public Health Bloomington, Bloomington, IN, USA
| |
Collapse
|
2
|
Kroeger CM, Ejima K, Hannon BA, Halliday TM, McComb B, Teran-Garcia M, Dawson JA, King DB, Brown AW, Allison DB. Persistent confusion in nutrition and obesity research about the validity of classic nonparametric tests in the presence of heteroscedasticity: evidence of the problem and valid alternatives. Am J Clin Nutr 2021; 113:517-524. [PMID: 33515017 PMCID: PMC7948897 DOI: 10.1093/ajcn/nqaa357] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2019] [Accepted: 11/06/2020] [Indexed: 12/22/2022] Open
Abstract
The use of classic nonparametric tests (cNPTs), such as the Kruskal-Wallis and Mann-Whitney U tests, in the presence of unequal variance for between-group comparisons of means and medians may lead to marked increases in the rate of falsely rejecting null hypotheses and decreases in statistical power. Yet, this practice remains prevalent in the scientific literature, including nutrition and obesity literature. Some nutrition and obesity studies use a cNPT in the presence of unequal variance (i.e., heteroscedasticity), sometimes because of the mistaken rationale that the test corrects for heteroscedasticity. Herein, we discuss misconceptions of using cNPTs in the presence of heteroscedasticity. We then discuss assumptions, purposes, and limitations of 3 common tests used to test for mean differences between multiple groups, including 2 parametric tests: Fisher's ANOVA and Welch's ANOVA; and 1 cNPT: the Kruskal-Wallis test. To document the impact of heteroscedasticity on the validity of these tests under conditions similar to those used in nutrition and obesity research, we conducted simple simulations and assessed type I error rates (i.e., false positives, defined as incorrectly rejecting the null hypothesis). We demonstrate that type I error rates for Fisher's ANOVA, which does not account for heteroscedasticity, and Kruskal-Wallis, which tests for differences in distributions rather than means, deviated from the expected significance level. Greater deviation from the expected type I error rate was observed as the heterogeneity increased, especially in the presence of an imbalanced sample size. We provide brief tutorial guidance for authors, editors, and reviewers to identify appropriate statistical tests when test assumptions are violated, with a particular focus on cNPTs.
Collapse
Affiliation(s)
- Cynthia M Kroeger
- Charles Perkins Centre, Central Clinical School, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Keisuke Ejima
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Bridget A Hannon
- Division of Nutritional Sciences, University of Illinois, Urbana-Champaign, Urbana, IL, USA
| | - Tanya M Halliday
- Department of Health, Kinesiology, and Recreation, University of Utah, Salt Lake City, UT, USA
| | - Bryan McComb
- Division of Biostatistics, New York University School of Medicine, New York, NY, USA
| | - Margarita Teran-Garcia
- Division of Nutritional Sciences, University of Illinois, Urbana-Champaign, Urbana, IL, USA
| | - John A Dawson
- Department of Nutritional Sciences, Texas Tech University, Lubbock, TX, USA
| | - David B King
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - Andrew W Brown
- Department of Applied Health Science, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| | - David B Allison
- Department of Epidemiology and Biostatistics, Indiana University School of Public Health-Bloomington, Bloomington, IN, USA
| |
Collapse
|
3
|
Spence ND, Skelton JA, Ball GDC. A proposed standardized approach to studying attrition in pediatric weight management. Obes Res Clin Pract 2019; 14:60-65. [PMID: 31818674 DOI: 10.1016/j.orcp.2019.11.004] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/18/2018] [Revised: 07/29/2019] [Accepted: 11/20/2019] [Indexed: 11/19/2022]
Abstract
Pediatric obesity is a major public health issue. Lifestyle and behavioral interventions are the foundation of pediatric weight management; however, intervention effectiveness is compromised when families (children, youth, and/or parent[s]) discontinue care prematurely. Intervention attrition minimizes the potential health benefits derived from interventions, results in inefficient use of health services resources, and can magnify health disparities. Most attrition research in pediatric weight management has been descriptive, highlighting the need to advance the field, both academically and clinically. Herein, we propose a standard approach to studying attrition in pediatric weight management interventions to enhance our understanding, elevate the quality of research, enable study-to-study comparisons, and inform strategies designed to mitigate its impact. We focus on three issues. First, "Conceptualization and operationalization," whereby the processes underlying attrition from interventions should be decomposed into clinically important phases that are defined based on intervention characteristics. Relatedly, theoretically relevant variables should be identified with different mechanisms driving attrition in each phase. We propose a matrix of attrition, a tool designed to delineate the relevant stages of attrition and associated variables of analytical value. Second, "Pre-study" underscores the value of developing a plan to study attrition a priori rather than post hoc, including variable and sample size considerations, which broadens the range and quality of analysis. Finally, "Post-study" emphasizes comprehensive reporting of attrition, outlines typical comparisons of analytical interest, and statistical techniques used to handle missing data. Implications for clinical practice in pediatric weight management are discussed. Clinical Trial Registration: Not applicable.
Collapse
Affiliation(s)
- Nicholas D Spence
- Department of Sociology, University of Toronto, Toronto, ON, Canada; Interdisciplinary Center for Health and Society, University of Toronto, Toronto, ON, Canada
| | - Joseph A Skelton
- Department of Pediatrics, Wake Forest School of Medicine, Medical Center Boulevard, Winston-Salem, NC, United States; Brenner FIT (Families in Training) Program, Brenner Children's Hospital, Medical Center Boulevard, Winston-Salem, NC, United States; Division of Public Health Sciences, Department of Epidemiology and Prevention, Wake Forest School of Medicine, Winston-Salem, NC, United States
| | - Geoff D C Ball
- Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada; Pediatric Centre for Weight & Health, Stollery Children's Hospital, Alberta Health Services, Edmonton, AB, Canada.
| |
Collapse
|
4
|
Reprint of "Methodological and reporting quality in laboratory studies of human eating behavior". Appetite 2018; 130:321-326. [DOI: 10.1016/j.appet.2018.06.037] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2017] [Revised: 02/07/2018] [Accepted: 02/08/2018] [Indexed: 01/16/2023]
|
5
|
Ball GDC, Kebbe M. Preventing and managing paediatric obesity: a special edition on randomized controlled trials. Pediatr Obes 2018; 13:635-638. [PMID: 30362290 DOI: 10.1111/ijpo.12482] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/06/2018] [Accepted: 09/08/2018] [Indexed: 12/18/2022]
Affiliation(s)
- G D C Ball
- Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada
| | - M Kebbe
- Department of Pediatrics, Faculty of Medicine & Dentistry, University of Alberta, Edmonton, AB, Canada
| |
Collapse
|
6
|
Trinquart L, Dunn AG, Bourgeois FT. Registration of published randomized trials: a systematic review and meta-analysis. BMC Med 2018; 16:173. [PMID: 30322399 PMCID: PMC6190546 DOI: 10.1186/s12916-018-1168-6] [Citation(s) in RCA: 44] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/21/2018] [Accepted: 09/07/2018] [Indexed: 12/19/2022] Open
Abstract
BACKGROUND Prospective trial registration is a powerful tool to prevent reporting bias. We aimed to determine the extent to which published randomized controlled trials (RCTs) were registered and registered prospectively. METHODS We searched MEDLINE and EMBASE from January 2005 to October 2017; we also screened all articles cited by or citing included and excluded studies, and the reference lists of related reviews. We included studies that examined published RCTs and evaluated their registration status, regardless of medical specialty or language. We excluded studies that assessed RCT registration status only through mention of registration in the published RCT, without searching registries or contacting the trial investigators. Two independent reviewers blinded to the other's work performed the selection. Following PRISMA guidelines, two investigators independently extracted data, with discrepancies resolved by consensus. We calculated pooled proportions and 95% confidence intervals using random-effects models. RESULTS We analyzed 40 studies examining 8773 RCTs across a wide range of clinical specialties. The pooled proportion of registered RCTs was 53% (95% confidence interval 44% to 58%), with considerable between-study heterogeneity. A subset of 24 studies reported data on prospective registration across 5529 RCTs. The pooled proportion of prospectively registered RCTs was 20% (95% confidence interval 15% to 25%). Subgroup analyses showed that registration was higher for industry-supported and larger RCTs. A meta-regression analysis across 19 studies (5144 RCTs) showed that the proportion of registered trials significantly increased over time, with a mean proportion increase of 27%, from 25 to 52%, between 2005 and 2015. CONCLUSIONS The prevalence of trial registration has increased over time, but only one in five published RCTs is prospectively registered, undermining the validity and integrity of biomedical research.
Collapse
Affiliation(s)
- Ludovic Trinquart
- Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts USA
| | - Adam G. Dunn
- Centre for Health Informatics, Australian Institute of Health Innovation, Macquarie University, Sydney, Australia
| | - Florence T. Bourgeois
- Department of Pediatrics, Harvard Medical School, Boston, Massachusetts USA
- Center for Pediatric Therapeutics and Regulatory Science, and Computational Health Informatics Program, Boston Children’s Hospital, Boston, MA USA
| |
Collapse
|
7
|
Robinson E, Bevelander KE, Field M, Jones A. Methodological and reporting quality in laboratory studies of human eating behavior. Appetite 2018; 125:486-491. [PMID: 29452224 PMCID: PMC5890731 DOI: 10.1016/j.appet.2018.02.008] [Citation(s) in RCA: 36] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2017] [Revised: 02/07/2018] [Accepted: 02/08/2018] [Indexed: 01/31/2023]
Abstract
The methodological quality and reporting practices of laboratory studies of human eating behavior determine the validity and replicability of nutrition science. The aim of this research was to examine basic methodology and reporting practices in recent representative laboratory studies of human eating behavior. We examined laboratory studies of human eating behavior (N = 140 studies) published during 2016. Basic methodology (e.g., sample size, use of participant blinding) and reporting practices (e.g., information on participant characteristics) were assessed for each study. Some information relating to participant characteristics (e.g., age, gender) and study methodology (e.g., length of washout periods in within-subjects studies) were reported in the majority of studies. However, other aspects of study reporting, including participant eligibility criteria and how sample size was determined were frequently not reported. Studies often did not appear to standardize pre-test meal appetite or attempt to blind participants to study aims. The average sample size of studies was small (between-subjects design studies in particular) and the primary statistical analyses in a number of studies (24%) were reliant on very small sample sizes that would be likely to produce unreliable results. There are basic methodology and reporting practices in the laboratory study of human eating behavior that are sub-optimal and this is likely to be affecting the validity and replicability of research. Recommendations to address these issues are discussed.
Collapse
Affiliation(s)
- Eric Robinson
- Institute of Psychology, Health & Society, University of Liverpool, L69 7ZA, UK.
| | - Kirsten E Bevelander
- Behavioural Science Institute, Communication Science, Radboud University, Nijmegen, The Netherlands
| | - Matt Field
- Institute of Psychology, Health & Society, University of Liverpool, L69 7ZA, UK
| | - Andrew Jones
- Institute of Psychology, Health & Society, University of Liverpool, L69 7ZA, UK
| |
Collapse
|