Misleading comparisons: the fallacy of comparing statistical significance

Sex contextualism in laboratory research: Enhancing rigor and precision in the study of sex-related variables

Understanding sex-related variation in health and illness requires rigorous and precise approaches to revealing underlying mechanisms. A first step is to recognize that sex is not in and of itself a causal mechanism; rather, it is a classification system comprising a set of categories, usually assigned according to a range of varying traits. Moving beyond sex as a system of classification to working with concrete and measurable sex-related variables is necessary for precision. Whether and how these sex-related variables matter-and what patterns of difference they contribute to-will vary in context-specific ways. Second, when researchers incorporate these sex-related variables into research designs, rigorous analytical methods are needed to allow strongly supported conclusions. Third, the interpretation and reporting of sex-related variation require care to ensure that basic and preclinical research advance health equity for all.

Evidence of subgroup differences in meta-analyses evaluating medications for alcohol use disorder: An umbrella review

Randomized controlled trials (RCTs) evaluating medications for alcohol use disorder (AUD) often examine heterogeneity of treatment effects through subgroup analyses that contrast effect estimates in groups of patients across individual demographic, clinical, and study design-related characteristics. However, these analyses are often not prespecified or adequately powered, highlighting the potential role of subgroup analyses in meta-analysis. Here, we conducted an umbrella review (i.e., a systematic review of meta-analyses) to determine the range and characteristics of reported subgroup analyses in meta-analyses of AUD medications. We searched PubMed to identify meta-analyses of RCTs evaluating medications for the management of AUD, alcohol abuse, or alcohol dependence in adults. We sought studies that measured drinking-related outcomes; quality of life, function, and rates of mortality; adverse events; and dropout. We considered meta-analyses that reported the results from formal subgroup analyses (comparing the summary effects across subgroup levels); summary effect estimates stratified across subgroup levels; and meta-regression, regression, or correlation-based subgroup analyses. We analyzed nine meta-analyses that included 61 formal subgroup analyses (median = 6 per meta-analysis), of which 33 (54%) were based on baseline participant-level and 28 (46%) were based on trial-level characteristics. Of the 58 subgroup analyses with either a p-value from a subgroup test or a statement by the authors that the subgroup analyses were not statistically significant, eight (14%) were statistically significant at the p < 0.05 level. Twelve meta-analyses reported the results of 102 meta-regression analyses, of which 25 (25%) identified statistically significant predictors of the relevant outcome of interest; nine (9%) were based on baseline participant-level and 93 (91%) were based on trial characteristics. Subgroup analyses across meta-analyses of AUD medications often focus on study-level characteristics, which may not be as clinically informative as subgroup analyses based on participant-level characteristics. Opportunities exist for future meta-analyses to standardize their subgroup methodology, focus on more clinically informative participant-level characteristics, and use predictive approaches to account for multiple relevant variables.

Why and How to Account for Sex and Gender in Brain and Behavioral Research

Long overlooked in neuroscience research, sex and gender are increasingly included as key variables potentially impacting all levels of neurobehavioral analysis. Still, many neuroscientists do not understand the difference between the terms "sex" and "gender," the complexity and nuance of each, or how to best include them as variables in research designs. This TechSights article outlines rationales for considering the influence of sex and gender across taxa, and provides technical guidance for strengthening the rigor and reproducibility of such analyses. This guidance includes the use of appropriate statistical methods for comparing groups as well as controls for key covariates of sex (e.g., total intracranial volume) and gender (e.g., income, caregiver stress, bias). We also recommend approaches for interpreting and communicating sex- and gender-related findings about the brain, which have often been misconstrued by neuroscientists and the lay public alike.

Considering Sex as a Variable at a Research University: Knowledge, Attitudes, and Practices

Biomedical research has a history of excluding females as research subjects, which threatens rigor, reproducibility, and inclusivity. In 2016, to redress this bias, the U.S. National Institutes of Health (NIH) implemented a policy requiring the consideration of sex as a biological variable (SABV) in all studies involving vertebrate animals, including humans. Unless strongly justified, females and males must be included in all studies and results reported disaggregated by sex. Recent evidence indicates, however, that misunderstandings of the policy and other significant barriers impede its implementation. To shed light on those barriers at our home institution, we conducted a study funded by the Emory University Specialized Center of Research Excellence on Sex Differences (SCORE). In semistructured interviews of Emory principal investigators in the biological sciences, we noted their knowledge of what the policy entails and why it was implemented, their attitudes toward it, and the extent to which it has or has not changed their research practices. Although attitudes toward SABV were generally positive, most researchers face challenges with respect to its implementation. We suggest interventions that can be mounted at the level of home institutions, such as raising awareness of locally available core facilities, to help address these challenges. More training is needed on what the policy asks of researchers, how sex is defined, the nonhormonal ways that sex differences can manifest, and best practices for statistical analysis of sex-based data. Home institutions may also want to explore ways to lessen the stress associated with rollout of SABV policy.

Graduate Student Literature Review: A systematic review on the associations between nonsteroidal anti-inflammatory drug use at the time of diagnosis and treatment of claw horn lameness in dairy cattle and lameness scores, algometer readings, and lying times

The objectives of this systematic review were to investigate the association between nonsteroidal anti-inflammatory drug (NSAID) use during the treatment of claw horn lameness in dairy cattle and locomotion score (LS), nociceptive threshold, and lying times. A total of 229 studies were initially identified and had their title and abstract screened. From this, we screened the full text of 23 articles, identifying 6 articles for inclusion in the systematic review. Of these 6, 5 reported LS, 2 reported nociceptor thresholds, and 1 reported lying times. The quality of evidence was assessed using a Cochrane risk-of-bias tool and CONSORT items reported for each included study. Due to heterogeneity between the studies, data were reported following Cochrane's Synthesis without meta-analysis guidelines. Identified heterogeneity between the studies included differences in LS systems and statistical analyses, length of time from enrollment to outcome reported, the NSAID used, concomitant treatments administered, and severity and chronicity of lameness. Recommendations are made with respect to consistency of LS reporting and analysis, along with improvements that may be noted with compulsory reporting guidelines. There were at least some concerns over the risk of bias in 4 of the studies, with risks of bias present in missing outcome data between the study groups. Within the 5 studies included with LS outcomes, there were 22 different pairwise comparisons with either NSAID or NSAID + block as the intervention, with measures of association with presence or absence of lameness as the outcome available for 20 of these comparisons. Animals in the NSAID intervention groups had a lower point estimate lameness risk than animals in the comparison groups in 3 of 8 and 9 of 14 analyses for LS outcomes <10 and ≥10 d post-treatment, respectively. However, there was no difference identified between animals in the NSAID intervention groups compared with the animals in the control group in any of these pairwise comparisons with lameness as the outcome. Twelve pairwise comparisons were reported in the 2 studies with nociceptor threshold as an outcome. Animals in the NSAID intervention groups had a greater nociceptor threshold point estimate compared with animals in the comparison groups in 6 of 6 and 1 of 6 analyses for outcomes <10 and ≥10 d post-treatment, respectively. However, no differences were identified between animals in the NSAID intervention groups and those in the comparison groups. All 4 pairwise comparisons reported in the study with lying times as an outcome found no differences between animals in the NSAID groups and those in the comparison groups. Despite the widespread use of NSAID in the treatment of claw horn lameness, there is a lack of studies of NSAID association with LS, nociceptive thresholds, or lying times. The limited evidence is consistent with no association with NSAID use and those parameters, but comparability across studies was limited by heterogeneity.

Subgroup Analysis in Pulmonary Hypertension-Specific Therapy Clinical Trials: A Systematic Review

Pulmonary hypertension (PH) treatment decisions are driven by the results of randomized controlled trials (RCTs). Subgroup analyses are often performed to assess whether the intervention effect will change due to the patient's characteristics, thus allowing for individualized decisions. This review aimed to evaluate the appropriateness and interpretation of subgroup analyses performed in PH-specific therapy RCTs published between 2000 and 2020. Claims of subgroup effects were evaluated with prespecified criteria. Overall, 30 RCTs were included. Subgroup analyses presented: a high number of subgroup analyses reported, lack of prespecification, and lack of interaction tests. The trial protocol was not available for most RCTs; significant differences were found in those articles that published the protocol. Authors reported 13 claims of subgroup effect, with 12 claims meeting four or fewer of Sun's criteria. Even when most RCTs were generally at low risk of bias and were published in high-impact journals, the credibility and general quality of subgroup analyses and subgroup claims were low due to methodological flaws. Clinicians should be skeptical of claims of subgroup effects and interpret subgroup analyses with caution, as due to their poor quality, these analyses may not serve as guidance for personalized care.

Adequate enrollment of women in cardiovascular drug trials and the need for sex-specific assessment and reporting

Cardiovascular disease (CVD) is the leading cause of death for women in the United States and globally. There is an abundance of evidence-based trials evaluating the efficacy of drug therapies to reduce morbidity and mortality in CVD. Additionally, there are well-established influences of sex, through a variety of mechanisms, on pharmacologic treatments in CVD. Despite this, the majority of drug trials are not powered to evaluate sex-specific outcomes, and much of the data that exists is gathered post hoc and through meta-analysis. The FDA established a committee in 1993 to increase the enrollment of women in clinical trials to improve this situation. Several authors, reviewing committees, and professional societies have highlighted the importance of sex-specific analysis and reporting. Despite these statements, there has not been a major improvement in representation or reporting. There are ongoing efforts to assess trial design, female representation on steering committees, and clinical trial processes to improve the representation of women. This review will describe the pharmacologic basis for the need for sex-specific assessment of cardiovascular drug therapies. It will also review the sex-specific reporting of landmark drug trials in hypertension, coronary artery disease (CAD), hyperlipidemia, and heart failure (HF). In reporting enrollment of women, several therapeutic areas like antihypertensives and newer anticoagulation trials fare better than therapeutics for HF and acute coronary syndromes. Further, drug trials and cardiometabolic or lifestyle intervention trials had a higher percentage of female participants than the device or procedural trials.

From Model Organisms to Humans, the Opportunity for More Rigor in Methodologic and Statistical Analysis, Design, and Interpretation of Aging and Senescence Research

This review identifies frequent design and analysis errors in aging and senescence research and discusses best practices in study design, statistical methods, analyses, and interpretation. Recommendations are offered for how to avoid these problems. The following issues are addressed: (a) errors in randomization, (b) errors related to testing within-group instead of between-group differences, (c) failing to account for clustering, (d) failing to consider interference effects, (e) standardizing metrics of effect size, (f) maximum life-span testing, (g) testing for effects beyond the mean, (h) tests for power and sample size, (i) compression of morbidity versus survival curve squaring, and (j) other hot topics, including modeling high-dimensional data and complex relationships and assessing model assumptions and biases. We hope that bringing increased awareness of these topics to the scientific community will emphasize the importance of employing sound statistical practices in all aspects of aging and senescence research.

Sex difference analyses under scrutiny

A survey reveals that many researchers do not use appropriate statistical analyses to evaluate sex differences in biomedical research.

Errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research: a guide to their avoidance

Randomization is an important tool used to establish causal inferences in studies designed to further our understanding of questions related to obesity and nutrition. To take advantage of the inferences afforded by randomization, scientific standards must be upheld during the planning, execution, analysis, and reporting of such studies. We discuss ten errors in randomized experiments from real-world examples from the literature and outline best practices for their avoidance. These ten errors include: representing nonrandom allocation as random, failing to adequately conceal allocation, not accounting for changing allocation ratios, replacing subjects in nonrandom ways, failing to account for non-independence, drawing inferences by comparing statistical significance from within-group comparisons instead of between-groups, pooling data and breaking the randomized design, failing to account for missing data, failing to report sufficient information to understand study methods, and failing to frame the causal question as testing the randomized assignment per se. We hope that these examples will aid researchers, reviewers, journal editors, and other readers to endeavor to a high standard of scientific rigor in randomized experiments within obesity and nutrition research.

Misleading Epidemiological and Statistical Evidence in the Presence of Simpson's Paradox: An Illustrative Study Using Simulated Scenarios of Observational Study Designs

This study empirically illustrates the mechanism by which epidemiological effect measures and statistical evidence can be misleading in the presence of Simpson's paradox and identify possible alternative methods of analysis to manage the paradox.

Three scenarios of observational study designs, including cross-sectional, cohort, and case-control approaches, are simulated. In each scenario, data are generated, and various methods of epidemiological and statistical analyses are undertaken to obtain empirical results that illustrate Simpson's paradox and mislead conclusions. Rational methods of analysis are also performed to illustrate how to avoid pitfalls and obtain valid results.

In the presence of Simpson's paradox, results from analyses in overall data contradict the findings from all subgroups of the same data. This paradox occurs when distributions of confounding characteristics are unequal in the groups being compared. Data analysis methods which do not take confounding factor into account, including epidemiological 2×2 table analysis, independent samples t-test, Wilcoxon rank-sum test, chi-square test, and univariable regression analysis, cannot manage the problem of Simpson's paradox and mislead research conclusions. Mantel-Haenszel procedure and multivariable regression methods are examples of rational analysis methods leading to valid results.

Therefore, Simpson's paradox arises as a consequence of extreme unequal distributions of a specific inherent characteristic in groups being compared. Analytical methods which take control of confounding effect must be applied to manage the paradox and obtain valid research evidence regarding the causal association.

Perspective: Limiting Dependence on Nonrandomized Studies and Improving Randomized Trials in Human Nutrition Research: Why and How

A large majority of human nutrition research uses nonrandomized observational designs, but this has led to little reliable progress. This is mostly due to many epistemologic problems, the most important of which are as follows: difficulty detecting small (or even tiny) effect sizes reliably for nutritional risk factors and nutrition-related interventions; difficulty properly accounting for massive confounding among many nutrients, clinical outcomes, and other variables; difficulty measuring diet accurately; and suboptimal research reporting. Tiny effect sizes and massive confounding are largely unfixable problems that narrowly confine the scenarios in which nonrandomized observational research is useful. Although nonrandomized studies and randomized trials have different priorities (assessment of long-term causality compared with assessment of treatment effects), the odds for obtaining reliable information with the former are limited. Randomized study designs should therefore largely replace nonrandomized studies in human nutrition research going forward. To achieve this, many of the limitations that have traditionally plagued most randomized trials in nutrition, such as small sample size, short length of follow-up, high cost, and selective reporting, among others, must be overcome. Pivotal megatrials with tens of thousands of participants and lifelong follow-up are possible in nutrition science with proper streamlining of operational costs. Fixable problems that have undermined observational research, such as dietary measurement error and selective reporting, need to be addressed in randomized trials. For focused questions in which dietary adherence is important to maximize, trials with direct observation of participants in experimental in-house settings may offer clean answers on short-term metabolic outcomes. Other study designs of randomized trials to consider in nutrition include registry-based designs and "N-of-1" designs. Mendelian randomization designs may also offer some more reliable leads for testing interventions in trials. Collectively, an improved randomized agenda may clarify many things in nutrition science that might never be answered credibly with nonrandomized observational designs.

Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials

Importance

Many published randomized clinical trials (RCTs) make claims for subgroup differences.

Objective

To evaluate how often subgroup claims reported in the abstracts of RCTs are actually supported by statistical evidence (P < .05 from an interaction test) and corroborated by subsequent RCTs and meta-analyses.

Data Sources

This meta-epidemiological survey examines data sets of trials with at least 1 subgroup claim, including Subgroup Analysis of Trials Is Rarely Easy (SATIRE) articles and Discontinuation of Randomized Trials (DISCO) articles. We used Scopus (updated July 2016) to search for English-language articles citing each of the eligible index articles with at least 1 subgroup finding in the abstract.

Study Selection

Articles with a subgroup claim in the abstract with or without evidence of statistical heterogeneity (P < .05 from an interaction test) in the text and articles attempting to corroborate the subgroup findings.

Data Extraction and Synthesis

Study characteristics of trials with at least 1 subgroup claim in the abstract were recorded. Two reviewers extracted the data necessary to calculate subgroup-level effect sizes, standard errors, and the P values for interaction. For individual RCTs and meta-analyses that attempted to corroborate the subgroup findings from the index articles, trial characteristics were extracted. Cochran Q test was used to reevaluate heterogeneity with the data from all available trials.

Main Outcomes and Measures

The number of subgroup claims in the abstracts of RCTs, the number of subgroup claims in the abstracts of RCTs with statistical support (subgroup findings), and the number of subgroup findings corroborated by subsequent RCTs and meta-analyses.

Results

Sixty-four eligible RCTs made a total of 117 subgroup claims in their abstracts. Of these 117 claims, only 46 (39.3%) in 33 articles had evidence of statistically significant heterogeneity from a test for interaction. In addition, out of these 46 subgroup findings, only 16 (34.8%) ensured balance between randomization groups within the subgroups (eg, through stratified randomization), 13 (28.3%) entailed a prespecified subgroup analysis, and 1 (2.2%) was adjusted for multiple testing. Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect. In addition, all effect sizes from meta-analyses were attenuated toward the null.

Conclusions and Relevance

A minority of subgroup claims made in the abstracts of RCTs are supported by their own data (ie, a significant interaction effect). For those that have statistical support (P < .05 from an interaction test), most fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing. Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.