Big little lies: a compendium and simulation of p-hacking strategies

Evaluating nomogram models for predicting survival outcomes in gastric gastrointestinal stromal tumors with SEER database analysis

Gastrointestinal stromal tumors (GISTs) predominantly develop in the stomach. While nomogram offer tremendous therapeutic promise, there is yet no ideal nomogram comparison customized specifically for handling categorical data and model selection related gastric GISTs. (1) We selected 5463 patients with gastric GISTs from the SEER Research Plus database spanning from 2000 to 2020; (2) We proposed an advanced missing data imputation algorithm specifically designed for categorical variables; (3) We constructed five Cox nomogram models, each employing distinct methods for the selection and modeling of categorical variables, including Cox (Two-Stage), Lasso-Cox, Ridge-Cox, Elastic Net-Cox, and Cox With Lasso; (4) We conducted a comprehensive comparison of both overall survival (OS) and cancer-specific survival (CSS) tasks at six different time points; (5) To ensure robustness, we performed 50 randomized splits for each task, maintaining a 7:3 ratio between the training and test cohorts with no discernible statistical differences. Among the five models, the Cox (Two-Stage) nomogram contains the fewest features. Notably, at Near-term, Mid-term, and Long-term intervals, the Cox (Two-Stage) model attains the highest Area Under the Curve (AUC), top-1 ratio, and top-3 ratio in both OS and CSS tasks. For the prediction of survival in patients with gastric GISTs, the Cox (Two-Stage) nomogram stands as a simple, stable, and accurate predictive model with substantial promise for clinical application. To enhance the clinical utility and accessibility of our findings, we have deployed the nomogram model online, allowing healthcare professionals and researchers worldwide to access and utilize this predictive tool.

Detection of PRRSV-1 in tongue fluids under experimental and field conditions and comparison of different sampling material for PRRSV sow herd monitoring

BACKGROUND

Infection with porcine reproductive and respiratory syndrome virus (PRRSV) leads to significant economic losses worldwide. One of the initial measures following an outbreak is to stabilise the herd and to prevent vertical transmission of PRRSV. The objective of this study was to detect PRRSV in different sampling material, both in an experimental model and on a commercial piglet producing farm, with a focus on evaluating the suitability of tongue fluid samples.

RESULTS

In the experimental model, PRRSV negative pregnant gilts were infected with PRRSV-1 AUT15-33 on gestation day 85 and necropsy of gilts and foetuses was performed three weeks later. 38.3% of individual foetal serum and 39.4% of individual foetal thymus samples were considered PRRSV RT-qPCR positive. Tongue fluids from individual foetuses showed a 33.0% positivity rate. PRRSV RNA was detected in all but one sample of litter-wise pooled processing fluids and tongue fluids. In the field study, the investigated farm remained PRRSV positive and unstable for five consecutive farrowing groups after the start of the sampling process. Tongue fluid samples pooled by litter in the first investigated farrowing group had a 54.5% positivity rate, with the overall highest viral load obtained in the field study. In this farrowing group, 33.3% of investigated litter-wise pooled processing fluid samples and all investigated serum samples (pools of 4-6 individuals, two piglets per litter) were considered positive. Across all investigated farrowing groups, tongue fluid samples consistently showed the highest viral load. Moreover, tongue fluid samples contained the virus in moderate amounts for the longest time compared to the other investigated sampling material.

CONCLUSION

It can be concluded that the viral load in individual foetuses is higher in serum or thymus compared to tongue fluid samples. However, litter-wise pooled tongue fluid samples are well-suited for detecting vertical transmission within the herd, even when the suspected prevalence of vertical transmission events is low.

P-hacking in meta-analyses: A formalization and new meta-analytic methods

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.

Demonstrating the utility of Instrumented Gait Analysis in the treatment of children with cerebral palsy

BACKGROUND

Instrumented gait analysis (IGA) has been around for a long time but has never been shown to be useful for improving patient outcomes. In this study we demonstrate the potential utility of IGA by showing that machine learning models are better able to estimate treatment outcomes when they include both IGA and clinical (CLI) features compared to when they include CLI features alone.

DESIGN

We carried out a retrospective analysis of data from ambulatory children diagnosed with cerebral palsy who were seen at least twice at our gait analysis center. Individuals underwent a variety of treatments (including no treatment) between sequential gait analyses. We fit Bayesian Additive Regression Tree (BART) models that estimated outcomes for mean stance foot progression to demonstrate the approach. We built two models: one using CLI features only, and one using CLI and IGA features. We then compared the models' performance in detail. We performed similar, but less detailed, analyses for a number of other outcomes. All results were based on independent test data from a 70%/30% training/testing split.

RESULTS

The IGA model was more accurate than the CLI model for mean stance-phase foot progression outcomes (RMSEIGA = 11∘, RMSECLI = 13∘) and explained more than 1.5 × as much of the variance (R2IGA = .45, R2CLI = .28). The IGA model outperformed the CLI model for every level of treatment complexity, as measured by number of simultaneous surgeries. The IGA model also exhibited superior performance for estimating outcomes of mean stance-phase knee flexion, mean stance-phase ankle dorsiflexion, maximum swing-phase knee flexion, gait deviation index (GDI), and dimensionless speed.

INTERPRETATION

The results show that IGA has the potential to be useful in the treatment planning process for ambulatory children diagnosed with cerebral palsy. We propose that the results of machine learning outcome estimators-including estimates of uncertainty-become the primary IGA tool utilized in the clinical process, complementing the standard medical practice of conducting a through patient history and physical exam, eliciting patient goals, reviewing relevant imaging data, and so on.

Looking for a broader mindset in psychometrics: the case for more participatory measurement practices

Psychometrics and the consequences of its use as the method of quantitative empirical psychology has been continuously criticized by both psychologists and psychometrists. However, the scope of the possible solutions to these issues has been mostly focused on the establishment of methodological-statistical best practices for researchers, without any regard to the pitfalls of previous stages of measurement as well as theory development of the targeted phenomenon. Conversely, other researchers advance the idea that, since psychometrics is riddled with many issues, the best way forward is a complete rework of the discipline even if it leaves psychologists and other practitioners without any way to measure quantitatively for a long period of time. Given these tensions, we therefore advocate for an alternative path to consider while we work on making substantive change in measurement. We propose a set of research practices focusing on the inclusion and active participation of groups involved in measurement activities, such as psychometrists, researchers but most importantly practitioners and potential participants. Involving a wider community while measuring in psychology could tackle some key issues that would take us closer to a more authentic approach to our phenomenon of interest.

Implementation Documentation and Process Assessment of the PharmNet Intervention: Observational Report

BACKGROUND

The number of overdose deaths in the United States involving opioids continues to exceed 100,000 per year. This has precipitated ongoing declarations of a public health emergency. Harm reduction approaches, such as promoting awareness of, ensuring access to, and fostering willingness to use naloxone to reverse opioid overdose, are a key component of a larger national strategy to address the crisis. In addition, overdose reversal with naloxone directly and immediately saves lives. Because of pharmacies' ubiquity and pharmacists' extensive clinical training, community pharmacies are well-positioned, in principle, to facilitate naloxone access and education.

OBJECTIVE

In 2022, a single-site pilot study of PharmNet, a community pharmacy intervention incorporating naloxone distribution, awareness building, and referral, showed promising outcomes for both naloxone and resource distribution in the community. As a next step, this study was intended to be a pilot randomized controlled trial of PharmNet in 7 pharmacies. However, due to circumstances outside of the study team's control, data collection was unable to be fully completed as planned. In keeping with open research standards, we transparently report all available data from the study and discuss trial barriers and processes. We do so both to provide insights that may inform similar studies and to avoid the "file-drawer" (publication bias) problem, which can skew the aggregated scholarly literature through nonpublication of registered trial results or selective publication of findings affirming authors' hypotheses.

METHODS

This paper reports an in-depth implementation study assessment, provides the available observational data, and discusses implementation considerations for similar studies in independent (eg, nonchain) community pharmacies.

RESULTS

Retrospective assessment of study outcomes and fidelity data provided for robust discussion around how resource differences in independent community pharmacies (vs well-resourced chain pharmacies), as well as high demands on staff, can affect intervention implementation, even when leadership is highly supportive.

CONCLUSIONS

Community pharmacies, particularly independent community pharmacies, may require more support than anticipated to be successful when implementing a new intervention into practice, even if it might affect estimates of real-world effectiveness. Further implementation science research is needed specific to independent community pharmacies. All study elements are outlined in the International Registered Report Identifier (IRRID) PRR1-10.2196/42373. Although this paper reports results associated with that registration, results and conclusions should not be given the weight assigned to findings from a preregistered study.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

RR2-10.2196/42373.

Exploring open science practices in behavioural public policy research

In their book 'Nudge: Improving Decisions About Health, Wealth and Happiness', Thaler & Sunstein (2009) argue that choice architectures are promising public policy interventions. This research programme motivated the creation of 'nudge units', government agencies which aim to apply insights from behavioural science to improve public policy. We closely examine a meta-analysis of the evidence gathered by two of the largest and most influential nudge units (DellaVigna & Linos (2022 Econometrica 90, 81-116 (doi:10.3982/ECTA18709))) and use statistical techniques to detect reporting biases. Our analysis shows evidence suggestive of selective reporting. We additionally evaluate the public pre-analysis plans from one of the two nudge units (Office of Evaluation Sciences). We identify several instances of excellent practice; however, we also find that the analysis plans and reporting often lack sufficient detail to evaluate (unintentional) reporting biases. We highlight several improvements that would enhance the effectiveness of the pre-analysis plans and reports as a means to combat reporting biases. Our findings and suggestions can further improve the evidence base for policy decisions.

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta-analyses

Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).

Assessing the Evidential Value of Mental Fatigue and Exercise Research

It has often been reported that mental exertion, presumably leading to mental fatigue, can negatively affect exercise performance; however, recent findings have questioned the strength of the effect. To further complicate this issue, an overlooked problem might be the presence of publication bias in studies using underpowered designs, which is known to inflate false positive report probability and effect size estimates. Altogether, the presence of bias is likely to reduce the evidential value of the published literature on this topic, although it is unknown to what extent. The purpose of the current work was to assess the evidential value of studies published to date on the effect of mental exertion on exercise performance by assessing the presence of publication bias and the observed statistical power achieved by these studies. A traditional meta-analysis revealed a Cohen's dz effect size of - 0.54, 95% CI [- 0.68, - 0.40], p < .001. However, when we applied methods for estimating and correcting for publication bias (based on funnel plot asymmetry and observed p-values), we found that the bias-corrected effect size became negligible with most of publication-bias methods and decreased to - 0.36 in the more optimistic of all the scenarios. A robust Bayesian meta-analysis found strong evidence in favor of publication bias, BFpb > 1000, and inconclusive evidence in favor of the effect, adjusted dz = 0.01, 95% CrI [- 0.46, 0.37], BF10 = 0.90. Furthermore, the median observed statistical power assuming the unadjusted meta-analytic effect size (i.e., - 0.54) as the true effect size was 39% (min = 19%, max = 96%), indicating that, on average, these studies only had a 39% chance of observing a significant result if the true effect was Cohen's dz = - 0.54. If the more optimistic adjusted effect size (- 0.36) was assumed as the true effect, the median statistical power was just 20%. We conclude that the current literature is a useful case study for illustrating the dangers of conducting underpowered studies to detect the effect size of interest.

An observational study on the adherence to study registrations in German interventional and observational studies from various fields

Background

The registration of studies, especially in the case of clinical trials, is required by the declaration of Helsinki and the policies of various scientific journals. However, numerous analyses have found considerable discrepancies between published articles and accompanying trial registrations. The aim of this study is to assess such discrepancies for a sample of studies with recruiting locations in Germany. Additionally, the association between the adherence to registrations and possible involvement of Coordinating Centers for Clinical Studies (KKS) as well as Universities of Excellence was tested.

Methods

For a sample of 376 interventional or observational study registrations, we found 115 published articles. Subsequently, we searched for discrepancies in the study design, the key inclusion criteria, the interventions, the blinding, and a primary and a secondary outcome.

Results

We found discrepancies in 26% of all studies, most frequently in the secondary outcomes, where 16.5% of the secondary outcomes per study that were registered in most detail had discrepancies. When running regression models for adherence discrepancies, the only variable that had a statistically significant association with better adherence was registration on ClinicalTrials.gov. The association of potential involvement of a KKS with adherence ratings was positive, too, but statistically insignificant.

Conclusions

In summary, the amount of discrepancies between registrations and published articles remains elevated.

Evidence of questionable research practices in clinical prediction models

BACKGROUND

Clinical prediction models are widely used in health and medical research. The area under the receiver operating characteristic curve (AUC) is a frequently used estimate to describe the discriminatory ability of a clinical prediction model. The AUC is often interpreted relative to thresholds, with "good" or "excellent" models defined at 0.7, 0.8 or 0.9. These thresholds may create targets that result in "hacking", where researchers are motivated to re-analyse their data until they achieve a "good" result.

METHODS

We extracted AUC values from PubMed abstracts to look for evidence of hacking. We used histograms of the AUC values in bins of size 0.01 and compared the observed distribution to a smooth distribution from a spline.

RESULTS

The distribution of 306,888 AUC values showed clear excesses above the thresholds of 0.7, 0.8 and 0.9 and shortfalls below the thresholds.

CONCLUSIONS

The AUCs for some models are over-inflated, which risks exposing patients to sub-optimal clinical decision-making. Greater modelling transparency is needed, including published protocols, and data and code sharing.

Publication bias, statistical power and reporting practices in the Journal of Sports Sciences: potential barriers to replicability

Two factors that decrease the replicability of studies in the scientific literature are publication bias and studies with underpowered desgins. One way to ensure that studies have adequate statistical power to detect the effect size of interest is by conducting a-priori power analyses. Yet, a previous editorial published in the Journal of Sports Sciences reported a median sample size of 19 and the scarce usage of a-priori power analyses. We meta-analysed 89 studies from the same journal to assess the presence and extent of publication bias, as well as the average statistical power, by conducting a z-curve analysis. In a larger sample of 174 studies, we also examined a) the usage, reporting practices and reproducibility of a-priori power analyses; and b) the prevalence of reporting practices of t-statistic or F-ratio, degrees of freedom, exact p-values, effect sizes and confidence intervals. Our results indicate that there was some indication of publication bias and the average observed power was low (53% for significant and non-significant findings and 61% for only significant findings). Finally, the usage and reporting practices of a-priori power analyses as well as statistical results including test statistics, effect sizes and confidence intervals were suboptimal.