Statistical Significance and the Dichotomization of Evidence

For a proper use of frequentist inferential statistics in public health

As widely noted in the literature and by international bodies such as the American Statistical Association, severe misinterpretations of P-values, confidence intervals, and statistical significance are sadly common in public health. This scenario poses serious risks concerning terminal decisions such as the approval or rejection of therapies. Cognitive distortions about statistics likely stem from poor teaching in schools and universities, overly simplified interpretations, and - as we suggest - the reckless use of calculation software with predefined standardized procedures. In light of this, we present a framework to recalibrate the role of frequentist-inferential statistics within clinical and epidemiological research. In particular, we stress that statistics is only a set of rules and numbers that make sense only when properly placed within a well-defined scientific context beforehand. Practical examples are discussed for educational purposes. Alongside this, we propose some tools to better evaluate statistical outcomes, such as multiple compatibility or surprisal intervals or tuples of various point hypotheses. Lastly, we emphasize that every conclusion must be informed by different kinds of scientific evidence (e.g., biochemical, clinical, statistical, etc.) and must be based on a careful examination of costs, risks, and benefits.

The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

BACKGROUND

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

PURPOSE

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

STUDY DESIGN

Meta-analysis; Level of evidence, 2.

METHODS

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

RESULTS

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial's sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

CONCLUSION

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

LFK index does not reliably detect small-study effects in meta-analysis: A simulation study

The LFK index has been promoted as an improved method to detect bias in meta-analysis. Putatively, its performance does not depend on the number of studies in the meta-analysis. We conducted a simulation study, comparing the LFK index test to three standard tests for funnel plot asymmetry in settings with smaller or larger group sample sizes. In general, false positive rates of the LFK index test markedly depended on the number and size of studies as well as the between-study heterogeneity with values between 0% and almost 30%. Egger's test adhered well to the pre-specified significance level of 5% under homogeneity, but was too liberal (smaller groups) or conservative (larger groups) under heterogeneity. The rank test was too conservative for most simulation scenarios. The Thompson-Sharp test was too conservative under homogeneity, but adhered well to the significance level in case of heterogeneity. The true positive rate of the LFK index test was only larger compared with classic tests if the false positive rate was inflated. The power of classic tests was similar or larger than the LFK index test if the false positive rate of the LFK index test was used as significance level for the classic tests. Under ideal conditions, the false positive rate of the LFK index test markedly and unpredictably depends on the number and sample size of studies as well as the extent of between-study heterogeneity. The LFK index test in its current implementation should not be used to assess funnel plot asymmetry in meta-analysis.

P-hacking in meta-analyses: A formalization and new meta-analytic methods

As traditionally conceived, publication bias arises from selection operating on a collection of individually unbiased estimates. A canonical form of such selection across studies (SAS) is the preferential publication of affirmative studies (i.e., those with significant, positive estimates) versus nonaffirmative studies (i.e., those with nonsignificant or negative estimates). However, meta-analyses can also be compromised by selection within studies (SWS), in which investigators "p-hack" results within their study to obtain an affirmative estimate. Published estimates can then be biased even conditional on affirmative status, which comprises the performance of existing methods that only consider SAS. We propose two new analysis methods that accommodate joint SAS and SWS; both analyze only the published nonaffirmative estimates. First, we propose estimating the underlying meta-analytic mean by fitting "right-truncated meta-analysis" (RTMA) to the published nonaffirmative estimates. This method essentially imputes the entire underlying distribution of population effects. Second, we propose conducting a standard meta-analysis of only the nonaffirmative studies (MAN); this estimate is conservative (negatively biased) under weakened assumptions. We provide an R package (phacking) and website (metabias.io). Our proposed methods supplement existing methods by assessing the robustness of meta-analyses to joint SAS and SWS.

How to Use and Report on p-values

The use of the p-value in quantitative research, particularly its threshold of "P < 0.05" for determining "statistical significance," has long been a cornerstone of statistical analysis in research. However, this standard has been increasingly scrutinized for its potential to mislead findings, especially when the practical significance, the number of comparisons, or the suitability of statistical tests are not properly considered. In response to controversy around use of p-values, the American Statistical Association published a statement in 2016 that challenged the research community to abandon the term "statistically significant". This stance has been echoed by leading scientific journals to urge a significant reduction or complete elimination in the reliance on p-values when reporting results. To provide guidance to researchers in health professions education, this paper provides a succinct overview of the ongoing debate regarding the use of p-values and the definition of p-values. It reflects on the controversy by highlighting the common pitfalls associated with p-value interpretation and usage, such as misinterpretation, overemphasis, and false dichotomization between "significant" and "non-significant" results. This paper also outlines specific recommendations for the effective use of p-values in statistical reporting including the importance of reporting effect sizes, confidence intervals, the null hypothesis, and conducting sensitivity analyses for appropriate interpretation. These considerations aim to guide researchers toward a more nuanced and informative use of p-values.

Evidence for Infant-directed Speech Preference Is Consistent Across Large-scale, Multi-site Replication and Meta-analysis

There is substantial evidence that infants prefer infant-directed speech (IDS) to adult-directed speech (ADS). The strongest evidence for this claim has come from two large-scale investigations: i) a community-augmented meta-analysis of published behavioral studies and ii) a large-scale multi-lab replication study. In this paper, we aim to improve our understanding of the IDS preference and its boundary conditions by combining and comparing these two data sources across key population and design characteristics of the underlying studies. Our analyses reveal that both the meta-analysis and multi-lab replication show moderate effect sizes (d ≈ 0.35 for each estimate) and that both of these effects persist when relevant study-level moderators are added to the models (i.e., experimental methods, infant ages, and native languages). However, while the overall effect size estimates were similar, the two sources diverged in the effects of key moderators: both infant age and experimental method predicted IDS preference in the multi-lab replication study, but showed no effect in the meta-analysis. These results demonstrate that the IDS preference generalizes across a variety of experimental conditions and sampling characteristics, while simultaneously identifying key differences in the empirical picture offered by each source individually and pinpointing areas where substantial uncertainty remains about the influence of theoretically central moderators on IDS preference. Overall, our results show how meta-analyses and multi-lab replications can be used in tandem to understand the robustness and generalizability of developmental phenomena.

Sensitivity analysis for the interactive effects of internal bias and publication bias in meta-analyses

Meta-analyses can be compromised by studies' internal biases (e.g., confounding in nonrandomized studies) as well as publication bias. These biases often operate nonadditively: publication bias that favors significant, positive results selects indirectly for studies with more internal bias. We propose sensitivity analyses that address two questions: (1) "For a given severity of internal bias across studies and of publication bias, how much could the results change?"; and (2) "For a given severity of publication bias, how severe would internal bias have to be, hypothetically, to attenuate the results to the null or by a given amount?" These methods consider the average internal bias across studies, obviating specifying the bias in each study individually. The analyst can assume that internal bias affects all studies, or alternatively that it only affects a known subset (e.g., nonrandomized studies). The internal bias can be of unknown origin or, for certain types of bias in causal estimates, can be bounded analytically. The analyst can specify the severity of publication bias or, alternatively, consider a "worst-case" form of publication bias. Robust estimation methods accommodate non-normal effects, small meta-analyses, and clustered estimates. As we illustrate by re-analyzing published meta-analyses, the methods can provide insights that are not captured by simply considering each bias in turn. An R package implementing the methods is available (multibiasmeta).

Interpreting Randomized Controlled Trials

This article describes rationales and limitations for making inferences based on data from randomized controlled trials (RCTs). We argue that obtaining a representative random sample from a patient population is impossible for a clinical trial because patients are accrued sequentially over time and thus comprise a convenience sample, subject only to protocol entry criteria. Consequently, the trial's sample is unlikely to represent a definable patient population. We use causal diagrams to illustrate the difference between random allocation of interventions within a clinical trial sample and true simple or stratified random sampling, as executed in surveys. We argue that group-specific statistics, such as a median survival time estimate for a treatment arm in an RCT, have limited meaning as estimates of larger patient population parameters. In contrast, random allocation between interventions facilitates comparative causal inferences about between-treatment effects, such as hazard ratios or differences between probabilities of response. Comparative inferences also require the assumption of transportability from a clinical trial's convenience sample to a targeted patient population. We focus on the consequences and limitations of randomization procedures in order to clarify the distinctions between pairs of complementary concepts of fundamental importance to data science and RCT interpretation. These include internal and external validity, generalizability and transportability, uncertainty and variability, representativeness and inclusiveness, blocking and stratification, relevance and robustness, forward and reverse causal inference, intention to treat and per protocol analyses, and potential outcomes and counterfactuals.

Bayesian analysis of the Substrate Ablation vs. Antiarrhythmic Drug Therapy for Symptomatic Ventricular Tachycardia trial

BACKGROUND AND AIMS

Bayesian analyses can provide additional insights into the results of clinical trials, aiding in the decision-making process. We analysed the Substrate Ablation vs. Antiarrhythmic Drug Therapy for Symptomatic Ventricular Tachycardia (SURVIVE-VT) trial using Bayesian survival models.

METHODS AND RESULTS

The SURVIVE-VT trial randomized patients with ischaemic cardiomyopathy and monomorphic ventricular tachycardia (VT) to catheter ablation or antiarrhythmic drugs (AAD) as a first-line strategy. The primary outcome was a composite of cardiovascular death, appropriate implantable cardioverter-defibrillator shocks, unplanned heart failure hospitalizations, or severe treatment-related complications. We used informative, skeptical, and non-informative priors with different probabilities of large effects to compute the posterior distributions using Markov Chain Monte Carlo methods. We calculated the probabilities of hazard ratios (HR) being <1, <0.9, and <0.75, as well as 2-year survival estimates. Of the 144 randomized patients, 71 underwent catheter ablation and 73 received AAD. Regardless of the prior, catheter ablation had a >98% probability of reducing the primary outcome (HR < 1) and a >96% probability of achieving a reduction of >10% (HR < 0.9). The probability of a >25% (HR < 0.75) reduction of treatment-related complications was >90%. Catheter ablation had a high probability (>93%) of reducing incessant/slow undetected VT/electric storm, unplanned hospitalizations for ventricular arrhythmias, and overall cardiovascular admissions > 25%, with absolute differences of 15.2%, 21.2%, and 20.2%, respectively.

CONCLUSION

In patients with ischaemic cardiomyopathy and VT, catheter ablation as a first-line therapy resulted in a high probability of reducing several clinical outcomes compared to AAD. Our study highlights the value of Bayesian analysis in clinical trials and its potential for guiding treatment decisions.

TRIAL REGISTRATION

ClinicalTrials.gov identifier: NCT03734562.

Problems and alternatives of testing significance using null hypothesis and P-value in food research

A testing method to identify statistically significant differences by comparing the significance level and the probability value based on the Null Hypothesis Significance Test (NHST) has been used in food research. However, problems with this testing method have been discussed. Several alternatives to the NHST and the P-value test methods have been proposed including lowering the P-value threshold and using confidence interval (CI), effect size, and Bayesian statistics. The CI estimates the extent of the effect or difference and determines the presence or absence of statistical significance. The effect size index determines the degree of effect difference and allows for the comparison of various statistical results. Bayesian statistics enable predictions to be made even when only a small amount of data is available. In conclusion, CI, effect size, and Bayesian statistics can complement or replace traditional statistical tests in food research by replacing the use of NHST and P-value.

Has the degree of outcome reporting bias in surgical randomized trials changed? A meta-regression analysis

BACKGROUND

Outcome reporting bias in individual trials can compromise the validity of pooled estimates within systematic reviews. Recent strategies have attempted to address outcome reporting bias, which favours the full reporting of statistically significant outcomes over non-significant outcomes. We examined whether the association between full outcome reporting and statistical significance in surgical trials has changed from 2009 to 2019.

METHODS

We systematically searched for 350 surgical randomized controlled trials (RCTs) from 2009 and 350 surgical RCTs from 2019. Outcomes were classified as fully reported, partially reported, qualitatively reported or unreported. For each outcome, a contingency table was populated with full outcome reporting (yes/no) and statistical significance (yes/no). We combined odds ratios in random effects meta-analysis to estimate the association between full outcome reporting and statistical significance in 2009 compared with 2019.

RESULTS

Twenty-eight percent of outcomes in 2009 were incompletely reported, compared with 30% in 2019. In 2009, significant outcomes were more likely to be fully reported than non-significant outcomes (OR = 2.4, 95% CI 1.7-3.4, I²  = 35%), but the opposite association was seen in 2019 (OR = 0.51, 95% CI 0.34-0.77, I²  = 43%). RCTs from 2019 were less likely to demonstrate outcome reporting bias favouring significant outcomes (OR = 0.21, 95% CI 0.12-0.35, P < 0.001).

CONCLUSION

Outcome reporting bias favouring the full reporting of significant over non-significant outcomes was demonstrated in 2009, but the opposite association was seen in 2019. There remains a high prevalence of incomplete outcome reporting. We recommend ongoing adherence to trial protocol guidelines to improve outcome reporting transparency and completeness.

Are most published research findings false in a continuous universe?

Diagnostic screening models for the interpretation of null hypothesis significance test (NHST) results have been influential in highlighting the effect of selective publication on the reproducibility of the published literature, leading to John Ioannidis' much-cited claim that most published research findings are false. These models, however, are typically based on the assumption that hypotheses are dichotomously true or false, without considering that effect sizes for different hypotheses are not the same. To address this limitation, we develop a simulation model that overcomes this by modeling effect sizes explicitly using different continuous distributions, while retaining other aspects of previous models such as publication bias and the pursuit of statistical significance. Our results show that the combination of selective publication, bias, low statistical power and unlikely hypotheses consistently leads to high proportions of false positives, irrespective of the effect size distribution assumed. Using continuous effect sizes also allows us to evaluate the degree of effect size overestimation and prevalence of estimates with the wrong sign in the literature, showing that the same factors that drive false-positive results also lead to errors in estimating effect size direction and magnitude. Nevertheless, the relative influence of these factors on different metrics varies depending on the distribution assumed for effect sizes. The model is made available as an R ShinyApp interface, allowing one to explore features of the literature in various scenarios.

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger's test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors "statistically significant" results, and the methods also directly accommodate effect heterogeneity. Selection models have been established for decades in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use a previously published meta-analysis to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.

Losing Sight of Piecemeal Progress: People Lump and Dismiss Improvement Efforts That Fall Short of Categorical Change-Despite Improving

Fourteen experiments (N = 10,556 adult participants, including more than 20,000 observed choices across 25 issues) documented how people perceive and respond to relative progress out in the world, revealing a robust "negative-lumping" effect. As problematic entities worked to better their ways, participants shifted to dismiss them if they fell short of categorical reform-despite distinctions in improvement. This increased dismissal of relative gains as "all the same" was driven by the belief that falling short signals an eschewal of doing the bare minimum and lacking serious intent to change, making these gains seem less deserving of recognition. Critically, participants then "checked out": They underrewarded and underinvested in efforts toward "merely" incremental improvement. Finally, in all experiments, participants lumped together absolute failures but not absolute successes, highlighting a unique blindness to gradations of badness. When attempts to eradicate a problem fail, people might dismiss smaller but critical steps that were and can still be made.

Non-inferiority and clinical superiority of glucagon-like peptide-1 receptor agonists and sodium-glucose co-transporter-2 inhibitors: Systematic analysis of cardiorenal outcome trials in type 2 diabetes

AIMS

Most trials leading to the approval of glucagon-like peptide receptor agonists (GLP-1RAs) and sodium-glucose co-transporter-2 inhibitors (SGLT2is) were primarily designed to confirm their non-inferiority to placebo (commonly using an upper 95% confidence limit threshold of 1.3) and, if confirmed, superiority (threshold 1): this asymmetry of margins (1 vs. 1.3) favours the active intervention. We aimed to quantify the probability of clinical superiority of the active treatment by applying the same threshold used to claim non-inferiority.

MATERIALS AND METHODS

We searched PubMed and Cochrane CENTRAL for cardiorenal outcome trials in subjects with type 2 diabetes published before 5 December 2021, to reconstruct from Kaplan-Meier plots individual-level data for the primary outcome or all-cause mortality. We calculated Bayesian posterior densities to obtain the probability for a treatment effect (hazard ratio) <0.769, which is symmetric to the 1.3 threshold (i.e. its reciprocal 1/1.3), emulating a scenario where the active treatment is placebo and placebo is the active treatment.

RESULTS

We extracted data from 27 Kaplan-Meier plots (18 for the primary outcome, nine for mortality). Probabilities of clinical superiority to placebo varied significantly: for GLP-1RAs, from a minimum of 0% to a maximum of 69% for the primary outcome and from 0% to 8% for mortality; corresponding estimates for SGLT2is were 0% to 96% and 0% to 93%. Probabilities were on average greater for SGLT2is, particularly in trials investigating kidney or heart failure outcomes.

CONCLUSIONS

The probability of clinical superiority to placebo varies widely across trials previously reported as showing superiority of GLP-1RAs or SGLT2is compared with placebo. These results showed within- and between-class differences, highlight the drawbacks of a binary interpretation of the results, particularly in the context of the current designs of non-inferiority trials, and have implications for decision makers and future clinical recommendations.

A genome-wide association and prediction study in grapevine deciphers the genetic architecture of multiple traits and identifies genes under many new QTLs

To cope with the challenges facing agriculture, speeding-up breeding programs is a worthy endeavor, especially for perennial species such as grapevine, but requires understanding the genetic architecture of target traits. To go beyond the mapping of quantitative trait loci in bi-parental crosses, we exploited a diversity panel of 279 Vitis vinifera L. cultivars planted in 5 blocks in the vineyard. This panel was phenotyped over several years for 127 traits including yield components, organic acids, aroma precursors, polyphenols, and a water stress indicator. The panel was genotyped for 63k single nucleotide polymorphisms by combining an 18K microarray and genotyping-by-sequencing. The experimental design allowed to reliably assess the genotypic values for most traits. Marker densification via genotyping-by-sequencing markedly increased the proportion of genetic variance explained by single nucleotide polymorphisms, and 2 multi-single nucleotide polymorphism models identified quantitative trait loci not found by a single nucleotide polymorphism-by-single nucleotide polymorphism model. Overall, 489 reliable quantitative trait loci were detected for 41% more response variables than by a single nucleotide polymorphism-by-single nucleotide polymorphism model with microarray-only single nucleotide polymorphisms, many new ones compared with the results from bi-parental crosses. A prediction accuracy higher than 0.42 was obtained for 50% of the response variables. Our overall approach as well as quantitative trait locus and prediction results provide insights into the genetic architecture of target traits. New candidate genes and the application into breeding are discussed.

Making Sense of Uncertainty in the Science Classroom: A Bayesian Approach

Uncertainty is ubiquitous in science, but scientific knowledge is often represented to the public and in educational contexts as certain and immutable. This contrast can foster distrust when scientific knowledge develops in a way that people perceive as a reversals, as we have observed during the ongoing COVID-19 pandemic. Drawing on research in statistics, child development, and several studies in science education, we argue that a Bayesian approach can support science learners to make sense of uncertainty. We provide a brief primer on Bayes' theorem and then describe three ways to make Bayesian reasoning practical in K-12 science education contexts. There are a) using principles informed by Bayes' theorem that relate to the nature of knowing and knowledge, b) interacting with a web-based application (or widget-Confidence Updater) that makes the calculations needed to apply Bayes' theorem more practical, and c) adopting strategies for supporting even young learners to engage in Bayesian reasoning. We conclude with directions for future research and sum up how viewing science and scientific knowledge from a Bayesian perspective can build trust in science.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s11191-022-00341-3.

Misinterpretations of P-values and statistical tests persists among researchers and professionals working with statistics and epidemiology

BACKGROUND

The aim was to investigate inferences of statistically significant test results among persons with more or less statistical education and research experience.

METHODS

A total of 75 doctoral students and 64 statisticians/epidemiologist responded to a web questionnaire about inferences of statistically significant findings. Participants were asked about their education and research experience, and also whether a 'statistically significant' test result (P = 0.024, α-level 0.05) could be inferred as proof or probability statements about the truth or falsehood of the null hypothesis (H₀) and the alternative hypothesis (H₁).

RESULTS

Almost all participants reported having a university degree, and among statisticians/epidemiologist, most reported having a university degree in statistics and were working professionally with statistics. Overall, 9.4% of statisticians/epidemiologist and 24.0% of doctoral students responded that the statistically significant finding proved that H₀ is not true, and 73.4% of statisticians/epidemiologists and 53.3% of doctoral students responded that the statistically significant finding indicated that H₀ is improbable. Corresponding numbers about inferences about the alternative hypothesis (H₁) were 12.0% and 6.2% about proving H₁ being true and 62.7 and 62.5% for the conclusion that H₁ is probable. Correct inferences to both questions, which is that a statistically significant finding cannot be inferred as either proof or a measure of a hypothesis' probability, were given by 10.7% of doctoral students and 12.5% of statisticians/epidemiologists.

CONCLUSIONS

Misinterpretation of P-values and statistically significant test results persists also among persons who have substantial statistical education and who work professionally with statistics.

Teaching: confidence, prediction and tolerance intervals in scientific practice: a tutorial on binary variables

BACKGROUND

One of the emerging themes in epidemiology is the use of interval estimates. Currently, three interval estimates for confidence (CI), prediction (PI), and tolerance (TI) are at a researcher's disposal and are accessible within the open access framework in R. These three types of statistical intervals serve different purposes. Confidence intervals are designed to describe a parameter with some uncertainty due to sampling errors. Prediction intervals aim to predict future observation(s), including some uncertainty present in the actual and future samples. Tolerance intervals are constructed to capture a specified proportion of a population with a defined confidence. It is well known that interval estimates support a greater knowledge gain than point estimates. Thus, a good understanding and the use of CI, PI, and TI underlie good statistical practice. While CIs are taught in introductory statistical classes, PIs and TIs are less familiar.

RESULTS

In this paper, we provide a concise tutorial on two-sided CI, PI and TI for binary variables. This hands-on tutorial is based on our teaching materials. It contains an overview of the meaning and applicability from both a classical and a Bayesian perspective. Based on a worked-out example from veterinary medicine, we provide guidance and code that can be directly applied in R.

CONCLUSIONS

This tutorial can be used by others for teaching, either in a class or for self-instruction of students and senior researchers.

Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect

Common reporting styles for statistical results in scientific articles, such as p-values and confidence intervals (CI), have been reported to be prone to dichotomous interpretations, especially with respect to the null hypothesis significance testing framework. For example when the p-value is small enough or the CIs of the mean effects of a studied drug and a placebo are not overlapping, scientists tend to claim significant differences while often disregarding the magnitudes and absolute differences in the effect sizes. This type of reasoning has been shown to be potentially harmful to science. Techniques relying on the visual estimation of the strength of evidence have been recommended to reduce such dichotomous interpretations but their effectiveness has also been challenged. We ran two experiments on researchers with expertise in statistical analysis to compare several alternative representations of confidence intervals and used Bayesian multilevel models to estimate the effects of the representation styles on differences in researchers' subjective confidence in the results. We also asked the respondents' opinions and preferences in representation styles. Our results suggest that adding visual information to classic CI representation can decrease the tendency towards dichotomous interpretations - measured as the 'cliff effect': the sudden drop in confidence around p-value 0.05 - compared with classic CI visualization and textual representation of the CI with p-values. All data and analyses are publicly available at https://github.com/helske/statvis.

Rapid statistical discrimination of fluorescence images of T cell receptors on immobilizing surfaces with different coating conditions

The spatial organization of T cell receptors (TCRs) correlates with membrane-associated signal amplification, dispersion, and regulation during T cell activation. Despite its potential clinical importance, quantitative analysis of the spatial arrangement of TCRs from standard fluorescence images remains difficult. Here, we report Statistical Classification Analyses of Membrane Protein Images or SCAMPI as a technique capable of analyzing the spatial arrangement of TCRs on the plasma membrane of T cells. We leveraged medical image analysis techniques that utilize pixel-based values. We transformed grayscale pixel values from fluorescence images of TCRs into estimated model parameters of partial differential equations. The estimated model parameters enabled an accurate classification using linear discrimination techniques, including Fisher Linear Discriminant (FLD) and Logistic Regression (LR). In a proof-of-principle study, we modeled and discriminated images of fluorescently tagged TCRs from Jurkat T cells on uncoated cover glass surfaces (Null) or coated cover glass surfaces with either positively charged poly-L-lysine (PLL) or TCR cross-linking anti-CD3 antibodies (OKT3). Using 80 training images and 20 test images per class, our statistical technique achieved 85% discrimination accuracy for both OKT3 versus PLL and OKT3 versus Null conditions. The run time of image data download, model construction, and image discrimination was 21.89 s on a laptop computer, comprised of 20.43 s for image data download, 1.30 s on the FLD-SCAMPI analysis, and 0.16 s on the LR-SCAMPI analysis. SCAMPI represents an alternative approach to morphology-based qualifications for discriminating complex patterns of membrane proteins conditioned on a small sample size and fast runtime. The technique paves pathways to characterize various physiological and pathological conditions using the spatial organization of TCRs from patient T cells.

Cardiovascular effects of sodium-glucose co-transporter-2 inhibitors and glucagon-like peptide-1 receptor agonists: The P value and beyond

Despite growing awareness of the dangers of a dichotomous interpretation of trial results based on the 'statistical significance' of a treatment effect, the uptake of new approaches has been slow in diabetes medicine. We showcase a number of ways to interpret the evidence for a treatment effect applied to the cardiovascular outcome trials of glucagon-like peptide-1 receptor agonists (GLP-1RAs) and sodium-glucose co-transporter-2 inhibitors (SGLT-2is): the P value function (or confidence curves), which depicts the treatment effect across the whole spectrum of confidence levels; the counternull value, which is the hazard ratio (i.e. treatment effect size) supported by the same amount of evidence as the null value (i.e. no treatment effect); and the S value, which quantifies the strength of the evidence against the null hypothesis in terms of the number of coin tosses yielding the same side. We show how this approach identifies potential treatment effects, highlights similarities among trials straddling the threshold of statistical significance, and quantifies differences in the strength of the evidence from trials reporting statistically significant results. For example, while REWIND, CANVAS and CREDENCE failed to reach statistical significance at the .05 level for all-cause mortality, their counternull values indicate that reduced death rates by 19%, 24% and 31%, respectively, are supported by the same amount of evidence as that indicating no treatment effect. Moreover, similarities among results emerge in trials of GLP-1RAs (REWIND, EXSCEL and LEADER) lying closely around the threshold of 'statistical significance'. Lastly, several S values, such as for the primary outcome in HARMONY Outcomes (S value 10.9) and all-cause death in EMPAREG-OUTCOME (S value 15.0), stand out compared with values for other outcomes and other trials, suggesting much larger differences in the evidence between these studies and several others that cluster around the .05 significance threshold. P value functions, counternull values and S values should complement the standard reporting of the treatment effect to help interpret clinical trials and make decisions among competing glucose-lowering medications.

Reflecting on "A Statistician in Medicine" in 2020

In this commentary, we revisit Sir Austin Bradford Hill's seminal Alfred Watson Memorial Lecture in 1962 through the eyes of two practicing biostatisticians of the current era. We summarize some eternal takeaway messages from Hill's lecture regarding observations and experiments translated through the modern lexicon of causal inference. Finally, we pose a series of questions that we would have liked to pose to Sir Austin Bradford Hill if he were to deliver the lecture in 2020.

Reflection on modern methods: Statistics education beyond 'significance': novel plain English interpretations to deepen understanding of statistics and to steer away from misinterpretations

Concerns have been expressed over standards of statistical interpretation. Results with P <0.05 are often referred to as 'significant' which, in plain English, implies important. This leads some people directly into the misconception that this provides proof that associations are clinically relevant. There are calls for statistics educators to respond to these concerns. This article provides novel plain English interpretations that are designed to deepen understanding. Experience teaching postgraduates at Imperial College is discussed. A key issue with focusing on 'significance' is the common inappropriate practice of implying no association exists, simply because P >0.05. Referring to strengths of association in 'study participants' gives them gravitas, which may help to avoid this. This contrasts with the common practice of focusing on imprecision, by referring to the 'sample' and to 'point estimates'. Unlike formal statistical definitions, interpretations developed and presented here are rooted in the application of statistics. They are based on one set of study participants (not many random samples). Precision of strengths of association are based on using strengths in study participants to estimate strengths of association in the population (from which participants were selected by probability random sampling). Reference to 'compatibility with study data, dependent on statistical modelling assumptions' reminds us of the importance of data quality and modelling assumptions. A straightforward graph shows the relationship between P-values and test statistics. This figure and associated interpretations were developed to illuminate the continuous nature of P-values. This is designed to discourage focus on whether P <0.05, and encourage interpretation of exact P-values.

Détente: A Practical Understanding of P values and Bayesian Posterior Probabilities

Null hypothesis significance testing (NHST) with its benchmark P value < 0.05 has long been a stalwart of scientific reporting and such statistically significant findings have been used to imply scientifically or clinically significant findings. Challenges to this approach have arisen over the past 6 decades, but they have largely been unheeded. There is a growing movement for using Bayesian statistical inference to quantify the probability that a scientific finding is credible. There have been differences of opinion between the frequentist (i.e., NHST) and Bayesian schools of inference, and warnings about the use or misuse of P values have come from both schools of thought spanning many decades. Controversies in this arena have been heightened by the American Statistical Association statement on P values and the further denouncement of the term "statistical significance" by others. My experience has been that many scientists, including many statisticians, do not have a sound conceptual grasp of the fundamental differences in these approaches, thereby creating even greater confusion and acrimony. If we let A represent the observed data, and B represent the hypothesis of interest, then the fundamental distinction between these two approaches can be described as the frequentist approach using the conditional probability pr(A | B) (i.e., the P value), and the Bayesian approach using pr(B | A) (the posterior probability). This paper will further explain the fundamental differences in NHST and Bayesian approaches and demonstrate how they can co-exist harmoniously to guide clinical trial design and inference.

Beyond the p-value: Bayesian Statistics and Causation

Statistical paradigms limit the perspective and tools social work researchers use to study the world and answer questions impacting people and policy. Currently, quantitative social work researchers overwhelmingly rely on the frequentist paradigm of statistics. This paper discusses foundational differences between the frequentist and Bayesian statistical paradigms, describes basic concepts of Bayesian analysis, compares Bayesian and frequentist statistical analysis for a sample social work problem, and introduces two types of causal analyses built on Bayesian statistical thinking: counterfactual causality, and causality based on work by computer scientist Judea Pearl. Implications for social work research are discussed.

Estimating publication bias in meta-analyses of peer-reviewed studies: A meta-meta-analysis across disciplines and journal tiers

Selective publication and reporting in individual papers compromise the scientific record, but are meta-analyses as compromised as their constituent studies? We systematically sampled 63 meta-analyses (each comprising at least 40 studies) in PLoS One, top medical journals, top psychology journals, and Metalab, an online, open-data database of developmental psychology meta-analyses. We empirically estimated publication bias in each, including only the peer-reviewed studies. Across all meta-analyses, we estimated that "statistically significant" results in the expected direction were only 1.17 times more likely to be published than "nonsignificant" results or those in the unexpected direction (95% CI: [0.93, 1.47]), with a confidence interval substantially overlapping the null. Comparable estimates were 0.83 for meta-analyses in PLoS One, 1.02 for top medical journals, 1.54 for top psychology journals, and 4.70 for Metalab. The severity of publication bias did differ across individual meta-analyses; in a small minority (10%; 95% CI: [2%, 21%]), publication bias appeared to favor "significant" results in the expected direction by more than threefold. We estimated that for 89% of meta-analyses, the amount of publication bias that would be required to attenuate the point estimate to the null exceeded the amount of publication bias estimated to be actually present in the vast majority of meta-analyses from the relevant scientific discipline (exceeding the 95th percentile of publication bias). Study-level measures ("statistical significance" with a point estimate in the expected direction and point estimate size) did not indicate more publication bias in higher-tier versus lower-tier journals, nor in the earliest studies published on a topic versus later studies. Overall, we conclude that the mere act of performing a meta-analysis with a large number of studies (at least 40) and that includes non-headline results may largely mitigate publication bias in meta-analyses, suggesting optimism about the validity of meta-analytic results.

Analysis goals, error-cost sensitivity, and analysis hacking: Essential considerations in hypothesis testing and multiple comparisons

The "replication crisis" has been attributed to perverse incentives that lead to selective reporting and misinterpretations of P-values and confidence intervals. A crude fix offered for this problem is to lower testing cut-offs (α levels), either directly or in the form of null-biased multiple comparisons procedures such as naïve Bonferroni adjustments. Methodologists and statisticians have expressed positions that range from condemning all such procedures to demanding their application in almost all analyses. Navigating between these unjustifiable extremes requires defining analysis goals precisely enough to separate inappropriate from appropriate adjustments. To meet this need, I here review issues arising in single-parameter inference (such as error costs and loss functions) that are often skipped in basic statistics, yet are crucial to understanding controversies in testing and multiple comparisons. I also review considerations that should be made when examining arguments for and against modifications of decision cut-offs and adjustments for multiple comparisons. The goal is to provide researchers a better understanding of what is assumed by each side and to enable recognition of hidden assumptions. Basic issues of goal specification and error costs are illustrated with simple fixed cut-off hypothesis testing scenarios. These illustrations show how adjustment choices are extremely sensitive to implicit decision costs, making it inevitable that different stakeholders will vehemently disagree about what is necessary or appropriate. Because decisions cannot be justified without explicit costs, resolution of inference controversies is impossible without recognising this sensitivity. Pre-analysis statements of funding, scientific goals, and analysis plans can help counter demands for inappropriate adjustments, and can provide guidance as to what adjustments are advisable. Hierarchical (multilevel) regression methods (including Bayesian, semi-Bayes, and empirical-Bayes methods) provide preferable alternatives to conventional adjustments, insofar as they facilitate use of background information in the analysis model, and thus can provide better-informed estimates on which to base inferences and decisions.

Semantic and cognitive tools to aid statistical science: replace confidence and significance by compatibility and surprise

BACKGROUND

Researchers often misinterpret and misrepresent statistical outputs. This abuse has led to a large literature on modification or replacement of testing thresholds and P-values with confidence intervals, Bayes factors, and other devices. Because the core problems appear cognitive rather than statistical, we review some simple methods to aid researchers in interpreting statistical outputs. These methods emphasize logical and information concepts over probability, and thus may be more robust to common misinterpretations than are traditional descriptions.

METHODS

We use the Shannon transform of the P-value p, also known as the binary surprisal or S-value s = -log2(p), to provide a measure of the information supplied by the testing procedure, and to help calibrate intuitions against simple physical experiments like coin tossing. We also use tables or graphs of test statistics for alternative hypotheses, and interval estimates for different percentile levels, to thwart fallacies arising from arbitrary dichotomies. Finally, we reinterpret P-values and interval estimates in unconditional terms, which describe compatibility of data with the entire set of analysis assumptions. We illustrate these methods with a reanalysis of data from an existing record-based cohort study.

CONCLUSIONS

In line with other recent recommendations, we advise that teaching materials and research reports discuss P-values as measures of compatibility rather than significance, compute P-values for alternative hypotheses whenever they are computed for null hypotheses, and interpret interval estimates as showing values of high compatibility with data, rather than regions of confidence. Our recommendations emphasize cognitive devices for displaying the compatibility of the observed data with various hypotheses of interest, rather than focusing on single hypothesis tests or interval estimates. We believe these simple reforms are well worth the minor effort they require.

Sensitivity analysis for publication bias in meta-analyses

We propose sensitivity analyses for publication bias in meta‐analyses. We consider a publication process such that ‘statistically significant’ results are more likely to be published than negative or “non‐significant” results by an unknown ratio, η. Our proposed methods also accommodate some plausible forms of selection based on a study's standard error. Using inverse probability weighting and robust estimation that accommodates non‐normal population effects, small meta‐analyses, and clustering, we develop sensitivity analyses that enable statements such as ‘For publication bias to shift the observed point estimate to the null, “significant” results would need to be at least 30 fold more likely to be published than negative or “non‐significant” results’. Comparable statements can be made regarding shifting to a chosen non‐null value or shifting the confidence interval. To aid interpretation, we describe empirical benchmarks for plausible values of η across disciplines. We show that a worst‐case meta‐analytic point estimate for maximal publication bias under the selection model can be obtained simply by conducting a standard meta‐analysis of only the negative and ‘non‐significant’ studies; this method sometimes indicates that no amount of such publication bias could ‘explain away’ the results. We illustrate the proposed methods by using real meta‐analyses and provide an R package: PublicationBias.

The P Value Line Dance: When Does the Music Stop?

When should a trial stop? Such a seemingly innocent question evokes concerns of type I and II errors among those who believe that certainty can be the product of uncertainty and among researchers who have been told that they need to carefully calculate sample sizes, consider multiplicity, and not spend P values on interim analyses. However, the endeavor to dichotomize evidence into significant and nonsignificant has led to the basic driving force of science, namely uncertainty, to take a back seat. In this viewpoint we discuss that if testing the null hypothesis is the ultimate goal of science, then we need not worry about writing protocols, consider ethics, apply for funding, or run any experiments at all-all null hypotheses will be rejected at some point-everything has an effect. The job of science should be to unearth the uncertainties of the effects of treatments, not to test their difference from zero. We also show the fickleness of P values, how they may one day point to statistically significant results; and after a few more participants have been recruited, the once statistically significant effect suddenly disappears. We show plots which we hope would intuitively highlight that all assessments of evidence will fluctuate over time. Finally, we discuss the remedy in the form of Bayesian methods, where uncertainty leads; and which allows for continuous decision making to stop or continue recruitment, as new data from a trial is accumulated.

Effect of Medetomidine, Dexmedetomidine, and Their Reversal with Atipamezole on the Nociceptive Withdrawal Reflex in Beagles

The objectives were: (1) to compare the antinociceptive activity of dexmedetomidine and medetomidine, and (2) to investigate its modulation by atipamezole. This prospective, randomized, blinded experimental trial was carried out on eight beagles. During the first session, dogs received either medetomidine (MED) (0.02 mg kg-1 intravenously (IV)] or dexmedetomidine (DEX) [0.01 mg kg-1 IV), followed by either atipamezole (ATI) (0.1 mg kg-1) or an equivalent volume of saline (SAL) administered intramuscularly 45 min later. The opposite treatments were administered in a second session 10-14 days later. The nociceptive withdrawal reflex (NWR) threshold was determined using a continuous tracking approach. Sedation was scored (0 to 21) every 10 min. Both drugs (MED and DEX) increased the NWR thresholds significantly up to 5.0 (3.7-5.9) and 4.4 (3.9-4.8) times the baseline (p = 0.547), at seven (3-11) and six (4-9) minutes (p = 0.938), respectively. Sedation scores were not different between MED and DEX during the first 45 min (15 (12-17), p = 0.67). Atipamezole antagonized sedation within 25 (15-25) minutes (p = 0.008) and antinociception within five (3-6) minutes (p = 0.008). Following atipamezole, additional analgesics may be needed to maintain pain relief.

Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003)

The attentional spatial-numerical association of response codes (Att-SNARC) effect (Fischer, Castel, Dodd, & Pratt, 2003)—the finding that participants are quicker to detect left-side targets when the targets are preceded by small numbers and quicker to detect right-side targets when they are preceded by large numbers—has been used as evidence for embodied number representations and to support strong claims about the link between number and space (e.g., a mental number line). We attempted to replicate Experiment 2 of Fischer et al. by collecting data from 1,105 participants at 17 labs. Across all 1,105 participants and four interstimulus-interval conditions, the proportion of times the effect we observed was positive (i.e., directionally consistent with the original effect) was .50. Further, the effects we observed both within and across labs were minuscule and incompatible with those observed by Fischer et al. Given this, we conclude that we failed to replicate the effect reported by Fischer et al. In addition, our analysis of several participant-level moderators (finger-counting habits, reading and writing direction, handedness, and mathematics fluency and mathematics anxiety) revealed no substantial moderating effects. Our results indicate that the Att-SNARC effect cannot be used as evidence to support strong claims about the link between number and space.

In Defense of P Values

Recently, a controversy has erupted regarding the use of statistical significance tests and the associated P values. Prominent academic statisticians have recommended that the use of statistical tests be discouraged or not used at all. This has naturally led to a lot of confusion among research investigators about the support in the academic statistical community for statistical methods in general. In fact, the controversy surrounding the use of P values has a long history. Critics of P values argue that their use encourages bad scientific practice, leading to the publication of far more false-positive and false-negative findings than the methodology would imply. The thesis of this commentary is that the problem is really human nature, the natural proclivity of scientists to believe their own theories and present data in the most favorable light. This is strongly encouraged by a celebrity culture that is fueled by academic institutions, the scientific journals, and the media. The importance of the truth-seeking tradition of the scientific method needs to be reinforced, and this is being helped by current initiatives to improve transparency in science and to encourage reproducible and replicable research. Statistical testing, used correctly, has an important and valuable place in the scientific tradition.

Examination of CIs in health and medical journals from 1976 to 2019: an observational study

OBJECTIVES

Previous research has shown clear biases in the distribution of published p values, with an excess below the 0.05 threshold due to a combination of p-hacking and publication bias. We aimed to examine the bias for statistical significance using published confidence intervals.

DESIGN

Observational study.

SETTING

Papers published in Medline since 1976.

PARTICIPANTS

Over 968 000 confidence intervals extracted from abstracts and over 350 000 intervals extracted from the full-text.

OUTCOME MEASURES

Cumulative distributions of lower and upper confidence interval limits for ratio estimates.

RESULTS

We found an excess of statistically significant results with a glut of lower intervals just above one and upper intervals just below 1. These excesses have not improved in recent years. The excesses did not appear in a set of over 100 000 confidence intervals that were not subject to p-hacking or publication bias.

CONCLUSIONS

The huge excesses of published confidence intervals that are just below the statistically significant threshold are not statistically plausible. Large improvements in research practice are needed to provide more results that better reflect the truth.

Electronic Screening for Alcohol Use and Brief Intervention by Email for University Students: Reanalysis of Findings From a Randomized Controlled Trial Using a Bayesian Framework

BACKGROUND

Almost a decade ago, Sweden became the first country to implement a national system enabling student health care centers across all universities to routinely administer (via email) an electronic alcohol screening and brief intervention to their students. The Alcohol email assessment and feedback study dismantling effectiveness for university students (AMADEUS-1) trial aimed to assess the effect of the student health care centers' routine practices by exploiting the lack of any standard timing for the email invitation and by masking trial participation from students. The original analyses adopted the conventional null hypothesis framework, and the results were consistently in the expected direction. However, since for some tests the P values did not pass the conventional .05 threshold, some of the analyses were necessarily inconclusive.

OBJECTIVE

The outcomes of the AMADEUS-1 trial were derived from the first 3 items of the Alcohol Use Disorders Identification Test (AUDIT-C). The aim of this paper was to reanalyze the two primary outcomes of the AMADEUS-1 trial (AUDIT-C scores and prevalence of risky drinking), using the same models used in the original publication but applying a Bayesian inference framework and interpretation.

METHODS

The same regression models used in the original analysis were employed in this reanalysis (linear and logistic regression). Model parameters were given uniform priors. Markov chain Monte Carlo was used for Bayesian inference, and posterior probabilities were calculated for prespecified thresholds of interest.

RESULTS

Where the null hypothesis tests showed inconclusive results, the Bayesian analysis showed that offering an intervention at baseline was preferable compared to offering nothing. At follow-up, the probability of a lower AUDIT-C score among those who had been offered an intervention at baseline was greater than 95%, as was the case when comparing the prevalence of risky drinking.

CONCLUSIONS

The Bayesian analysis allows for a more consistent perspective of the data collected in the trial, since dichotomization of evidence is not looked for at some arbitrary threshold. Results are presented that represent the data collected in the trial rather than trying to make conclusions about the existence of a population effect. Thus, policy makers can think about the value of keeping the national system without having to navigate the treacherous landscape of statistical significance.

TRIAL REGISTRATION

ISRCTN Registry ISRCTN28328154; http://www.isrctn.com/ISRCTN28328154.

Nicht replizieren: publizieren!?

Zusammenfassung. In der (Pädagogischen) Psychologie sind Replikationsstudien bislang extrem seltene Ausnahmen. Dieser Artikel legt dar, dass und warum Wiederholungsstudien unentbehrlich sind. Weiterhin wird der Frage nachgegangen, warum – trotz des enormen Mehrwerts – nahezu keine Replikationen publiziert werden und warum viele „Ergebnisse“ der psychologischen Forschung nicht replizierbar sind. Dass es sich bei diesen Sachverhalten nicht um Vermutungen handelt, wird durch vorliegende Untersuchungen belegt. Die Ursachen dafür liegen in verschiedenen – teilweise voneinander abhängigen – Ebenen des Wissenschaftssystems: die verbreitete – aber abwegige – Ansicht, „statistische Signifikanz“ indiziere auch die Wahrscheinlichkeit, einen Befund replizieren zu können; die Verwechslung von „statistisch signifikant“ mit relevant; die Unsitte, getestete Untersuchungshypothesen erst im Nachhinein (ex post), also in Kenntnis der Resultate einer Studie, aufgestellt zu haben, aber in der Publikation als theoretisch abgeleiteten Ausgangspunkt (d. h. a priori formuliert) auszugeben; die α-Fehler-Inflationierung durch multiple statistische Signifikanztestungen; das exklusive Berichten von Ergebnissen, welche die Forschungshypothesen stützen, verbunden mit dem Unterschlagen abweichender Befunde; mangelnde Konstruktvalidität der verwendeten Messinstrumente; Lug und Betrug in der Wissenschaft; die Geringschätzung von Replikationen durch Zeitschriftenherausgeber, Gutachter und Drittmittelgeber. All das führt dazu, dass fast ausschließlich „statistisch signifikante“ und „neue“ Ergebnisse veröffentlicht werden und falsche Theorien persistieren. Als Gegenmaßnahmen werden beispielhaft genannt: eine großzügige finanzielle Förderung von Replikationsprojekten und ihrer Publikation; die nachdrückliche gutachterliche Befürwortung der Veröffentlichung methodisch adäquater Wiederholungsstudien; die Bereitschaft von Fachzeitschriften, dafür genug Platz bereitzustellen; die Anerkennung des großen wissenschaftlichen Werts von Wiederholungsstudien, auch in Berufungsverfahren. Daraus ergibt sich, dass mit den aufgezeigten Möglichkeiten und Forderungen zur Etablierung und Förderung von Replikationsstudien unterschiedliche Adressaten parallel angesprochen werden müssen. Nachhaltige Veränderungen sind allerdings nur erreichbar, wenn die einzelnen Akteure (Forscher; Gutachter; Zeitschriftenherausgeber; Berufungskommissionen; Drittmittelgeber) ihre individuelle Verantwortung anerkennen und entsprechende Taten folgen lassen.

Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals

OBJECTIVES

To describe and summarise how the results of randomised controlled trials (RCTs) that did not find a significant treatment effect are reported, and to estimate how commonly trial reports make unwarranted claims.

DESIGN

We performed a retrospective survey of published RCTs, published in four high impact factor general medical journals between June 2016 and June 2017.

SETTING

Trials conducted in all settings were included.

PARTICIPANTS

94 reports of RCTs that did not find a difference in their main comparison or comparisons were included.

INTERVENTIONS

All interventions.

PRIMARY AND SECONDARY OUTCOMES

We recorded the way the results of each trial for its primary outcome or outcomes were described in Results and Conclusions sections of the Abstract, using a 10-category classification. Other outcomes were whether confidence intervals (CIs) and p values were presented for the main treatment comparisons, and whether the results and conclusions referred to measures of uncertainty. We estimated the proportion of papers that made claims that were not justified by the results, or were open to multiple interpretations.

RESULTS

94 trial reports (120 treatment comparisons) were included. In Results sections, for 58/120 comparisons (48.3%) the results of the study were re-stated, without interpretation, and 38/120 (31.7%) stated that there was no statistically significant difference. In Conclusions, 65/120 treatment comparisons (54.2%) stated that there was no treatment benefit, 14/120 (11.7%) that there was no significant benefit and 16/120 (13.3%) that there was no significant difference. CIs and p values were both presented by 84% of studies (79/94), but only 3/94 studies referred to uncertainty when drawing conclusions.

CONCLUSIONS

The majority of trials (54.2%) inappropriately interpreted a result that was not statistically significant as indicating no treatment benefit. Very few studies interpreted the result as indicating a lack of evidence against the null hypothesis of zero difference between the trial arms.

A Bayesian analysis of longitudinal farm surveys in Central Malawi reveals yield determinants and site-specific management strategies

Understanding the challenges to increasing maize productivity in sub-Saharan Africa, especially agronomic factors that reduce on-farm crop yield, has important implications for policies to reduce national and global food insecurity. Previous research on the maize yield gap has tended to emphasize the size of the gap (theoretical vs. achievable yields), rather than what determines maize yield in specific contexts. As a result, there is insufficient evidence on the key agronomic and environmental factors that influence maize yield in a smallholder farm environment. In this study, we implemented a Bayesian analysis with plot-level longitudinal household survey data covering 1,197 plots and 320 farms in Central Malawi. Households were interviewed and monitored three times per year, in 2015 and 2016, to document farmer management practices and seasonal rainfall, and direct measurements were taken of plant and soil characteristics to quantify impact on plot-level maize yield stability. The results revealed a high positive association between a leaf chlorophyll indicator and maize yield, with significance levels exceeding 95% Bayesian credibility at all sites and a regression coefficient posterior mean from 28% to 42% on a relative scale. A parasitic weed, Striga asiatica, was the variable most consistently negatively associated with maize yield, exceeding 95% credibility in most cases, of high intensity, with regression means ranging from 23% to 38% on a relative scale. The influence of rainfall, either directly or indirectly, varied by site and season. We conclude that the factors preventing Striga infestation and enhancing nitrogen fertility will lead to higher maize yield in Malawi. To improve plant nitrogen status, fertilizer was effective at higher productivity sites, whereas soil carbon and organic inputs were important at marginal sites. Uniquely, a Bayesian approach allowed differentiation of response by site for a relatively modest sample size study (given the complexity of farm environments and management practices). Considering the biophysical constraints, our findings highlight management strategies for crop yields, and point towards area-specific recommendations for nitrogen management and crop yield.