Manipulating the Alpha Level Cannot Cure Significance Testing

The null hypothesis significance test and the dichotomization of the p-value: Errare Humanum Est

Decision-making in healthcare is complex and needs to be based on the best scientific evidence. In this process, information derived from statistical analysis of data is crucial, which can be developed from either frequentist or Bayesian perspectives. When it comes to the frequentist field, the null hypothesis significance test (NHST) and its p-value is one of the most widely used techniques in different disciplines. However, NHST has been subjected to questioning from different academic points of view, which has led to it being considered as one of the causes of the so-called replicability crisis in science. In this review article, we provide a brief historical account of its development, summarize the underlying methods, describe some controversies and limitations, address misuse and misinterpretation, and finally give some scopes and reflections in the context of biomedical research.

Some Contributions from Embodied Cognition to Psychonarratology

Psychonarratology is a discipline that combines classic frameworks in narratology and psycholinguistics. The goal of this review article is to highlight the contributions that recent evidence from causality processing studies, conceptual metaphor theory and embodied cognition can make to Psychonarratology, in order to promote a more comprehensive study of narrative. We argue that, in order to increase its descriptive and explanatory power, Psychonarratology would benefit from a more strongly interdisciplinary approach. This approach would integrate grounded theoretical cognition and recent methods from different disciplines. With this aim, we review study evidence that highlights the fundamental link between cognition and causal connections in narratives, as well as proposals from conceptualization theories. These approaches indicate that metaphorical mappings and discourse connections play a crucial role in the establishment of narrative sequences. Moreover, we elaborate on the contributions that recent advances in research on embodied cognition, causality processing and Conceptual Metaphor Theory can make to the theoretical and methodological framework of Psychonarratology, such as promoting the design of more ecologically valid tasks, and the study of narrative production and comprehension by non-neurotypical participants.

Wings of Change: aPKC/FoxP-dependent plasticity in steering motor neurons underlies operant self-learning in Drosophila

Background

Motor learning is central to human existence, such as learning to speak or walk, sports moves, or rehabilitation after injury. Evidence suggests that all forms of motor learning share an evolutionarily conserved molecular plasticity pathway. Here, we present novel insights into the neural processes underlying operant self-learning, a form of motor learning in the fruit fly Drosophila.

Methods

We operantly trained wild type and transgenic Drosophila fruit flies, tethered at the torque meter, in a motor learning task that required them to initiate and maintain turning maneuvers around their vertical body axis (yaw torque). We combined this behavioral experiment with transgenic peptide expression, CRISPR/Cas9-mediated, spatio-temporally controlled gene knock-out and confocal microscopy.

Results

We find that expression of atypical protein kinase C (aPKC) in direct wing steering motoneurons co-expressing the transcription factor FoxP is necessary for this type of motor learning and that aPKC likely acts via non-canonical pathways. We also found that it takes more than a week for CRISPR/Cas9-mediated knockout of FoxP in adult animals to impair motor learning, suggesting that adult FoxP expression is required for operant self-learning.

Conclusions

Our experiments suggest that, for operant self-learning, a type of motor learning in Drosophila, co-expression of atypical protein kinase C (aPKC) and the transcription factor FoxP is necessary in direct wing steering motoneurons. Some of these neurons control the wing beat amplitude when generating optomotor responses, and we have discovered modulation of optomotor behavior after operant self-learning. We also discovered that aPKC likely acts via non-canonical pathways and that FoxP expression is also required in adult flies.

The hazards of dealing with response time outliers

The presence of outliers in response times can affect statistical analyses and lead to incorrect interpretation of the outcome of a study. Therefore, it is a widely accepted practice to try to minimize the effect of outliers by preprocessing the raw data. There exist numerous methods for handling outliers and researchers are free to choose among them. In this article, we use computer simulations to show that serious problems arise from this flexibility. Choosing between alternative ways for handling outliers can result in the inflation of p-values and the distortion of confidence intervals and measures of effect size. Using Bayesian parameter estimation and probability distributions with heavier tails eliminates the need to deal with response times outliers, but at the expense of opening another source of flexibility.

The replication crisis has led to positive structural, procedural, and community changes

The emergence of large-scale replication projects yielding successful rates substantially lower than expected caused the behavioural, cognitive, and social sciences to experience a so-called 'replication crisis'. In this Perspective, we reframe this 'crisis' through the lens of a credibility revolution, focusing on positive structural, procedural and community-driven changes. Second, we outline a path to expand ongoing advances and improvements. The credibility revolution has been an impetus to several substantive changes which will have a positive, long-term impact on our research environment.

Trials and tribulations of statistical significance in biochemistry and omics

Over recent years many statisticians and researchers have highlighted that statistical inference would benefit from a better use and understanding of hypothesis testing, p-values, and statistical significance. We highlight three recommendations in the context of biochemical sciences. First recommendation: to improve the biological interpretation of biochemical data, do not use p-values (or similar test statistics) as thresholded values to select biomolecules. Second recommendation: to improve comparison among studies and to achieve robust knowledge, perform complete reporting of data. Third recommendation: statistical analyses should be reported completely with exact numbers (not as asterisks or inequalities). Owing to the high number of variables, a better use of statistics is of special importance in omic studies.

Analysis of proportions using arcsine transform with any experimental design

Introduction

Exact tests on proportions exist for single-group and two-group designs, but no general test on proportions exists that is appropriate for any experimental design involving more than two groups, repeated measures, and/or factorial designs.

Method

Herein, we extend the analysis of proportions using arcsine transform to any sort of design. The resulting framework, which we have called Analysis of Proportions Using Arcsine Transform (ANOPA), is completely analogous to the analysis of variance for means of continuous data, allowing the examination of interactions, main and simple effects, post-hoc tests, orthogonal contrasts, et cetera.

Result

We illustrate the method with a few examples (single-factor design, two-factor design, within-subject design, and mixed design) and explore type I error rates with Monte Carlo simulations. We also examine power computation and confidence intervals for proportions.

Discussion

ANOPA is a complete series of analyses for proportions, applicable to any design.

Why and how we should join the shift from significance testing to estimation

A paradigm shift away from null hypothesis significance testing seems in progress. Based on simulations, we illustrate some of the underlying motivations. First, p-values vary strongly from study to study, hence dichotomous inference using significance thresholds is usually unjustified. Second, 'statistically significant' results have overestimated effect sizes, a bias declining with increasing statistical power. Third, 'statistically non-significant' results have underestimated effect sizes, and this bias gets stronger with higher statistical power. Fourth, the tested statistical hypotheses usually lack biological justification and are often uninformative. Despite these problems, a screen of 48 papers from the 2020 volume of the Journal of Evolutionary Biology exemplifies that significance testing is still used almost universally in evolutionary biology. All screened studies tested default null hypotheses of zero effect with the default significance threshold of p = 0.05, none presented a pre-specified alternative hypothesis, pre-study power calculation and the probability of 'false negatives' (beta error rate). The results sections of the papers presented 49 significance tests on average (median 23, range 0-390). Of 41 studies that contained verbal descriptions of a 'statistically non-significant' result, 26 (63%) falsely claimed the absence of an effect. We conclude that studies in ecology and evolutionary biology are mostly exploratory and descriptive. We should thus shift from claiming to 'test' specific hypotheses statistically to describing and discussing many hypotheses (possible true effect sizes) that are most compatible with our data, given our statistical model. We already have the means for doing so, because we routinely present compatibility ('confidence') intervals covering these hypotheses.

Beyond p < .05: a critical review of new Bayesian proposals for assessing the p-value

This paper reviews recent contributions from a Bayesian-oriented perspective, after the ASA statement on p-values (2016). We classify proposals that (i) supplement the p-value; (ii) modify the p-value itself. In the first group, we review the Bayes factor, the False Positive risk, the rejection odds and the analysis of credibility from both Matthews' and Held's point of view. We also put forth and discuss a new index of credibility, about which we conduct a delimited simulation study. In the second group, we discuss Gannon's modification of the p-value based on the Bayes factor and the second-generation p-value. The theory is illustrated with two case studies on pharmacotherapy in infectious diseases. Contemporary authors still refer to the p-value as a statistical indicator but have abandoned the perspective of evaluating p-values with fixed thresholds. Statistical societies worldwide should target new strategies to disseminate the debate on p-values in all applied fields of knowledge, as well as they may promote the use of different statistical procedures to supplement p-values.

Testing the Reproducibility of the Effects of Transcranial Direct Current Stimulation: Failure to Modulate Beauty Perception by Brain Stimulation

Transcranial direct current stimulation (tDCS) has been recognized as a promising tool for investigating the causal relationship between specific brain areas of interest and behavior. However, the reproducibility of previous tDCS studies is often questioned because of failures in replication. This study focused on the effects of tDCS on one cognitive domain: beauty perception. To date, the modulation of beauty perception by tDCS has been shown in two studies: Cattaneo et al. (2014) and Nakamura and Kawabata (2015). Here, we aimed at replicating their studies and investigating the effects of tDCS on beauty perception using the following parameters: (1) cathodal stimulation over the medial prefrontal cortex (mPFC) (Nakamura and Kawabata, 2015); (2) anodal stimulation over the left dorsolateral prefrontal cortex (lDLPFC) (Cattaneo et al., 2014). We also performed a more focal stimulation targeting the orbitofrontal cortex (OFC) to determine the optimal stimulation site for modulating beauty perception (3). Participants rated the subjectively-perceived beauty of the images before and after the tDCS administration. We divided images into four clusters according to the obtained scores in our preliminary study and examined changes in beauty ratings in each image cluster separately to exclude factors, such as stimuli attributions that may reduce tDCS effects. The results showed no strong effects of tDCS with the same parameters as in previous studies on beauty rating scores in any image cluster. Likewise, anodal stimulation over the OFC did not result in a change in rating scores. In contrast to previous studies, the current study did not corroborate the effects of tDCS on beauty perception. Our findings provide evidence regarding the recent reproducibility issue of tDCS effects and suggest the possible inflation of its effects on cognitive domains.

Additive genetic variance for traits least related to fitness increases with environmental stress in the desert locust, Schistocerca gregaria

Under environmental stress, previously hidden additive genetic variation can be unmasked and exposed to selection. The amount of hidden variation is expected to be higher for life history traits, which strongly correlate to individual fitness, than for morphological traits, in which fitness effects are more ambiguous. However, no consensual pattern has been recovered yet, and this idea is still debated in the literature. Here, we hypothesize that the classical categorization of traits (i.e., life history and morphology) may fail to capture their proximity to fitness. In the desert locust, Schistocerca gregaria, a model organism for the study of insect polyphenism, we quantified changes in additive genetic variation elicited by lifetime thermal stress for ten traits, in which evolutionary significance is known. Irrespective of their category, traits under strong stabilizing selection showed genetic invariance with environmental stress, while traits more loosely associated with fitness showed a marked increase in additive genetic variation in the stressful environment. Furthermore, traits involved in adaptive phenotypic plasticity (growth compensation) showed either no change in additive genetic variance or a change of moderate magnitude across thermal environments. We interpret this mitigated response of plastic traits in the context of integrated evolution to adjust the entire phenotype in heterogeneous environments (i.e., adaptiveness of initial plasticity, compromise of phenotypic compensation with stress, and shared developmental pathway). Altogether, our results indicate, in agreement with theoretical expectations, that environmental stress can increase available additive genetic variance in some desert locust traits, but those closely linked to fitness are largely unaffected. Our study also highlights the importance of assessing the proximity to fitness of a trait on a case-by-case basis and in an ecologically relevant context, as well as considering the processes of canalization and plasticity, involved in the control of phenotypic variation.

New relevance and significance measures to replace p-values

The p-value has been debated exorbitantly in the last decades, experiencing fierce critique, but also finding some advocates. The fundamental issue with its misleading interpretation stems from its common use for testing the unrealistic null hypothesis of an effect that is precisely zero. A meaningful question asks instead whether the effect is relevant. It is then unavoidable that a threshold for relevance is chosen. Considerations that can lead to agreeable conventions for this choice are presented for several commonly used statistical situations. Based on the threshold, a simple quantitative measure of relevance emerges naturally. Statistical inference for the effect should be based on the confidence interval for the relevance measure. A classification of results that goes beyond a simple distinction like “significant / non-significant” is proposed. On the other hand, if desired, a single number called the “secured relevance” may summarize the result, like the p-value does it, but with a scientifically meaningful interpretation.

Permutation tests are robust and powerful at 0.5% and 5% significance levels

Recent replication crisis has led to a number of ad hoc suggestions to decrease the chance of making false positive findings. Among them, Johnson (Proceedings of the National Academy of Sciences, 110, 19313-19317, 2013) and Benjamin et al. (Nature Human Behaviour, 2, 6-10 2018) recommend using the significance level of α = 0.005 (0.5%) as opposed to the conventional 0.05 (5%) level. Even though their suggestion is easy to implement, it is unclear whether or not the commonly used statistical tests are robust and/or powerful at such a small significance level. Therefore, the main aim of our study is to investigate the robustness and power curve behaviors of independent (unpaired) two-sample tests for metric and ordinal data at nominal significance levels of α = 0.005 and α = 0.05. Through an extensive simulation study, it is found that the permutation versions of the Welch t-test and the Brunner-Munzel test are particularly robust and powerful while the commonly used two-sample tests which utilize t-distribution tend to be either liberal or conservative, and have peculiar power curve behaviors under skewed distributions with variance heterogeneity.

Teachers as co-regulators of children's emotions: A descriptive study of teacher-child emotion dialogues in special education

BACKGROUND

The study examined how teachers and children with emotional and behavioral disturbances engage in dialogues about children's emotional experiences. Dialogues about emotions are an important strategy for teachers to co-regulate children's emotions but have remained understudied.

AIMS

This study aimed to explore whether the Autobiographical Emotional Events Dialogue (AEED) can help to assess the quality of teacher-child emotion dialogues about past emotional events and examined associations with child behavior and teacher-child relationship quality.

METHOD

The sample included 85 children and 70 teachers from special education schools serving children with emotional and behavioral disturbances. Teacher-child dialogues were videotaped and coded using the 16 rating scales of the AEED coding system (Koren-Karie, Oppenheim, Carasso, & Haimovich, 2003).

RESULTS

The scales (except child boundary dissolution) could be reliably assessed. A Principal Component Analysis yielded 4 factors: Adequate task completion (coherent dialogues and positive child task behavior), Negativity (hostility and teacher boundary dissolution), Teacher Guidance (involvement, structuring, and acceptance), and Resolution (positive closure of negative stories). Child age, verbal intelligence, prosocial behavior, and higher teacher-child relationship scores (higher closeness, lower conflict) were positively associated with the quality of the dialogues but behavior problems were not.

CONCLUSIONS AND IMPLICATIONS

The study provides first insight in teachers' scaffolding of dialogues with children about negative emotional events in special education serving children with emotional and behavioral disturbances.

Farewell to Bright-Line: A Guide to Reporting Quantitative Results Without the S-Word

Recent calls to end the practice of categorizing findings based on statistical significance have focused on what not to do. Practitioners who subscribe to the conceptual basis behind these calls may be unaccustomed to presenting results in the nuanced and integrative manner that has been recommended as an alternative. This alternative is often presented as a vague proposal. Here, we provide practical guidance and examples for adopting a research evaluation posture and communication style that operates without bright-line significance testing. Characteristics of the structure of results communications that are based on conventional significance testing are presented. Guidelines for writing results without the use of bright-line significance testing are then provided. Examples of conventional styles for communicating results are presented. These examples are then modified to conform to recent recommendations. These examples demonstrate that basic modifications to written scientific communications can increase the information content of scientific reports without a loss of rigor. The adoption of alternative approaches to results presentations can help researchers comply with multiple recommendations and standards for the communication and reporting of statistics in the psychological sciences.

Statistical significance: p value, 0.05 threshold, and applications to radiomics-reasons for a conservative approach

Here, we summarise the unresolved debate about p value and its dichotomisation. We present the statement of the American Statistical Association against the misuse of statistical significance as well as the proposals to abandon the use of p value and to reduce the significance threshold from 0.05 to 0.005. We highlight reasons for a conservative approach, as clinical research needs dichotomic answers to guide decision-making, in particular in the case of diagnostic imaging and interventional radiology. With a reduced p value threshold, the cost of research could increase while spontaneous research could be reduced. Secondary evidence from systematic reviews/meta-analyses, data sharing, and cost-effective analyses are better ways to mitigate the false discovery rate and lack of reproducibility associated with the use of the 0.05 threshold. Importantly, when reporting p values, authors should always provide the actual value, not only statements of "p < 0.05" or "p ≥ 0.05", because p values give a measure of the degree of data compatibility with the null hypothesis. Notably, radiomics and big data, fuelled by the application of artificial intelligence, involve hundreds/thousands of tested features similarly to other "omics" such as genomics, where a reduction in the significance threshold, based on well-known corrections for multiple testing, has been already adopted.

The Power of Replicated Measures to Increase Statistical Power

When running statistical tests, researchers can commit a Type II error, that is, fail to reject the null hypothesis when it is false. To diminish the probability of committing a Type II error (β), statistical power must be augmented. Typically, this is done by increasing sample size, as more participants provide more power. When the estimated effect size is small, however, the sample size required to achieve sufficient statistical power can be prohibitive. To alleviate this lack of power, a common practice is to measure participants multiple times under the same condition. Here, we show how to estimate statistical power by taking into account the benefit of such replicated measures. To that end, two additional parameters are required: the correlation between the multiple measures within a given condition and the number of times the measure is replicated. An analysis of a sample of 15 studies (total of 298 participants and 38,404 measurements) suggests that in simple cognitive tasks, the correlation between multiple measures is approximately .14. Although multiple measurements increase statistical power, this effect is not linear, but reaches a plateau past 20 to 50 replications (depending on the correlation). Hence, multiple measurements do not replace the added population representativeness provided by additional participants.

A Frequentist Alternative to Significance Testing, p-Values, and Confidence Intervals

There has been much debate about null hypothesis significance testing, p-values without null hypothesis significance testing, and confidence intervals. The first major section of the present article addresses some of the main reasons these procedures are problematic. The conclusion is that none of them are satisfactory. However, there is a new procedure, termed the a priori procedure (APP), that validly aids researchers in obtaining sample statistics that have acceptable probabilities of being close to their corresponding population parameters. The second major section provides a description and review of APP advances. Not only does the APP avoid the problems that plague other inferential statistical procedures, but it is easy to perform too. Although the APP can be performed in conjunction with other procedures, the present recommendation is that it be used alone.

Sparse Sampling of Silence Type I Errors With an Emphasis on Primary Auditory Cortex

Sparse sampling functional MRI (ssfMRI) enables stronger primary auditory cortex blood oxygen level-dependent (BOLD) signal by acquiring volumes interspersed with silence, reducing the physiological artifacts associated with scanner noise. Recent calculations of type I error rates associated with resting-state fMRI suggest that the techniques used to model the hemodynamic response function (HRF) might be resulting in higher false positives than is generally acceptable. In the present study, we analyze ssfMRI to determine type I error rates associated with whole brain and primary auditory cortex voxel-wise activation patterns. Study participants (n = 15, age 27.62 ± 3.21 years, range: 22–33 years; 6 females) underwent ssfMRI. An optimized paradigm was used to determine the HRF to auditory stimuli, which was then substituted for silent stimuli to ascertain false positives. We report that common techniques used for analyzing ssfMRI result in high type I error rates. The whole brain and primary auditory cortex voxel-wise analysis resulted in similar error distributions. The number of type I errors for P < 0.05, P < 0.01, and P < 0.001 for the whole brain was 7.88 ± 9.29, 2.37 ± 3.54, and 0.53 ± 0.96% and for the auditory cortex was 9.02 ± 1.79, 2.95 ± 0.91, and 0.58 ± 0.21%, respectively. When conducting a ssfMRI analysis, conservative α level should be employed (α < 0.001) to bolster the results in the face of false positive results.

In-hive variation of the gut microbial composition of honey bee larvae and pupae from the same oviposition time

BACKGROUND

Knowledge of microbiota composition, persistence, and transmission as well as the overall function of the bacterial community is important and may be linked to honey bee health. This study aimed to investigate the inter-individual variation in the gut microbiota in honey bee larvae and pupae.

RESULTS

Individual larvae differed in the composition of major bacterial groups. In the majority of 5th instar bees, Firmicutes showed predominance (70%); however, after larval defecation and during pupation, the abundance decreased to 40%, in favour of Gammaproteobacteria. The 5th instar larvae hosted significantly more (P < 0.001) Firmicutes than black pupae. Power calculations revealed that 11 and 18 replicate-individuals, respectively, were required for the detection of significant differences (P < 0.05) in the Bacteroidetes and Firmicutes abundance between stages, while higher numbers of replicates were required for Actinobacteria (478 replicates) and Gammaproteobacteria (111 replicates).

CONCLUSIONS

Although sample processing and extraction protocols may have had a significant influence, sampling is very important for studying the bee microbiome, and the importance of the number of individuals pooled in samples used for microbiome studies should not be underestimated.

An Introduction to Bayesian Multilevel Models Using brms: A Case Study of Gender Effects on Vowel Variability in Standard Indonesian

Purpose Bayesian multilevel models are increasingly used to overcome the limitations of frequentist approaches in the analysis of complex structured data. This tutorial introduces Bayesian multilevel modeling for the specific analysis of speech data, using the brms package developed in R. Method In this tutorial, we provide a practical introduction to Bayesian multilevel modeling by reanalyzing a phonetic data set containing formant (F1 and F2) values for 5 vowels of standard Indonesian (ISO 639-3:ind), as spoken by 8 speakers (4 females and 4 males), with several repetitions of each vowel. Results We first give an introductory overview of the Bayesian framework and multilevel modeling. We then show how Bayesian multilevel models can be fitted using the probabilistic programming language Stan and the R package brms, which provides an intuitive formula syntax. Conclusions Through this tutorial, we demonstrate some of the advantages of the Bayesian framework for statistical modeling and provide a detailed case study, with complete source code for full reproducibility of the analyses ( https://osf.io/dpzcb /). Supplemental Material https://doi.org/10.23641/asha.7973822.

Effect of fine and gross motor training or motor imagery, delivered via novel or routine modes, on cognitive function

There is extensive literature linking motor activity to cognitive effects at various stages in life, promoting both development and the reduction of aging associated pathologies. It is unclear whether the benefits of this activity on the cognitive level are associated with brain functions that are necessary for their performance or recurrence of activity or type of activity itself. The aim of this study was to evaluate whether the type of motor activity (fine, gross, and motor imagery) in two modes (novel and routine) can affect cognitive functions such as attention, executive functions, and praxis in college students. A 2 × 3 factorial design with repeated measures was used without a control group and pre- and post-training evaluation. Fifty-three young people (14 men and 39 women) participated, with mean age of 18.94 years (SD = 1.61 years) and were divided into six groups. Each of the groups performed relevant training 20 minutes per day for five days depending on the group. Measures were taken pre and post-training for attention tests, attention span, working memory, visual constructive skills, procedural memory, and motor skills. The results show a "learning effect" from the exposure to the tests in measurements after training. It was also found that between groups, there is a difference in some of the variables of procedural memory (number of errors) and working memory. More extensive training could better reflect the effects of the training, and longitudinal evaluation could show the rate of change of functions. The main clinical implication could be the evaluation of training programs for recovery and motor training in cerebral plasticity having effect on the cognitive aspects.

Presidential address: Six open questions to genetic epidemiologists

Given the rapid pace with which genomics and other ‐omics disciplines are evolving, it is sometimes necessary to shift down a gear to consider more general scientific questions. In this line, in my presidential address I formulate six questions for genetic epidemiologists to ponder on. These cover the areas of reproducibility, statistical significance, chance findings, precision medicine and related fields such as bioinformatics and data science. Possible hints at responses are presented to foster our further discussion of these topics.

Paving the Way to Eureka-Introducing "Dira" as an Experimental Paradigm to Observe the Process of Creative Problem Solving

"Dira" is a novel experimental paradigm to record combinations of behavioral and metacognitive measures for the creative process. This task allows assessing chronological and chronometric aspects of the creative process directly and without a detour through creative products or proxy phenomena. In a study with 124 participants we show that (a) people spend more time attending to selected vs. rejected potential solutions, (b) there is a clear connection between behavioral patterns and self-reported measures, (c) the reported intensity of Eureka experiences is a function of interaction time with potential solutions, and (d) experiences of emerging solutions can happen immediately after engaging with a problem, before participants explore all potential solutions. The conducted study exemplifies how "Dira" can be used as an instrument to narrow down the moment when solutions emerge. We conclude that the "Dira" experiment is paving the way to study the process, as opposed to the product, of creative problem solving.

The replication crisis in epidemiology: snowball, snow job, or winter solstice?

PURPOSE OF REVIEW

Like a snowball rolling down a steep hill, the most recent crisis over the perceived lack of reproducibility of scientific results has outpaced the evidence of crisis. It has led to new actions and new guidelines that have been rushed to market without plans for evaluation, metrics for success, or due consideration of the potential for unintended consequences.

RECENT FINDINGS

The perception of the crisis is at least partly a snow job, heavily influenced by a small number of centers lavishly funded by a single foundation, with undue and unsupported attention to preregistration as a solution to the perceived crisis. At the same time, the perception of crisis provides an opportunity for introspection. Two studies' estimates of association may differ because of undue attention on null hypothesis statistical testing, because of differences in the distribution of effect modifiers, because of differential susceptibility to threats to validity, or for other reasons. Perhaps the expectation of what reproducible epidemiology ought to look like is more misguided than the practice of epidemiology. We advocate for the idea of "replication and advancement." Studies should not only replicate earlier work, but also improve on it in by enhancing the design or analysis.

SUMMARY

Abandoning blind reliance on null hypothesis significance testing for statistical inference, finding consensus on when pre-registration of non-randomized study protocols has merit, and focusing on replication and advance are the most certain ways to emerge from this solstice for the better.