Ethics and Statistics: The AAA Tranche of Subprime Science

Relationship between self-reported sleep and cognitive function: a specification curve analysis

The relationship between self-reported sleep and cognitive function is complex; it is unclear whether self-reported sleep is a robust correlate of people's cognitive function. We address this gap by using a comprehensive large-scale dataset (N = 1054) coupled with a novel modeling approach, specification curve analysis (SCA), to test the association between self-reported sleep and cognitive function. The results of the SCA showed robust correlations between self-reported sleep and cognitive function, with poorer sleep associated with worse cognitive function. Furthermore, the correlations between sleep components and cognitive function were heterogeneous, with differences emerging across cognitive measures and domains. Specifically, daytime dysfunction was associated with the strongest effect on subjective cognitive function, whereas sleep duration and sleep efficiency had the strongest effect on objective cognitive function. Therefore, the relationship between self-reported sleep and cognition depends largely on what and how cognitive function is measured. Our findings guide measurement and domain selection for future research on the role of sleep in cognitive function.

Advances in the field of intranasal oxytocin research: lessons learned and future directions for clinical research

Reports on the modulatory role of the neuropeptide oxytocin on social cognition and behavior have steadily increased over the last two decades, stimulating considerable interest in its psychiatric application. Basic and clinical research in humans primarily employs intranasal application protocols. This approach assumes that intranasal administration increases oxytocin levels in the central nervous system via a direct nose-to-brain route, which in turn acts upon centrally-located oxytocin receptors to exert its behavioral effects. However, debates have emerged on whether intranasally administered oxytocin enters the brain via the nose-to-brain route and whether this route leads to functionally relevant increases in central oxytocin levels. In this review we outline recent advances from human and animal research that provide converging evidence for functionally relevant effects of the intranasal oxytocin administration route, suggesting that direct nose-to-brain delivery underlies the behavioral effects of oxytocin on social cognition and behavior. Moreover, advances in previously debated methodological issues, such as pre-registration, reproducibility, statistical power, interpretation of non-significant results, dosage, and sex differences are discussed and integrated with suggestions for the next steps in translating intranasal oxytocin into psychiatric applications.

Changing the logic of replication: A case from infant studies

Among infant researchers there is growing concern regarding the widespread practice of undertaking studies that have small sample sizes and employ tests with low statistical power (to detect a wide range of possible effects). For many researchers, issues of confidence may be partially resolved by relying on replications. Here, we bring further evidence that the classical logic of confirmation, according to which the result of a replication study confirms the original finding when it reaches statistical significance, could be usefully abandoned. With real examples taken from the infant literature and Monte Carlo simulations, we show that a very wide range of possible replication results would in a formal statistical sense constitute confirmation as they can be explained simply due to sampling error. Thus, often no useful conclusion can be derived from a single or small number of replication studies. We suggest that, in order to accumulate and generate new knowledge, the dichotomous view of replication as confirmatory/disconfirmatory can be replaced by an approach that emphasizes the estimation of effect sizes via meta-analysis. Moreover, we discuss possible solutions for reducing problems affecting the validity of conclusions drawn from meta-analyses in infant research.

A web-based automated machine learning platform to analyze liquid biopsy data

Liquid biopsy (LB) technologies continue to improve in sensitivity, specificity, and multiplexing and can measure an ever growing library of disease biomarkers. However, clinical interpretation of the increasingly large sets of data these technologies generate remains a challenge. Machine learning is a popular approach to discover and detect signatures of disease. However, limited machine learning expertise in the LB field has kept the discipline from fully leveraging these tools and risks improper analyses and irreproducible results. In this paper, we develop a web-based automated machine learning tool tailored specifically for LB, where machine learning models can be built without the user's input. We also incorporate a differential privacy algorithm, designed to limit the effects of overfitting that can arise from users iteratively developing a panel with feedback from our platform. We validate our approach by performing a meta-analysis on 11 published LB datasets, and found that we had similar or better performance compared to those reported in the literature. Moreover, we show that our platform's performance improved when incorporating information from prior LB datasets, suggesting that this approach can continue to improve with increased access to LB data. Finally, we show that by using our platform the results achieved in the literature can be matched using 40% of the number of subjects in the training set, potentially reducing study cost and time. This self-improving and overfitting-resistant automatic machine learning platform provides a new standard that can be used to validate machine learning works in the LB field.

For and Against Methodologies: Some Perspectives on Recent Causal and Statistical Inference Debates

I present an overview of two methods controversies that are central to analysis and inference: That surrounding causal modeling as reflected in the "causal inference" movement, and that surrounding null bias in statistical methods as applied to causal questions. Human factors have expanded what might otherwise have been narrow technical discussions into broad philosophical debates. There seem to be misconceptions about the requirements and capabilities of formal methods, especially in notions that certain assumptions or models (such as potential-outcome models) are necessary or sufficient for valid inference. I argue that, once these misconceptions are removed, most elements of the opposing views can be reconciled. The chief problem of causal inference then becomes one of how to teach sound use of formal methods (such as causal modeling, statistical inference, and sensitivity analysis), and how to apply them without generating the overconfidence and misinterpretations that have ruined so many statistical practices.

Replication, Communication, and the Population Dynamics of Scientific Discovery

Many published research results are false (Ioannidis, 2005), and controversy continues over the roles of replication and publication policy in improving the reliability of research. Addressing these problems is frustrated by the lack of a formal framework that jointly represents hypothesis formation, replication, publication bias, and variation in research quality. We develop a mathematical model of scientific discovery that combines all of these elements. This model provides both a dynamic model of research as well as a formal framework for reasoning about the normative structure of science. We show that replication may serve as a ratchet that gradually separates true hypotheses from false, but the same factors that make initial findings unreliable also make replications unreliable. The most important factors in improving the reliability of research are the rate of false positives and the base rate of true hypotheses, and we offer suggestions for addressing each. Our results also bring clarity to verbal debates about the communication of research. Surprisingly, publication bias is not always an obstacle, but instead may have positive impacts—suppression of negative novel findings is often beneficial. We also find that communication of negative replications may aid true discovery even when attempts to replicate have diminished power. The model speaks constructively to ongoing debates about the design and conduct of science, focusing analysis and discussion on precise, internally consistent models, as well as highlighting the importance of population dynamics.