Between the Devil and the Deep Blue Sea: Tensions Between Scientific Judgement and Statistical Model Selection

A tutorial on fitting joint models of M/EEG and behavior to understand cognition

We present motivation and practical steps necessary to find parameter estimates of joint models of behavior and neural electrophysiological data. This tutorial is written for researchers wishing to build joint models of human behavior and scalp and intracranial electroencephalographic (EEG) or magnetoencephalographic (MEG) data, and more specifically those researchers who seek to understand human cognition. Although these techniques could easily be applied to animal models, the focus of this tutorial is on human participants. Joint modeling of M/EEG and behavior requires some knowledge of existing computational and cognitive theories, M/EEG artifact correction, M/EEG analysis techniques, cognitive modeling, and programming for statistical modeling implementation. This paper seeks to give an introduction to these techniques as they apply to estimating parameters from neurocognitive models of M/EEG and human behavior, and to evaluate model results and compare models. Due to our research and knowledge on the subject matter, our examples in this paper will focus on testing specific hypotheses in human decision-making theory. However, most of the motivation and discussion of this paper applies across many modeling procedures and applications. We provide Python (and linked R) code examples in the tutorial and appendix. Readers are encouraged to try the exercises at the end of the document.

Recent approaches in computational modelling for controlling pathogen threats

In this review, we assess the status of computational modelling of pathogens. We focus on three disparate but interlinked research areas that produce models with very different spatial and temporal scope. First, we examine antimicrobial resistance (AMR). Many mechanisms of AMR are not well understood. As a result, it is hard to measure the current incidence of AMR, predict the future incidence, and design strategies to preserve existing antibiotic effectiveness. Next, we look at how to choose the finite number of bacterial strains that can be included in a vaccine. To do this, we need to understand what happens to vaccine and non-vaccine strains after vaccination programmes. Finally, we look at within-host modelling of antibody dynamics. The SARS-CoV-2 pandemic produced huge amounts of antibody data, prompting improvements in this area of modelling. We finish by discussing the challenges that persist in understanding these complex biological systems.

Power to Detect What? Considerations for Planning and Evaluating Sample Size

ACADEMIC ABSTRACT

In the wake of the replication crisis, social and personality psychologists have increased attention to power analysis and the adequacy of sample sizes. In this article, we analyze current controversies in this area, including choosing effect sizes, why and whether power analyses should be conducted on already-collected data, how to mitigate the negative effects of sample size criteria on specific kinds of research, and which power criterion to use. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings), given the limitations of interest-based minimums or field-wide effect sizes. We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

PUBLIC ABSTRACT

Recently, social-personality psychology has been criticized for basing some of its conclusions on studies with low numbers of participants. As a result, power analysis, a mathematical way to ensure that a study has enough participants to reliably "detect" a given size of psychological effect, has become popular. This article describes power analysis and discusses some controversies about it, including how researchers should derive assumptions about effect size, and how the requirements of power analysis can be applied without harming research on hard-to-reach and marginalized communities. For novel research questions, we advocate that researchers base sample sizes on effects that are likely to be cost-effective for other people to implement (in applied settings) or to study (in basic research settings). We discuss two alternatives to power analysis, precision analysis and sequential analysis, and end with recommendations for improving the practices of researchers, reviewers, and journal editors in social-personality psychology.

Simulation-based prior knowledge elicitation for parametric Bayesian models

A central characteristic of Bayesian statistics is the ability to consistently incorporate prior knowledge into various modeling processes. In this paper, we focus on translating domain expert knowledge into corresponding prior distributions over model parameters, a process known as prior elicitation. Expert knowledge can manifest itself in diverse formats, including information about raw data, summary statistics, or model parameters. A major challenge for existing elicitation methods is how to effectively utilize all of these different formats in order to formulate prior distributions that align with the expert's expectations, regardless of the model structure. To address these challenges, we develop a simulation-based elicitation method that can learn the hyperparameters of potentially any parametric prior distribution from a wide spectrum of expert knowledge using stochastic gradient descent. We validate the effectiveness and robustness of our elicitation method in four representative simulation studies covering linear models, generalized linear models, and hierarchical models. Our results support the claim that our method is largely independent of the underlying model structure and adaptable to various elicitation techniques, including quantile-based, moment-based, and histogram-based methods.

Self-Controlled Feedback in Motor Learning: The Effects Depend on the Frequency of Request

The benefits of allowing learners to control when to receive knowledge of results (KR) compared to a yoked group has been recently challenged and postulated to be mild at best. A potential explanation for such dissident findings is that individuals differentially utilize the autonomy provided by the self-controlled condition, which, in its turn, affects the outcomes. Therefore, the present study investigated the effects of self-controlled KR on motor learning focusing on the frequency of KR requests when performing an anticipatory timing task. Self-controlled groups were created based on participants' KR frequency of request (High, Medium, and Low referring to fifth, third, and first quintile) and, then, Yoked groups were created self-control condition pairing the KR request of the Self-controlled groups. We also measured self-efficacy and processing time as means to verify potential correlates. The results supported the expected interaction. While no difference between self-controlled and yoked groups were found for low frequencies of KR, a moderate amount of KR request was related to better results for the self-controlled group. Nonetheless, the opposite trend was observed for high frequencies of KR; the yoked group was superior to the self-controlled group. The results of this study allow us to conclude that the choices made, and not just the possibility of choosing, seem to define the benefits of KR self-control in motor learning.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

No position-specific interference from prior lists in cued recognition: A challenge for position coding (and other) theories of serial memory

Position-specific intrusions of items from prior lists are rare but important phenomena that distinguish broad classes of theory in serial memory. They are uniquely predicted by position coding theories, which assume items on all lists are associated with the same set of codes representing their positions. Activating a position code activates items associated with it in current and prior lists in proportion to their distance from the activated position. Thus, prior list intrusions are most likely to come from the coded position. Alternative "item dependent" theories based on associations between items and contexts built from items have difficulty accounting for the position specificity of prior list intrusions. We tested the position coding account with a position-cued recognition task designed to produce prior list interference. Cuing a position should activate a position code, which should activate items in nearby positions in the current and prior lists. We presented lures from the prior list to test for position-specific activation in response time and error rate; lures from nearby positions should interfere more. We found no evidence for such interference in 10 experiments, falsifying the position coding prediction. We ran two serial recall experiments with the same materials and found position-specific prior list intrusions. These results challenge all theories of serial memory: Position coding theories can explain the prior list intrusions in serial recall and but not the absence of prior list interference in cued recognition. Item dependent theories can explain the absence of prior list interference in cued recognition but cannot explain the occurrence of prior list intrusions in serial recall.

Better Accuracy for Better Science . . . Through Random Conclusions

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

The Social Route to Abstraction: Interaction and Diversity Enhance Performance and Transfer in a Rule-Based Categorization Task

Capacities for abstract thinking and problem-solving are central to human cognition. Processes of abstraction allow the transfer of experiences and knowledge between contexts helping us make informed decisions in new or changing contexts. While we are often inclined to relate such reasoning capacities to individual minds and brains, they may in fact be contingent on human-specific modes of collaboration, dialogue, and shared attention. In an experimental study, we test the hypothesis that social interaction enhances cognitive processes of rule-induction, which in turn improves problem-solving performance. Through three sessions of increasing complexity, individuals and groups were presented with a problem-solving task requiring them to categorize a set of visual stimuli. To assess the character of participants' problem representations, after each training session, they were presented with a transfer task involving stimuli that differed in appearance, but shared relations among features with the training set. Besides, we compared participants' categorization behaviors to simulated agents relying on exemplar learning. We found that groups performed superior to individuals and agents in the training sessions and were more likely to correctly generalize their observations in the transfer phase, especially in the high complexity session, suggesting that groups more effectively induced underlying categorization rules from the stimuli than individuals and agents. Crucially, variation in performance among groups was predicted by semantic diversity in members' dialogical contributions, suggesting a link between social interaction, cognitive diversity, and abstraction.

Explaining the description-experience gap in risky decision-making: learning and memory retention during experience as causal mechanisms

When making decisions based on probabilistic outcomes, people guide their behavior using knowledge gathered through both indirect descriptions and direct experience. Paradoxically, how people obtain information significantly impacts apparent preferences. A ubiquitous example is the description-experience gap: individuals seemingly overweight low probability events when probabilities are described yet underweight them when probabilities must be experienced firsthand. A leading explanation for this fundamental gap in decision-making is that probabilities are weighted differently when learned through description relative to experience, yet a formal theoretical account of the mechanism responsible for such weighting differences remains elusive. We demonstrate how various learning and memory retention models incorporating neuroscientifically motivated learning mechanisms can explain why probability weighting and valuation parameters often are found to vary across description and experience. In a simulation study, we show how learning through experience can lead to systematically biased estimates of probability weighting when using a traditional cumulative prospect theory model. We then use hierarchical Bayesian modeling and Bayesian model comparison to show how various learning and memory retention models capture participants' behavior over and above changes in outcome valuation and probability weighting, accounting for description and experience-based decisions in a within-subject experiment. We conclude with a discussion of how substantive models of psychological processes can lead to insights that heuristic statistical models fail to capture.

Explaining human interactions on the road by large-scale integration of computational psychological theory

When humans share space in road traffic, as drivers or as vulnerable road users, they draw on their full range of communicative and interactive capabilities. Much remains unknown about these behaviors, but they need to be captured in models if automated vehicles are to coexist successfully with human road users. Empirical studies of human road user behavior implicate a large number of underlying cognitive mechanisms, which taken together are well beyond the scope of existing computational models. Here, we note that for all of these putative mechanisms, computational theories exist in different subdisciplines of psychology, for more constrained tasks. We demonstrate how these separate theories can be generalized from abstract laboratory paradigms and integrated into a computational framework for modeling human road user interaction, combining Bayesian perception, a theory of mind regarding others' intentions, behavioral game theory, long-term valuation of action alternatives, and evidence accumulation decision-making. We show that a model with these assumptions-but not simpler versions of the same model-can account for a number of previously unexplained phenomena in naturalistic driver-pedestrian road-crossing interactions, and successfully predicts interaction outcomes in an unseen data set. Our modeling results contribute to demonstrating the real-world value of the theories from which we draw, and address calls in psychology for cumulative theory-building, presenting human road use as a suitable setting for work of this nature. Our findings also underscore the formidable complexity of human interaction in road traffic, with strong implications for the requirements to set on development and testing of vehicle automation.

How Do People Come to Judge What Is "Reasonable"? Effects of Legal and Sociological Systems on Human Psychology

How do people decide what is reasonable? People often have to make those judgments, judgments that can influence tremendously consequential decisions-such as whether to indict someone in a legal proceeding. In this article, we take a situated cognition lens to review and integrate findings from social psychology, judgment and decision-making, communication, law, and sociology to generate a new framework for conceptualizing judgments of reasonableness and their implications for how people make decisions, particularly in the context of the legal system. We theorize that differences in structural and social contexts create information asymmetries that shape people's priors about what is and is not reasonable and how they update their priors in the face of new information. We use the legal system as a context for exploring the implications of the framework for both individual and collective decision-making and for considering the practical implications of the framework for inequities in law and social policy.

Successes and critical failures of neural networks in capturing human-like speech recognition

Natural and artificial audition can in principle acquire different solutions to a given problem. The constraints of the task, however, can nudge the cognitive science and engineering of audition to qualitatively converge, suggesting that a closer mutual examination would potentially enrich artificial hearing systems and process models of the mind and brain. Speech recognition - an area ripe for such exploration - is inherently robust in humans to a number transformations at various spectrotemporal granularities. To what extent are these robustness profiles accounted for by high-performing neural network systems? We bring together experiments in speech recognition under a single synthesis framework to evaluate state-of-the-art neural networks as stimulus-computable, optimized observers. In a series of experiments, we (1) clarify how influential speech manipulations in the literature relate to each other and to natural speech, (2) show the granularities at which machines exhibit out-of-distribution robustness, reproducing classical perceptual phenomena in humans, (3) identify the specific conditions where model predictions of human performance differ, and (4) demonstrate a crucial failure of all artificial systems to perceptually recover where humans do, suggesting alternative directions for theory and model building. These findings encourage a tighter synergy between the cognitive science and engineering of audition.

Constructs, Tape Measures, and Mercury

This is a Lewinian-field-theory approach to understanding the Graduate Record Examination (GRE) in the context of racism to contribute to the debate about whether graduate schools should remove GRE scores from admissions processes. Woo and colleagues (this issue) review the empirical literature on bias from a psychometric perspective. In this commentary, I challenge the definition of the underlying construct measured by the GRE and offer alternative definitions of what is measured. Next, drawing on an analogy from genome-wide association studies, I discuss how genomic models predicting height that are trained on data from European ancestral populations systematically underpredict the height of West Africans. Our access to data from tape measures, and their correlation with height, provide objective opportunities to audit our prediction. I discuss the implications of this when the criterion variable for validating the GRE is first-year grades. I then probe an analogy used by Woo and colleagues in which they assert that blaming the GRE for disparities in scores across groups is akin to blaming the thermometer for global warming. I describe racism as context for a field-theory approach to thinking about the limitations of this misguided analogy. Finally, I suggest pathways forward.

Linking research of biomedical datasets

Biomedical data preprocessing and efficient computing can be as important as the statistical methods used to fit the data; data processing needs to consider application scenarios, data acquisition and individual rights and interests. We review common principles, knowledge and methods of integrated research according to the whole-pipeline processing mechanism diverse, coherent, sharing, auditable and ecological. First, neuromorphic and native algorithms integrate diverse datasets, providing linear scalability and high visualization. Second, the choice mechanism of different preprocessing, analysis and transaction methods from raw to neuromorphic was summarized on the node and coordinator platforms. Third, combination of node, network, cloud, edge, swarm and graph builds an ecosystem of cohort integrated research and clinical diagnosis and treatment. Looking forward, it is vital to simultaneously combine deep computing, mass data storage and massively parallel communication.

The interpretation of computational model parameters depends on the context

Reinforcement Learning (RL) models have revolutionized the cognitive and brain sciences, promising to explain behavior from simple conditioning to complex problem solving, to shed light on developmental and individual differences, and to anchor cognitive processes in specific brain mechanisms. However, the RL literature increasingly reveals contradictory results, which might cast doubt on these claims. We hypothesized that many contradictions arise from two commonly-held assumptions about computational model parameters that are actually often invalid: That parameters generalize between contexts (e.g. tasks, models) and that they capture interpretable (i.e. unique, distinctive) neurocognitive processes. To test this, we asked 291 participants aged 8-30 years to complete three learning tasks in one experimental session, and fitted RL models to each. We found that some parameters (exploration / decision noise) showed significant generalization: they followed similar developmental trajectories, and were reciprocally predictive between tasks. Still, generalization was significantly below the methodological ceiling. Furthermore, other parameters (learning rates, forgetting) did not show evidence of generalization, and sometimes even opposite developmental trajectories. Interpretability was low for all parameters. We conclude that the systematic study of context factors (e.g. reward stochasticity; task volatility) will be necessary to enhance the generalizability and interpretability of computational cognitive models.

Formal models for the study of the relationship between fluctuating asymmetry and fitness in humans

OBJECTIVES

To evaluate three of the main verbal models that have been proposed to explain the relationship between fluctuating asymmetry and fitness in humans: the "good genes," the "good development," and the "growth" hypotheses.

MATERIALS AND METHODS

A formal model was generated for each verbal model following three steps. First, based on the literature, a theoretical causal model and the theoretical object of inquiry were outlined. Second, an empirical causal model and the targets of inference were defined using observational data of facial asymmetries and life-history traits related to fitness. Third, generalized linear models and causal inference were used as the estimation strategy.

RESULTS

The results suggest that the theoretical and empirical assumptions of the "good genes" hypothesis should be reformulated. The results were compatible with most of the empirical assumptions of "the good development" hypothesis but suggest that further discussion of its theoretical assumptions is needed. The results were less informative about the "growth" hypothesis, both theoretically and empirically. There was a positive association between facial fluctuating asymmetry and the number of offspring that was not compatible with any of the empirical causal models evaluated.

CONCLUSIONS

Although the three hypotheses focus on different aspects of the link between asymmetry and fitness, their overlap opens the possibility of a unified theory on the subject. The results of this study make explicit which assumptions need to be updated and discussed, facilitating the advancement of this area of research. Overall, this study elucidates the potential benefit of using formal models for theory revision and development.

Toward a more comprehensive modeling of sequential lineups

Sequential lineups are one of the most commonly used procedures in police departments across the USA. Although this procedure has been the target of much experimental research, there has been comparatively little work formally modeling it, especially the sequential nature of the judgments that it elicits. There are also important gaps in our understanding of how informative different types of judgments can be (binary responses vs. confidence ratings), and the severity of the inferential risks incurred when relying on different aggregate data structures. Couched in a signal detection theory (SDT) framework, the present work directly addresses these issues through a reanalysis of previously published data alongside model simulations. Model comparison results show that SDT modeling can provide elegant characterizations of extant data, despite some discrepancies across studies, which we attempt to address. Additional analyses compare the merits of sequential lineups (with and without a stopping rule) relative to showups and delineate the conditions in which distinct modeling approaches can be informative. Finally, we identify critical issues with the removal of the stopping rule from sequential lineups as an approach to capture within-subject differences and sidestep the risk of aggregation biases.

Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal

During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8-30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants' mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.

Context Matters: Recovering Human Semantic Structure from Machine Learning Analysis of Large-Scale Text Corpora

Applying machine learning algorithms to automatically infer relationships between concepts from large‐scale collections of documents presents a unique opportunity to investigate at scale how human semantic knowledge is organized, how people use it to make fundamental judgments (“How similar are cats and bears?”), and how these judgments depend on the features that describe concepts (e.g., size, furriness). However, efforts to date have exhibited a substantial discrepancy between algorithm predictions and human empirical judgments. Here, we introduce a novel approach to generating embeddings for this purpose motivated by the idea that semantic context plays a critical role in human judgment. We leverage this idea by constraining the topic or domain from which documents used for generating embeddings are drawn (e.g., referring to the natural world vs. transportation apparatus). Specifically, we trained state‐of‐the‐art machine learning algorithms using contextually‐constrained text corpora (domain‐specific subsets of Wikipedia articles, 50+ million words each) and showed that this procedure greatly improved predictions of empirical similarity judgments and feature ratings of contextually relevant concepts. Furthermore, we describe a novel, computationally tractable method for improving predictions of contextually‐unconstrained embedding models based on dimensionality reduction of their internal representation to a small number of contextually relevant semantic features. By improving the correspondence between predictions derived automatically by machine learning methods using vast amounts of data and more limited, but direct empirical measurements of human judgments, our approach may help leverage the availability of online corpora to better understand the structure of human semantic representations and how people make judgments based on those.

The computational challenge of social learning

The complex reward structure of the social world and the uncertainty endemic to social contexts poses a challenge for modeling. For example, during social interactions, the actions of one person influence the internal states of another. These social dependencies make it difficult to formalize social learning problems in a mathematically tractable way. While it is tempting to dispense with these complexities, they are a defining feature of social life. Because the structure of social interactions challenges the simplifying assumptions often made in models, they make an ideal testbed for computational models of cognition. By adopting a framework that embeds existing social knowledge into the model, we can go beyond explaining behaviors in laboratory tasks to explaining those observed in the wild.

When Apologies become Meaningful: Perceptions of Apologies in Environmental Justice Communities

In the United States, people of color from low income and working-class backgrounds are at disproportionate risk to pollution and other environmental stressors. These environmental justice communities (EJCs) can also experience increased risk when a natural disaster collides with a preexisting environmental risk. The current research is an exploratory field study that examines perceptions of environmental risk after a natural disaster and how meaningful a public apology would be in three communities. Residents (N=161) in two EJCs and a community without documented risks reported their environmental concerns and perceptions of public apologies. Overall, EJC residents reported greater concern about chemical hazard exposure than did residents with decreased risk. Furthermore, chemical exposure concerns facilitated public apology meaningfulness within the EJCs, but not in the decreased risk community.

What do Reinforcement Learning Models Measure? Interpreting Model Parameters in Cognition and Neuroscience

Reinforcement learning (RL) is a concept that has been invaluable to fields including machine learning, neuroscience, and cognitive science. However, what RL entails differs between fields, leading to difficulties when interpreting and translating findings. After laying out these differences, this paper focuses on cognitive (neuro)science to discuss how we as a field might over-interpret RL modeling results. We too often assume-implicitly-that modeling results generalize between tasks, models, and participant populations, despite negative empirical evidence for this assumption. We also often assume that parameters measure specific, unique (neuro)cognitive processes, a concept we call interpretability, when evidence suggests that they capture different functions across studies and tasks. We conclude that future computational research needs to pay increased attention to implicit assumptions when using RL models, and suggest that a more systematic understanding of contextual factors will help address issues and improve the ability of RL to explain brain and behavior.

A Bayesian meta-analysis of infants' ability to perceive audio-visual congruence for speech

This paper quantifies the extent to which infants can perceive audio-visual congruence for speech information and assesses whether this ability changes with native language exposure over time. A hierarchical Bayesian robust regression model of 92 separate effect sizes extracted from 24 studies indicates a moderate effect size in a positive direction (0.35, CI [0.21: 0.50]). This result suggests that infants possess a robust ability to detect audio-visual congruence for speech. Moderator analyses, moreover, suggest that infants' audio-visual matching ability for speech emerges at an early point in the process of language acquisition and remains stable for both native and non-native speech throughout early development. A sensitivity analysis of the meta-analytic data, however, indicates that a moderate publication bias for significant results could shift the lower credible interval to include null effects. Based on these findings, we outline recommendations for new lines of enquiry and suggest ways to improve the replicability of results in future investigations.

When numbers fail: do researchers agree on operationalization of published research?

Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalization of phenomena. Operationalization is the process of translating theoretical constructs into measurable laboratory quantities. Thus, the validity of operationalization is central for the quality of empirical studies. But do differences in the validity of operationalization affect the way scientists evaluate scientific literature? To investigate this, we manipulated the strength of operationalization of three published studies and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalization is more valid, but (2) the different validity is only to some extent reflected in a judgement of the study's quality. These results combined give partial corroboration to the notion that researchers' evaluations of research results are not affected by operationalization validity.

When the Going Gets Tough, How Do We Perceive the Future?

People struggle to stay motivated to work toward difficult goals. Sometimes the feeling of difficulty signals that the goal is important and worth pursuing; other times, it signals that the goal is impossible and should be abandoned. In this article, we argue that how difficulty is experienced depends on how we perceive and experience the timing of difficult events. We synthesize research from across the social and behavioral sciences and propose a new, integrated model to explain how components of time perception interact with interpretations of experienced difficulty to influence motivation and goal-directed behavior. Although these constructs have been studied separately in previous research, we suggest that these factors are inseparable and that an integrated model will help us to better understand motivation and predict behavior. We conclude with new empirical questions to guide future research and by discussing the implications of this research for both theory and intervention practice.

Theory Before the Test: How to Build High-Verisimilitude Explanatory Theories in Psychological Science

Drawing on the philosophy of psychological explanation, we suggest that psychological science, by focusing on effects, may lose sight of its primary explananda: psychological capacities. We revisit Marr's levels-of-analysis framework, which has been remarkably productive and useful for cognitive psychological explanation. We discuss ways in which Marr's framework may be extended to other areas of psychology, such as social, developmental, and evolutionary psychology, bringing new benefits to these fields. We then show how theoretical analyses can endow a theory with minimal plausibility even before contact with empirical data: We call this the theoretical cycle. Finally, we explain how our proposal may contribute to addressing critical issues in psychological science, including how to leverage effects to understand capacities better.

Model independent feature attributions: Shapley values that uncover non-linear dependencies

Shapley values have become increasingly popular in the machine learning literature, thanks to their attractive axiomatisation, flexibility, and uniqueness in satisfying certain notions of 'fairness'. The flexibility arises from the myriad potential forms of the Shapley value game formulation. Amongst the consequences of this flexibility is that there are now many types of Shapley values being discussed, with such variety being a source of potential misunderstanding. To the best of our knowledge, all existing game formulations in the machine learning and statistics literature fall into a category, which we name the model-dependent category of game formulations. In this work, we consider an alternative and novel formulation which leads to the first instance of what we call model-independent Shapley values. These Shapley values use a measure of non-linear dependence as the characteristic function. The strength of these Shapley values is in their ability to uncover and attribute non-linear dependencies amongst features. We introduce and demonstrate the use of the energy distance correlations, affine-invariant distance correlation, and Hilbert-Schmidt independence criterion as Shapley value characteristic functions. In particular, we demonstrate their potential value for exploratory data analysis and model diagnostics. We conclude with an interesting expository application to a medical survey data set.

The case for formal methodology in scientific reform

Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.

Making the Black Box Transparent: A Template and Tutorial for Registration of Studies Using Experience-Sampling Methods

A growing interest in understanding complex and dynamic psychological processes as they occur in everyday life has led to an increase in studies using ambulatory assessment techniques, including the experience-sampling method (ESM) and ecological momentary assessment. These methods, however, tend to involve numerous forking paths and researcher degrees of freedom, even beyond those typically encountered with other research methodologies. Although a number of researchers working with ESM techniques are actively engaged in efforts to increase the methodological rigor and transparency of research that uses them, currently there is little routine implementation of open-science practices in ESM research. In this article, we discuss the ways in which ESM research is especially vulnerable to threats to transparency, reproducibility, and replicability. We propose that greater use of study registration, a cornerstone of open science, may address some of these threats to the transparency of ESM research. Registration of ESM research is not without challenges, including model selection, accounting for potential model-convergence issues, and the use of preexisting data sets. As these may prove to be significant barriers for ESM researchers, we also discuss ways of overcoming these challenges and of documenting them in a registration. A further challenge is that current general preregistration templates do not adequately capture the unique features of ESM. We present a registration template for ESM research and also discuss registration of studies using preexisting data.

How Computational Modeling Can Force Theory Building in Psychological Science

Psychology endeavors to develop theories of human capacities and behaviors on the basis of a variety of methodologies and dependent measures. We argue that one of the most divisive factors in psychological science is whether researchers choose to use computational modeling of theories (over and above data) during the scientific-inference process. Modeling is undervalued yet holds promise for advancing psychological science. The inherent demands of computational modeling guide us toward better science by forcing us to conceptually analyze, specify, and formalize intuitions that otherwise remain unexamined-what we dub open theory. Constraining our inference process through modeling enables us to build explanatory and predictive theories. Here, we present scientific inference in psychology as a path function in which each step shapes the next. Computational modeling can constrain these steps, thus advancing scientific inference over and above the stewardship of experimental practice (e.g., preregistration). If psychology continues to eschew computational modeling, we predict more replicability crises and persistent failure at coherent theory building. This is because without formal modeling we lack open and transparent theorizing. We also explain how to formalize, specify, and implement a computational model, emphasizing that the advantages of modeling can be achieved by anyone with benefit to all.

People as Intuitive Scientists: Reconsidering Statistical Explanations of Decision Making

A persistent metaphor in decision-making research casts people as intuitive statisticians. Popular explanations based on this metaphor assume that the way in which people represent the environment is specified and fixed a priori. A major flaw in this account is that it is not clear how people know what aspects of an environment are important, how to interpret those aspects, and how to make decisions based on them. We suggest a theoretical reorientation away from assuming people's representations towards a focus on explaining how people themselves specify what is important to represent. This perspective casts decision makers as intuitive scientists able to flexibly construct, modify, and replace the representations of the decision problems they face.

Formalizing Verbal Theories

Abstract. We present a tutorial for formalizing verbal theories of psychological phenomena – social or otherwise. The approach builds on concepts and tools from the mathematics of computation. We use intuitive examples and illustrate the intrinsic dialectical nature of the formalization process by presenting dialogues between two fictive characters, called Verbal and Formal. These characters’ conversations and thought experiments serve to highlight important lessons in theoretical modeling.

Moving beyond Ordinary Factor Analysis in Studies of Personality and Personality Disorder: A Computational Modeling Perspective

Almost all forms of psychopathology, including personality disorders, are arrived at through complex interactions among neurobiological vulnerabilities and environmental risk factors across development. Yet despite increasing recognition of etiological complexity, psychopathology research is still dominated by searches for large main effects causes. This derives in part from reliance on traditional inferential methods, including ordinary factor analysis, regression, ANCOVA, and other techniques that use statistical partialing to isolate unique effects. In principle, some of these methods can accommodate etiological complexity, yet as typically applied they are insensitive to interactive functional dependencies (modulating effects) among etiological influences. Here, we use our developmental model of antisocial and borderline traits to illustrate challenges faced when modeling complex etiological mechanisms of psychopathology. We then consider how computational models, which are rarely used in the personality disorders literature, remedy some of these challenges when combined with hierarchical Bayesian analysis.

Using qualitative approaches to improve quantitative inferences in environmental psychology

This article describes the qualitative approach used to generate and interpret the quantitative study reported by Song and colleagues' (2020) in their article, "What counts as an 'environmental' issue? Differences in environmental issue conceptualization across race, ethnicity, and socioeconomic status." Song and colleagues (2020) describe the results of a survey documenting that, in the United States, White and high-SES respondents perceive environmental issues differently than their non-White and lower-SES counterparts, reflecting structural differences in environmental risks. While Song and colleagues (2020) discuss the survey results in detail, the discussion of the qualitative research that led to the creation of that survey was limited due to space constraints. The current article provides a more holistic account of the methods behind the Song and colleagues (2020) study by discussing the qualitative component of the research in detail. In addition to discussing how the qualitative research complements and critically informs the findings reported by Song et al., we also consider the broader implications and value of integrating qualitative and quantitative methods in environmental psychology.•Conduct qualitative study to inform quantitative design.•Use qualitative patterns to make inferences about quantitative indicators.

Asymptotic estimates of SARS-CoV-2 infection counts and their sensitivity to stochastic perturbation

Despite the importance of having robust estimates of the time-asymptotic total number of infections, early estimates of COVID-19 show enormous fluctuations. Using COVID-19 data from different countries, we show that predictions are extremely sensitive to the reporting protocol and crucially depend on the last available data point before the maximum number of daily infections is reached. We propose a physical explanation for this sensitivity, using a susceptible-exposed-infected-recovered model, where the parameters are stochastically perturbed to simulate the difficulty in detecting patients, different confinement measures taken by different countries, as well as changes in the virus characteristics. Our results suggest that there are physical and statistical reasons to assign low confidence to statistical and dynamical fits, despite their apparently good statistical scores. These considerations are general and can be applied to other epidemics.

Asymptotic estimates of SARS-CoV-2 infection counts and their sensitivity to stochastic perturbation

Despite the importance of having robust estimates of the time-asymptotic total number of infections, early estimates of COVID-19 show enormous fluctuations. Using COVID-19 data from different countries, we show that predictions are extremely sensitive to the reporting protocol and crucially depend on the last available data point before the maximum number of daily infections is reached. We propose a physical explanation for this sensitivity, using a susceptible-exposed-infected-recovered model, where the parameters are stochastically perturbed to simulate the difficulty in detecting patients, different confinement measures taken by different countries, as well as changes in the virus characteristics. Our results suggest that there are physical and statistical reasons to assign low confidence to statistical and dynamical fits, despite their apparently good statistical scores. These considerations are general and can be applied to other epidemics.

Double responding: A new constraint for models of speeded decision making

Evidence accumulation models (EAMs) have become the dominant models of speeded decision making, which are able to decompose choices and response times into cognitive parameters that drive the decision process. Several models within the EAM framework contain fundamentally different ideas of how the decision making process operates, though previous assessments have found that these models display a high level of mimicry, which has hindered the ability of researchers to contrast these different theoretical viewpoints. Our study introduces a neglected phenomenon that we term "double responding", which can help to further constrain these models. We show that double responding produces several interesting benchmarks, and that the predictions of different EAMs can be distinguished in standard experiment paradigms when they are constrained to account for the choice response time distributions and double responding behaviour in unison. Our findings suggest that lateral inhibition (e.g., the leaky-competing accumulator) provides models with a universal ability to make accurate predictions for these data. Furthermore, only models containing feed-forward inhibition (e.g., the diffusion model) performed poorly under both of our proposed extensions of the standard EAM framework to double responding, suggesting a general inability of feed-forward inhibition to accurately predict these data. We believe that our study provides an important step forward in further constraining models of speeded decision making, though additional research on double responding is required before broad conclusions are made about which models provide the best explanation of the underlying decision-making process.

Ten simple rules for the computational modeling of behavioral data

Computational modeling of behavior has revolutionized psychology and neuroscience. By fitting models to experimental data we can probe the algorithms underlying behavior, find neural correlates of computational variables and better understand the effects of drugs, illness and interventions. But with great power comes great responsibility. Here, we offer ten simple rules to ensure that computational modeling is used with care and yields meaningful insights. In particular, we present a beginner-friendly, pragmatic and details-oriented introduction on how to relate models to data. What, exactly, can a model tell us about the mind? To answer this, we apply our rules to the simplest modeling techniques most accessible to beginning modelers and illustrate them with examples and code available online. However, most rules apply to more advanced techniques. Our hope is that by following our guidelines, researchers will avoid many pitfalls and unleash the power of computational modeling on their own data.

Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods

Typical Bayesian methods for models with latent variables (or random effects) involve directly sampling the latent variables along with the model parameters. In high-level software code for model definitions (using, e.g., BUGS, JAGS, Stan), the likelihood is therefore specified as conditional on the latent variables. This can lead researchers to perform model comparisons via conditional likelihoods, where the latent variables are considered model parameters. In other settings, however, typical model comparisons involve marginal likelihoods where the latent variables are integrated out. This distinction is often overlooked despite the fact that it can have a large impact on the comparisons of interest. In this paper, we clarify and illustrate these issues, focusing on the comparison of conditional and marginal Deviance Information Criteria (DICs) and Watanabe-Akaike Information Criteria (WAICs) in psychometric modeling. The conditional/marginal distinction corresponds to whether the model should be predictive for the clusters that are in the data or for new clusters (where "clusters" typically correspond to higher-level units like people or schools). Correspondingly, we show that marginal WAIC corresponds to leave-one-cluster out cross-validation, whereas conditional WAIC corresponds to leave-one-unit out. These results lead to recommendations on the general application of the criteria to models with latent variables.