A parameter recovery assessment of time-variant models of decision-making

Cognitive mechanisms of learning in sequential decision-making under uncertainty: an experimental and theoretical approach

Learning to make adaptive decisions involves making choices, assessing their consequence, and leveraging this assessment to attain higher rewarding states. Despite vast literature on value-based decision-making, relatively little is known about the cognitive processes underlying decisions in highly uncertain contexts. Real world decisions are rarely accompanied by immediate feedback, explicit rewards, or complete knowledge of the environment. Being able to make informed decisions in such contexts requires significant knowledge about the environment, which can only be gained via exploration. Here we aim at understanding and formalizing the brain mechanisms underlying these processes. To this end, we first designed and performed an experimental task. Human participants had to learn to maximize reward while making sequences of decisions with only basic knowledge of the environment, and in the absence of explicit performance cues. Participants had to rely on their own internal assessment of performance to reveal a covert relationship between their choices and their subsequent consequences to find a strategy leading to the highest cumulative reward. Our results show that the participants' reaction times were longer whenever the decision involved a future consequence, suggesting greater introspection whenever a delayed value had to be considered. The learning time varied significantly across participants. Second, we formalized the neurocognitive processes underlying decision-making within this task, combining mean-field representations of competing neural populations with a reinforcement learning mechanism. This model provided a plausible characterization of the brain dynamics underlying these processes, and reproduced each aspect of the participants' behavior, from their reaction times and choices to their learning rates. In summary, both the experimental results and the model provide a principled explanation to how delayed value may be computed and incorporated into the neural dynamics of decision-making, and to how learning occurs in these uncertain scenarios.

Uncovering the cognitive mechanisms underlying the gaze cueing effect

The gaze cueing effect is the tendency for people to respond faster to targets appearing at locations gazed at by others, compared with locations gazed away from by others. The effect is robust, widely studied, and is an influential finding within social cognition. Formal evidence accumulation models provide the dominant theoretical account of the cognitive processes underlying speeded decision-making, but they have rarely been applied to social cognition research. In this study, using a combination of individual-level and hierarchical computational modelling techniques, we applied evidence accumulation models to gaze cueing data (three data sets total, N = 171, 139,001 trials) for the first time to assess the relative capacity that an attentional orienting mechanism and information processing mechanisms have for explaining the gaze cueing effect. We found that most participants were best described by the attentional orienting mechanism, such that response times were slower at gazed away from locations because they had to reorient to the target before they could process the cue. However, we found evidence for individual differences, whereby the models suggested that some gaze cueing effects were driven by a short allocation of information processing resources to the gazed at location, allowing for a brief period where orienting and processing could occur in parallel. There was exceptionally little evidence to suggest any sustained reallocation of information processing resources neither at the group nor individual level. We discuss how this individual variability might represent credible individual differences in the cognitive mechanisms that subserve behaviourally observed gaze cueing effects.

Neural correlates of confidence during decision formation in a perceptual judgment task

When we make a decision, we also estimate the probability that our choice is correct or accurate. This probability estimate is termed our degree of decision confidence. Recent work has reported event-related potential (ERP) correlates of confidence both during decision formation (the centro-parietal positivity component; CPP) and after a decision has been made (the error positivity component; Pe). However, there are several measurement confounds that complicate the interpretation of these findings. More recent studies that overcome these issues have so far produced conflicting results. To better characterise the ERP correlates of confidence we presented participants with a comparative brightness judgment task while recording electroencephalography. Participants judged which of two flickering squares (varying in luminance over time) was brighter on average. Participants then gave confidence ratings ranging from "surely incorrect" to "surely correct". To elicit a range of confidence ratings we manipulated both the mean luminance difference between the brighter and darker squares (relative evidence) and the overall luminance of both squares (absolute evidence). We found larger CPP amplitudes in trials with higher confidence ratings. This association was not simply a by-product of differences in relative evidence (which covaries with confidence) across trials. We did not identify postdecisional ERP correlates of confidence, except when they were artificially produced by pre-response ERP baselines. These results provide further evidence for neural correlates of processes that inform confidence judgments during decision formation.

PyBEAM: A Bayesian approach to parameter inference for a wide class of binary evidence accumulation models

Many decision-making theories are encoded in a class of processes known as evidence accumulation models (EAM). These assume that noisy evidence stochastically accumulates until a set threshold is reached, triggering a decision. One of the most successful and widely used of this class is the Diffusion Decision Model (DDM). The DDM however is limited in scope and does not account for processes such as evidence leakage, changes of evidence, or time varying caution. More complex EAMs can encode a wider array of hypotheses, but are currently limited by computational challenges. In this work, we develop the Python package PyBEAM (Bayesian Evidence Accumulation Models) to fill this gap. Toward this end, we develop a general probabilistic framework for predicting the choice and response time distributions for a general class of binary decision models. In addition, we have heavily computationally optimized this modeling process and integrated it with PyMC, a widely used Python package for Bayesian parameter estimation. This 1) substantially expands the class of EAM models to which Bayesian methods can be applied, 2) reduces the computational time to do so, and 3) lowers the entry fee for working with these models. Here we demonstrate the concepts behind this methodology, its application to parameter recovery for a variety of models, and apply it to a recently published data set to demonstrate its practical use.

Multiphasic value biases in fast-paced decisions

Perceptual decisions are biased toward higher-value options when overall gains can be improved. When stimuli demand immediate reactions, the neurophysiological decision process dynamically evolves through distinct phases of growing anticipation, detection, and discrimination, but how value biases are exerted through these phases remains unknown. Here, by parsing motor preparation dynamics in human electrophysiology, we uncovered a multiphasic pattern of countervailing biases operating in speeded decisions. Anticipatory preparation of higher-value actions began earlier, conferring a 'starting point' advantage at stimulus onset, but the delayed preparation of lower-value actions was steeper, conferring a value-opposed buildup-rate bias. This, in turn, was countered by a transient deflection toward the higher-value action evoked by stimulus detection. A neurally-constrained process model featuring anticipatory urgency, biased detection, and accumulation of growing stimulus-discriminating evidence, successfully captured both behavior and motor preparation dynamics. Thus, an intricate interplay of distinct biasing mechanisms serves to prioritise time-constrained perceptual decisions.

Normative decision rules in changing environments

Models based on normative principles have played a major role in our understanding of how the brain forms decisions. However, these models have typically been derived for simple, stable conditions, and their relevance to decisions formed under more naturalistic, dynamic conditions is unclear. We previously derived a normative decision model in which evidence accumulation is adapted to fluctuations in the evidence-generating process that occur during a single decision (Glaze et al., 2015), but the evolution of commitment rules (e.g. thresholds on the accumulated evidence) under dynamic conditions is not fully understood. Here, we derive a normative model for decisions based on changing contexts, which we define as changes in evidence quality or reward, over the course of a single decision. In these cases, performance (reward rate) is maximized using decision thresholds that respond to and even anticipate these changes, in contrast to the static thresholds used in many decision models. We show that these adaptive thresholds exhibit several distinct temporal motifs that depend on the specific predicted and experienced context changes and that adaptive models perform robustly even when implemented imperfectly (noisily). We further show that decision models with adaptive thresholds outperform those with constant or urgency-gated thresholds in accounting for human response times on a task with time-varying evidence quality and average reward. These results further link normative and neural decision-making while expanding our view of both as dynamic, adaptive processes that update and use expectations to govern both deliberation and commitment.

Beyond Drift Diffusion Models: Fitting a Broad Class of Decision and Reinforcement Learning Models with HDDM

Computational modeling has become a central aspect of research in the cognitive neurosciences. As the field matures, it is increasingly important to move beyond standard models to quantitatively assess models with richer dynamics that may better reflect underlying cognitive and neural processes. For example, sequential sampling models (SSMs) are a general class of models of decision-making intended to capture processes jointly giving rise to RT distributions and choice data in n-alternative choice paradigms. A number of model variations are of theoretical interest, but empirical data analysis has historically been tied to a small subset for which likelihood functions are analytically tractable. Advances in methods designed for likelihood-free inference have recently made it computationally feasible to consider a much larger spectrum of SSMs. In addition, recent work has motivated the combination of SSMs with reinforcement learning models, which had historically been considered in separate literatures. Here, we provide a significant addition to the widely used HDDM Python toolbox and include a tutorial for how users can easily fit and assess a (user-extensible) wide variety of SSMs and how they can be combined with reinforcement learning models. The extension comes batteries included, including model visualization tools, posterior predictive checks, and ability to link trial-wise neural signals with model parameters via hierarchical Bayesian regression.

Efficient numerical approximation of a non-regular Fokker-Planck equation associated with first-passage time distributions

In neuroscience, the distribution of a decision time is modelled by means of a one-dimensional Fokker-Planck equation with time-dependent boundaries and space-time-dependent drift. Efficient approximation of the solution to this equation is required, e.g., for model evaluation and parameter fitting. However, the prescribed boundary conditions lead to a strong singularity and thus to slow convergence of numerical approximations. In this article we demonstrate that the solution can be related to the solution of a parabolic PDE on a rectangular space-time domain with homogeneous initial and boundary conditions by transformation and subtraction of a known function. We verify that the solution of the new PDE is indeed more regular than the solution of the original PDE and proceed to discretize the new PDE using a space-time minimal residual method. We also demonstrate that the solution depends analytically on the parameters determining the boundaries as well as the drift. This justifies the use of a sparse tensor product interpolation method to approximate the PDE solution for various parameter ranges. The predicted convergence rates of the minimal residual method and that of the interpolation method are supported by numerical simulations.

Think fast! The implications of emphasizing urgency in decision-making

Evidence accumulation models (EAMs) have become the dominant explanation of how the decision-making process operates, proposing that decisions are the result of a process of evidence accumulation. The primary use of EAMs has been as "measurement tools" of the underlying decision-making process, where researchers apply EAMs to empirical data to estimate participants' task ability (i.e., the "drift rate"), response caution (i.e., the "decision threshold"), and the time taken for other processes (i.e., the "non-decision time"), making EAMs a powerful tool for discriminating between competing psychological theories. Recent studies have brought into question the mapping between the latent parameters of EAMs and the theoretical constructs that they are thought to represent, showing that emphasizing urgent responding - which intuitively should selectively influence decision threshold - may also influence drift rate and/or non-decision time. However, these findings have been mixed, leading to differences in opinion between experts in the field. The current study aims to provide a more conclusive answer to the implications of emphasizing urgent responding, providing a re-analysis of 6 data sets from previous studies using two different EAMs - the diffusion model and the linear ballistic accumulator (LBA) - with state-of-the-art methods for model selection based inference. The findings display clear evidence for a difference in conclusions between the two models, with the diffusion model suggesting that decision threshold and non-decision time decrease when urgency is emphasized, and the LBA suggesting that decision threshold and drift rate decrease when urgency is emphasized. Furthermore, although these models disagree regarding whether non-decision time or drift rate decrease under urgency emphasis, both show clear evidence that emphasizing urgency does not selectively influence decision threshold. These findings suggest that researchers should revise their assumptions about certain experimental manipulations, the specification of certain EAMs, or perhaps both.