Extracting the dynamics of behavior in sensory decision-making experiments

Pre-existing visual responses in a projection-defined dopamine population explain individual learning trajectories

A key challenge of learning a new task is that the environment is high dimensional-there are many different sensory features and possible actions, with typically only a small reward-relevant subset. Although animals can learn to perform complex tasks that involve arbitrary associations between stimuli, actions, and rewards,1,2,3,4,5,6 a consistent and striking result across varied experimental paradigms is that in initially acquiring such tasks, large differences between individuals are apparent in the learning process.7,8,9,10,11,12 What neural mechanisms contribute to initial task acquisition, and why do some individuals learn a new task much more quickly than others? To address these questions, we recorded longitudinally from dopaminergic (DA) axon terminals in mice learning a visual decision-making task.7 Across striatum, DA responses tracked idiosyncratic and side-specific learning trajectories, consistent with widespread reward prediction error coding across DA terminals. However, even before any rewards were delivered, contralateral-side-specific visual responses were present in DA terminals, primarily in the dorsomedial striatum (DMS). These pre-existing responses predicted the extent of learning for contralateral stimuli. Moreover, activation of these terminals improved contralateral performance. Thus, the initial conditions of a projection-specific and feature-specific DA signal help explain individual learning trajectories. More broadly, this work suggests that functional heterogeneity across DA projections may serve to bias target regions toward learning about different subsets of task features, providing a potential mechanism to address the dimensionality of the initial task learning problem.

Dynamic reinforcement learning reveals time-dependent shifts in strategy during reward learning

Different brain systems have been hypothesized to subserve multiple "experts" that compete to generate behavior. In reinforcement learning, two general processes, one model-free (MF) and one model-based (MB), are often modeled as a mixture of agents (MoA) and hypothesized to capture differences between automaticity vs. deliberation. However, shifts in strategy cannot be captured by a static MoA. To investigate such dynamics, we present the mixture-of-agents hidden Markov model (MoA-HMM), which simultaneously learns inferred action values from a set of agents and the temporal dynamics of underlying "hidden" states that capture shifts in agent contributions over time. Applying this model to a multi-step, reward-guided task in rats reveals a progression of within-session strategies: a shift from initial MB exploration to MB exploitation, and finally to reduced engagement. The inferred states predict changes in both response time and OFC neural encoding during the task, suggesting that these states are capturing real shifts in dynamics.

A Bayesian Hierarchical Model of Trial-To-Trial Fluctuations in Decision Criterion

Classical decision models assume that the parameters giving rise to choice behavior are stable, yet emerging research suggests these parameters may fluctuate over time. Such fluctuations, observed in neural activity and behavioral strategies, have significant implications for understanding decision-making processes. However, empirical studies on fluctuating human decision-making strategies have been limited due to the extensive data requirements for estimating these fluctuations. Here, we introduce hMFC (Hierarchical Model for Fluctuations in Criterion), a Bayesian framework designed to estimate slow fluctuations in the decision criterion from limited data. We first showcase the importance of considering fluctuations in decision criterion: incorrectly assuming a stable criterion gives rise to apparent history effects and underestimates perceptual sensitivity. We then present a hierarchical estimation procedure capable of reliably recovering the underlying state of the fluctuating decision criterion with as few as 500 trials per participant, offering a robust tool for researchers with typical human datasets. Critically, hMFC does not only accurately recover the state of the underlying decision criterion, it also effectively deals with the confounds caused by criterion fluctuations. Lastly, we provide code and a comprehensive demo at www.github.com/robinvloeberghs/hMFC to enable widespread application of hMFC in decision-making research.

Reward history guides focal attention in whisker somatosensory cortex

Prior reward is a potent cue for attentional capture, but the underlying neurobiology is largely unknown. In a novel whisker touch detection task, we show that mice flexibly shift attention between specific whiskers on a trial-by-trial timescale, guided by the recent history of stimulus-reward association. Two-photon calcium imaging and spike recordings revealed a robust neurobiological correlate of attention in the somatosensory cortex (S1), boosting sensory responses to the attended whisker in L2/3 and L5, but not L4. Attentional boosting in L2/3 pyramidal cells was topographically precise and whisker-specific, and shifted receptive fields toward the attended whisker. L2/3 VIP interneurons were broadly activated by whisker stimuli, motion, and arousal but did not carry a whisker-specific attentional signal, and thus did not mediate spatially focused tactile attention. Together, these findings establish a new model of focal attention in the mouse whisker tactile system, showing that the history of stimuli and rewards in the recent past can dynamically engage local modulation in cortical sensory maps to guide flexible shifts in ongoing behavior.

Sensory choices as logistic classification

Logistic classification is a simple way to make choices based on a set of factors: give each factor a weight, sum the results, and use the sum to set the log odds of a random draw. This operation is known to describe human and animal choices based on value (economic decisions). There is increasing evidence that it also describes choices based on sensory inputs (perceptual decisions), presented across sensory modalities (multisensory integration) and combined with non-sensory factors such as prior probability, expected value, overall motivation, and recent actions. Logistic classification can also capture the effects of brain manipulations such as local inactivations. The brain may implement it by thresholding stochastic inputs (as in signal detection theory) acquired over time (as in the drift diffusion model). It is the optimal strategy under certain conditions, and the brain appears to use it as a heuristic in a wider set of conditions.

Spatially organized striatal neuromodulator release encodes trajectory errors

Goal-directed navigation requires animals to continuously evaluate their current direction and speed of travel relative to landmarks to discern whether they are approaching or deviating from their goal. Striatal dopamine and acetylcholine are powerful modulators of goal-directed behavior, but it is unclear whether and how neuromodulator dynamics at landmarks incorporate relative motion for effective behavioral guidance. Using optical measurements in mice, we demonstrate that cue-evoked striatal dopamine release encodes bi-directional 'trajectory errors' reflecting relationships between ongoing speed and direction of locomotion and visual flow relative to optimal goal trajectories. Striatum-wide micro-fiber array recordings resolved an anatomical gradient of trajectory error signaling across the anterior-posterior axis, distinct from trajectory error independent cue signals. Dynamic regression modeling revealed that positive and negative trajectory error encoding emerges early and late respectively during learning and over different time courses in the medial and lateral striatum, enabling region specific contributions to learning. Striatal acetylcholine release also encodes trajectory errors, but encoding is more spatially restricted, opposite polarity, and delayed relative to dopamine, supporting distinct roles in modulating striatal output and behavior. Dopamine trajectory error signaling and task performance were reproduced in a reinforcement learning model incorporating a conjunctive state space representation, suggesting a potential neural substrate for trajectory error generation. Our results establish region specific neuromodulator signals positioned to guide the speed and direction of locomotion to reach goals based on environmental landmarks during navigation.

Goal-directed learning is multidimensional and accompanied by diverse and widespread changes in neocortical signaling

New tasks are often learned in stages with each stage reflecting a different learning challenge. Accordingly, each learning stage is likely mediated by distinct neuronal processes. And yet, most rodent studies of the neuronal correlates of goal-directed learning focus on individual outcome measures and individual brain regions. Here, we longitudinally studied mice from naïve to expert performance in a head-fixed, operant conditioning whisker discrimination task. In addition to tracking the primary behavioral outcome of stimulus discrimination, we tracked and compared an array of object-based and temporal-based behavioral measures. These behavioral analyses identify multiple, partially overlapping learning stages in this task, consistent with initial response implementation, early stimulus-response generalization, and late response inhibition. To begin to understand the neuronal foundations of these learning processes, we performed widefield Ca2+ imaging of dorsal neocortex throughout learning and correlated behavioral measures with neuronal activity. We found distinct and widespread correlations between neocortical activation patterns and various behavioral measures. For example, improvements in sensory discrimination correlated with target stimulus evoked activations of response-related cortices along with distractor stimulus evoked global cortical suppression. Our study reveals multidimensional learning for a simple goal-directed learning task and generates hypotheses for the neuronal modulations underlying these various learning processes.

How to reward animals based on their subjective percepts: A Bayesian approach to online estimation of perceptual biases

Elucidating the neural basis of perceptual biases, such as those produced by visual illusions, can provide powerful insights into the neural mechanisms of perceptual inference. However, studying the subjective percepts of animals poses a fundamental challenge: unlike human participants, animals cannot be verbally instructed to report what they see, hear, or feel. Instead, they must be trained to perform a task for reward, and researchers must infer from their responses what the animal perceived. However, animals' responses are shaped by reward feedback, thus raising the major concern that the reward regimen may alter the animal's decision strategy or even intrinsic perceptual biases. We developed a method that estimates perceptual bias during task performance and then computes the reward for each trial based on the evolving estimate of the animal's perceptual bias. Our approach makes use of multiple stimulus contexts to dissociate perceptual biases from decision-related biases. Starting with an informative prior, our Bayesian method updates a posterior over the perceptual bias after each trial. The prior can be specified based on data from past sessions, thus reducing the variability of the online estimates and allowing it to converge to a stable estimate over a small number of trials. After validating our method on synthetic data, we apply it to estimate perceptual biases of monkeys in a motion direction discrimination task in which varying background optic flow induces robust perceptual biases. This method overcomes an important challenge to understanding the neural basis of subjective percepts.

A Comparison of Rapid Rule-Learning Strategies in Humans and Monkeys

Interspecies comparisons are key to deriving an understanding of the behavioral and neural correlates of human cognition from animal models. We perform a detailed comparison of the strategies of female macaque monkeys to male and female humans on a variant of the Wisconsin Card Sorting Test (WCST), a widely studied and applied task that provides a multiattribute measure of cognitive function and depends on the frontal lobe. WCST performance requires the inference of a rule change given ambiguous feedback. We found that well-trained monkeys infer new rules three times more slowly than minimally instructed humans. Input-dependent hidden Markov model-generalized linear models were fit to their choices, revealing hidden states akin to feature-based attention in both species. Decision processes resembled a win-stay, lose-shift strategy with interspecies similarities as well as key differences. Monkeys and humans both test multiple rule hypotheses over a series of rule-search trials and perform inference-like computations to exclude candidate choice options. We quantitatively show that perseveration, random exploration, and poor sensitivity to negative feedback account for the slower task-switching performance in monkeys.

Sensory choices as logistic classification

Logistic classification is a simple way to make choices based on a set of factors: give each factor a weight, sum the results, and use the sum to set the log odds of a random draw. This operation is known to describe human and animal choices based on value (economic decisions). There is increasing evidence that it also describes choices based on sensory inputs (perceptual decisions), presented across sensory modalities (multisensory integration) and combined with non-sensory factors such as prior probability, expected value, overall motivation, and recent actions. Logistic classification can also capture the effects of brain manipulations such as local inactivations. The brain may implement by thresholding stochastic inputs (as in signal detection theory) acquired over time (as in the drift diffusion model). It is the optimal strategy under certain conditions, and the brain appears to use it as a heuristic in a wider set of conditions.

A cognitive process model captures near-optimal confidence-guided waiting in rats

Rational decision-makers invest more time pursuing rewards they are more confident they will eventually receive. A series of studies have therefore used willingness to wait for delayed rewards as a proxy for decision confidence. However, interpretation of waiting behavior is limited because it is unclear how environmental statistics influence optimal waiting, and how sources of internal variability influence subjects' behavior. We trained rats to perform a confidence-guided waiting task, and derived expressions for optimal waiting that make relevant environmental statistics explicit, including travel time incurred traveling from one reward opportunity to another. We found that rats waited longer than fully optimal agents, but that their behavior was closely matched by optimal agents with travel times constrained to match their own. We developed a process model describing the decision to stop waiting as an accumulation to bound process, which allowed us to compare the effects of multiple sources of internal variability on waiting. Surprisingly, although mean wait times grew with confidence, variability did not, inconsistent with scalar invariant timing, and best explained by variability in the stopping bound. Our results describe a tractable process model that can capture the influence of environmental statistics and internal sources of variability on subjects' decision process during confidence-guided waiting.

Dissociation of attentional state and behavioral outcome using local field potentials

Successful behavior depends on attentional state and other factors related to decision-making, which may modulate neuronal activity differently. Here, we investigated whether attentional state and behavioral outcome (i.e., whether a target is detected or missed) are distinguishable using the power and phase of local field potential (LFP) recorded bilaterally from area V4 of monkeys performing a cued visual attention task. To link each trial's outcome to pairwise measures of attention that are typically averaged across trials, we used several methods to obtain single-trial estimates of spike count correlation and phase consistency. Surprisingly, while attentional location was best discriminated using gamma and high-gamma power, behavioral outcome was best discriminated by alpha power and steady-state visually evoked potential. Power outperformed absolute phase in attentional/behavioral discriminability, although single-trial gamma phase consistency provided reasonably high attentional discriminability. Our results suggest a dissociation between the neuronal mechanisms that regulate attentional focus and behavioral outcome.

Behavioral strategy shapes activation of the Vip-Sst disinhibitory circuit in visual cortex

In complex environments, animals can adopt diverse strategies to find rewards. How distinct strategies differentially engage brain circuits is not well understood. Here, we investigate this question, focusing on the cortical Vip-Sst disinhibitory circuit between vasoactive intestinal peptide-postive (Vip) interneurons and somatostatin-positive (Sst) interneurons. We characterize the behavioral strategies used by mice during a visual change detection task. Using a dynamic logistic regression model, we find that individual mice use mixtures of a visual comparison strategy and a statistical timing strategy. Separately, mice also have periods of task engagement and disengagement. Two-photon calcium imaging shows large strategy-dependent differences in neural activity in excitatory, Sst inhibitory, and Vip inhibitory cells in response to both image changes and image omissions. In contrast, task engagement has limited effects on neural population activity. We find that the diversity of neural correlates of strategy can be understood parsimoniously as the increased activation of the Vip-Sst disinhibitory circuit during the visual comparison strategy, which facilitates task-appropriate responses.

Performance errors during rodent learning reflect a dynamic choice strategy

Humans, even as infants, use cognitive strategies, such as exploration and hypothesis testing, to learn about causal interactions in the environment. In animal learning studies, however, it is challenging to disentangle higher-order behavioral strategies from errors arising from imperfect task knowledge or inherent biases. Here, we trained head-fixed mice on a wheel-based auditory two-choice task and exploited the intra- and inter-animal variability to understand the drivers of errors during learning. During learning, performance errors are dominated by a choice bias, which, despite appearing maladaptive, reflects a dynamic strategy. Early in learning, mice develop an internal model of the task contingencies such that violating their expectation of reward on correct trials (by using short blocks of non-rewarded "probe" trials) leads to an abrupt shift in strategy. During the probe block, mice behave more accurately with less bias, thereby using their learned stimulus-action knowledge to test whether the outcome contingencies have changed. Despite having this knowledge, mice continued to exhibit a strong choice bias during reinforced trials. This choice bias operates on a timescale of tens to hundreds of trials with a dynamic structure, shifting between left, right, and unbiased epochs. Biased epochs also coincided with faster motor kinematics. Although bias decreased across learning, expert mice continued to exhibit short bouts of biased choices interspersed with longer bouts of unbiased choices and higher performance. These findings collectively suggest that during learning, rodents actively probe their environment in a structured manner to refine their decision-making and maintain long-term flexibility.

A cross-species framework for investigating perceptual evidence accumulation

Cross-species studies are important for a comprehensive understanding of brain functions. However, direct quantitative comparison of behaviors across species presents a significant challenge. To enable such comparisons in perceptual decision-making, we developed a synchronized evidence accumulation task for rodents and humans, by aligning mechanics, stimuli, and training. Rats, mice and humans readily learned the task and exhibited qualitatively similar performance. Quantitative model comparison revealed that all three species employed an evidence accumulation strategy, but differed in speed, accuracy, and key decision parameters. Human performance prioritized accuracy, whereas rodent performance was limited by internal time-pressure. Rats optimized reward rate, while mice appeared to switch between evidence accumulation and other strategies trial-to-trial. Together, these results reveal striking similarities and species-specific priorities in decision-making. Furthermore, the synchronized behavioral framework we present may facilitate future studies involving cross-species comparisons, such as evaluating the face validity of animal models of neuropsychiatric disorders.

Highlights

Development of a free response evidence accumulation task for rats and miceSynchronized video game allows direct comparisons with humansRat, mouse and human behavior are well fit by the same decision modelsModel parameters reveal species-specific priorities in accumulation strategy.

Striatal dopamine signals reflect perceived cue-action-outcome associations in mice

Striatal dopamine drives associative learning by acting as a teaching signal. Much work has focused on simple learning paradigms, including Pavlovian and instrumental learning. However, higher cognition requires that animals generate internal concepts of their environment, where sensory stimuli, actions and outcomes become flexibly associated. Here, we performed fiber photometry dopamine measurements across the striatum of male mice as they learned cue-action-outcome associations based on implicit and changing task rules. Reinforcement learning models of the behavioral and dopamine data showed that rule changes lead to adjustments of learned cue-action-outcome associations. After rule changes, mice discarded learned associations and reset outcome expectations. Cue- and outcome-triggered dopamine signals became uncoupled and dependent on the adopted behavioral strategy. As mice learned the new association, coupling between cue- and outcome-triggered dopamine signals and task performance re-emerged. Our results suggest that dopaminergic reward prediction errors reflect an agent's perceived locus of control.

Gradient boosted decision trees reveal nuances of auditory discrimination behavior

Animal psychophysics can generate rich behavioral datasets, often comprised of many 1000s of trials for an individual subject. Gradient-boosted models are a promising machine learning approach for analyzing such data, partly due to the tools that allow users to gain insight into how the model makes predictions. We trained ferrets to report a target word's presence, timing, and lateralization within a stream of consecutively presented non-target words. To assess the animals' ability to generalize across pitch, we manipulated the fundamental frequency (F0) of the speech stimuli across trials, and to assess the contribution of pitch to streaming, we roved the F0 from word token to token. We then implemented gradient-boosted regression and decision trees on the trial outcome and reaction time data to understand the behavioral factors behind the ferrets' decision-making. We visualized model contributions by implementing SHAPs feature importance and partial dependency plots. While ferrets could accurately perform the task across all pitch-shifted conditions, our models reveal subtle effects of shifting F0 on performance, with within-trial pitch shifting elevating false alarms and extending reaction times. Our models identified a subset of non-target words that animals commonly false alarmed to. Follow-up analysis demonstrated that the spectrotemporal similarity of target and non-target words rather than similarity in duration or amplitude waveform was the strongest predictor of the likelihood of false alarming. Finally, we compared the results with those obtained with traditional mixed effects models, revealing equivalent or better performance for the gradient-boosted models over these approaches.

Specificity and sensitivity of the fixed-point test for binary mixture distributions

When two cognitive processes contribute to a behavioral output-each process producing a specific distribution of the behavioral variable of interest-and when the mixture proportion of these two processes varies as a function of an experimental condition, a common density point should be present in the observed distributions of the data across said conditions. In principle, one can statistically test for the presence (or absence) of a fixed point in experimental data to provide evidence in favor of (or against) the presence of a mixture of processes, whose proportions are affected by an experimental manipulation. In this paper, we provide an empirical diagnostic of this test to detect a mixture of processes. We do so using resampling of real experimental data under different scenarios, which mimic variations in the experimental design suspected to affect the sensitivity and specificity of the fixed-point test (i.e., mixture proportion, time on task, and sample size). Resampling such scenarios with real data allows us to preserve important features of data which are typically observed in real experiments while maintaining tight control over the properties of the resampled scenarios. This is of particular relevance considering such stringent assumptions underlying the fixed-point test. With this paper, we ultimately aim at validating the fixed-point property of binary mixture data and at providing some performance metrics to researchers aiming at testing the fixed-point property on their experimental data.

Tracking subjects' strategies in behavioural choice experiments at trial resolution

Investigating how, when, and what subjects learn during decision-making tasks requires tracking their choice strategies on a trial-by-trial basis. Here, we present a simple but effective probabilistic approach to tracking choice strategies at trial resolution using Bayesian evidence accumulation. We show this approach identifies both successful learning and the exploratory strategies used in decision tasks performed by humans, non-human primates, rats, and synthetic agents. Both when subjects learn and when rules change the exploratory strategies of win-stay and lose-shift, often considered complementary, are consistently used independently. Indeed, we find the use of lose-shift is strong evidence that subjects have latently learnt the salient features of a new rewarded rule. Our approach can be extended to any discrete choice strategy, and its low computational cost is ideally suited for real-time analysis and closed-loop control.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Perceptual confirmation bias and decision bias underlie adaptation to sequential regularities

Our perception does not depend exclusively on the immediate sensory input. It is also influenced by our internal predictions derived from prior observations and the temporal regularities of the environment, which can result in choice history biases. However, it is unclear how this flexible use of prior information to predict the future influences perceptual decisions. Prior information may bias decisions independently of the current sensory input, or it may modulate the weight of current sensory input based on its consistency with the expectation. To address this question, we used a visual decision-making task and manipulated the transitional probabilities between successive noisy grating stimuli. Using a reverse correlation analysis, we evaluated the contribution of stimulus-independent decision bias and stimulus-dependent sensitivity modulations to choice history biases. We found that both effects coexist, whereby there was increased bias to respond in line with the predicted orientation alongside modulations in perceptual sensitivity to favor perceptual information consistent with the prediction, akin to selective attention. Furthermore, at the individual differences level, we investigated the relationship between autistic-like traits and the adaptation of choice history biases to the sequential statistics of the environment. Over two studies, we found no convincing evidence of reduced adaptation to sequential regularities in individuals with high autistic-like traits. In sum, we present robust evidence for both perceptual confirmation bias and decision bias supporting adaptation to sequential regularities in the environment.

Trial-history biases in evidence accumulation can give rise to apparent lapses in decision-making

Trial history biases and lapses are two of the most common suboptimalities observed during perceptual decision-making. These suboptimalities are routinely assumed to arise from distinct processes. However, previous work has suggested that they covary in their prevalence and that their proposed neural substrates overlap. Here we demonstrate that during decision-making, history biases and apparent lapses can both arise from a common cognitive process that is optimal under mistaken beliefs that the world is changing i.e. nonstationary. This corresponds to an accumulation-to-bound model with history-dependent updates to the initial state of the accumulator. We test our model's predictions about the relative prevalence of history biases and lapses, and show that they are robustly borne out in two distinct decision-making datasets of male rats, including data from a novel reaction time task. Our model improves the ability to precisely predict decision-making dynamics within and across trials, by positing a process through which agents can generate quasi-stochastic choices.

Signatures of task learning in neural representations

While neural plasticity has long been studied as the basis of learning, the growth of large-scale neural recording techniques provides a unique opportunity to study how learning-induced activity changes are coordinated across neurons within the same circuit. These distributed changes can be understood through an evolution of the geometry of neural manifolds and latent dynamics underlying new computations. In parallel, studies of multi-task and continual learning in artificial neural networks hint at a tradeoff between non-interference and compositionality as guiding principles to understand how neural circuits flexibly support multiple behaviors. In this review, we highlight recent findings from both biological and artificial circuits that together form a new framework for understanding task learning at the population level.

Sensory processing in humans and mice fluctuates between external and internal modes

Perception is known to cycle through periods of enhanced and reduced sensitivity to external information. Here, we asked whether such slow fluctuations arise as a noise-related epiphenomenon of limited processing capacity or, alternatively, represent a structured mechanism of perceptual inference. Using 2 large-scale datasets, we found that humans and mice alternate between externally and internally oriented modes of sensory analysis. During external mode, perception aligns more closely with the external sensory information, whereas internal mode is characterized by enhanced biases toward perceptual history. Computational modeling indicated that dynamic changes in mode are enabled by 2 interlinked factors: (i) the integration of subsequent inputs over time and (ii) slow antiphase oscillations in the impact of external sensory information versus internal predictions that are provided by perceptual history. We propose that between-mode fluctuations generate unambiguous error signals that enable optimal inference in volatile environments.

Continuous multiplexed population representations of task context in the mouse primary visual cortex

Effective task execution requires the representation of multiple task-related variables that determine how stimuli lead to correct responses. Even the primary visual cortex (V1) represents other task-related variables such as expectations, choice, and context. However, it is unclear how V1 can flexibly accommodate these variables without interfering with visual representations. We trained mice on a context-switching cross-modal decision task, where performance depends on inferring task context. We found that the context signal that emerged in V1 was behaviorally relevant as it strongly covaried with performance, independent from movement. Importantly, this signal was integrated into V1 representation by multiplexing visual and context signals into orthogonal subspaces. In addition, auditory and choice signals were also multiplexed as these signals were orthogonal to the context representation. Thus, multiplexing allows V1 to integrate visual inputs with other sensory modalities and cognitive variables to avoid interference with the visual representation while ensuring the maintenance of task-relevant variables.

Mixtures of strategies underlie rodent behavior during reversal learning

In reversal learning tasks, the behavior of humans and animals is often assumed to be uniform within single experimental sessions to facilitate data analysis and model fitting. However, behavior of agents can display substantial variability in single experimental sessions, as they execute different blocks of trials with different transition dynamics. Here, we observed that in a deterministic reversal learning task, mice display noisy and sub-optimal choice transitions even at the expert stages of learning. We investigated two sources of the sub-optimality in the behavior. First, we found that mice exhibit a high lapse rate during task execution, as they reverted to unrewarded directions after choice transitions. Second, we unexpectedly found that a majority of mice did not execute a uniform strategy, but rather mixed between several behavioral modes with different transition dynamics. We quantified the use of such mixtures with a state-space model, block Hidden Markov Model (block HMM), to dissociate the mixtures of dynamic choice transitions in individual blocks of trials. Additionally, we found that blockHMM transition modes in rodent behavior can be accounted for by two different types of behavioral algorithms, model-free or inference-based learning, that might be used to solve the task. Combining these approaches, we found that mice used a mixture of both exploratory, model-free strategies and deterministic, inference-based behavior in the task, explaining their overall noisy choice sequences. Together, our combined computational approach highlights intrinsic sources of noise in rodent reversal learning behavior and provides a richer description of behavior than conventional techniques, while uncovering the hidden states that underlie the block-by-block transitions.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Prior experience modifies acquisition trajectories via response-strategy sampling

Few studies have considered how signal detection parameters evolve during acquisition periods. We addressed this gap by training mice with differential prior experience in a conditional discrimination, auditory signal detection task. Naïve mice, mice given separate experience with each of the later correct choice options (Correct Choice Response Transfer, CCRT), and mice experienced in conditional discriminations (Conditional Discrimination Transfer, CDT) were trained to detect the presence or absence of a tone in white noise. We analyzed data assuming a two-period model of acquisition: a pre-solution and solution period (Heinemann EG (1983) in The Presolution period and the detection of statistical associations. In: Quantitative analyses of behavior: discrimination processes, vol. 4, pp. 21-36). Ballinger. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.536.1978andrep=rep1andtype=pdf ). The pre-solution period was characterized by a selective sampling of biased response strategies until adoption of a conditional responding strategy in the solution period. Correspondingly, discriminability remained low until the solution period; criterion took excursions reflecting response-strategy sampling. Prior experience affected the length and composition of the pre-solution period. Whereas CCRT and CDT mice had shorter pre-solution periods than naïve mice, CDT and Naïve mice developed substantial criterion biases and acquired asymptotic discriminability faster than CCRT mice. To explain these data, we propose a learning model in which mice selectively sample and test different response-strategies and corresponding task structures until they exit the pre-solution period. Upon exit, mice adopt the conditional responding strategy and task structure, with action values updated via inference and generalization from the other task structures. Simulations of representative mouse data illustrate the viability of this model.

Environmental dynamics shape perceptual decision bias

To interpret the sensory environment, the brain combines ambiguous sensory measurements with knowledge that reflects context-specific prior experience. But environmental contexts can change abruptly and unpredictably, resulting in uncertainty about the current context. Here we address two questions: how should context-specific prior knowledge optimally guide the interpretation of sensory stimuli in changing environments, and do human decision-making strategies resemble this optimum? We probe these questions with a task in which subjects report the orientation of ambiguous visual stimuli that were drawn from three dynamically switching distributions, representing different environmental contexts. We derive predictions for an ideal Bayesian observer that leverages knowledge about the statistical structure of the task to maximize decision accuracy, including knowledge about the dynamics of the environment. We show that its decisions are biased by the dynamically changing task context. The magnitude of this decision bias depends on the observer's continually evolving belief about the current context. The model therefore not only predicts that decision bias will grow as the context is indicated more reliably, but also as the stability of the environment increases, and as the number of trials since the last context switch grows. Analysis of human choice data validates all three predictions, suggesting that the brain leverages knowledge of the statistical structure of environmental change when interpreting ambiguous sensory signals.

Trial-history biases in evidence accumulation can give rise to apparent lapses

Trial history biases and lapses are two of the most common suboptimalities observed during perceptual decision-making. These suboptimalities are routinely assumed to arise from distinct processes. However, several hints in the literature suggest that they covary in their prevalence and that their proposed neural substrates overlap - what could underlie these links? Here we demonstrate that history biases and apparent lapses can both arise from a common cognitive process that is normative under misbeliefs about non-stationarity in the world. This corresponds to an accumulation-to-bound model with history-dependent updates to the initial state of the accumulator. We test our model's predictions about the relative prevalence of history biases and lapses, and show that they are robustly borne out in two distinct rat decision-making datasets, including data from a novel reaction time task. Our model improves the ability to precisely predict decision-making dynamics within and across trials, by positing a process through which agents can generate quasi-stochastic choices.

Modelling human behaviour in cognitive tasks with latent dynamical systems

Response time data collected from cognitive tasks are a cornerstone of psychology and neuroscience research, yet existing models of these data either make strong assumptions about the data-generating process or are limited to modelling single trials. We introduce task-DyVA, a deep learning framework in which expressive dynamical systems are trained to reproduce sequences of response times observed in data from individual human subjects. Models fitted to a large task-switching dataset captured subject-specific behavioural differences with high temporal precision, including task-switching costs. Through perturbation experiments and analyses of the models' latent dynamics, we find support for a rational account of switch costs in terms of a stability-flexibility trade-off. Thus, our framework can be used to discover interpretable cognitive theories that explain how the brain dynamically gives rise to behaviour.

Comparing rapid rule-learning strategies in humans and monkeys

Inter-species comparisons are key to deriving an understanding of the behavioral and neural correlates of human cognition from animal models. We perform a detailed comparison of macaque monkey and human strategies on an analogue of the Wisconsin Card Sort Test, a widely studied and applied multi-attribute measure of cognitive function, wherein performance requires the inference of a changing rule given ambiguous feedback. We found that well-trained monkeys rapidly infer rules but are three times slower than humans. Model fits to their choices revealed hidden states akin to feature-based attention in both species, and decision processes that resembled a Win-stay lose-shift strategy with key differences. Monkeys and humans test multiple rule hypotheses over a series of rule-search trials and perform inference-like computations to exclude candidates. An attention-set based learning stage categorization revealed that perseveration, random exploration and poor sensitivity to negative feedback explain the under-performance in monkeys.

Strategically managing learning during perceptual decision making

Making optimal decisions in the face of noise requires balancing short-term speed and accuracy. But a theory of optimality should account for the fact that short-term speed can influence long-term accuracy through learning. Here, we demonstrate that long-term learning is an important dynamical dimension of the speed-accuracy trade-off. We study learning trajectories in rats and formally characterize these dynamics in a theory expressed as both a recurrent neural network and an analytical extension of the drift-diffusion model that learns over time. The model reveals that choosing suboptimal response times to learn faster sacrifices immediate reward, but can lead to greater total reward. We empirically verify predictions of the theory, including a relationship between stimulus exposure and learning speed, and a modulation of reaction time by future learning prospects. We find that rats' strategies approximately maximize total reward over the full learning epoch, suggesting cognitive control over the learning process.

Cell-type-specific population dynamics of diverse reward computations

Computational analysis of cellular activity has developed largely independently of modern transcriptomic cell typology, but integrating these approaches may be essential for full insight into cellular-level mechanisms underlying brain function and dysfunction. Applying this approach to the habenula (a structure with diverse, intermingled molecular, anatomical, and computational features), we identified encoding of reward-predictive cues and reward outcomes in distinct genetically defined neural populations, including TH+ cells and Tac1+ cells. Data from genetically targeted recordings were used to train an optimized nonlinear dynamical systems model and revealed activity dynamics consistent with a line attractor. High-density, cell-type-specific electrophysiological recordings and optogenetic perturbation provided supporting evidence for this model. Reverse-engineering predicted how Tac1+ cells might integrate reward history, which was complemented by in vivo experimentation. This integrated approach describes a process by which data-driven computational models of population activity can generate and frame actionable hypotheses for cell-type-specific investigation in biological systems.

Predicting brain activation maps for arbitrary tasks with cognitive encoding models

A deep understanding of the neural architecture of mental function should enable the accurate prediction of a specific pattern of brain activity for any psychological task, based only on the cognitive functions known to be engaged by that task. Encoding models (EMs), which predict neural responses from known features (e.g., stimulus properties), have succeeded in circumscribed domains (e.g., visual neuroscience), but implementing domain-general EMs that predict brain-wide activity for arbitrary tasks has been limited mainly by availability of datasets that 1) sufficiently span a large space of psychological functions, and 2) are sufficiently annotated with such functions to allow robust EM specification. We examine the use of EMs based on a formal specification of psychological function, to predict cortical activation patterns across a broad range of tasks. We utilized the Multi-Domain Task Battery, a dataset in which 24 subjects completed 32 ten-minute fMRI scans, switching tasks every 35 s and engaging in 44 total conditions of diverse psychological manipulations. Conditions were annotated by a group of experts using the Cognitive Atlas ontology to identify putatively engaged functions, and region-wise cognitive EMs (CEMs) were fit, for individual subjects, on neocortical responses. We found that CEMs predicted cortical activation maps of held-out tasks with high accuracy, outperforming a permutation-based null model while approaching the noise ceiling of the data, without being driven solely by either cognitive or perceptual-motor features. Hierarchical clustering on the similarity structure of CEM generalization errors revealed relationships amongst psychological functions. Spatial distributions of feature importances systematically overlapped with large-scale resting-state functional networks (RSNs), supporting the hypothesis of functional specialization within RSNs while grounding their function in an interpretable data-driven manner. Our implementation and validation of CEMs provides a proof of principle for the utility of formal ontologies in cognitive neuroscience and motivates the use of CEMs in the further testing of cognitive theories.

Functional neuroimaging in psychiatry and the case for failing better

Psychiatric disorders encompass complex aberrations of cognition and affect and are among the most debilitating and poorly understood of any medical condition. Current treatments rely primarily on interventions that target brain function (drugs) or learning processes (psychotherapy). A mechanistic understanding of how these interventions mediate their therapeutic effects remains elusive. From the early 1990s, non-invasive functional neuroimaging, coupled with parallel developments in the cognitive neurosciences, seemed to signal a new era of neurobiologically grounded diagnosis and treatment in psychiatry. Yet, despite three decades of intense neuroimaging research, we still lack a neurobiological account for any psychiatric condition. Likewise, functional neuroimaging plays no role in clinical decision making. Here, we offer a critical commentary on this impasse and suggest how the field might fare better and deliver impactful neurobiological insights.

Lateralization of short-term memory in the frontal cortex

Despite essentially symmetric structures in mammalian brains, the left and right hemispheres do not contribute equally to certain cognitive functions. How both hemispheres interact to cause this asymmetry remains unclear. Here, we study this question in the anterior lateral motor cortex (ALM) of mice performing five versions of a tactile-based decision-making task with a short-term memory (STM) component. Unilateral inhibition of ALM produces variable behavioral deficits across tasks, with the left, right, or both ALMs playing critical roles in STM. Neural activity and its encoding capability are similar across hemispheres, despite that only one hemisphere dominates in behavior. Inhibition of the dominant ALM disrupts encoding capability in the non-dominant ALM, but not vice versa. Variable behavioral deficits are predicted by the influence on contralateral activity across sessions, mice, and tasks. Together, these results reveal that the left and right ALM interact asymmetrically, leading to their differential contributions to STM.

Shared and specialized coding across posterior cortical areas for dynamic navigation decisions

Animals adaptively integrate sensation, planning, and action to navigate toward goal locations in ever-changing environments, but the functional organization of cortex supporting these processes remains unclear. We characterized encoding in approximately 90,000 neurons across the mouse posterior cortex during a virtual navigation task with rule switching. The encoding of task and behavioral variables was highly distributed across cortical areas but differed in magnitude, resulting in three spatial gradients for visual cue, spatial position plus dynamics of choice formation, and locomotion, with peaks respectively in visual, retrosplenial, and parietal cortices. Surprisingly, the conjunctive encoding of these variables in single neurons was similar throughout the posterior cortex, creating high-dimensional representations in all areas instead of revealing computations specialized for each area. We propose that, for guiding navigation decisions, the posterior cortex operates in parallel rather than hierarchically, and collectively generates a state representation of the behavior and environment, with each area specialized in handling distinct information modalities.

Information normally considered task-irrelevant drives decision-making and affects premotor circuit recruitment

Decision-making is a continuous and dynamic process with prior experience reflected in and used by the brain to guide adaptive behavior. However, most neurobiological studies constrain behavior and/or analyses to task-related variables, not accounting for the continuous internal and temporal space in which they occur. We show mice rely on information learned through recent and longer-term experience beyond just prior actions and reward - including checking behavior and the passage of time - to guide self-initiated, self-paced, and self-generated actions. These experiences are represented in secondary motor cortex (M2) activity and its projections into dorsal medial striatum (DMS). M2 integrates this information to bias strategy-level decision-making, and DMS projections reflect specific aspects of this recent experience to guide actions. This suggests diverse aspects of experience drive decision-making and its neural representation, and shows premotor corticostriatal circuits are crucial for using selective aspects of experiential information to guide adaptive behavior.

Mice exhibit stochastic and efficient action switching during probabilistic decision making

In probabilistic and nonstationary environments, individuals must use internal and external cues to flexibly make decisions that lead to desirable outcomes. To gain insight into the process by which animals choose between actions, we trained mice in a task with time-varying reward probabilities. In our implementation of such a two-armed bandit task, thirsty mice use information about recent action and action–outcome histories to choose between two ports that deliver water probabilistically. Here we comprehensively modeled choice behavior in this task, including the trial-to-trial changes in port selection, i.e., action switching behavior. We find that mouse behavior is, at times, deterministic and, at others, apparently stochastic. The behavior deviates from that of a theoretically optimal agent performing Bayesian inference in a hidden Markov model (HMM). We formulate a set of models based on logistic regression, reinforcement learning, and sticky Bayesian inference that we demonstrate are mathematically equivalent and that accurately describe mouse behavior. The switching behavior of mice in the task is captured in each model by a stochastic action policy, a history-dependent representation of action value, and a tendency to repeat actions despite incoming evidence. The models parsimoniously capture behavior across different environmental conditionals by varying the stickiness parameter, and like the mice, they achieve nearly maximal reward rates. These results indicate that mouse behavior reaches near-maximal performance with reduced action switching and can be described by a set of equivalent models with a small number of relatively fixed parameters.

Opponent control of behavior by dorsomedial striatal pathways depends on task demands and internal state

A classic view of the striatum holds that activity in direct and indirect pathways oppositely modulates motor output. Whether this involves direct control of movement, or reflects a cognitive process underlying movement, remains unresolved. Here we find that strong, opponent control of behavior by the two pathways of the dorsomedial striatum depends on the cognitive requirements of a task. Furthermore, a latent state model (a hidden Markov model with generalized linear model observations) reveals that-even within a single task-the contribution of the two pathways to behavior is state dependent. Specifically, the two pathways have large contributions in one of two states associated with a strategy of evidence accumulation, compared to a state associated with a strategy of repeating previous choices. Thus, both the demands imposed by a task, as well as the internal state of mice when performing a task, determine whether dorsomedial striatum pathways provide strong and opponent control of behavior.

Different Forms of Variability Could Explain a Difference Between Human and Rat Decision Making

When observers make rapid, difficult perceptual decisions, their response time is highly variable from trial to trial. In a visual motion discrimination task, it has been reported that human accuracy declines with increasing response time, whereas rat accuracy increases with response time. This is of interest because different mathematical theories of decision-making differ in their predictions regarding the correlation of accuracy with response time. On the premise that perceptual decision-making mechanisms are likely to be conserved among mammals, we seek to unify the rodent and primate results in a common theoretical framework. We show that a bounded drift diffusion model (DDM) can explain both effects with variable parameters: trial-to-trial variability in the starting point of the diffusion process produces the pattern typically observed in rats, whereas variability in the drift rate produces the pattern typically observed in humans. We further show that the same effects can be produced by deterministic biases, even in the absence of parameter stochasticity or parameter change within a trial.

Assessing the functions underlying learning using by-trial and by-participant models: Evidence from two visual perceptual learning paradigms

Inferred mechanisms of learning, such as those involved in improvements resulting from perceptual training, are reliant on (and reflect) the functional forms that models of learning take. However, previous investigations of the functional forms of perceptual learning have been limited in ways that are incompatible with the known mechanisms of learning. For instance, previous work has overwhelmingly aggregated learning data across learning participants, learning trials, or both. Here we approach the study of the functional form of perceptual learning on the by-person and by-trial levels at which the mechanisms of learning are expected to act. Each participant completed one of two visual perceptual learning tasks over the course of two days, with the first 75% of task performance using a single reference stimulus (i.e., "training") and the last 25% using an orthogonal reference stimulus (to test generalization). Five learning functions, coming from either the exponential or the power family, were fit to each participant's data. The exponential family was uniformly supported by Bayesian Information Criteria (BIC) model comparisons. The simplest exponential function was the best fit to learning on a texture oddball detection task, while a Weibull (augmented exponential) function tended to be the best fit to learning on a dot-motion discrimination task. The support for the exponential family corroborated previous by-person investigations of the functional form of learning, while the novel evidence supporting the Weibull learning model has implications for both the analysis and the mechanistic bases of the learning.

Standardized and reproducible measurement of decision-making in mice

Progress in science requires standardized assays whose results can be readily shared, compared, and reproduced across laboratories. Reproducibility, however, has been a concern in neuroscience, particularly for measurements of mouse behavior. Here, we show that a standardized task to probe decision-making in mice produces reproducible results across multiple laboratories. We adopted a task for head-fixed mice that assays perceptual and value-based decision making, and we standardized training protocol and experimental hardware, software, and procedures. We trained 140 mice across seven laboratories in three countries, and we collected 5 million mouse choices into a publicly available database. Learning speed was variable across mice and laboratories, but once training was complete there were no significant differences in behavior across laboratories. Mice in different laboratories adopted similar reliance on visual stimuli, on past successes and failures, and on estimates of stimulus prior probability to guide their choices. These results reveal that a complex mouse behavior can be reproduced across multiple laboratories. They establish a standard for reproducible rodent behavior, and provide an unprecedented dataset and open-access tools to study decision-making in mice. More generally, they indicate a path toward achieving reproducibility in neuroscience through collaborative open-science approaches.

Standardized and reproducible measurement of decision-making in mice

Progress in science requires standardized assays whose results can be readily shared, compared, and reproduced across laboratories. Reproducibility, however, has been a concern in neuroscience, particularly for measurements of mouse behavior. Here, we show that a standardized task to probe decision-making in mice produces reproducible results across multiple laboratories. We adopted a task for head-fixed mice that assays perceptual and value-based decision making, and we standardized training protocol and experimental hardware, software, and procedures. We trained 140 mice across seven laboratories in three countries, and we collected 5 million mouse choices into a publicly available database. Learning speed was variable across mice and laboratories, but once training was complete there were no significant differences in behavior across laboratories. Mice in different laboratories adopted similar reliance on visual stimuli, on past successes and failures, and on estimates of stimulus prior probability to guide their choices. These results reveal that a complex mouse behavior can be reproduced across multiple laboratories. They establish a standard for reproducible rodent behavior, and provide an unprecedented dataset and open-access tools to study decision-making in mice. More generally, they indicate a path toward achieving reproducibility in neuroscience through collaborative open-science approaches.

Neuroscience needs behavior: inferring psychophysical strategy trial by trial

Tools quantifying dynamic behavior are important to understand brain function. In this issue of Neuron, Roy et al. (2021) extended the available repertoire with PsyTrack, which tracks, trial by trial, how subjects performing psychophysical tasks adjust the way they weigh stimuli and task covariates or change their biases.

Inferring learning rules from animal decision-making

How do animals learn? This remains an elusive question in neuroscience. Whereas reinforcement learning often focuses on the design of algorithms that enable artificial agents to efficiently learn new tasks, here we develop a modeling framework to directly infer the empirical learning rules that animals use to acquire new behaviors. Our method efficiently infers the trial-to-trial changes in an animal's policy, and decomposes those changes into a learning component and a noise component. Specifically, this allows us to: (i) compare different learning rules and objective functions that an animal may be using to update its policy; (ii) estimate distinct learning rates for different parameters of an animal's policy; (iii) identify variations in learning across cohorts of animals; and (iv) uncover trial-to-trial changes that are not captured by normative learning rules. After validating our framework on simulated choice data, we applied our model to data from rats and mice learning perceptual decision-making tasks. We found that certain learning rules were far more capable of explaining trial-to-trial changes in an animal's policy. Whereas the average contribution of the conventional REINFORCE learning rule to the policy update for mice learning the International Brain Laboratory's task was just 30%, we found that adding baseline parameters allowed the learning rule to explain 92% of the animals' policy updates under our model. Intriguingly, the best-fitting learning rates and baseline values indicate that an animal's policy update, at each trial, does not occur in the direction that maximizes expected reward. Understanding how an animal transitions from chance-level to high-accuracy performance when learning a new task not only provides neuroscientists with insight into their animals, but also provides concrete examples of biological learning algorithms to the machine learning community.