Explicit neural signals reflecting reward uncertainty

Syncopation as structure bootstrapping: the role of asymmetry in rhythm and language

Syncopation - the occurrence of a musical event on a metrically weak position preceding a rest on a metrically strong position - represents an important challenge in the study of the mapping between rhythm and meter. In this contribution, we present the hypothesis that syncopation is an effective strategy to elicit the bootstrapping of a multi-layered, hierarchically organized metric structure from a linear rhythmic surface. The hypothesis is inspired by a parallel with the problem of linearization in natural language syntax, which is the problem of how hierarchically organized phrase-structure markers are mapped onto linear sequences of words. The hypothesis has important consequences for the role of meter in music perception and cognition and, more particularly, for its role in the relationship between rhythm and bodily entrainment.

Resolving the Delusion Paradox

BACKGROUND AND HYPOTHESIS

The neurocomputational framework of predictive processing (PP) provides a promising approach to explaining delusions, a key symptom of psychotic disorders. According to PP, the brain makes inferences about the world by weighing prior beliefs against the available sensory data. Mismatches between prior beliefs and sensory data result in prediction errors that may update the brain's model of the world. Psychosis has been associated with reduced weighting of priors relative to the sensory data. However, delusional beliefs are highly resistant to change, suggesting increased rather than decreased weighting of priors. We propose that this "delusion paradox" can be resolved within a hierarchical PP model: Reduced weighting of prior beliefs at low hierarchical levels may be compensated by an increased influence of higher-order beliefs represented at high hierarchical levels, including delusional beliefs. This may sculpt perceptual processing into conformity with delusions and foster their resistance to contradictory evidence.

STUDY DESIGN

We review several lines of experimental evidence on low- and high-level processes, and their neurocognitive underpinnings in delusion-related phenotypes and link them to predicted processing.

STUDY RESULTS

The reviewed evidence supports the notion of decreased weighting of low-level priors and increased weighting of high-level priors, in both delusional and delusion-prone individuals. Moreover, we highlight the role of prefrontal cortex as a neural basis for the increased weighting of high-level prior beliefs and discuss possible clinical implications of the proposed hierarchical predictive-processing model.

CONCLUSIONS

Our review suggests the delusion paradox can be resolved within a hierarchical PP model.

Studying the neural representations of uncertainty

The study of the brain's representations of uncertainty is a central topic in neuroscience. Unlike most quantities of which the neural representation is studied, uncertainty is a property of an observer's beliefs about the world, which poses specific methodological challenges. We analyze how the literature on the neural representations of uncertainty addresses those challenges and distinguish between 'code-driven' and 'correlational' approaches. Code-driven approaches make assumptions about the neural code for representing world states and the associated uncertainty. By contrast, correlational approaches search for relationships between uncertainty and neural activity without constraints on the neural representation of the world state that this uncertainty accompanies. To compare these two approaches, we apply several criteria for neural representations: sensitivity, specificity, invariance and functionality. Our analysis reveals that the two approaches lead to different but complementary findings, shaping new research questions and guiding future experiments.

Decision-making under ambiguity and risk and executive functions in Parkinson's disease patients: A scoping review of the studies investigating the Iowa Gambling Task and the Game of Dice

Evidence shows that patients affected by Parkinson's disease (PD) display the tendency toward making risky choices. This is due, at least in part, to the pathophysiological characteristics of the disease that affects neural areas underlying decision making (DM), in which a pivotal role is played by nonmotor corticostriatal circuits and dopamine. Executive functions (EFs), which can be impaired by PD as well, may sustain optimal choices in DM processes. However, few studies have investigated whether EFs can support PD patients to make good decisions. Adopting the scoping review approach, the present article is designed to deepen the cognitive mechanisms of DM under conditions of ambiguity and risk (that are conditions common to everyday life decisions) in PD patients without impulse control disorders. We focused our attention on the Iowa Gambling Task and the Game of Dice Task, because they are the most commonly used and reliable tasks to assess DM under ambiguity and under risk, respectively, and analyzed the performances in such tasks and their relationships with EFs tests in PD patients. The analysis supported the relationships between EFs and DM performance, especially when a higher cognitive load is required to make optimal decisions, as it happens under conditions of risk. Possible knowledge gaps and further research directions are suggested to better understand DM mechanisms in PD sustaining patients' cognitive functioning and preventing negative consequences in everyday life derived from suboptimal decisions.

Blocking D2/D3 dopamine receptors in male participants increases volatility of beliefs when learning to trust others

The ability to learn about other people is crucial for human social functioning. Dopamine has been proposed to regulate the precision of beliefs, but direct behavioural evidence of this is lacking. In this study, we investigate how a high dose of the D2/D3 dopamine receptor antagonist sulpiride impacts learning about other people's prosocial attitudes in a repeated Trust game. Using a Bayesian model of belief updating, we show that in a sample of 76 male participants sulpiride increases the volatility of beliefs, which leads to higher precision weights on prediction errors. This effect is driven by participants with genetically conferred higher dopamine availability (Taq1a polymorphism) and remains even after controlling for working memory performance. Higher precision weights are reflected in higher reciprocal behaviour in the repeated Trust game but not in single-round Trust games. Our data provide evidence that the D2 receptors are pivotal in regulating prediction error-driven belief updating in a social context.

Mechanisms of adjustments to different types of uncertainty in the reward environment across mice and monkeys

Despite being unpredictable and uncertain, reward environments often exhibit certain regularities, and animals navigating these environments try to detect and utilize such regularities to adapt their behavior. However, successful learning requires that animals also adjust to uncertainty associated with those regularities. Here, we analyzed choice data from two comparable dynamic foraging tasks in mice and monkeys to investigate mechanisms underlying adjustments to different types of uncertainty. In these tasks, animals selected between two choice options that delivered reward probabilistically, while baseline reward probabilities changed after a variable number (block) of trials without any cues to the animals. To measure adjustments in behavior, we applied multiple metrics based on information theory that quantify consistency in behavior, and fit choice data using reinforcement learning models. We found that in both species, learning and choice were affected by uncertainty about reward outcomes (in terms of determining the better option) and by expectation about when the environment may change. However, these effects were mediated through different mechanisms. First, more uncertainty about the better option resulted in slower learning and forgetting in mice, whereas it had no significant effect in monkeys. Second, expectation of block switches accompanied slower learning, faster forgetting, and increased stochasticity in choice in mice, whereas it only reduced learning rates in monkeys. Overall, while demonstrating the usefulness of metrics based on information theory in examining adaptive behavior, our study provides evidence for multiple types of adjustments in learning and choice behavior according to uncertainty in the reward environment.

Pupil size predicts the onset of exploration in brain and behavior

In uncertain environments, intelligent decision-makers exploit actions that have been rewarding in the past, but also explore actions that could be even better. Several neuromodulatory systems are implicated in exploration, based, in part, on work linking exploration to pupil size-a peripheral correlate of neuromodulatory tone and index of arousal. However, pupil size could instead track variables that make exploration more likely, like volatility or reward, without directly predicting either exploration or its neural bases. Here, we simultaneously measured pupil size, exploration, and neural population activity in the prefrontal cortex while two rhesus macaques explored and exploited in a dynamic environment. We found that pupil size under constant luminance specifically predicted the onset of exploration, beyond what could be explained by reward history. Pupil size also predicted disorganized patterns of prefrontal neural activity at both the single neuron and population levels, even within periods of exploitation. Ultimately, our results support a model in which pupil-linked mechanisms promote the onset of exploration via driving the prefrontal cortex through a critical tipping point where prefrontal control dynamics become disorganized and exploratory decisions are possible.

Distinct neural dynamics underlying risk and ambiguity during valued-based decision making

Uncertainty can be fractioned into risk and ambiguity psychologically and neurobiologically. However, whether and how risk and ambiguity are dissociated in terms of neural dynamics during value-based decision making remain elusive. The present event-related potential (ERP) study addressed these issues by asking participants to perform a wheel-of-fortune task either during a risky context (Experiment 1; N = 30) where outcome probability was known or during an ambiguous context (Experiment 2; N = 30) where outcome probability was unknown. Results revealed that the cue-P3 was more enhanced for risk versus ambiguity during the anticipatory phase, whereas the RewP was more increased for ambiguity than risk during the consummatory phase. Moreover, the SPN and the fb-P3 components were further modulated by the levels of risk and ambiguity, respectively. These findings demonstrate a neural dissociation between risk and ambiguity, which unfolds from the anticipatory phase to the consummatory phase.

Uncertainty modulates visual maps during noninstrumental information demand

Animals are intrinsically motivated to obtain information independently of instrumental incentives. This motivation depends on two factors: a desire to resolve uncertainty by gathering accurate information and a desire to obtain positively-valenced observations, which predict favorable rather than unfavorable outcomes. To understand the neural mechanisms, we recorded parietal cortical activity implicated in prioritizing stimuli for spatial attention and gaze, in a task in which monkeys were free (but not trained) to obtain information about probabilistic non-contingent rewards. We show that valence and uncertainty independently modulated parietal neuronal activity, and uncertainty but not reward-related enhancement consistently correlated with behavioral sensitivity. The findings suggest uncertainty-driven and valence-driven information demand depend on partially distinct pathways, with the former being consistently related to parietal responses and the latter depending on additional mechanisms implemented in downstream structures.

Curiosity is motivated by uncertainty and valence, but how uncertainty and valence are encoded in the brain remains poorly understood. Here, the authors show that parietal neurons are enhanced by both factors, but that they specifically predict visual information seeking based on uncertainty.

Executive Functions in Decision Making under Ambiguity and Risk in Healthy Adults: A Scoping Review Adopting the Hot and Cold Executive Functions Perspective

Decision making (DM) has a pivotal role in supporting individual autonomy and well-being. It is considered a complex ability exploiting many cognitive functions, among which executive functions (EFs) are crucial. Few studies analyzed the role played by EFs in DM in healthy adults under ambiguity and risk, which are common conditions for most decisions in daily life. This scoping review aims to analyze the relationships between two individual tasks widely used to assess DM under these conditions (Iowa Gambling Task and Game of Dice Task) and EFs. According to the organizing principle that conceptualizes hot and cold EFs, DM under such conditions mainly implies hot EFs, but the relationship with cold EFs is still unclear. Using such an approach, a comprehensive framework is provided, highlighting main findings and identifying possible gaps in the literature. The results suggest different roles played by cold EFs in DM under ambiguity and risk, according to the characteristics of the tasks. The findings can offer guidance to further studies and to design interventions to support DM in healthy adults.

Meta-analysis of human prediction error for incentives, perception, cognition, and action

Prediction errors (PEs) are a keystone for computational neuroscience. Their association with midbrain neural firing has been confirmed across species and has inspired the construction of artificial intelligence that can outperform humans. However, there is still much to learn. Here, we leverage the wealth of human PE data acquired in the functional neuroimaging setting in service of a deeper understanding, using an MKDA (multi-level kernel-based density) meta-analysis. Studies were identified with Google Scholar, and we included studies with healthy adult participants that reported activation coordinates corresponding to PEs published between 1999-2018. Across 264 PE studies that have focused on reward, punishment, action, cognition, and perception, consistent with domain-general theoretical models of prediction error we found midbrain PE signals during cognitive and reward learning tasks, and an insula PE signal for perceptual, social, cognitive, and reward prediction errors. There was evidence for domain-specific error signals--in the visual hierarchy during visual perception, and the dorsomedial prefrontal cortex during social inference. We assessed bias following prior neuroimaging meta-analyses and used family-wise error correction for multiple comparisons. This organization of computation by region will be invaluable in building and testing mechanistic models of cognitive function and dysfunction in machines, humans, and other animals. Limitations include small sample sizes and ROI masking in some included studies, which we addressed by weighting each study by sample size, and directly comparing whole brain vs. ROI-based results.

How Stress Can Change Our Deepest Preferences: Stress Habituation Explained Using the Free Energy Principle

People who habituate to stress show a repetition-induced response attenuation—neuroendocrine, cardiovascular, neuroenergetic, and emotional—when exposed to a threatening environment. But the exact dynamics underlying stress habituation remain obscure. The free energy principle offers a unifying account of self-organising systems such as the human brain. In this paper, we elaborate on how stress habituation can be explained and modelled using the free energy principle. We introduce habituation priors that encode the agent’s tendency for stress habituation and incorporate them in the agent’s decision-making process. Using differently shaped goal priors—that encode the agent’s goal preferences—we illustrate, in two examples, the optimising (and thus habituating) behaviour of agents. We show that habituation minimises free energy by reducing the precision (inverse variance) of goal preferences. Reducing the precision of goal priors means that the agent accepts adverse (previously unconscionable) states (e.g., lower social status and poverty). Acceptance or tolerance of adverse outcomes may explain why habituation causes people to exhibit an attenuation of the stress response. Given that stress habituation occurs in brain regions where goal priors are encoded, i.e., in the ventromedial prefrontal cortex and that these priors are encoded as sufficient statistics of probability distributions, our approach seems plausible from an anatomical-functional and neuro-statistical point of view. The ensuing formal and generalisable account—based on the free energy principle—further motivate our novel treatment of stress habituation. Our analysis suggests that stress habituation has far-reaching consequences, protecting against the harmful effects of toxic stress, but on the other hand making the acceptability of precarious living conditions and the development of the obese type 2 diabetes mellitus phenotype more likely.

Computational Neuroscience Approach to Psychiatry: A Review on Theory-driven Approaches

Translating progress in neuroscience into clinical benefits for patients with psychiatric disorders is challenging because it involves the brain as the most complex organ and its interaction with a complex environment and condition. Dealing with such complexity requires powerful techniques. Computational neuroscience approach to psychiatry integrates multiple levels and types of simulation, analysis and computation according to the different types of computational models to enhance comprehending, prediction and treatment of psychiatric disorder. This approach comprises two approaches: theory-driven and data-driven. In this review, we focus on recent advances in theory-driven approaches that mathematically and mechanistically examine the relationships between disorder-related changes and behavior at different level of brain organization. We discuss recent progresses in computational neuroscience models that relate to psychiatry and show how principles of neural computational modeling can be employed to explain psychopathology.

Characterizing reward system neural trajectories from adolescence to young adulthood

Mixed findings exist in studies comparing brain responses to reward in adolescents and adults. Here we examined the trajectories of brain response, functional connectivity and task-modulated network properties during reward processing with a large-sample longitudinal design. Participants from the IMAGEN study performed a Monetary Incentive Delay task during fMRI at timepoint 1 (T1; n = 1304, mean age=14.44 years old) and timepoint 2 (T2; n = 1241, mean age=19.09 years). The Alcohol Use Disorders Identification Test (AUDIT) was administrated at both T1 and T2 to assess a participant’s alcohol use during the past year. Voxel-wise linear mixed effect models were used to compare whole brain response as well as functional connectivity of the ventral striatum (VS) during reward anticipation (large reward vs no-reward cue) between T1 and T2. In addition, task-modulated networks were constructed using generalized psychophysiological interaction analysis and summarized with graph theory metrics. To explore alcohol use in relation to development, participants with no/low alcohol use at T1 but increased alcohol use to hazardous use level at T2 (i.e., participants with AUDIT≤2 at T1 and ≥8 at T2) were compared against those with consistently low scores (i.e., participants with AUDIT≤2 at T1 and ≤7 at T2). Across the whole sample, lower brain response during reward anticipation was observed at T2 compared with T1 in bilateral caudate nucleus, VS, thalamus, midbrain, dorsal anterior cingulate as well as left precentral and postcentral gyrus. Conversely, greater response was observed bilaterally in the inferior and middle frontal gyrus and right precentral and postcentral gyrus at T2 (vs. T1). Increased functional connectivity with VS was found in frontal, temporal, parietal and occipital regions at T2. Graph theory metrics of the task-modulated network showed higher inter-regional connectivity and topological efficiency at T2. Interactive effects between time (T1 vs. T2) and alcohol use group (low vs. high) on the functional connectivity were observed between left middle temporal gyrus and right VS and the characteristic shortest path length of the task-modulated networks. Collectively, these results demonstrate the utility of the MID task as a probe of typical brain response and network properties during development and of differences in these features related to adolescent drinking, a reward-related behaviour associated with heightened risk for future negative health outcomes.

•

Imaging data during reward anticipation at T1 (age 14) and T2 (age 19) was compared.

•

Brain response decreased in subcortical areas and increased in cortical areas at T2.

•

Functional connectivity (FC) with the ventral striatum increased at T2.

•

Topological efficiency of task-modulated network increased at T2.

•

The developmental pattern was altered in those who increased drinking most at T2.

Impaired Perception of Unintentional Transgression of Social Norms after Prefrontal Cortex Damage: Relationship to Decision Making, Emotion Recognition, and Executive Functions

OBJECTIVE

Patients with prefrontal cortex damage often transgress social rules and show lower accuracy in identifying and explaining inappropriate social behavior. The objective of this study was to examine the relationship between the ability to perceive other unintentional transgressions of social norms and both decision making and emotion recognition as these abilities are critical for appropriate social behavior.

METHOD

We examined a group of patients with focal prefrontal cortex damage (N = 28) and a group of matched control participants (N = 28) for their abilities to detect unintentional transgression of social norms using the "Faux-Pas" task of theory of mind, to make advantageous decisions on the Iowa gambling task, and to recognize basic emotions on the Ekman facial affect test.

RESULTS

The group of patients with frontal lobe damage was impaired in all of these tasks compared with control participants. Moreover, all the "Faux-Pas", Iowa gambling, and emotion recognition tasks were significantly associated and predicted by executive measures of inhibition, flexibility, or planning. However, only measures from the Iowa gambling task were associated and predicted performance on the "Faux-Pas" task. These tasks were not associated with performance in recognition of basic emotions. These findings suggest that theory of mind, executive functions, and decision-making abilities act in an interdependent way for appropriate social behavior. However, theory of mind and emotion recognition seem to have distinct but additive effects upon social behavior. Results from VLSM analysis also corroborate these data by showing a partially overlapped prefrontal circuitry underlying these cognitive domains.

Tonic dopamine, uncertainty and basal ganglia action selection

To make optimal decisions in uncertain circumstances flexible adaption of behaviour is required; exploring alternatives when the best choice is unknown, exploiting what is known when that is best. Using a computational model of the basal ganglia, we propose that switches between exploratory and exploitative decisions are mediated by the interaction between tonic dopamine and cortical input to the basal ganglia. We show that a biologically detailed action selection circuit model, endowed with dopamine dependant striatal plasticity, can optimally solve the explore-exploit problem, estimating the true underlying state of a noisy Gaussian diffusion process. Critical to the model's performance was a fluctuating level of tonic dopamine which increased under conditions of uncertainty. With an optimal range of tonic dopamine, explore-exploit decisions were mediated by the effects of tonic dopamine on the precision of the model action selection mechanism. Under conditions of uncertain reward pay-out, the model's reduced selectivity allowed disinhibition of multiple alternative actions to be explored at random. Conversely, when uncertainly about reward pay-out was low, enhanced selectivity of the action selection circuit facilitated exploitation of the high value choice. Model performance was at the level of a Kalman filter which provides an optimal solution for the task. These simulations support the idea that this subcortical neural circuit may have evolved to facilitate decision making in non-stationary reward environments. The model generates several experimental predictions with relevance to abnormal decision making in neuropsychiatric and neurological disease.

Two sources of uncertainty independently modulate temporal expectancy

The environment is shaped by two sources of temporal uncertainty: the discrete probability of whether an event will occur and-if it does-the continuous probability of when it will happen. These two types of uncertainty are fundamental to every form of anticipatory behavior including learning, decision-making, and motor planning. It remains unknown how the brain models the two uncertainty parameters and how they interact in anticipation. It is commonly assumed that the discrete probability of whether an event will occur has a fixed effect on event expectancy over time. In contrast, we first demonstrate that this pattern is highly dynamic and monotonically increases across time. Intriguingly, this behavior is independent of the continuous probability of when an event will occur. The effect of this continuous probability on anticipation is commonly proposed to be driven by the hazard rate (HR) of events. We next show that the HR fails to account for behavior and propose a model of event expectancy based on the probability density function of events. Our results hold for both vision and audition, suggesting independence of the representation of the two uncertainties from sensory input modality. These findings enrich the understanding of fundamental anticipatory processes and have provocative implications for many aspects of behavior and its neural underpinnings.

Dissociating brain systems that respond to contingency and valence during monetary loss avoidance in adolescence

Negative reinforcement processes allow individuals to avoid negative and/or harmful outcomes. They depend on the brain's ability to differentiate; (i) contingency from non-contingency, separately from (ii) judgements about positive and negative valence. Thirty-three males (8-18 years) performed a cued reaction-time task during fMRI scanning to differentiate the brain's responses to contingency and valence during loss avoidance. In two conditions, cues indicated no -contingency between participants' responses and monetary loss - (1) CERTAIN LOSS (negative valence) of €0.20, €1 or €5 or (2) CERTAIN LOSS AVOIDANCE (positive valence). In a third condition, cues indicated a contingency between short reaction times and avoidance of monetary loss. As expected participants had shorter reaction times in this latter condition where CONDITIONAL LOSS AVOIDANCE cues activated salience and motor-response-preparation brain networks - independent of the relative valence of the contrast (CERTAIN LOSS or CERTAIN LOSS AVOIDANCE). Effects of valence were seen toward the session's end where CERTAIN LOSS AVOIDANCE cues activated ventral striatum, medial-orbitofrontal cortex and medial-temporal areas more than CERTAIN LOSS. CONDITIONAL LOSS AVOIDANCE trials with feedback indicating "success" activated ventral striatum more than "failure feedback". The findings support the hypothesis that brain networks controlling contingency and valence processes during negative reinforcement are dissociable.

Shifting uncertainty intolerance: methylphenidate and attention-deficit hyperactivity disorder

Risk evaluation is a critical component of decision making. Risk tolerance is relevant in both daily decisions and pathological disorders such as attention-deficit hyperactivity disorder (ADHD), where impulsivity is a cardinal symptom. Methylphenidate, a commonly prescribed drug in ADHD, improves attention but has mixed reports on risk-based decision making. Using a double-blinded placebo protocol, we studied the risk attitudes of ADHD patients and age-matched healthy volunteers while performing the 2-step sequential learning task and examined the effect of methylphenidate on their choices. We then applied a novel computational analysis using the hierarchical drift-diffusion model to extract parameters such as threshold ('a'-amount of evidence accumulated before making a decision), drift rate ('v'-information processing speed) and response bias ('z' apriori bias towards a specific choice) focusing specifically on risky choice preference. Critically, we show that ADHD patients on placebo have an apriori bias towards risky choices compared to controls. Furthermore, methylphenidate enhanced preference towards risky choices (higher apriori bias) in both groups but had a significantly greater effect in the patient population independent of clinical scores. Thus, methylphenidate appears to shift tolerance towards risky uncertain choices possibly mediated by prefrontal dopaminergic and noradrenergic modulation. We emphasise the utility of computational models in detecting underlying processes. Our findings have implications for subtle yet differential effects of methylphenidate on ADHD compared to healthy population.

Between Action and Emotional Survival During the COVID-19 era: Sensorimotor Pathways as Control Systems of Transdiagnostic Anxiety-Related Intolerance to Uncertainty

Objectives: The COVID-19 pandemic and aligned social and physical distancing regulations increase the sense of uncertainty, intensifying the risk for psychopathology globally. Anxiety disorders are associated with intolerance to uncertainty. In this review we describe brain circuits and sensorimotor pathways involved in human reactions to uncertainty. We present the healthy mode of coping with uncertainty and discuss deviations from this mode. Methods: Literature search of PubMed and Google Scholar. Results: As manifestation of anxiety disorders includes peripheral reactions and negative cognitions, we suggest an integrative model of threat cognitions modulated by sensorimotor regions: "The Sensorimotor-Cognitive-Integration-Circuit." The model emphasizes autonomic nervous system coupling with the cortex, addressing peripheral anxious reactions to uncertainty, pathways connecting cortical regions and cost-reward evaluation circuits to sensorimotor regions, filtered by the amygdala and basal ganglia. Of special interest are the ascending and descending tracts for sensory-motor crosstalk in healthy and pathological conditions. We include arguments regarding uncertainty in anxiety reactions to the pandemic and derive from our model treatment suggestions which are supported by scientific evidence. Our model is based on systematic control theories and emphasizes the role of goal conflict regulation in health and pathology. We also address anxiety reactions as a spectrum ranging from healthy to pathological coping with uncertainty, and present this spectrum as a transdiagnostic entity in accordance with recent claims and models. Conclusions: The human need for controllability and predictability suggests that anxiety disorders reactive to the pandemic's uncertainties reflect pathological disorganization of top-down bottom-up signaling and neural noise resulting from non-pathological human needs for coherence in life.

Distributional Reinforcement Learning in the Brain

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Uncertainty modulates value-driven attentional capture

The majority of previous studies on the value modulation of attention have shown that the magnitude of value-driven attentional bias correlates with the strength of reward association. However, relatively little is known about how uncertainty affects value-based attentional bias. We investigated whether attentional capture by previously rewarded stimuli is modulated by the uncertainty of the learned value without the influence of the strength of reward association. Participants were instructed to identify the line orientation in the target color circle. Importantly, each target color was associated with a different level of uncertainty by tuning the variation in reward delivery (Experiment 1) or reward magnitude (Experiment 2). Attentional interference for uncertainty-related distractors was greater than that for certainty distractors in Experiments 1 and 2. In addition, uncertainty-induced attentional bias disappeared earlier than attentional bias for certainty. The study demonstrated that uncertainty modulates value-based attentional capture in terms of strength and persistence, even when the effect of expected value remains constant.

How Outcome Uncertainty Mediates Attention, Learning, and Decision-Making

Animals and humans evolved sophisticated nervous systems that endowed them with the ability to form internal-models or beliefs and make predictions about the future to survive and flourish in a world in which future outcomes are often uncertain. Crucial to this capacity is the ability to adjust behavioral and learning policies in response to the level of uncertainty. Until recently, the neuronal mechanisms that could underlie such uncertainty-guided control have been largely unknown. In this review, I discuss newly discovered neuronal circuits in primates that represent uncertainty about future rewards and propose how they guide information-seeking, attention, decision-making, and learning to help us survive in an uncertain world. Lastly, I discuss the possible relevance of these findings to learning in artificial systems.

Aberrant Salience, Information Processing, and Dopaminergic Signaling in People at Clinical High Risk for Psychosis

The aberrant salience hypothesis proposes that striatal dopamine dysregulation causes misattribution of salience to irrelevant stimuli leading to psychosis. Recently, new lines of preclinical evidence on information coding by subcortical dopamine coupled with computational models of the brain's ability to predict and make inferences about the world (predictive processing) provide a new perspective on this hypothesis. We review these and summarize the evidence for dopamine dysfunction, reward processing, and salience abnormalities in people at clinical high risk of psychosis (CHR) relative to findings in patients with psychosis. This review identifies consistent evidence for dysregulated subcortical dopamine function in people at CHR, but also indicates a number of areas where neurobiological processes are different in CHR subjects relative to patients with psychosis, particularly in reward processing. We then consider how predictive processing models may explain psychotic symptoms in terms of alterations in prediction error and precision signaling using Bayesian approaches. We also review the potential role of environmental risk factors, particularly early adverse life experiences, in influencing the prior expectations that individuals have about their world in terms of computational models of the progression from being at CHR to frank psychosis. We identify a number of key outstanding questions, including the relative roles of prediction error or precision signaling in the development of symptoms and the mechanism underlying dopamine dysfunction. Finally, we discuss how the integration of computational psychiatry with biological investigation may inform the treatment for people at CHR of psychosis.

Neural arbitration between social and individual learning systems

Decision making requires integrating knowledge gathered from personal experiences with advice from others. The neural underpinnings of the process of arbitrating between information sources has not been fully elucidated. In this study, we formalized arbitration as the relative precision of predictions, afforded by each learning system, using hierarchical Bayesian modeling. In a probabilistic learning task, participants predicted the outcome of a lottery using recommendations from a more informed advisor and/or self-sampled outcomes. Decision confidence, as measured by the number of points participants wagered on their predictions, varied with our definition of arbitration as a ratio of precisions. Functional neuroimaging demonstrated that arbitration signals were independent of decision confidence and involved modality-specific brain regions. Arbitrating in favor of self-gathered information activated the dorsolateral prefrontal cortex and the midbrain, whereas arbitrating in favor of social information engaged the ventromedial prefrontal cortex and the amygdala. These findings indicate that relative precision captures arbitration between social and individual learning systems at both behavioral and neural levels.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

Maladaptive consequences of repeated intermittent exposure to uncertainty

Recently we reported that nucleus accumbens (NAcc) dopamine (DA) tracks uncertainty during operant responding for non-caloric saccharin. We also showed that repeated intermittent exposure to this uncertainty, like exposure to drugs of abuse, leads to sensitization of the locomotor and NAcc DA effects of amphetamine and promotes the subsequent self-administration of the drug. Here we review these findings together with others showing that NAcc glutamate signaling is similarly affected by uncertainty. Extracellular levels of glutamate in this site also track uncertainty in a task in which nose poking for saccharin on an escalating variable ratio schedule of reinforcement is associated with progressively increasing variance between performance of the operant and payout. Furthermore, sensitized behavioral responding to and for amphetamine following exposure to uncertainty is accompanied by increased levels of Ca2+/calmodulin-dependent protein kinase II (CaMKII) and protein kinase C (PKC) phosphorylation as well as altered protein levels of the transcription factor ∆FosB (increased) and glutamate transporter 1 (GLT1; decreased) in NAcc tissues. Notably, phosphorylation by CaMKII and PKC regulates AMPA receptor trafficking and function in this site, is elevated following psychostimulant exposure, and is necessary for the expression of enhanced drug taking. Increased ∆FosB and decreased GLT1 levels are observed following psychostimulant exposure, are associated with increased drug taking and seeking, and are known to modulate AMPA receptors and extracellular glutamate levels respectively. These adaptations in glutamate transmission as well as those observed with DA following repeated intermittent exposure to uncertainty are similar to those produced by exposure to abused drugs. Together, they point to the recruitment of both DA and glutamate signaling pathways in the NAcc in both drug and behavioral addictions. As uncertainty is central to games of chance, these findings have particular relevance for gambling disorders known to exhibit comorbidity with drug abuse.

The ease and sureness of a decision: evidence accumulation of conflict and uncertainty

The likelihood of an outcome (uncertainty or sureness) and the similarity between choices (conflict or ease of a decision) are often critical to decision-making. We often ask ourselves: how likely are we to win or lose? And how different is this option's likelihood from the other? Uncertainty is a characteristic of the stimulus and conflict between stimuli, but these dissociable processes are often confounded. Here, applying a novel hierarchical drift diffusion approach, we study their interaction using a sequential learning task in healthy volunteers and pathological groups characterized by compulsive behaviours, by posing it as an evidence accumulation problem. The variables, Conflict (difficult or easy; difference between reward probabilities of the stimuli) and Uncertainty (low, medium or high; inverse U-shaped probability-uncertainty function) were then used to extract threshold ('a', amount of evidence accumulated before making a decision) and drift rate ('v', information processing speed) parameters. Critically, when a decision was both difficult (high conflict) and uncertain, relative to other conditions, healthy volunteers unexpectedly accumulated less evidence with lower decision thresholds and accuracy rates at chance levels. In contrast, patients with obsessive-compulsive disorder had slower processing speeds during these difficult uncertain decisions; yet, despite this more cautious approach, performed suboptimally with poorer accuracy relative to healthy volunteers below that of chance level. Thus, faced with a difficult uncertain decision, healthy controls are capable of rapid possibly random decisions, displaying almost a willingness to 'walk away', whereas those with obsessive compulsive disorder become more deliberative and cautious but despite appearing to learn the differential contingencies, still perform poorly. These observations might underlie disordered behaviours characterized by pathological uncertainty or doubt despite compulsive checking with impaired performance. In contrast, alcohol-dependent subjects show a different pattern relative to healthy controls with difficulties in adjusting their behavioural patterns with slower drift rates or processing speed despite decisions being easy or low conflict. We emphasize the multidimensional nature of compulsive behaviours and the utility of computational models in detecting subtle underlying processes relative to behavioural measures. These observations have implications for targeted behavioural interventions for specific cognitive impairments across psychiatric disorders.

The anticipatory dopamine response in addiction: A common neurobiological underpinning of gambling disorder and substance use disorder?

The dopamine system is associated with reward processes in both gambling disorder and substance use disorder, and may constitute a common neurobiological underpinning in addiction. The present review examines differences and similarities of dopaminergic reward processes in gambling disorder and substance use disorder. First, it is suggested that baseline binding potentials of the dopamine system may not be a common pathway, since substance use disorder is associated with reduced binding potentials, whereas gambling disorder is not. Second, it is suggested that dopaminergic reward response may be not a common pathway, since substance use disorder is associated with a blunted dopamine response toward drugs, while conflicting findings of reward response has been reported in gambling disorder. Instead, it is suggested that the anticpatory dopamine response may constitute a common underpinning of gambling disorder and substance use disorder, which may be associated with increased dopamine activity in both types of disorder, and does not involve the intake of substances. The notion of the anticipatory dopamine response as a common underpinning of gambling disorder and substance use disorder is consistent with dopaminergic models of addictions such as the incentive-sensitization model, the ingrative neurodevelopmental model of vulnerability toward addiction and the reward prediction error model.

A neural network for information seeking

Humans and other animals often show a strong desire to know the uncertain rewards their future has in store, even when they cannot use this information to influence the outcome. However, it is unknown how the brain predicts opportunities to gain information and motivates this information-seeking behavior. Here we show that neurons in a network of interconnected subregions of primate anterior cingulate cortex and basal ganglia predict the moment of gaining information about uncertain rewards. Spontaneous increases in their information prediction signals are followed by gaze shifts toward objects associated with resolving uncertainty, and pharmacologically disrupting this network reduces the motivation to seek information. These findings demonstrate a cortico-basal ganglia mechanism responsible for motivating actions to resolve uncertainty by seeking knowledge about the future.

Animals resolve uncertainty by seeking knowledge about the future. How the brain controls this is unclear. The authors show that a network including primate anterior cingulate cortex and basal ganglia encodes opportunities to gain information about uncertain rewards and mediates information seeking.

A computational account of threat-related attentional bias

Visual selective attention acts as a filter on perceptual information, facilitating learning and inference about important events in an agent's environment. A role for visual attention in reward-based decisions has previously been demonstrated, but it remains unclear how visual attention is recruited during aversive learning, particularly when learning about multiple stimuli concurrently. This question is of particular importance in psychopathology, where enhanced attention to threat is a putative feature of pathological anxiety. Using an aversive reversal learning task that required subjects to learn, and exploit, predictions about multiple stimuli, we show that the allocation of visual attention is influenced significantly by aversive value but not by uncertainty. Moreover, this relationship is bidirectional in that attention biases value updates for attended stimuli, resulting in heightened value estimates. Our findings have implications for understanding biased attention in psychopathology and support a role for learning in the expression of threat-related attentional biases in anxiety.

Parietal neurons encode information sampling based on decision uncertainty

In natural behavior animals actively gather information that is relevant for learning or actions, but the mechanisms of active sampling are rarely investigated. We tested parietal neurons involved in oculomotor control in a task in which monkeys made saccades to gather visual information before reporting a decision based on the information. We show that the neurons encode, before the saccade, the information gains (reduction in decision uncertainty) that the saccade was expected to bring, correlating with the monkeys’ efficiency in processing the information in the post-saccadic fixation. Informational sensitivity is independent of the neurons’ reward sensitivity, which is unreliable across task contexts, inconsistent with the view that the cells encode economic utility. Instead, we suggest that parietal cells are involved in implementing active sampling policies, showing uncertainty-dependent boosts of neural gain that facilitate the selection of relevant cues and the efficient use of the information delivered by these cues.

Diverse motives for human curiosity

Curiosity-our desire to know-is a fundamental drive in human behaviour, but its mechanisms are poorly understood. A classical question concerns the curiosity motives. What drives individuals to become curious about some but not other sources of information?¹ Here we show that curiosity about probabilistic events depends on multiple aspects of the distribution of these events. Participants (n = 257) performed a task in which they could demand advance information about only one of two randomly selected monetary prizes that contributed to their income. Individuals differed markedly in the extent to which they requested information as a function of the ex ante uncertainty or ex ante value of an individual prize. This heterogeneity was not captured by theoretical models describing curiosity as a desire to learn about the total rewards of a situation^2,3. Instead, it could be explained by an extended model that allowed for attribute-specific anticipatory utility-the savouring of individual components of the eventual reward-and postulates that this utility increased nonlinearly with the certainty of receiving the reward. Parameter values fitting individual choices were consistent for information about gains or losses, suggesting that attribute-specific anticipatory utility captures fundamental heterogeneity in the determinants of curiosity.

Exposure to conditions of uncertainty promotes the pursuit of amphetamine

Prior exposure to abused drugs leads to long-lasting neuroadaptations culminating in excessive drug intake. Given the comorbidity between substance use and gambling disorders, surprisingly little is known about the effects of exposure to reinforcement contingencies experienced during games of chance. As it is a central feature of these games, we characterized the effects of exposure to uncertainty on biochemical and behavioral effects normally observed in rats exposed to amphetamine. Rats in different groups were trained to nose-poke for saccharin under certain [fixed-ratio (FR)] or uncertain conditions [variable-ratio (VR)] for 55 1-h sessions. Ratios were escalated on successive sessions and rats maintained on the last ratio (FR/VR 20) for 20-25 days. Two to three weeks later, rats were tested for their locomotor or nucleus accumbens dopamine (NAcc DA) response to amphetamine or self-administration of the drug using a lever press operant. NAcc DA overflow was also assessed in additional rats during the saccharin sessions. Rats exposed to uncertainty subsequently showed a higher locomotor and NAcc DA response to amphetamine and self-administered more drug infusions relative to rats exposed to predictable reinforcement. NAcc DA levels during the saccharin sessions tracked the variance of the scheduled ratios (a measure of uncertainty). VR rats showed escalating DA overflow with increasing ratios. Exposure to uncertainty triggered neuroadaptations similar to those produced by exposure to abused drugs. As these were produced in drug naive rats both during and after exposure to uncertainty, they provide a novel common pathway to drug and behavioral addictions.

Expected Value of Reward Predicts Episodic Memory for Incidentally Learnt Reward-Item Associations

In this paper, we draw connections between reward processing and cognition by behaviourally testing the implications of neurobiological theories of reward processing on memory. Single-cell neurophysiology in non-human primates and imaging work in humans suggests that the dopaminergic reward system responds to different components of reward: expected value; outcome or prediction error; and uncertainty of reward (Schultz et al., 2008). The literature on both incidental and motivated learning has focused on understanding how expected value and outcome—linked to increased activity in the reward system—lead to consolidation-related memory enhancements. In the current study, we additionally investigate the impact of reward uncertainty on human memory. The contribution of reward uncertainty—the spread of the reward probability distribution irrespective of the magnitude—has not been previously examined. To examine the effects of uncertainty on memory, a word-learning task was introduced, along with a surprise delayed recognition memory test. Using Bayesian model selection, we found evidence only for expected value as a predictor of memory performance. Our findings suggest that reward uncertainty does not enhance memory for individual items. This supports emerging evidence that an effect of uncertainty on memory is only observed in high compared to low risk environments.

Disentangling Risk and Uncertainty: When Risk-Taking Measures Are Not About Risk

Many studies claim to measure decision-making under risk by employing the Domain-Specific Risk-Taking (DOSPERT) scale, a self-report measure, or the Balloon Analogue Risk Task (BART), a behavioural task. However, these tasks do not measure decision-making under risk but decision-making under uncertainty, a related but distinct concept. The present commentary discusses both the theoretical and empirical basis of the distinction between uncertainty and risk from the viewpoint of several scientific disciplines and reports how many studies wrongfully employ the DOSPERT scale and BART as risk-taking measures. Importantly, we call for proper distinguishing between (tasks measuring) decision-making under uncertainty and decision-making under risk in psychology, and related fields. We believe this is vital as research has shown that people’s attitudes, behaviour, and brain activity differ between both concepts, indicating that confusing the concepts may lead researchers to erroneous conclusions.

Cognitive neuroscience can support public health approaches to minimise the harm of 'losses disguised as wins' in multiline slot machines

Video slot machines are associated with both accelerated transition into problematic forms of gambling, as well as psychosocial harm above and beyond other forms of gambling. A growing body of evidence is uncovering how key design features of multiline slot machines produce an inflated experience of reward, despite the fact that these features offer no overall financial benefit to the player. A pernicious example of this are 'losses disguised as wins' (LDWs), which occur when simultaneous bets placed on multiple lines result in a winning combination that returns an amount greater than zero, but less the total wager. These events are usually accompanied by the same celebratory sounds and animations that accompany true wins. We argue that LDWs may leverage neuropsychological phenomena that underlie reinforcement learning and contribute to extended or repetitive use and gambling-related harm. While other characteristics of slot machine gambling have been examined by cognitive neuroscientists, this feature has not yet received attention. Neuroscientific methods can be used to assess the impact of LDWs on the human reward system, to assess the claim that these events are a reinforcing and contributing factor in the development of harmful play. Positive findings would provide further persuasive evidence in support of strategies to minimise gambling harm through the regulation of machine design.

Planning and navigation as active inference

This paper introduces an active inference formulation of planning and navigation. It illustrates how the exploitation-exploration dilemma is dissolved by acting to minimise uncertainty (i.e. expected surprise or free energy). We use simulations of a maze problem to illustrate how agents can solve quite complicated problems using context sensitive prior preferences to form subgoals. Our focus is on how epistemic behaviour-driven by novelty and the imperative to reduce uncertainty about the world-contextualises pragmatic or goal-directed behaviour. Using simulations, we illustrate the underlying process theory with synthetic behavioural and electrophysiological responses during exploration of a maze and subsequent navigation to a target location. An interesting phenomenon that emerged from the simulations was a putative distinction between 'place cells'-that fire when a subgoal is reached-and 'path cells'-that fire until a subgoal is reached.

Now you hear it: a predictive coding model for understanding rhythmic incongruity

Rhythmic incongruity in the form of syncopation is a prominent feature of many contemporary musical styles. Syncopations afford incongruity between rhythmic patterns and the meter, giving rise to mental models of differently accented isochronous beats. Syncopations occur either in isolation or as part of rhythmic patterns, so-called grooves. On the basis of the predictive coding framework, we discuss how brain processing of rhythm can be seen as a special case of predictive coding. We present a simple, yet powerful model for how the brain processes rhythmic incongruity: the model for predictive coding of rhythmic incongruity. Our model proposes that a given rhythm's syncopation and its metrical uncertainty (precision) is at the heart of how the brain models rhythm and meter based on priors, predictions, and prediction error. Our minimal model can explain prominent features of brain processing of syncopation: why isolated syncopations lead to stronger prediction error in the brains of musicians, as evidenced by larger event-related potentials to rhythmic incongruity, and why we all experience a stronger urge to move to grooves with a medium level of syncopation compared with low and high levels of syncopation.

Relationships between executive function, working memory, and decision-making on the Iowa Gambling Task: Evidence from ventromedial patients, dorsolateral patients, and normal subjects

The results of previous studies are inconsistent in regard to the relationship between the Iowa Gambling Task (IGT), working-memory (WM), and executive tasks, and whether these cognitive processes could be considered as mechanisms underlying a decision-making deficit. Moreover, the relationship between the IGT and executive measures is examined based on a limited number of executive tasks, within different populations showing diffuse damage. In addition, there are fewer studies carried out within control participants, with those studies also being inconclusive. It is also suggested that the association of the IGT performance with executive tasks depends on whether the IGT was running under ambiguity or under risk. In this work, all of these issues are studied. Results showed that both patients with ventromedial (VMPFC, N = 10) and dorsolateral (DLPFC, N = 10) prefrontal cortex lesions are significantly impaired on almost all executive tasks, WM tasks, and the IGT. Furthermore, when the IGT is run under risk, there are significant correlations between executive measures and the IGT for the DLPFC patients and the control participants (N = 34) but not the VMPFC patients. No correlation was found between WM tasks and the IGT for both frontal subgroups and control participants. These findings suggested that the mechanisms underlying the IGT deficit differ according to the lesion locations.

A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum

Dopamine‐dependent synaptic plasticity is a candidate mechanism for reinforcement learning. A silent eligibility trace – initiated by synaptic activity and transformed into synaptic strengthening by later action of dopamine – has been hypothesized to explain the retroactive effect of dopamine in reinforcing past behaviour. We tested this hypothesis by measuring time‐dependent modulation of synaptic plasticity by dopamine in adult mouse striatum, using whole‐cell recordings. Presynaptic activity followed by postsynaptic action potentials (pre–post) caused spike‐timing‐dependent long‐term depression in D1‐expressing neurons, but not in D2 neurons, and not if postsynaptic activity followed presynaptic activity. Subsequent experiments focused on D1 neurons. Applying a dopamine D1 receptor agonist during induction of pre–post plasticity caused long‐term potentiation. This long‐term potentiation was hidden by long‐term depression occurring concurrently and was unmasked when long‐term depression blocked an L‐type calcium channel antagonist. Long‐term potentiation was blocked by a Ca²⁺‐permeable AMPA receptor antagonist but not by an NMDA antagonist or an L‐type calcium channel antagonist. Pre–post stimulation caused transient elevation of rectification – a marker for expression of Ca²⁺‐permeable AMPA receptors – for 2–4‐s after stimulation. To test for an eligibility trace, dopamine was uncaged at specific time points before and after pre‐ and postsynaptic conjunction of activity. Dopamine caused potentiation selectively at synapses that were active 2‐s before dopamine release, but not at earlier or later times. Our results provide direct evidence for a silent eligibility trace in the synapses of striatal neurons. This dopamine‐timing‐dependent plasticity may play a central role in reinforcement learning.

Amygdala Functional and Structural Connectivity Predicts Individual Risk Tolerance

Risk tolerance, the degree to which an individual is willing to tolerate risk in order to achieve a greater expected return, influences a variety of financial choices and health behaviors. Here we identify intrinsic neural markers for risk tolerance in a large (n = 108) multimodal imaging dataset of healthy young adults, which includes anatomical and resting-state functional MRI and diffusion tensor imaging. Using a data-driven approach, we found that higher risk tolerance was most strongly associated with greater global functional connectivity (node strength) of and greater gray matter volume in bilateral amygdala. Further, risk tolerance was positively associated with functional connectivity between amygdala and medial prefrontal cortex and negatively associated with structural connectivity between these regions. These findings show how the intrinsic functional and structural architecture of the amygdala, and amygdala-medial prefrontal pathways, which have previously been implicated in anxiety, are linked to individual differences in risk tolerance during economic decision making.

Risky Decision Making in Juvenile Myoclonic Epilepsy

It is not known whether patients with juvenile myoclonic epilepsy (JME) differ from healthy people in decision making under risk, i.e., when the decision-making context offers explicit information about options, probabilities, and consequences already from the beginning. In this study, we adopted the Game of Dice Task-Double to investigate decision making under risk in a group of 36 patients with JME (mean age 25.25/SD 5.29 years) and a group of 38 healthy controls (mean age 26.03/SD 4.84 years). Participants also underwent a comprehensive neuropsychological assessment focused on frontal executive functions. Significant group differences were found in tests of psychomotor speed and divided attention, with the patients scoring lower than the controls. Importantly, patients made risky decisions more frequently than controls. In the patient group, poor decision making was associated with poor executive control, poor response inhibition, and a short interval since the last seizure episode. Executive control and response inhibition could predict 42% of variance in the frequency of risky decisions. This study indicates that patients with JME with poorer executive functions are more likely to make risky decisions than healthy controls. Decision making under risk is of major importance in every-day life, especially with regard to treatment decisions and adherence to long-term medical therapy. Since even a single disadvantageous decision may have long-lasting consequences, this finding is of high relevance.

Exploration Disrupts Choice-Predictive Signals and Alters Dynamics in Prefrontal Cortex

In uncertain environments, decision-makers must balance two goals: they must "exploit" rewarding options but also "explore" in order to discover rewarding alternatives. Exploring and exploiting necessarily change how the brain responds to identical stimuli, but little is known about how these states, and transitions between them, change how the brain transforms sensory information into action. To address this question, we recorded neural activity in a prefrontal sensorimotor area while monkeys naturally switched between exploring and exploiting rewarding options. We found that exploration profoundly reduced spatially selective, choice-predictive activity in single neurons and delayed choice-predictive population dynamics. At the same time, reward learning was increased in brain and behavior. These results indicate that exploration is related to sudden disruptions in prefrontal sensorimotor control and rapid, reward-dependent reorganization of control dynamics. This may facilitate discovery through trial and error.

Working memory, attention, and salience in active inference

The psychological concepts of working memory and attention are widely used in the cognitive and neuroscientific literatures. Perhaps because of the interdisciplinary appeal of these concepts, the same terms are often used to mean very different things. Drawing on recent advances in theoretical neurobiology, this paper tries to highlight the correspondence between these established psychological constructs and the formal processes implicit in mathematical descriptions of brain function. Here, we consider attention and salience from the perspective offered by active inference. Using variational principles and simulations, we use active inference to demonstrate how attention and salience can be disambiguated in terms of message passing between populations of neurons in cortical and subcortical structures. In brief, we suggest that salience is something that is afforded to actions that realise epistemic affordance, while attention per se is afforded to precise sensory evidence - or beliefs about the causes of sensations.

Novelty and fear conditioning induced gene expression in high and low states of anxiety

Emotional states influence how stimuli are interpreted. High anxiety states in humans lead to more negative, threatening interpretations of novel information, typically accompanied by activation of the amygdala. We developed a handling protocol that induces long-lasting high and low anxiety-like states in rats to explore the role of state anxiety on brain activation during exposure to a novel environment and fear conditioning. In situ hybridization of the inducible transcription factor Egr-1 found increased gene expression in the lateral nucleus of the amygdala (LA) following exposure to a novel environment and contextual fear conditioning in high anxiety-like rats. In contrast, low state anxiety-like rats did not generate Egr-1 increases in LA when placed in a novel chamber. Egr-1 expression was also examined in the dorsal hippocampus and prefrontal cortex. In CA1 of the hippocampus and medial prefrontal cortex (mPFC), Egr-1 expression increased in response to novel context exposure and fear conditioning, independent of state anxiety level. Furthermore, in mPFC, Egr-1 in low anxiety-like rats was increased more with fear conditioning than novel exposure. The current series of experiments show that brain areas involved in fear and anxiety-like states do not respond uniformly to novelty during high and low states of anxiety.