Neural Circuitry of Reward Prediction Error

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

Dopamine Release in the Nucleus Accumbens Core Encodes the General Excitatory Components of Learning

Dopamine release in the nucleus accumbens core (NAcC) is generally considered to be a proxy for phasic firing of the ventral tegmental area dopamine (VTADA) neurons. Thus, dopamine release in NAcC is hypothesized to reflect a unitary role in reward prediction error signaling. However, recent studies reveal more diverse roles of dopamine neurons, which support an emerging idea that dopamine regulates learning differently in distinct circuits. To understand whether the NAcC might regulate a unique component of learning, we recorded dopamine release in NAcC while male rats performed a backward conditioning task where a reward is followed by a neutral cue. We used this task because we can delineate different components of learning, which include sensory-specific inhibitory and general excitatory components. Furthermore, we have shown that VTADA neurons are necessary for both the specific and general components of backward associations. Here, we found that dopamine release in NAcC increased to the reward across learning while reducing to the cue that followed as it became more expected. This mirrors the dopamine prediction error signal seen during forward conditioning and cannot be accounted for temporal-difference reinforcement learning. Subsequent tests allowed us to dissociate these learning components and revealed that dopamine release in NAcC reflects the general excitatory component of backward associations, but not their sensory-specific component. These results emphasize the importance of examining distinct functions of different dopamine projections in reinforcement learning.

Mating proximity blinds threat perception

Romantic engagement can bias sensory perception. This 'love blindness' reflects a common behavioural principle across organisms: favouring pursuit of a coveted reward over potential risks1. In the case of animal courtship, such sensory biases may support reproductive success but can also expose individuals to danger, such as predation2,3. However, how neural networks balance the trade-off between risk and reward is unknown. Here we discover a dopamine-governed filter mechanism in male Drosophila that reduces threat perception as courtship progresses. We show that during early courtship stages, threat-activated visual neurons inhibit central courtship nodes via specific serotonergic neurons. This serotonergic inhibition prompts flies to abort courtship when they see imminent danger. However, as flies advance in the courtship process, the dopaminergic filter system reduces visual threat responses, shifting the balance from survival to mating. By recording neural activity from males as they approach mating, we demonstrate that progress in courtship is registered as dopaminergic activity levels ramping up. This dopamine signalling inhibits the visual threat detection pathway via Dop2R receptors, allowing male flies to focus on courtship when they are close to copulation. Thus, dopamine signalling biases sensory perception based on perceived goal proximity, to prioritize between competing behaviours.

Pavlovian safety learning: An integrative theoretical review

Safety learning involves associating stimuli with the absence of threats, enabling the inhibition of fear and anxiety. Despite growing interest in psychology, psychiatry, and neuroscience, safety learning lacks a formal consensus definition, leading to inconsistent methodologies and varied results. Conceptualized as a form of inhibitory learning (conditioned inhibition), safety learning can be understood through formal learning theories, such as the Rescorla-Wagner and Pearce-Hall models. This review aims to establish a principled conceptualization of 'Pavlovian safety learning', identifying cognitive mechanisms that generate safety and the boundary conditions that constrain it. Based on these observations, we define Pavlovian safety learning as an active associative process, where surprising threat-omission (safety prediction error) acts as a salient reinforcing event. Instead of producing merely neutral or nonaversive states, safety learning endows stimuli with active positive associations to 'safety'. The resulting stimulus-safety memories counteract the influence of fear memories, promoting fear regulation, positive affect, and relief. We critically analyze traditional criteria of conditioned inhibition for their relevance to safety and propose areas for future innovation. A principled concept of Pavlovian safety learning may reduce methodological inconsistencies, stimulate translational research, and facilitate a comprehensive understanding of an indispensable psychological construct.

Prediction-error signals in anterior cingulate cortex drive task-switching

Task-switching is a fundamental cognitive ability that allows animals to update their knowledge of current rules or contexts. Detecting discrepancies between predicted and observed events is essential for this process. However, little is known about how the brain computes cognitive prediction-errors and whether neural prediction-error signals are causally related to task-switching behaviours. Here we trained mice to use a prediction-error to switch, in a single trial, between responding to the same stimuli using two distinct rules. Optogenetic silencing and un-silencing, together with widefield and two-photon calcium imaging revealed that the anterior cingulate cortex (ACC) was specifically required for this rapid task-switching, but only when it exhibited neural prediction-error signals. These prediction-error signals were projection-target dependent and were larger preceding successful behavioural transitions. An all-optical approach revealed a disinhibitory interneuron circuit required for successful prediction-error computation. These results reveal a circuit mechanism for computing prediction-errors and transitioning between distinct cognitive states.

Predictive sequence learning in the hippocampal formation

The hippocampus receives sequences of sensory inputs from the cortex during exploration and encodes the sequences with millisecond precision. We developed a predictive autoencoder model of the hippocampus including the trisynaptic and monosynaptic circuits from the entorhinal cortex (EC). CA3 was trained as a self-supervised recurrent neural network to predict its next input. We confirmed that CA3 is predicting ahead by analyzing the spike coupling between simultaneously recorded neurons in the dentate gyrus, CA3, and CA1 of the mouse hippocampus. In the model, CA1 neurons signal prediction errors by comparing CA3 predictions to the next direct EC input. The model exhibits the rapid appearance and slow fading of CA1 place cells and displays replay and phase precession from CA3. The model could be learned in a biologically plausible way with error-encoding neurons. Similarities between the hippocampal and thalamocortical circuits suggest that such computation motif could also underlie self-supervised sequence learning in the cortex.

Model of a striatal circuit exploring biological mechanisms underlying decision-making during normal and disordered states

Decision-making requires continuous adaptation to internal and external contexts. Changes in decision-making are reliable transdiagnostic symptoms of neuropsychiatric disorders. We created a computational model demonstrating how the striosome compartment of the striatum constructs a mathematical space for decision-making computations depending on context, and how the matrix compartment defines action value depending on the space. The model explains multiple experimental results and unifies other theories like reward prediction error, roles of the direct versus indirect pathways, and roles of the striosome versus matrix, under one framework. We also found, through new analyses, that striosome and matrix neurons increase their synchrony during difficult tasks, caused by a necessary increase in dimensionality of the space. The model makes testable predictions about individual differences in disorder susceptibility, decision-making symptoms shared among neuropsychiatric disorders, and differences in neuropsychiatric disorder symptom presentation. The model reframes the role of the striosomal circuit in neuroeconomic and disorder-affected decision-making.

Highlights

Striosomes prioritize decision-related data used by matrix to set action values. Striosomes and matrix have different roles in the direct and indirect pathways. Abnormal information organization/valuation alters disorder presentation. Variance in data prioritization may explain individual differences in disorders.

eTOC

Beck et al. developed a computational model of how a striatal circuit functions during decision-making. The model unifies and extends theories about the direct versus indirect pathways. It further suggests how aberrant circuit function underlies decision-making phenomena observed in neuropsychiatric disorders.

Inductive biases of neural network modularity in spatial navigation

The brain may have evolved a modular architecture for daily tasks, with circuits featuring functionally specialized modules that match the task structure. We hypothesize that this architecture enables better learning and generalization than architectures with less specialized modules. To test this, we trained reinforcement learning agents with various neural architectures on a naturalistic navigation task. We found that the modular agent, with an architecture that segregates computations of state representation, value, and action into specialized modules, achieved better learning and generalization. Its learned state representation combines prediction and observation, weighted by their relative uncertainty, akin to recursive Bayesian estimation. This agent's behavior also resembles macaques' behavior more closely. Our results shed light on the possible rationale for the brain's modularity and suggest that artificial systems can use this insight from neuroscience to improve learning and generalization in natural tasks.

Natural phasic inhibition of dopamine neurons signals cognitive rigidity

When animals unexpectedly fail, their dopamine neurons undergo phasic inhibition that canonically drives extinction learning-a cognitive-flexibility mechanism for discarding outdated strategies. However, the existing evidence equates natural and artificial phasic inhibition, despite their spatiotemporal differences. Addressing this gap, we targeted a GABAA-receptor antagonist precisely to dopamine neurons, yielding three unexpected findings. First, this intervention blocked natural phasic inhibition selectively, leaving tonic activity unaffected. Second, blocking natural phasic inhibition accelerated extinction learning-opposite to canonical mechanisms. Third, our approach selectively benefitted perseverative mice, restoring rapid extinction without affecting new reward learning. Our findings reveal that extinction learning is rapid by default and slowed by natural phasic inhibition-challenging foundational learning theories, while delineating a synaptic mechanism and therapeutic target for cognitive rigidity.

Reinforcement Learning May Demystify the Limited Human Motor Learning Efficacy Due to Visual-Proprioceptive Mismatch

Vision and proprioception have fundamental sensory mismatches in delivering locational information, and such mismatches are critical factors limiting the efficacy of motor learning. However, it is still not clear how and to what extent this mismatch limits motor learning outcomes. To further the understanding of the effect of sensory mismatch on motor learning outcomes, a reinforcement learning algorithm and the simplified biomechanical elbow joint model were employed to mimic the motor learning process in a computational environment. By applying a reinforcement learning algorithm to the motor learning of elbow joint flexion task, simulation results successfully explained how visual-proprioceptive mismatch limits motor learning outcomes in terms of motor control accuracy and task completion speed. The larger the perceived angular offset between the two sensory modalities, the lower the motor control accuracy. Also, the more similar the peak reward amplitude of the two sensory modalities, the lower the motor control accuracy. In addition, simulation results suggest that insufficient exploration rate limits task completion speed, and excessive exploration rate limits motor control accuracy. Such a speed-accuracy trade-off shows that a moderate exploration rate could serve as another important factor in motor learning.

Dorsal raphe neurons integrate the values of reward amount, delay, and uncertainty in multi-attribute decision-making

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when confronting reward uncertainty. However, it has been unclear whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider these attributes to make a choice. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes, and this population tended to integrate the attributes in a manner that reflected monkeys' preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how the DRN participates in value computations, guiding theories about the role of the DRN in decision-making and psychiatric disease.

Salience signaling and stimulus scaling of ventral tegmental area glutamate neuron subtypes

Ventral tegmental area (VTA) glutamatergic neurons participate in reward, aversion, drug-seeking, and stress. Subsets of VTA VGluT2+ neurons are capable of co-transmitting glutamate and GABA (VGluT2+VGaT+ neurons), transmitting glutamate without GABA (VGluT2+VGaT- neurons), or co-transmitting glutamate and dopamine (VGluT2+TH+ neurons), but whether these molecularly distinct subpopulations show behavior-related differences is not wholly understood. We identified that neuronal activity of each VGluT2+ subpopulation is sensitive to reward value but signaled this in different ways. The phasic maximum activity of VGluT2+VGaT+ neurons increased with sucrose concentration, whereas VGluT2+VGaT- neurons increased maximum and sustained activity with sucrose concentration, and VGluT2+TH+ neurons increased sustained but not maximum activity with sucrose concentration. Additionally, VGluT2+ subpopulations signaled consummatory preferences in different ways. VGluT2+VGaT- neurons and VGluT2+TH+ neurons showed a signaling preference for a behaviorally-preferred fat reward over sucrose, but in temporally-distinct ways. In contrast, VGluT2+VGaT+ neurons uniquely signaled a less behaviorally-preferred sucrose reward compared with fat. Further experiments suggested that VGluT2+VGaT+ consummatory reward-related activity was related to sweetness, partially modulated by hunger state, and not dependent on caloric content or behavioral preference. All VGluT2+ subtypes increased neuronal activity following aversive stimuli but VGluT2+VGaT+ neurons uniquely scaled their magnitude and sustained activity with footshock intensity. Optogenetic activation of VGluT2+VGaT+ neurons during low intensity footshock enhanced fear-related behavior without inducing place preference or aversion. We interpret these data such that VTA glutamatergic subpopulations signal different elements of rewarding and aversive experiences and highlight the unique role of VTA VGluT2+VGaT+ neurons in enhancing the salience of behavioral experiences.

Loss of glycine receptors in the nucleus accumbens and ethanol reward in an Alzheimer´s Disease mouse model

Alterations in cognitive and non-cognitive cerebral functions characterize Alzheimer's disease (AD). Cortical and hippocampal impairments related to extracellular accumulation of Aβ in AD animal models have been extensively investigated. However, recent reports have also implicated intracellular Aβ in limbic regions, such as the nucleus accumbens (nAc). Accumbal neurons express high levels of inhibitory glycine receptors (GlyRs) that are allosterically modulated by ethanol and have a role in controlling its intake. In the present study, we investigated how GlyRs in the 2xTg mice (AD model) affect nAc functions and ethanol intake behavior. Using transgenic and control aged-matched litter mates, we found that the GlyRα2 subunit was significantly decreased in AD mice (6-month-old). We also examined intracellular calcium dynamics using the fluorescent calcium protein reporter GCaMP in slice photometry. We also found that the calcium signal mediated by GlyRs, but not GABAAR, was also reduced in AD neurons. Additionally, ethanol potentiation was significantly decreased in accumbal neurons in the AD mice. Finally, we performed drinking in the dark (DID) experiments and found that 2xTg mice consumed less ethanol on the last day of DID, in agreement with a lower blood ethanol concentration. 2xTg mice also showed lower sucrose consumption, indicating that overall food reward was altered. In conclusion, the data support the role of GlyRs in nAc neuron excitability and a decreased glycinergic activity in the 2xTg mice that might lead to impairment in reward processing at an early stage of the disease.

Human Substantia Nigra Neurons Encode Reward Expectations

Dopamine (DA) signals originating from substantia nigra (SN) neurons are centrally involved in the regulation of motor and reward processing. DA signals behaviorally relevant events where reward outcomes differ from expectations (reward prediction errors, RPEs). RPEs play a crucial role in learning optimal courses of action and in determining response vigor when an agent expects rewards. Nevertheless, how reward expectations, crucial for RPE calculations, are conveyed to and represented in the dopaminergic system is not fully understood, especially in the human brain where the activity of DA neurons is difficult to study. One possibility, suggested by evidence from animal models, is that DA neurons explicitly encode reward expectations. Alternatively, they may receive RPE information directly from upstream brain regions. To address whether SN neuron activity directly reflects reward expectation information, we directly examined the encoding of reward expectation signals in human putative DA neurons by performing single-unit recordings from the SN of patients undergoing neurosurgery. Patients played a two-armed bandit decision-making task in which they attempted to maximize reward. We show that neuronal firing rates (FR) of putative DA neurons during the reward expectation period explicitly encode reward expectations. First, activity in these neurons was modulated by previous trial outcomes, such that FR were greater after positive outcomes than after neutral or negative outcome trials. Second, this increase in FR was associated with shorter reaction times, consistent with an invigorating effect of DA neuron activity during expectation. These results suggest that human DA neurons explicitly encode reward expectations, providing a neurophysiological substrate for a signal critical for reward learning.

Machine learning reveals differential effects of depression and anxiety on reward and punishment processing

Recent studies suggest that depression and anxiety are associated with unique aspects of EEG responses to reward and punishment, respectively; also, abnormal responses to punishment in depressed individuals are related to anxiety, the symptoms of which are comorbid with depression. In a non-clinical sample, we aimed to investigate the relationships between reward processing and anxiety, between punishment processing and anxiety, between reward processing and depression, and between punishment processing and depression. Towards this aim, we separated feedback-related brain activity into delta and theta bands to isolate activity that indexes functionally distinct processes. Based on the delta/theta frequency and feedback valence, we then used machine learning (ML) to classify individuals with high severity of depressive symptoms and individuals with high severity of anxiety symptoms versus controls. The significant difference between the depression and control groups was driven mainly by delta activity; there were no differences between reward- and punishment-theta activities. The high severity of anxiety symptoms was marginally more strongly associated with the punishment- than the reward-theta feedback processing. The findings provide new insights into the differences in the impacts of anxiety and depression on reward and punishment processing; our study shows the utility of ML in testing brain-behavior hypotheses and emphasizes the joint effect of theta-RewP/FRN and delta frequency on feedback-related brain activity.

Striatal dopamine signals reflect perceived cue-action-outcome associations in mice

Striatal dopamine drives associative learning by acting as a teaching signal. Much work has focused on simple learning paradigms, including Pavlovian and instrumental learning. However, higher cognition requires that animals generate internal concepts of their environment, where sensory stimuli, actions and outcomes become flexibly associated. Here, we performed fiber photometry dopamine measurements across the striatum of male mice as they learned cue-action-outcome associations based on implicit and changing task rules. Reinforcement learning models of the behavioral and dopamine data showed that rule changes lead to adjustments of learned cue-action-outcome associations. After rule changes, mice discarded learned associations and reset outcome expectations. Cue- and outcome-triggered dopamine signals became uncoupled and dependent on the adopted behavioral strategy. As mice learned the new association, coupling between cue- and outcome-triggered dopamine signals and task performance re-emerged. Our results suggest that dopaminergic reward prediction errors reflect an agent's perceived locus of control.

Brain state-dependent responses of midbrain dopaminergic neurons to footshock under urethane anaesthesia

For a long time, it has been assumed that dopaminergic (DA) neurons in both the ventral tegmental area (VTA) and the substantia nigra pars compacta (SNc) uniformly respond to rewarding and aversive stimuli by either increasing or decreasing their activity, respectively. This response was believed to signal information about the perceived stimuli's values. The identification of VTA&SNc DA neurons that are excited by both rewarding and aversive stimuli has led to the categorisation of VTA&SNc DA neurons into two subpopulations: one signalling the value and the other signalling the salience of the stimuli. It has been shown that the general state of the brain can modulate the electrical activity of VTA&SNc DA neurons, but it remains unknown whether this factor may also influence responses to aversive stimuli, such as a footshock (FS). To address this question, we have recorded the responses of VTA&SNc DA neurons to FSs across cortical activation and slow wave activity brain states in urethane-anaesthetised rats. Adding to the knowledge of aversion signalling by midbrain DA neurons, we report that significant proportion of VTA&SNc DA neurons can change their responses to an aversive stimulus in a brain state-dependent manner. The majority of these neurons decreased their activity in response to FS during cortical activation but switched to increasing it during slow wave activity. It can be hypothesised that this subpopulation of DA neurons may be involved in the 'dual signalling' of both the value and the salience of the stimuli, depending on the general state of the brain.

Separating cognitive and motor processes in the behaving mouse

The cognitive processes supporting complex animal behavior are closely associated with ubiquitous movements responsible for our posture, facial expressions, ability to actively sample our sensory environments, and other critical processes. These movements are strongly related to neural activity across much of the brain and are often highly correlated with ongoing cognitive processes, making it challenging to dissociate the neural dynamics that support cognitive processes from those supporting related movements. In such cases, a critical issue is whether cognitive processes are separable from related movements, or if they are driven by common neural mechanisms. Here, we demonstrate how the separability of cognitive and motor processes can be assessed, and, when separable, how the neural dynamics associated with each component can be isolated. We establish a novel two-context behavioral task in mice that involves multiple cognitive processes and show that commonly observed dynamics taken to support cognitive processes are strongly contaminated by movements. When cognitive and motor components are isolated using a novel approach for subspace decomposition, we find that they exhibit distinct dynamical trajectories. Further, properly accounting for movement revealed that largely separate populations of cells encode cognitive and motor variables, in contrast to the 'mixed selectivity' often reported. Accurately isolating the dynamics associated with particular cognitive and motor processes will be essential for developing conceptual and computational models of neural circuit function and evaluating the function of the cell types of which neural circuits are composed.

Experimental biology can inform our understanding of food insecurity

Food insecurity is a major public health issue. Millions of households worldwide have intermittent and unpredictable access to food and this experience is associated with greater risk for a host of negative health outcomes. While food insecurity is a contemporary concern, we can understand its effects better if we acknowledge that there are ancient biological programs that evolved to respond to the experience of food scarcity and uncertainty, and they may be particularly sensitive to food insecurity during development. Support for this conjecture comes from common findings in several recent animal studies that have modeled insecurity by manipulating predictability of food access in various ways. Using different experimental paradigms in different species, these studies have shown that experience of insecure access to food can lead to changes in weight, motivation and cognition. Some of these studies account for changes in weight through changes in metabolism, while others observe increases in feeding and motivation to work for food. It has been proposed that weight gain is an adaptive response to the experience of food insecurity as 'insurance' in an uncertain future, while changes in motivation and cognition may reflect strategic adjustments in foraging behavior. Animal studies also offer the opportunity to make in-depth controlled studies of mechanisms and behavior. So far, there is evidence that the experience of food insecurity can impact metabolic efficiency, reproductive capacity and dopamine neuron synapses. Further work on behavior, the central and peripheral nervous system, the gut and liver, along with variation in age of exposure, will be needed to better understand the full body impacts of food insecurity at different stages of development.

Neural Response to Expecting a Caloric Sweet Taste Stimulus Predicts Body Mass Index Longitudinally Among Young Adult Women With Anorexia Nervosa

BACKGROUND

Anorexia nervosa (AN) is an often-chronic illness, and we lack biomarkers to predict long-term outcome. Recent neuroimaging studies using caloric taste stimuli suggest that paradigms that have tested conditioned neural responses to expectation or salient stimulus receipt may underpin behaviors. However, whether activation of those neural circuits can predict long-term outcome has not been studied.

METHODS

We followed women treated for AN (n = 35, mean age [SD] = 23 [7] years) and tested whether functional imaging brain response during a taste conditioning paradigm could predict posttreatment body mass index (BMI). We anticipated greater neural activity relative to caloric stimulus expectation and that dopamine-related receipt conditions would predict lower posttreatment BMI, indicating fear-associated arousal.

RESULTS

Follow-up occurred at mean (SD) = 1648 (1216) days after imaging. Stimulus expectation in orbitofrontal and striatal regions and BMI and BMI change at follow-up were negatively correlated, and these correlations remained significant for the right superior orbitofrontal cortex and BMI change after multiple comparison correction (r = -0.484, p = .003). This relationship remained significant after including time between brain scanning and follow-up in the model. Reward prediction error response did not predict long-term BMI.

CONCLUSIONS

The orbitofrontal cortex is involved in learning and conditioning, and these data implicate this region in learned caloric stimulus expectation and long-term prediction of weight outcomes in AN. Thus, conditioned elevated brain response to the anticipation of receiving a caloric stimulus may drive food avoidance, suggesting that breaking such associations is central for long-term recovery from AN.

Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks

Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.

State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways

The nervous system uses fast- and slow-adapting sensory detectors in parallel to enable neuronal representations of external states and their temporal dynamics. It is unknown whether this dichotomy also applies to internal representations that have no direct correlation in the physical world. Here we find that two distinct dopamine (DA) neuron subtypes encode either a state or its rate-of-change. In mice performing a reward-seeking task, we found that the animal's behavioral state and rate-of-change were encoded by the sustained activity of DA neurons in medial ventral tegmental area (VTA) DA neurons and transient activity in lateral VTA DA neurons, respectively. The neural activity patterns of VTA DA cell bodies matched DA release patterns within anatomically defined mesoaccumbal pathways. Based on these results, we propose a model in which the DA system uses two parallel lines for proportional-differential encoding of a state variable and its temporal dynamics.

Ventral pallidum neurons projecting to the ventral tegmental area reinforce but do not invigorate reward-seeking behavior

Reward-predictive cues acquire motivating and reinforcing properties that contribute to the escalation and relapse of drug use in addiction. The ventral pallidum (VP) and ventral tegmental area (VTA) are two key nodes in brain reward circuitry implicated in addiction and cue-driven behavior. In the current study, we use in vivo fiber photometry and optogenetics to record from and manipulate VP→VTA in rats performing a discriminative stimulus task to determine the role these neurons play in invigoration and reinforcement by reward cues. We find that VP→VTA neurons are active during reward consumption and that optogenetic stimulation of these neurons biases choice behavior and is reinforcing. Critically, we find no encoding of reward-seeking vigor, and optogenetic stimulation does not enhance the probability or vigor of reward seeking in response to cues. Our results suggest that VP→VTA activity is more important for reinforcement than for invigoration of reward seeking by cues.

Distinct processing of the state prediction error signals in frontal and parietal correlates in learning the environment model

Goal-directed reinforcement learning constructs a model of how the states in the environment are connected and prospectively evaluates action values by simulating experience. State prediction error (SPE) is theorized as a crucial signal for learning the environment model. However, the underlying neural mechanisms remain unclear. Here, using electroencephalogram, we verified in a two-stage Markov task two neural correlates of SPEs: an early negative correlate transferring from frontal to central electrodes and a late positive correlate over parietal regions. Furthermore, by investigating the effects of explicit knowledge about the environment model and rewards in the environment, we found that, for the parietal correlate, rewards enhanced the representation efficiency (beta values of regression coefficient) of SPEs, whereas explicit knowledge elicited a larger SPE representation (event-related potential activity) for rare transitions. However, for the frontal and central correlates, rewards increased activities in a content-independent way and explicit knowledge enhanced activities only for common transitions. Our results suggest that the parietal correlate of SPEs is responsible for the explicit learning of state transition structure, whereas the frontal and central correlates may be involved in cognitive control. Our study provides novel evidence for distinct roles of the frontal and the parietal cortices in processing SPEs.

Sex differences in the impact of social status on social reward and associated mesolimbic activation

Social stress plays an important role in the etiology of many neuropsychiatric disorders and can lead to a variety of behavioral deficits such as social withdrawal. One way that social stress may contribute to psychiatric disorders is by reducing social motivation and the rewarding properties of social interactions. We investigated the impact of social stress on social reward in the context of winning versus losing agonistic encounters in Syrian hamsters (Mesocricetus auratus). First, we tested the hypothesis that social stress resulting from either stable low, or subordinate, social status or from social defeat reduces the rewarding properties of social interactions. Using an Operant Social Preference (OSP) task to measure social reward/motivation, we found that both subordinate and socially defeated males made significantly fewer entries into chambers containing novel, same-sex conspecifics compared to males who were dominant (i.e., stably won the agonistic encounters). In females, however, there were no differences in social entries between winners and losers. In a second experiment, we found more activation of the mesolimbic dopamine system (MDS) as assessed with cFos immunohistochemistry in the lateral ventral tegmental area (lVTA) and the nucleus accumbens (NAc) shell of male winners compared to losers. In females, however, there were no differences in activation in the lVTA between winners and losers. Surprisingly, however, winning females displayed significantly more activation in the NAc shell as compared to losing females, despite the lack of behavioral differences. Thus, behavioral and histological data suggest that there are sex differences in the impact of social status on social reward and associated mesolimbic activation.

Inhibitory Pedunculopontine Neurons Gate Dopamine-Mediated Motor Actions of Unsigned Valence

BACKGROUND

The pedunculopontine nucleus (PPN) maintains a bidirectional connectivity with the basal ganglia that supports their shared roles in the selection and execution of motor actions. Previous studies identified a role for PPN neurons in goal-directed behavior, but the cellular substrates underlying this function have not been elucidated. We recently revealed the existence of a monosynaptic GABAergic input from the PPN that inhibits dopamine neurons of the substantia nigra. Activation of this pathway interferes with the execution of learned motor sequences when the actions are rewarded, even though the inhibition of dopamine neurons did not shift the value of the action, hence suggesting executive control over the gating of behavior.

OBJECTIVE

To test the attributes of the inhibition of dopamine neurons by the PPN in the context of goal-directed behavior regardless of whether the outcome is positively or negatively reinforced.

METHODS

We delivered optogenetic stimulation to PPN GABAergic axon terminals in the substantia nigra during a battery of behavioral tasks with positive and negative valence.

RESULTS

Inhibition of dopamine neurons by PPN optogenetic activation during an appetitive task impaired the initiation and overall execution of the behavioral sequence without affecting the consumption of reward. During an active avoidance task, the same activation impaired the ability of mice to avoid a foot shock, but their escape response was unaffected. In addition, responses to potential threats were significantly attenuated.

CONCLUSION

Our results show that PPN GABAergic neurons modulate learned, goal-directed behavior of unsigned valence without affecting overall motor behavior.

Distinct spatiotemporal brainstem pathways of outcome valence during reward- and punishment-based learning

Learning to seek rewards and avoid punishments, based on positive and negative choice outcomes, is essential for human survival. Yet, the neural underpinnings of outcome valence in the human brainstem and the extent to which they differ in reward and punishment learning contexts remain largely elusive. Here, using simultaneously acquired electroencephalography and functional magnetic resonance imaging data, we show that during reward learning the substantia nigra (SN)/ventral tegmental area (VTA) and locus coeruleus are initially activated following negative outcomes, while the VTA subsequently re-engages exhibiting greater responses for positive than negative outcomes, consistent with an early arousal/avoidance response and a later value-updating process, respectively. During punishment learning, we show that distinct raphe nucleus and SN subregions are activated only by negative outcomes with a sustained post-outcome activity across time, supporting the involvement of these brainstem subregions in avoidance behavior. Finally, we demonstrate that the coupling of these brainstem structures with other subcortical and cortical areas helps to shape participants' serial choice behavior in each context.

Asymmetric coding of reward prediction errors in human insula and dorsomedial prefrontal cortex

The signed value and unsigned salience of reward prediction errors (RPEs) are critical to understanding reinforcement learning (RL) and cognitive control. Dorsomedial prefrontal cortex (dMPFC) and insula (INS) are key regions for integrating reward and surprise information, but conflicting evidence for both signed and unsigned activity has led to multiple proposals for the nature of RPE representations in these brain areas. Recently developed RL models allow neurons to respond differently to positive and negative RPEs. Here, we use intracranially recorded high frequency activity (HFA) to test whether this flexible asymmetric coding strategy captures RPE coding diversity in human INS and dMPFC. At the region level, we found a bias towards positive RPEs in both areas which paralleled behavioral adaptation. At the local level, we found spatially interleaved neural populations responding to unsigned RPE salience and valence-specific positive and negative RPEs. Furthermore, directional connectivity estimates revealed a leading role of INS in communicating positive and unsigned RPEs to dMPFC. These findings support asymmetric coding across distinct but intermingled neural populations as a core principle of RPE processing and inform theories of the role of dMPFC and INS in RL and cognitive control.

Reduced neural encoding of utility prediction errors in cocaine addiction

Influential accounts of addiction posit alterations in adaptive behavior driven by deficient dopaminergic prediction errors (PEs), signaling the discrepancy between actual and expected reward. Dopamine neurons encode these error signals in subjective terms, calibrated by individual risk preferences, as "utility" PEs. It remains unclear, however, whether people with drug addiction have PE deficits or their computational source. Here, using an analogous task to prior single-unit studies with known expectancies, we show that fMRI-measured PEs similarly reflect utility PEs. Relative to control participants, people with chronic cocaine addiction demonstrate reduced utility PEs in the dopaminoceptive ventral striatum, with similar trends in orbitofrontal cortex. Dissecting this PE signal into its subcomponent terms attributed these reductions to weaker striatal responses to received reward/utility, whereas suppression of activity with reward expectation was unchanged. These findings support that addiction may fundamentally disrupt PE signaling and reveal an underappreciated role for perceived reward value in this mechanism.

Stress relief as a natural resilience mechanism against depression-like behaviors

Relief, the appetitive state after the termination of aversive stimuli, is evolutionarily conserved. Understanding the behavioral role of this well-conserved phenomenon and its underlying neurobiological mechanisms are open and important questions. Here, we discover that the magnitude of relief from physical stress strongly correlates with individual resilience to depression-like behaviors in chronic stressed mice. Notably, blocking stress relief causes vulnerability to depression-like behaviors, whereas natural rewards supplied shortly after stress promotes resilience. Stress relief is mediated by reward-related mesolimbic dopamine neurons, which show minute-long, persistent activation after stress termination. Circuitry-wise, activation or inhibition of circuits downstream of the ventral tegmental area during the transient relief period bi-directionally regulates depression resilience. These results reveal an evolutionary function of stress relief in depression resilience and identify the neural substrate mediating this effect. Importantly, our data suggest a behavioral strategy of augmenting positive valence of stress relief with natural rewards to prevent depression.

Altered neural activity in the mesoaccumbens pathway underlies impaired social reward processing in Shank3-deficient rats

Social behaviors are crucial for human connection and belonging, often impacted in conditions like Autism Spectrum Disorder (ASD). The mesoaccumbens pathway (VTA and NAc) plays a pivotal role in social behavior and is implicated in ASD. However, the impact of ASD-related mutations on social reward processing remains insufficiently explored. This study focuses on the Shank3 mutation, associated with a rare genetic condition and linked to ASD, examining its influence on the mesoaccumbens pathway during behavior, using the Shank3-deficient rat model. Our findings indicate that Shank3-deficient rats exhibit atypical social interactions and have difficulty adjusting behavior based on reward values, associated with modified neuronal activity of VTA dopaminergic and GABAergic neurons and reduced dopamine release in the NAc. Moreover, we demonstrate that manipulating VTA neuronal activity can normalize this behavior, providing insights into the effects of Shank3 mutations on social reward and behavior, and identify a potential neural pathway for intervention.

A nigroincertal projection mediates aversion and enhances coping responses to potential threat

Recent studies have shown that the non-DA neurons in the ventral tegmental area (VTA) and substantia nigra (SN) not only modulate motivational behaviors but also regulate defensive behaviors. While zona incerta (ZI) is a threat-responsive substrate and receives innervations from the ventral midbrain, the function of the ventral midbrain-to-ZI connection remains poorly defined. Here, we demonstrate that the ZI receives heterogenous innervations from the ventral midbrain. By utilizing a retrograde AAV preferentially labeling non-DA neurons in the ventral midbrain, we found that ZI-projecting non-DA cells in the ventral midbrain are activated by restraint stress. We focused on the SN and found that SN-to-ZI GABAergic input is engaged by a predatory odor. Sustained pan-neuronal SN-to-ZI activation results in aversion and enhances defensive behaviors, likely through a disinhibition mechanism to recruit downstream brain regions that regulate defensive behaviors. Collectively, our results reveal a novel role of nigroincertal projection in mediating negative valence and regulating defensive behaviors.

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

In the mammalian brain, midbrain dopamine neuron activity is hypothesized to encode reward prediction errors that promote learning and guide behavior by causing rapid changes in dopamine levels in target brain regions. This hypothesis (and alternatives regarding dopamine's role in punishment-learning) has limited direct evidence in humans. We report intracranial, subsecond measurements of dopamine release in human striatum measured, while volunteers (i.e., patients undergoing deep brain stimulation surgery) performed a probabilistic reward and punishment learning choice task designed to test whether dopamine release encodes only reward prediction errors or whether dopamine release may also encode adaptive punishment learning signals. Results demonstrate that extracellular dopamine levels can encode both reward and punishment prediction errors within distinct time intervals via independent valence-specific pathways in the human brain.

Dopamine receptor activation regulates reward expectancy signals during cognitive control in primate prefrontal neurons

Dopamine neurons respond to reward-predicting cues but also modulate information processing in the prefrontal cortex essential for cognitive control. Whether dopamine controls reward expectation signals in prefrontal cortex that motivate cognitive control is unknown. We trained two male macaques on a working memory task while varying the reward size earned for successful task completion. We recorded neurons in lateral prefrontal cortex while simultaneously stimulating dopamine D1 receptor (D1R) or D2 receptor (D2R) families using micro-iontophoresis. We show that many neurons predict reward size throughout the trial. D1R stimulation showed mixed effects following reward cues but decreased reward expectancy coding during the memory delay. By contrast, D2R stimulation increased reward expectancy coding in multiple task periods, including cueing and memory periods. Stimulation of either dopamine receptors increased the neurons' selective responses to reward size upon reward delivery. The differential modulation of reward expectancy by dopamine receptors suggests that dopamine regulates reward expectancy necessary for successful cognitive control.

Remembering unexpected beauty: Contributions of the ventral striatum to the processing of reward prediction errors regarding the facial attractiveness in face memory

The COVID-19 pandemic has led people to predict facial attractiveness from partially covered faces. Differences in the predicted and observed facial attractiveness (i.e., masked and unmasked faces, respectively) are defined as reward prediction error (RPE) in a social context. Cognitive neuroscience studies have elucidated the neural mechanisms underlying RPE-induced memory improvements in terms of monetary rewards. However, little is known about the mechanisms underlying RPE-induced memory modulation in terms of social rewards. To elucidate this, the present functional magnetic resonance imaging (fMRI) study investigated activity and functional connectivity during face encoding. In encoding trials, participants rated the predicted attractiveness of faces covered except for around the eyes (prediction phase) and then rated the observed attractiveness of these faces without any cover (outcome phase). The difference in ratings between these phases was defined as RPE in facial attractiveness, and RPE was categorized into positive RPE (increased RPE from the prediction to outcome phases), negative RPE (decreased RPE from the prediction to outcome phases), and non-RPE (no difference in RPE between the prediction and outcome phases). During retrieval, participants were presented with individual faces that had been seen and unseen in the encoding trials, and were required to judge whether or not each face had been seen in the encoding trials. Univariate activity in the ventral striatum (VS) exhibited a linear increase with increased RPE in facial attractiveness. In the multivariate pattern analysis (MVPA), activity patterns in the VS and surrounding areas (extended VS) significantly discriminated between positive/negative RPE and non-RPE. In the functional connectivity analysis, significant functional connectivity between the extended VS and the hippocampus was observed most frequently in positive RPE. Memory improvements by face-based RPE could be involved in functional networks between the extended VS (representing RPE) and the hippocampus, and the interaction could be modulated by RPE values in a social context.

Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system

The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.

Contexts facilitate dynamic value encoding in the mesolimbic dopamine system

Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.

The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate

Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.

Dopaminergic neurons dynamically update sensory values during olfactory maneuver

Dopaminergic neurons (DANs) drive associative learning to update the value of sensory cues, but their contribution to the assessment of sensory values outside the context of association remains largely unexplored. Here, we show in Drosophila that DANs in the mushroom body encode the innate value of odors and constantly update the current value by inducing plasticity during olfactory maneuver. Our connectome-based network model linking all the way from the olfactory neurons to DANs reproduces the characteristics of DAN responses, proposing a concrete circuit mechanism for computation. Downstream of DANs, odors alone induce value- and dopamine-dependent changes in the activity of mushroom body output neurons, which store the current value of odors. Consistent with this neural plasticity, specific sets of DANs bidirectionally modulate flies' steering in a virtual olfactory environment. Thus, the DAN circuit known for discrete, associative learning also continuously updates odor values in a nonassociative manner.

Biologically plausible local synaptic learning rules robustly implement deep supervised learning

In deep neural networks, representational learning in the middle layer is essential for achieving efficient learning. However, the currently prevailing backpropagation learning rules (BP) are not necessarily biologically plausible and cannot be implemented in the brain in their current form. Therefore, to elucidate the learning rules used by the brain, it is critical to establish biologically plausible learning rules for practical memory tasks. For example, learning rules that result in a learning performance worse than that of animals observed in experimental studies may not be computations used in real brains and should be ruled out. Using numerical simulations, we developed biologically plausible learning rules to solve a task that replicates a laboratory experiment where mice learned to predict the correct reward amount. Although the extreme learning machine (ELM) and weight perturbation (WP) learning rules performed worse than the mice, the feedback alignment (FA) rule achieved a performance equal to that of BP. To obtain a more biologically plausible model, we developed a variant of FA, FA_Ex-100%, which implements direct dopamine inputs that provide error signals locally in the layer of focus, as found in the mouse entorhinal cortex. The performance of FA_Ex-100% was comparable to that of conventional BP. Finally, we tested whether FA_Ex-100% was robust against rule perturbations and biologically inevitable noise. FA_Ex-100% worked even when subjected to perturbations, presumably because it could calibrate the correct prediction error (e.g., dopaminergic signals) in the next step as a teaching signal if the perturbation created a deviation. These results suggest that simplified and biologically plausible learning rules, such as FA_Ex-100%, can robustly facilitate deep supervised learning when the error signal, possibly conveyed by dopaminergic neurons, is accurate.

Dopamine release and negative valence gated by inhibitory neurons in the laterodorsal tegmental nucleus

GABAergic neurons in the laterodorsal tegmental nucleus (LDTGABA) encode aversion by directly inhibiting mesolimbic dopamine (DA). Yet, the detailed cellular and circuit mechanisms by which these cells relay unpleasant stimuli to DA neurons and regulate behavioral output remain largely unclear. Here, we show that LDTGABA neurons bidirectionally respond to rewarding and aversive stimuli in mice. Activation of LDTGABA neurons promotes aversion and reduces DA release in the lateral nucleus accumbens. Furthermore, we identified two molecularly distinct LDTGABA cell populations. Somatostatin-expressing (Sst+) LDTGABA neurons indirectly regulate the mesolimbic DA system by disinhibiting excitatory hypothalamic neurons. In contrast, Reelin-expressing LDTGABA neurons directly inhibit downstream DA neurons. The identification of separate GABAergic subpopulations in a single brainstem nucleus that relay unpleasant stimuli to the mesolimbic DA system through direct and indirect projections is critical for establishing a circuit-level understanding of how negative valence is encoded in the mammalian brain.

Dorsal raphe neurons signal integrated value during multi-attribute decision-making

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when grappling with reward uncertainty. However, whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider all these attributes to make a choice, is unclear. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes. Remarkably, these neurons commonly integrated offer attributes in a manner that reflected monkeys' overall preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how DRN participates in integrated value computations, guiding theories of DRN in decision-making and psychiatric disease.

Neural Networks for Navigation: From Connections to Computations

Many animals can navigate toward a goal they cannot see based on an internal representation of that goal in the brain's spatial maps. These maps are organized around networks with stable fixed-point dynamics (attractors), anchored to landmarks, and reciprocally connected to motor control. This review summarizes recent progress in understanding these networks, focusing on studies in arthropods. One factor driving recent progress is the availability of the Drosophila connectome; however, it is increasingly clear that navigation depends on ongoing synaptic plasticity in these networks. Functional synapses appear to be continually reselected from the set of anatomical potential synapses based on the interaction of Hebbian learning rules, sensory feedback, attractor dynamics, and neuromodulation. This can explain how the brain's maps of space are rapidly updated; it may also explain how the brain can initialize goals as stable fixed points for navigation.

Suicide prevention and ketamine: insights from computational modeling

Suicide is a pressing public health issue, with over 700,000 individuals dying each year. Ketamine has emerged as a promising treatment for suicidal thoughts and behaviors (STBs), yet the complex mechanisms underlying ketamine's anti-suicidal effect are not fully understood. Computational psychiatry provides a promising framework for exploring the dynamic interactions underlying suicidality and ketamine's therapeutic action, offering insight into potential biomarkers, treatment targets, and the underlying mechanisms of both. This paper provides an overview of current computational theories of suicidality and ketamine's mechanism of action, and discusses various computational modeling approaches that attempt to explain ketamine's anti-suicidal effect. More specifically, the therapeutic potential of ketamine is explored in the context of the mismatch negativity and the predictive coding framework, by considering neurocircuits involved in learning and decision-making, and investigating altered connectivity strengths and receptor densities targeted by ketamine. Theory-driven computational models offer a promising approach to integrate existing knowledge of suicidality and ketamine, and for the extraction of model-derived mechanistic parameters that can be used to identify patient subgroups and personalized treatment approaches. Future computational studies on ketamine's mechanism of action should optimize task design and modeling approaches to ensure parameter reliability, and external factors such as set and setting, as well as psychedelic-assisted therapy should be evaluated for their additional therapeutic value.

Neural correlates of immediate versus delayed extinction when simultaneously varying the time of the test in humans

Anxiety disorders are effectively treated with exposure therapy based on the extinction of Pavlovian fear conditioning. Animal research indicates that both the timing of extinction and test are important factors to reduce the return of fear. However, empirical evidence in humans is incomplete and inconsistent. In this neuroimaging study, we, therefore, tested 103 young, healthy participants in a 2-factorial between-subjects design with the factors extinction group (immediate, delayed) and test group (+1 day and +7 days). Immediate extinction led to greater retention of fear memory at the beginning of extinction training indicated by increased skin conductance responses. A return of fear was observed in both extinction groups, with a trend toward a greater return of fear in immediate extinction. The return of fear was generally higher in groups with an early test. Neuroimaging results show successful cross-group fear acquisition and retention, as well as activation of the left nucleus accumbens during extinction training. Importantly, the delayed extinction group showed a larger bilateral nucleus accumbens activation during test. This nucleus accumbens finding is discussed in terms of salience, contingency, relief, and prediction error processing. It may imply that the delayed extinction group benefits more from the test as a new learning opportunity.

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Midbrain and striatal dopamine signals have been extremely well characterized over the past several decades, yet novel dopamine signals and functions in reward learning and motivation continue to emerge. A similar characterization of real-time sub-second dopamine signals in areas outside of the striatum has been limited. Recent advances in fluorescent sensor technology and fiber photometry permit the measurement of dopamine binding correlates, which can divulge basic functions of dopamine signaling in non-striatal dopamine terminal regions, like the dorsal bed nucleus of the stria terminalis (dBNST). Here, we record GRABDA signals in the dBNST during a Pavlovian lever autoshaping task. We observe greater Pavlovian cue-evoked dBNST GRABDA signals in sign-tracking (ST) compared to goal-tracking/intermediate (GT/INT) rats and the magnitude of cue-evoked dBNST GRABDA signals decreases immediately following reinforcer-specific satiety. When we deliver unexpected rewards or omit expected rewards, we find that dBNST dopamine signals encode bidirectional reward prediction errors in GT/INT rats, but only positive prediction errors in ST rats. Since sign- and goal-tracking approach strategies are associated with distinct drug relapse vulnerabilities, we examined the effects of experimenter-administered fentanyl on dBNST dopamine associative encoding. Systemic fentanyl injections do not disrupt cue discrimination but generally potentiate dBNST dopamine signals. These results reveal multiple dBNST dopamine correlates of learning and motivation that depend on the Pavlovian approach strategy employed.

Understanding visual hallucinations: a new synthesis

Despite decades of research, we do not definitively know how people sometimes see things that are not there. Eight models of complex visual hallucinations have been published since 2000, including Deafferentation, Reality Monitoring, Perception and Attention Deficit, Activation, Input, and Modulation, Hodological, Attentional Networks, Active inference, and Thalamocortical Dysrhythmia Default Mode Network Decoupling. Each was derived from different understandings of brain organisation. To reduce this variability, representatives from each research group agreed an integrated Visual Hallucination Framework that is consistent with current theories of veridical and hallucinatory vision. The Framework delineates cognitive systems relevant to hallucinations. It allows a systematic, consistent, investigation of relationships between the phenomenology of visual hallucinations and changes in underpinning cognitive structures. The episodic nature of hallucinations highlights separate factors associated with the onset, persistence, and end of specific hallucinations suggesting a complex relationship between state and trait markers of hallucination risk. In addition to a harmonised interpretation of existing evidence, the Framework highlights new avenues of research, and potentially, new approaches to treating distressing hallucinations.

Computational models of behavioral addictions: State of the art and future directions

Non-pharmacological behavioral addictions, such as pathological gambling, videogaming, social networking, or internet use, are becoming major public health concerns. It is not yet clear how behavioral addictions could share many major neurobiological and behavioral characteristics with substance use disorders, despite the absence of direct pharmacological influences. A deeper understanding of the neurocognitive mechanisms of addictive behavior is needed, and computational modeling could be one promising approach to explain intricately entwined cognitive and neural dynamics. This review describes computational models of addiction based on reinforcement learning algorithms, Bayesian inference, and biophysical neural simulations. We discuss whether computational frameworks originally conceived to explain maladaptive behavior in substance use disorders can be effectively extended to non-substance-related behavioral addictions. Moreover, we introduce recent studies on behavioral addictions that exemplify the possibility of such extension and propose future directions.

Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model

Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.