Mesolimbic dopamine release conveys causal associations

Distinct catecholaminergic pathways projecting to hippocampal CA1 transmit contrasting signals during navigation in familiar and novel environments

Neuromodulatory inputs to the hippocampus play pivotal roles in modulating synaptic plasticity, shaping neuronal activity, and influencing learning and memory. Recently, it has been shown that the main sources of catecholamines to the hippocampus, ventral tegmental area (VTA) and locus coeruleus (LC), may have overlapping release of neurotransmitters and effects on the hippocampus. Therefore, to dissect the impacts of both VTA and LC circuits on hippocampal function, a thorough examination of how these pathways might differentially operate during behavior and learning is necessary. We therefore utilized two-photon microscopy to functionally image the activity of VTA and LC axons within the CA1 region of the dorsal hippocampus in head-fixed male mice navigating linear paths within virtual reality (VR) environments. We found that within familiar environments some VTA axons and the vast majority of LC axons showed a correlation with the animals' running speed. However, as mice approached previously learned rewarded locations, a large majority of VTA axons exhibited a gradual ramping-up of activity, peaking at the reward location. In contrast, LC axons displayed a pre-movement signal predictive of the animal's transition from immobility to movement. Interestingly, a marked divergence emerged following a switch from the familiar to novel VR environments. Many LC axons showed large increases in activity that remained elevated for over a minute, while the previously observed VTA axon ramping-to-reward dynamics disappeared during the same period. In conclusion, these findings highlight distinct roles of VTA and LC catecholaminergic inputs in the dorsal CA1 hippocampal region. These inputs encode unique information, with reward information in VTA inputs and novelty and kinematic information in LC inputs, likely contributing to differential modulation of hippocampal activity during behavior and learning.

Building and Breaking the Chain: A Model of Reward Prediction Error Integration and Segmentation of Memory

Prediction errors drive reinforcement learning and organize episodic memory into distinct contexts, but do these effects interact? Here, we review the roles of midbrain dopamine, the locus coeruleus, and the hippocampus in event cognition to propose and simulate the theoretical influence of two prediction error signals in integrating versus segmenting events in memory. We suggest that signed reward prediction errors can build mental models of reward environments, increasing the contextual similarity (integration) of experiences with stronger, more stable reward expectations. On the other hand, unsigned reward prediction errors can signal a new model of the environment, generating a contextual shift (segmentation) between experiences that crossed them. We moreover predicted that these differences in contextual similarity give rise to distinct patterns of temporal-order memory. We combined these ideas in a computational model to account for a seemingly paradoxical pattern of temporal-order memory where greater representational distance helps order memory within context but impairs it across contexts. We found that simulating signed reward prediction error integration and unsigned reward prediction error segmentation differentially enabled the model to perform associative chaining, which involved reactivating items between two tested probes to assist with sequential retrieval. In summary, our simulations provide a unifying explanation for the varied ways that neuromodulatory systems may alter event cognition and memory.

Dopamine dynamics are dispensable for movement but promote reward responses

Dopamine signalling modes differ in kinetics and spatial patterns of receptor activation1,2. How these modes contribute to motor function, motivation and learning has long been debated3-21. Here we show that action-potential-induced dopamine release is dispensable for movement initiation but supports reward-oriented behaviour. We generated mice with dopamine-neuron-specific knockout of the release site organizer protein RIM to disrupt action-potential-induced dopamine release. In these mice, rapid in vivo dopamine dynamics were strongly impaired, but baseline dopamine persisted and fully supported spontaneous movement. Conversely, reserpine-mediated dopamine depletion or blockade of dopamine receptors disrupted movement initiation. The dopamine precursor L-DOPA reversed reserpine-induced bradykinesia without restoring fast dopamine dynamics, a result that substantiated the conclusion that these dynamics are dispensable for movement initiation. In contrast to spontaneous movement, reward-oriented behaviour was impaired in dopamine-neuron-specific RIM knockout mice. In conditioned place preference and two-odour discrimination tasks, the mice effectively learned to distinguish the cues, which indicates that reward-based learning persists after RIM ablation. However, the performance vigour was reduced. During probabilistic cue-reward association, dopamine dynamics and conditioned responses assessed through anticipatory licking were disrupted. These results demonstrate that action-potential-induced dopamine release is dispensable for motor function and subsecond precision of movement initiation but promotes motivation and performance during reward-guided behaviours.

A Statistical Framework for Analysis of Trial-Level Temporal Dynamics in Fiber Photometry Experiments

Fiber photometry has become a popular technique to measure neural activity in vivo, but common analysis strategies can reduce detection of effects because they condense within-trial signals into summary measures, and discard trial-level information by averaging across-trials . We propose a novel photometry statistical framework based on functional linear mixed modeling, which enables hypothesis testing of variable effects at every trial time-point , and uses trial-level signals without averaging. This makes it possible to compare the timing and magnitude of signals across conditions while accounting for between-animal differences. Our framework produces a series of plots that illustrate covariate effect estimates and statistical significance at each trial time-point. By exploiting signal autocorrelation, our methodology yields joint 95% confidence intervals that account for inspecting effects across the entire trial and improve the detection of event-related signal changes over common multiple comparisons correction strategies. We reanalyze data from a recent study proposing a theory for the role of mesolimbic dopamine in reward learning, and show the capability of our framework to reveal significant effects obscured by standard analysis approaches. For example, our method identifies two dopamine components with distinct temporal dynamics in response to reward delivery. In simulation experiments, our methodology yields improved statistical power over common analysis approaches. Finally, we provide an open-source package and analysis guide for applying our framework.

Expectancy-related changes in firing of dopamine neurons depend on hippocampus

The orbitofrontal cortex (OFC) and hippocampus (HC) both contribute to the cognitive maps that support flexible behavior. Previously, we used the dopamine neurons to measure the functional role of OFC. We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which expected rewards were manipulated across blocks. We found that ipsilateral OFC lesions degraded dopaminergic prediction errors, consistent with reduced resolution of the task states. Here we have repeated this experiment in male rats with ipsilateral HC lesions. The results show HC also shapes the task states, however unlike OFC, which provides information local to the trial, the HC is necessary for estimating upper-level hidden states that distinguish blocks. The results contrast the roles of the OFC and HC in cognitive mapping and suggest that the dopamine neurons access rich information from distributed regions regarding the environment's structure, potentially enabling this teaching signal to support complex behaviors.

Dopamine release plateau and outcome signals in dorsal striatum contrast with classic reinforcement learning formulations

We recorded dopamine release signals in centromedial and centrolateral sectors of the striatum as mice learned consecutive versions of visual cue-outcome conditioning tasks. Dopamine release responses differed for the centromedial and centrolateral sites. In neither sector could these be accounted for by classic reinforcement learning alone as classically applied to the activity of nigral dopamine-containing neurons. Medially, cue responses ranged from initial sharp peaks to modulated plateau responses; outcome (reward) responses during cue conditioning were minimal or, initially, negative. At centrolateral sites, by contrast, strong, transient dopamine release responses occurred at both cue and outcome. Prolonged, plateau release responses to cues emerged in both regions when discriminative behavioral responses became required. At most sites, we found no evidence for a transition from outcome signaling to cue signaling, a hallmark of temporal difference reinforcement learning as applied to midbrain dopaminergic neuronal activity. These findings delineate a reshaping of striatal dopamine release activity during learning and suggest that current views of reward prediction error encoding need review to accommodate distinct learning-related spatial and temporal patterns of striatal dopamine release in the dorsal striatum.

Intriguing astrocyte responses in CA1 to reduced and rehabilitated masticatory function: Dorsal and ventral distinct perspectives in adult mice

OBJECTIVE

We sought to investigate the plasticity of diet-induced changes in astrocyte morphology of stratum lacunosum-moleculare (SLM) in CA1.

DESIGN

Three diet regimes were adopted in 15 mice, from the 21st postnatal day to 6 months. The first diet regimen was pellet feed, called Hard Diet (HD). The second, with reduced masticatory, received a pellet-diet followed by a powdered-diet, and it was identified as Hard Diet/Soft Diet (HD/SD). Finally, the group with rehabilitated masticatory was named Hard Diet/Soft Diet/Hard Diet (HD/SD/HD). In the end, euthanasia and brain histological processing were performed, in which astrocytic immunoreactivity to glial-fibrillary-acidic-protein (GFAP) was tested. In reconstructed astrocytes, morphometric analysis was performed.

RESULTS

Astrocyte morphometric revealed that changes in masticatory regimens impact astrocyte morphology. In the dorsal CA1, switching from a hard diet to a soft diet led to reductions in most variables, whereas in the ventral, fewer variables were affected, highlighting regional differences in astrocyte responses. Cluster analysis further showed that diet-induced changes in astrocyte morphology were reversible in the dorsal region, but not in the ventral region, indicating a persistent impact on astrocyte diversity and complexity in the ventral even after rehabilitation. Correlation tests between astrocyte morphology and behavioral performance demonstrated disrupted relationships under masticatory stress, with effects persisting after rehabilitation.

CONCLUSION

Changes in the diet result in significant alterations in astrocyte morphology, suggesting a direct link between dietary modulation and cellular structure. Morphometric analyses revealed distinct alterations in astrocyte morphology in response to changes in the masticatory regimen, with both dorsal/ventral regions displaying notable changes. Moreover, the regional differential effects on astrocytes underscore the complexity of mastication on neuroplasticity and cognitive function.

Frustrative Nonreward: Behavior, Circuits, Neurochemistry, and Disorders

The surprising omission or reduction of vital resources (food, fluid, social partners) can induce an aversive emotion known as frustrative nonreward (FNR), which can influence subsequent behavior and physiology. FNR is an integral mediator of irritability/aggression, motivation (substance use disorders, depression), anxiety/fear/threat, learning/conditioning, and social behavior. Despite substantial progress in the study of FNR during the twentieth century, research lagged in the later part of the century and into the early twenty-first century until the National Institute of Mental Health's Research Domain Criteria initiative included FNR and loss as components of the negative valence domain. This led to a renaissance of new research and paradigms relevant to basic and clinical science alike. The COVID-19 pandemic's extensive individual and social restrictions were correlated with increased drug and alcohol use, social conflict, irritability, and suicide, all potential consequences of FNR. This article highlights animal models related to these psychiatric disorders and symptoms and presents recent advances in identifying the brain regions and neurotransmitters implicated.

Dopamine dynamics in nucleus accumbens across reward-based learning of goal-directed whisker-to-lick sensorimotor transformations in mice

The synaptic and neuronal circuit mechanisms underlying reward-based learning remain to be fully determined. In the mammalian brain, dopamine release in nucleus accumbens is thought to contribute importantly to reward signals for learning and promoting synaptic plasticity. Here, through longitudinal fiber photometry of a genetically-encoded fluorescent sensor, we investigated dopamine signals in the nucleus accumbens of thirsty head-restrained mice as they learned to lick a liquid reward spout in response to single deflections of the C2 whisker across varying reward contingencies. Reward delivery triggered by well-timed licking drove fast transient dopamine increases in nucleus accumbens. On the other hand, unrewarded licking was overall associated with reduced dopamine sensor fluorescence, especially in trials where reward was unexpectedly omitted. The dopamine reward signal upon liquid delivery decreased within individual sessions as mice became sated. As mice learned to lick the reward spout in response to whisker deflection, a fast transient sensory-evoked dopamine signal developed, correlating with the learning of the whisker detection task across consecutive training days, as well as within single learning sessions. The well-defined behavioral paradigm involving a unitary stimulus of a single whisker as a reward-predicting cue along with precisely quantified licking allows untangling of sensory, motor and reward-related dopamine signals and how they evolve across the learning of whisker-dependent goal-directed sensorimotor transformations. Our longitudinal measurements of dopamine dynamics across reward-based learning are overall consistent with the hypothesis that dopamine could play an important role as a reward signal for reinforcement learning.

Learning temporal relationships between symbols with Laplace Neural Manifolds

Firing across populations of neurons in many regions of the mammalian brain maintains a temporal memory, a neural timeline of the recent past. Behavioral results demonstrate that people can both remember the past and anticipate the future over an analogous internal timeline. This paper presents a mathematical framework for building this timeline of the future. We assume that the input to the system is a time series of symbols-sparse tokenized representations of the present-in continuous time. The goal is to record pairwise temporal relationships between symbols over a wide range of time scales. We assume that the brain has access to a temporal memory in the form of the real Laplace transform. Hebbian associations with a diversity of synaptic time scales are formed between the past timeline and the present symbol. The associative memory stores the convolution between the past and the present. Knowing the temporal relationship between the past and the present allows one to infer relationships between the present and the future. With appropriate normalization, this Hebbian associative matrix can store a Laplace successor representation and a Laplace predecessor representation from which measures of temporal contingency can be evaluated. The diversity of synaptic time constants allows for learning of non-stationary statistics as well as joint statistics between triplets of symbols. This framework synthesizes a number of recent neuroscientific findings including results from dopamine neurons in the mesolimbic forebrain.

Dopamine transmission at D1 and D2 receptors in the nucleus accumbens contributes to the expression of incubation of cocaine craving

Relapse represents a consistent clinical problem for individuals with substance use disorder. In the incubation of craving model of persistent craving and relapse, cue-induced drug seeking progressively intensifies or "incubates" during the first weeks of abstinence from drug self-administration and then remains high for months. Previously, we and others have demonstrated that expression of incubated cocaine craving requires strengthening of excitatory synaptic transmission in the nucleus accumbens core (NAcc). However, despite the importance of dopaminergic signaling in the NAcc for motivated behavior, little is known about the role that dopamine (DA) plays in the incubation of cocaine craving. Here we used fiber photometry to measure DA transients in the NAcc of male and female rats during cue-induced seeking tests conducted in early abstinence from cocaine self-administration, prior to incubation, and late abstinence, after incubation of craving has plateaued. We observed DA transients time-locked to cue-induced responding but their magnitude did not differ significantly when measured during early versus late abstinence seeking tests. Next, we tested for a functional role of these DA transients by injecting DA receptor antagonists into the NAcc just before the cue-induced seeking test. Blockade of either D1 or D2 DA receptors reduced cue-induced cocaine seeking after but not before incubation. We found no main effect of sex or significant interaction of sex with other factors in our experiments. These results suggest that DA contributes to incubated cocaine seeking but the emergence of this role reflects changes in postsynaptic responsiveness to DA rather than presynaptic alterations.

Individual differences in decision-making shape how mesolimbic dopamine regulates choice confidence and change-of-mind

Nucleus accumbens dopamine signaling is an important neural substrate for decision-making. Dominant theories generally discretize and homogenize decision-making, when it is in fact a continuous process, with evaluation and re-evaluation components that extend beyond simple outcome prediction into consideration of past and future value. Extensive work has examined mesolimbic dopamine in the context of reward prediction error, but major gaps persist in our understanding of how dopamine regulates volitional and self-guided decision-making. Moreover, there is little consideration of individual differences in value processing that may shape how dopamine regulates decision-making. Here, using an economic foraging task in mice, we found that dopamine dynamics in the nucleus accumbens core reflected decision confidence during evaluation of decisions, as well as both past and future value during re-evaluation and change-of-mind. Optogenetic manipulations of mesolimbic dopamine release selectively altered evaluation and re-evaluation of decisions in mice whose dopamine dynamics and behavior reflected future value.

Learning depends on the information conveyed by temporal relationships between events and is reflected in the dopamine response to cues

Contemporary theories guiding the search for neural mechanisms of learning and memory assume that associative learning results from the temporal pairing of cues and reinforcers resulting in coincident activation of associated neurons, strengthening their synaptic connection. While enduring, this framework has limitations: Temporal pairing-based models of learning do not fit with many experimental observations and cannot be used to make quantitative predictions about behavior. Here, we present behavioral data that support an alternative, information-theoretic conception: The amount of information that cues provide about the timing of reward delivery predicts behavior. Furthermore, this approach accounts for the rate and depth of both inhibitory and excitatory learning across paradigms and species. We also show that dopamine release in the ventral striatum reflects cue-predicted changes in reinforcement rates consistent with subjects understanding temporal relationships between task events. Our results reshape the conceptual and biological framework for understanding associative learning.

Reward timescale controls the rate of behavioural and dopaminergic learning

Learning the causes of rewards is necessary for survival. Thus, it is critical to understand the mechanisms of such a vital biological process. Cue-reward learning is controlled by mesolimbic dopamine and improves with spacing of cue-reward pairings. However, whether a mathematical rule governs such improvements in learning rate, and if so, whether a unifying mechanism captures this rule and dopamine dynamics during learning remain unknown. Here, we investigate the behavioral, algorithmic, and dopaminergic mechanisms governing cue-reward learning rate. Across a range of conditions in mice, we show a strong, mathematically proportional relationship between both behavioral and dopaminergic learning rates and the duration between rewards. Due to this relationship, removing up to 19 out of 20 cue-reward pairings over a fixed duration has no influence on overall learning. These findings are explained by a dopamine-based model of retrospective learning, thereby providing a unified account of the biological mechanisms of learning.

Explaining dopamine through prediction errors and beyond

The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.

Humans adaptively deploy forward and backward prediction

The formation of predictions is essential to our ability to build models of the world and use them for intelligent decision-making. Here we challenge the dominant assumption that humans form only forward predictions, which specify what future events are likely to follow a given present event. We demonstrate that in some environments, it is more efficient to use backward prediction, which specifies what present events are likely to precede a given future event. This is particularly the case in diverging environments, where possible future events outnumber possible present events. Correspondingly, in six preregistered experiments (n = 1,299) involving both simple decision-making and more challenging planning tasks, we find that humans engage in backward prediction in divergent environments and use forward prediction in convergent environments. We thus establish that humans adaptively deploy forward and backward prediction in the service of efficient decision-making.

Spatially organized striatal neuromodulator release encodes trajectory errors

Goal-directed navigation requires animals to continuously evaluate their current direction and speed of travel relative to landmarks to discern whether they are approaching or deviating from their goal. Striatal dopamine and acetylcholine are powerful modulators of goal-directed behavior, but it is unclear whether and how neuromodulator dynamics at landmarks incorporate relative motion for effective behavioral guidance. Using optical measurements in mice, we demonstrate that cue-evoked striatal dopamine release encodes bi-directional 'trajectory errors' reflecting relationships between ongoing speed and direction of locomotion and visual flow relative to optimal goal trajectories. Striatum-wide micro-fiber array recordings resolved an anatomical gradient of trajectory error signaling across the anterior-posterior axis, distinct from trajectory error independent cue signals. Dynamic regression modeling revealed that positive and negative trajectory error encoding emerges early and late respectively during learning and over different time courses in the medial and lateral striatum, enabling region specific contributions to learning. Striatal acetylcholine release also encodes trajectory errors, but encoding is more spatially restricted, opposite polarity, and delayed relative to dopamine, supporting distinct roles in modulating striatal output and behavior. Dopamine trajectory error signaling and task performance were reproduced in a reinforcement learning model incorporating a conjunctive state space representation, suggesting a potential neural substrate for trajectory error generation. Our results establish region specific neuromodulator signals positioned to guide the speed and direction of locomotion to reach goals based on environmental landmarks during navigation.

The nucleus accumbens in reward and aversion processing: insights and implications

The nucleus accumbens (NAc), a central component of the brain's reward circuitry, has been implicated in a wide range of behaviors and emotional states. Emerging evidence, primarily drawing from recent rodent studies, suggests that the function of the NAc in reward and aversion processing is multifaceted. Prolonged stress or drug use induces maladaptive neuronal function in the NAc circuitry, which results in pathological conditions. This review aims to provide comprehensive and up-to-date insights on the role of the NAc in motivated behavior regulation and highlights areas that demand further in-depth analysis. It synthesizes the latest findings on how distinct NAc neuronal populations and pathways contribute to the processing of opposite valences. The review examines how a range of neuromodulators, especially monoamines, influence the NAc's control over various motivational states. Furthermore, it delves into the complex underlying mechanisms of psychiatric disorders such as addiction and depression and evaluates prospective interventions to restore NAc functionality.

Chronic stress deficits in reward behaviour co-occur with low nucleus accumbens dopamine activity during reward anticipation specifically

Whilst reward pathologies are major and common in stress-related neuropsychiatric disorders, their neurobiology and treatment are poorly understood. Imaging studies in human reward pathology indicate attenuated BOLD activity in nucleus accumbens (NAc) coincident with reward anticipation but not reinforcement; potentially, this is dopamine (DA) related. In mice, chronic social stress (CSS) leads to reduced reward learning and motivation. Here, DA-sensor fibre photometry is used to investigate whether these behavioural deficits co-occur with altered NAc DA activity during reward anticipation and/or reinforcement. In CSS mice relative to controls: (1) Reduced discriminative learning of the sequence, tone-on + appetitive behaviour = tone-on + sucrose reinforcement, co-occurs with attenuated NAc DA activity throughout tone-on and sucrose reinforcement. (2) Reduced motivation during the sequence, operant behaviour = tone-on + sucrose delivery + sucrose reinforcement, co-occurs with attenuated NAc DA activity at tone-on and typical activity at sucrose reinforcement. (3) Reduced motivation during the sequence, operant behaviour = appetitive behaviour + sociosexual reinforcement, co-occurs with typical NAc DA activity at female reinforcement. Therefore, in CSS mice, low NAc DA activity co-occurs with low reward anticipation and could account for deficits in learning and motivation, with important implications for understanding human reward pathology.

Synaptogyrin-3 Prevents Cocaine Addiction and Dopamine Deficits

Synaptogyrin-3, a functionally obscure synaptic vesicle protein, interacts with vesicular monoamine and dopamine transporters, bringing together dopamine release and reuptake sites. Synaptogyrin-3 was reduced by chronic cocaine exposure in both humans and rats, and synaptogyrin-3 levels inversely correlated with motivation to take cocaine in rats. Synaptogyrin-3 overexpression in dopamine neurons reduced cocaine self-administration, decreased anxiety-like behavior, and enhanced cognitive flexibility. Overexpression also enhanced nucleus accumbens dopamine signaling and prevented cocaine-induced deficits, suggesting a putative therapeutic role for synaptogyrin-3 in cocaine use disorder.

The causal structure and computational value of narratives

Many human behavioral and brain imaging studies have used narratively structured stimuli (e.g., written, audio, or audiovisual stories) to better emulate real-world experience in the laboratory. However, narratives are a special class of real-world experience, largely defined by their causal connections across time. Much contemporary neuroscience research does not consider this key property. We review behavioral and neuroscientific work that speaks to how causal structure shapes comprehension of and memory for narratives. We further draw connections between this work and reinforcement learning, highlighting how narratives help link causes to outcomes in complex environments. By incorporating the plausibility of causal connections between classes of actions and outcomes, reinforcement learning models may become more ecologically valid, while simultaneously elucidating the value of narratives.

The neurobiology of aesthetic chills: How bodily sensations shape emotional experiences

The phenomenon of aesthetic chills-shivers and goosebumps associated with either rewarding or threatening stimuli-offers a unique window into the brain basis of conscious reward because of their universal nature and simultaneous subjective and physical counterparts. Elucidating the neural mechanisms underlying aesthetic chills can reveal fundamental insights about emotion, consciousness, and the embodied mind. What is the precise timing and mechanism of bodily feedback in emotional experience? How are conscious feelings and motivations generated from interoceptive predictions? What is the role of uncertainty and precision signaling in shaping emotions? How does the brain distinguish and balance processing of rewards versus threats? We review neuroimaging evidence and highlight key questions for understanding how bodily sensations shape conscious feelings. This research stands to advance models of brain-body interactions shaping affect and may lead to novel nonpharmacological interventions for disorders of motivation and pleasure.

Distinct catecholaminergic pathways projecting to hippocampal CA1 transmit contrasting signals during navigation in familiar and novel environments

Neuromodulatory inputs to the hippocampus play pivotal roles in modulating synaptic plasticity, shaping neuronal activity, and influencing learning and memory. Recently it has been shown that the main sources of catecholamines to the hippocampus, ventral tegmental area (VTA) and locus coeruleus (LC), may have overlapping release of neurotransmitters and effects on the hippocampus. Therefore, to dissect the impacts of both VTA and LC circuits on hippocampal function, a thorough examination of how these pathways might differentially operate during behavior and learning is necessary. We therefore utilized 2-photon microscopy to functionally image the activity of VTA and LC axons within the CA1 region of the dorsal hippocampus in head-fixed male mice navigating linear paths within virtual reality (VR) environments. We found that within familiar environments some VTA axons and the vast majority of LC axons showed a correlation with the animals' running speed. However, as mice approached previously learned rewarded locations, a large majority of VTA axons exhibited a gradual ramping-up of activity, peaking at the reward location. In contrast, LC axons displayed a pre-movement signal predictive of the animal's transition from immobility to movement. Interestingly, a marked divergence emerged following a switch from the familiar to novel VR environments. Many LC axons showed large increases in activity that remained elevated for over a minute, while the previously observed VTA axon ramping-to-reward dynamics disappeared during the same period. In conclusion, these findings highlight distinct roles of VTA and LC catecholaminergic inputs in the dorsal CA1 hippocampal region. These inputs encode unique information, with reward information in VTA inputs and novelty and kinematic information in LC inputs, likely contributing to differential modulation of hippocampal activity during behavior and learning.

Time-scale invariant contingency yields one-shot reinforcement learning despite extremely long delays to reinforcement

Reinforcement learning inspires much theorizing in neuroscience, cognitive science, machine learning, and AI. A central question concerns the conditions that produce the perception of a contingency between an action and reinforcement-the assignment-of-credit problem. Contemporary models of associative and reinforcement learning do not leverage the temporal metrics (measured intervals). Our information-theoretic approach formalizes contingency by time-scale invariant temporal mutual information. It predicts that learning may proceed rapidly even with extremely long action-reinforcer delays. We show that rats can learn an action after a single reinforcement, even with a 16-min delay between the action and reinforcement (15-fold longer than any delay previously shown to support such learning). By leveraging metric temporal information, our solution obviates the need for windows of associability, exponentially decaying eligibility traces, microstimuli, or distributions over Bayesian belief states. Its three equations have no free parameters; they predict one-shot learning without iterative simulation.

The influence of emotion on temporal context models

Temporal context models (TCMs) have been influential in understanding episodic memory and its neural underpinnings. Recently, TCMs have been extended to explain emotional memory effects, one of the most clinically important findings in the field of memory research. This review covers recent advances in hypotheses for the neural representation of spatiotemporal context through the lens of TCMs, including their ability to explain the influence of emotion on episodic and temporal memory. In recent years, simplifying assumptions of "classical" TCMs - with exponential trace decay and the mechanism by which temporal context is recovered - have become increasingly clear. The review also outlines how recent advances could be incorporated into a future TCM, beyond classical assumptions, to integrate emotional modulation.

Natural phasic inhibition of dopamine neurons signals cognitive rigidity

When animals unexpectedly fail, their dopamine neurons undergo phasic inhibition that canonically drives extinction learning-a cognitive-flexibility mechanism for discarding outdated strategies. However, the existing evidence equates natural and artificial phasic inhibition, despite their spatiotemporal differences. Addressing this gap, we targeted a GABAA-receptor antagonist precisely to dopamine neurons, yielding three unexpected findings. First, this intervention blocked natural phasic inhibition selectively, leaving tonic activity unaffected. Second, blocking natural phasic inhibition accelerated extinction learning-opposite to canonical mechanisms. Third, our approach selectively benefitted perseverative mice, restoring rapid extinction without affecting new reward learning. Our findings reveal that extinction learning is rapid by default and slowed by natural phasic inhibition-challenging foundational learning theories, while delineating a synaptic mechanism and therapeutic target for cognitive rigidity.

Learning to express reward prediction error-like dopaminergic activity requires plastic representations of time

The dominant theoretical framework to account for reinforcement learning in the brain is temporal difference learning (TD) learning, whereby certain units signal reward prediction errors (RPE). The TD algorithm has been traditionally mapped onto the dopaminergic system, as firing properties of dopamine neurons can resemble RPEs. However, certain predictions of TD learning are inconsistent with experimental results, and previous implementations of the algorithm have made unscalable assumptions regarding stimulus-specific fixed temporal bases. We propose an alternate framework to describe dopamine signaling in the brain, FLEX (Flexibly Learned Errors in Expected Reward). In FLEX, dopamine release is similar, but not identical to RPE, leading to predictions that contrast to those of TD. While FLEX itself is a general theoretical framework, we describe a specific, biophysically plausible implementation, the results of which are consistent with a preponderance of both existing and reanalyzed experimental data.

A chemogenetic approach for dopamine imaging with tunable sensitivity

Genetically-encoded dopamine (DA) sensors enable high-resolution imaging of DA release, but their ability to detect a wide range of extracellular DA levels, especially tonic versus phasic DA release, is limited by their intrinsic affinity. Here we show that a human-selective dopamine receptor positive allosteric modulator (PAM) can be used to boost sensor affinity on-demand. The PAM enhances DA detection sensitivity across experimental preparations (in vitro, ex vivo and in vivo) via one-photon or two-photon imaging. In vivo photometry-based detection of optogenetically-evoked DA release revealed that DETQ administration produces a stable 31 minutes window of potentiation without effects on animal behavior. The use of the PAM revealed region-specific and metabolic state-dependent differences in tonic DA levels and enhanced single-trial detection of behavior-evoked phasic DA release in cortex and striatum. Our chemogenetic strategy can potently and flexibly tune DA imaging sensitivity and reveal multi-modal (tonic/phasic) DA signaling across preparations and imaging approaches.

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Generalized cue reactivity in dopamine neurons after opioids

Cue reactivity is the maladaptive neurobiological and behavioral response upon exposure to drug cues and is a major driver of relapse. The leading hypothesis is that dopamine release by addictive drugs represents a persistently positive reward prediction error that causes runaway enhancement of dopamine responses to drug cues, leading to their pathological overvaluation compared to non-drug reward alternatives. However, this hypothesis has not been directly tested. Here we developed Pavlovian and operant procedures to measure firing responses, within the same dopamine neurons, to drug versus natural reward cues, which we found to be similarly enhanced compared to cues predicting natural rewards in drug-naïve controls. This enhancement was associated with increased behavioral reactivity to the drug cue, suggesting that dopamine release is still critical to cue reactivity, albeit not as previously hypothesized. These results challenge the prevailing hypothesis of cue reactivity, warranting new models of dopaminergic function in drug addiction, and provide critical insights into the neurobiology of cue reactivity with potential implications for relapse prevention.

What Role Does Striatal Dopamine Play in Goal-directed Action?

Evidence suggests that dopamine activity provides a US-related prediction error for Pavlovian conditioning and the reinforcement signal supporting the acquisition of habits. However, its role in goal-directed action is less clear. There are currently few studies that have assessed dopamine release as animals acquire and perform self-paced instrumental actions. Here we briefly review the literature documenting the psychological, behavioral and neural bases of goal-directed actions in rats and mice, before turning to describe recent studies investigating the role of dopamine in instrumental learning and performance. Plasticity in dorsomedial striatum, a central node in the network supporting goal-directed action, clearly requires dopamine release, the timing of which, relative to cortical and thalamic inputs, determines the degree and form of that plasticity. Beyond this, bilateral release appears to reflect reward prediction errors as animals experience the consequences of an action. Such signals feedforward to update the value of the specific action associated with that outcome during subsequent performance, with dopamine release at the time of action reflecting the updated predicted action value. More recently, evidence has also emerged for a hemispherically lateralised signal associated with the action; dopamine release is greater in the hemisphere contralateral to the spatial target of the action. This effect emerges over the course of acquisition and appears to reflect the strength of the action-outcome association. Thus, during goal-directed action, dopamine release signals the action, the outcome and their association to shape the learning and performance processes necessary to support this form of behavioral control.

Dopamine neuron activity evoked by sucrose and sucrose-predictive cues is augmented by peripheral and central manipulations of glucose availability

Food deprivation drives eating through multiple signals and circuits. Decreased glucose availability (i.e., cytoglucopenia) drives eating and also increases the value of sucrose. Ventral tegmental area (VTA) dopamine neurons (DANs) contribute to the evaluation of taste stimuli, but their role in integrating glucoprivic signals remains unknown. We monitored VTA DAN activity via Cre-dependent expression of a calcium indicator with in vivo fibre photometry. In ad libitum fed rats, intraoral sucrose evoked a phasic increase in DAN activity. To manipulate glucose availability, we administered (intraperitoneal, lateral or fourth ventricular) the antiglycolytic agent 5-thio-D-glucose (5TG), which significantly augmented the phasic DAN activity to sucrose. 5TG failed to alter DAN activity to water or saccharin, suggesting the response was selective for caloric stimuli. 5TG enhancement of sucrose-evoked DAN activity was stronger after fourth ventricular administration, suggesting a critical node of action within the hindbrain. As 5TG also increases blood glucose, in a separate study, we used peripheral insulin, which stimulates eating, to decrease blood glucose-which was associated with increased DAN activity to intraoral sucrose. DAN activity developed to a cue predictive of intraoral sucrose. While 5TG augmented cue-evoked DAN activity, its action was most potent when delivered to the lateral ventricle. Together, the studies point to central glucose availability as a key modulator of phasic DAN activity to food and food-cues. As glucose sensing neurons are known to populate the hypothalamus and brainstem, results suggest differential modulation of cue-evoked and sucrose-evoked DAN activity.

Implications of Kynurenine Pathway Metabolism for the Immune System, Hypothalamic-Pituitary-Adrenal Axis, and Neurotransmission in Alcohol Use Disorder

In recent years, there has been a marked increase in interest in the role of the kynurenine pathway (KP) in mechanisms associated with addictive behavior. Numerous reports implicate KP metabolism in influencing the immune system, hypothalamic-pituitary-adrenal (HPA) axis, and neurotransmission, which underlie the behavioral patterns characteristic of addiction. An in-depth analysis of the results of these new studies highlights interesting patterns of relationships, and approaching alcohol use disorder (AUD) from a broader neuroendocrine-immune system perspective may be crucial to better understanding this complex phenomenon. In this review, we provide an up-to-date summary of information indicating the relationship between AUD and the KP, both in terms of changes in the activity of this pathway and modulation of this pathway as a possible pharmacological approach for the treatment of AUD.

Mesolimbic dopamine ramps reflect environmental timescales

Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.

Memory for rewards guides retrieval

Rewards paid out for successful retrieval motivate the formation of long-term memory. However, it has been argued that the Motivated Learning Task does not measure reward effects on memory strength but decision-making during retrieval. We report three large-scale online experiments in healthy participants (N = 200, N = 205, N = 187) that inform this debate. In experiment 1, we found that explicit stimulus-reward associations formed during encoding influence response strategies at retrieval. In experiment 2, reward affected memory strength and decision-making strategies. In experiment 3, reward affected decision-making strategies only. These data support a theoretical framework that assumes that promised rewards not only increase memory strength, but additionally lead to the formation of stimulus-reward associations that influence decisions at retrieval.

Understanding the heterogeneity of anxiety using a translational neuroscience approach

Anxiety disorders affect millions of people worldwide and present a challenge in neuroscience research because of their substantial heterogeneity in clinical presentation. While a great deal of progress has been made in understanding the neurobiology of fear and anxiety, these insights have not led to effective treatments. Understanding the relationship between phenotypic heterogeneity and the underlying biology is a critical first step in solving this problem. We show translation, reverse translation, and computational modeling can contribute to a refined, cross-species understanding of fear and anxiety as well as anxiety disorders. More specifically, we outline how animal models can be leveraged to develop testable hypotheses in humans by using targeted, cross-species approaches and ethologically informed behavioral paradigms. We discuss reverse translational approaches that can guide and prioritize animal research in nontraditional research species. Finally, we advocate for the use of computational models to harmonize cross-species and cross-methodology research into anxiety. Together, this translational neuroscience approach will help to bridge the widening gap between how we currently conceptualize and diagnose anxiety disorders, as well as aid in the discovery of better treatments for these conditions.

Dopamine across timescales and cell types: Relevance for phenotypes in Parkinson's disease progression

Dopamine neurons in the substantia nigra pars compacta (SNc) synthesize and release dopamine, a critical neurotransmitter for movement and learning. SNc dopamine neurons degenerate in Parkinson's Disease (PD), causing a host of motor and non-motor symptoms. Here, we review recent conceptual advances in our basic understanding of the dopamine system - including our rapidly advancing knowledge of dopamine neuron heterogeneity - with special attention to their importance for understanding PD. In PD patients, dopamine neuron degeneration progresses from lateral SNc to medial SNc, suggesting clinically relevant heterogeneity in dopamine neurons. With technical advances in dopamine system interrogation, we can understand the relevance of this heterogeneity for PD progression and harness it to develop new treatments.

An open-source behavior controller for associative learning and memory (B-CALM)

Associative learning and memory, i.e., learning and remembering the associations between environmental stimuli, self-generated actions, and outcomes such as rewards or punishments, are critical for the well-being of animals. Hence, the neural mechanisms underlying these processes are extensively studied using behavioral tasks in laboratory animals. Traditionally, these tasks have been controlled using commercial hardware and software, which limits scalability and accessibility due to their cost. More recently, due to the revolution in microcontrollers or microcomputers, several general-purpose and open-source solutions have been advanced for controlling neuroscientific behavioral tasks. While these solutions have great strength due to their flexibility and general-purpose nature, for the same reasons, they suffer from some disadvantages including the need for considerable programming expertise, limited online visualization, or slower than optimal response latencies for any specific task. Here, to mitigate these concerns, we present an open-source behavior controller for associative learning and memory (B-CALM). B-CALM provides an integrated suite that can control a host of associative learning and memory behaviors. As proof of principle for its applicability, we show data from head-fixed mice learning Pavlovian conditioning, operant conditioning, discrimination learning, as well as a timing task and a choice task. These can be run directly from a user-friendly graphical user interface (GUI) written in MATLAB that controls many independently running Arduino Mega microcontrollers in parallel (one per behavior box). In sum, B-CALM will enable researchers to execute a wide variety of associative learning and memory tasks in a scalable, accurate, and user-friendly manner.

Does phasic dopamine release cause policy updates?

Phasic dopamine activity is believed to both encode reward-prediction errors (RPEs) and to cause the adaptations that these errors engender. If so, a rat working for optogenetic stimulation of dopamine neurons will repeatedly update its policy and/or action values, thus iteratively increasing its work rate. Here, we challenge this view by demonstrating stable, non-maximal work rates in the face of repeated optogenetic stimulation of midbrain dopamine neurons. Furthermore, we show that rats learn to discriminate between world states distinguished only by their history of dopamine activation. Comparison of these results to reinforcement learning simulations suggests that the induced dopamine transients acted more as rewards than RPEs. However, pursuit of dopaminergic stimulation drifted upwards over a time scale of days and weeks, despite its stability within trials. To reconcile the results with prior findings, we consider multiple roles for dopamine signalling.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.

Sensory Cues Potentiate VTA Dopamine Mediated Reinforcement

Sensory cues are critical for shaping decisions and invigorating actions during reward seeking. Dopamine neurons in the ventral tegmental area (VTA) are central in this process, supporting associative learning in Pavlovian and instrumental settings. Studies of intracranial self-stimulation (ICSS) behavior, which show that animals will work hard to receive stimulation of dopamine neurons, support the notion that dopamine transmits a reward or value signal to support learning. Recent studies have begun to question this, however, emphasizing dopamine's value-free functions, leaving its contribution to behavioral reinforcement somewhat muddled. Here, we investigated the role of sensory stimuli in dopamine-mediated reinforcement, using an optogenetic ICSS paradigm in tyrosine hydroxylase (TH)-Cre rats. We find that while VTA dopamine neuron activation in the absence of explicit external cues is sufficient to maintain robust self-stimulation, the presence of cues dramatically potentiates ICSS behavior. Our results support a framework where dopamine can have some base value as a reinforcer, but the impact of this signal is modulated heavily by the sensory learning context.

Apathy and Motivation: Biological Basis and Drug Treatment

Apathy is a disabling syndrome associated with poor functional outcomes that is common across a broad range of neurological and psychiatric conditions. Currently, there are no established therapies specifically for the condition, and safe and effective treatments are urgently needed. Advances in the understanding of motivation and goal-directed behavior in humans and animals have shed light on the cognitive and neurobiological mechanisms contributing to apathy, providing an important foundation for the development of new treatments. Here, we review the cognitive components, neural circuitry, and pharmacology of apathy and motivation, highlighting converging evidence of shared transdiagnostic mechanisms. Though no pharmacological treatments have yet been licensed, we summarize trials of existing and novel compounds to date, identifying several promising candidates for clinical use and avenues of future drug development.

An opponent striatal circuit for distributional reinforcement learning

Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.

From reinforcement learning to agency: Frameworks for understanding basal cognition

Organisms play, explore, and mimic those around them. Is there a purpose to this behavior? Are organisms just behaving, or are they trying to achieve goals? We believe this is a false dichotomy. To that end, to understand organisms, we attempt to unify two approaches for understanding complex agents, whether evolved or engineered. We argue that formalisms describing multiscale competencies and goal-directedness in biology (e.g., TAME), and reinforcement learning (RL), can be combined in a symbiotic framework. While RL has been largely focused on higher-level organisms and robots of high complexity, TAME is naturally capable of describing lower-level organisms and minimal agents as well. We propose several novel questions that come from using RL/TAME to understand biology as well as ones that come from using biology to formulate new theory in AI. We hope that the research programs proposed in this piece shape future efforts to understand biological organisms and also future efforts to build artificial agents.

Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans

We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.

Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans

In the mammalian brain, midbrain dopamine neuron activity is hypothesized to encode reward prediction errors that promote learning and guide behavior by causing rapid changes in dopamine levels in target brain regions. This hypothesis (and alternatives regarding dopamine's role in punishment-learning) has limited direct evidence in humans. We report intracranial, subsecond measurements of dopamine release in human striatum measured, while volunteers (i.e., patients undergoing deep brain stimulation surgery) performed a probabilistic reward and punishment learning choice task designed to test whether dopamine release encodes only reward prediction errors or whether dopamine release may also encode adaptive punishment learning signals. Results demonstrate that extracellular dopamine levels can encode both reward and punishment prediction errors within distinct time intervals via independent valence-specific pathways in the human brain.

Prefrontal Dopamine in Flexible Adaptation to Environmental Changes: A Game for Two Players

Deficits in cognitive flexibility have been characterized in affective, anxiety, and neurodegenerative disorders. This paper reviews data, mainly from studies on animal models, that support the existence of a cortical-striatal brain circuit modulated by dopamine (DA), playing a major role in cognitive/behavioral flexibility. Moreover, we reviewed clinical findings supporting misfunctioning of this circuit in Parkinson's disease that could be responsible for some important non-motoric symptoms. The reviewed findings point to a role of catecholaminergic transmission in the medial prefrontal cortex (mpFC) in modulating DA's availability in the nucleus accumbens (NAc), as well as a role of NAc DA in modulating the motivational value of natural and conditioned stimuli. The review section is accompanied by a preliminary experiment aimed at testing weather the extinction of a simple Pavlovian association fosters increased DA transmission in the mpFC and inhibition of DA transmission in the NAc.

Contexts facilitate dynamic value encoding in the mesolimbic dopamine system

Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.

Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping

The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.

Dual credit assignment processes underlie dopamine signals in a complex spatial environment

Animals frequently make decisions based on expectations of future reward ("values"). Values are updated by ongoing experience: places and choices that result in reward are assigned greater value. Yet, the specific algorithms used by the brain for such credit assignment remain unclear. We monitored accumbens dopamine as rats foraged for rewards in a complex, changing environment. We observed brief dopamine pulses both at reward receipt (scaling with prediction error) and at novel path opportunities. Dopamine also ramped up as rats ran toward reward ports, in proportion to the value at each location. By examining the evolution of these dopamine place-value signals, we found evidence for two distinct update processes: progressive propagation of value along taken paths, as in temporal difference learning, and inference of value throughout the maze, using internal models. Our results demonstrate that within rich, naturalistic environments dopamine conveys place values that are updated via multiple, complementary learning algorithms.