Ventral Tegmental Dopamine Neurons Participate in Reward Identity Predictions

Leveraging Basic Science for the Clinic-From Bench to Bedside

Importance

The tools and insights of behavioral neuroscience grow apace, yet their clinical application is lagging.

Observations

This article suggests that associative learning theory may be the algorithmic bridge to connect a burgeoning understanding of the brain with the challenges to the mind with which all clinicians and researchers are concerned.

Conclusions and Relevance

Instead of giving up, talking past one another, or resting on the laurels of face validity, a consilient and collaborative approach is suggested: visiting laboratory meetings and clinical rounds and attempting to converse in the language of behavior and cognition to better understand and ultimately treat patients.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

Dopamine D2 receptor signaling on iMSNs is required for initiation and vigor of learned actions

Striatal dopamine D2 receptors (D2Rs) are important for motor output. Selective deletion of D2Rs from indirect pathway-projecting medium spiny neurons (iMSNs) impairs locomotor activities in a task-specific manner. However, the role of D2Rs in the initiation of motor actions in reward seeking and taking is not fully understood, and there is little information about how receptors contribute under different task demands and with different outcome types. The iMSN-D2Rs modulate neuronal activity and synaptic transmission, exerting control on circuit functions that may play distinct roles in action learning and performance. Selective deletion of D2Rs on iMSNs resulted in slower action initiation and response rate in an instrumental conditioning task, but only when performance demand was increased. The iMSN-Drd2KO mice were also slower to initiate swimming in a T-maze procedural learning task but were unimpaired in cognitive function and behavioral flexibility. In contrast, in a Pavlovian discrimination learning task, iMSN-Drd2KO mice exhibited normal acquisition and extinction of rewarded responding. The iMSN-Drd2KO mice showed performance deficits at all phases of rotarod skill learning. These findings reveal that dopamine modulation through iMSN-D2Rs influences the ability to self-initiate actions, as well as the willingness and/or vigor with which these responses are performed. However, these receptors seem to have little influence on simple associative learning or on stimulus-driven responding. The loss of normal D2R roles may contribute to disorders in which impaired dopamine signaling leads to hypokinesia or impaired initiation of specific voluntary actions.

Fetal growth restriction mice are more likely to exhibit depression-like behaviors due to stress-induced loss of dopaminergic neurons in the VTA

Fetal growth restriction (FGR) is a severe perinatal complication that can increase risk for mental illness. To investigate the mechanism by which FGR mice develop mental illness in adulthood, we established the FGR mouse model and the FGR mice did not display obvious depression-like behaviors, but after environmental stress exposure, FGR mice were more likely to exhibit depression-like behaviors than control mice. Moreover, FGR mice had significantly fewer dopaminergic neurons in the ventral tegmental area but no difference in serotoninergic neurons in the dorsal raphe. RNA-seq analysis showed that the downregulated genes in the midbrain of FGR mice were associated with many mental diseases and were especially involved in the regulation of NMDA-selective glutamate receptor (NMDAR) activity. Furthermore, the NMDAR antagonist memantine can relieve the stress-induced depression-like behaviors of FGR mice. In summary, our findings provide a theoretical basis for future research and treatment of FGR-related depression.

Heterogeneity in striatal dopamine circuits: Form and function in dynamic reward seeking

The striatal dopamine system has long been studied in the context of reward learning, motivation, and movement. Given the prominent role dopamine plays in a variety of adaptive behavioral states, as well as diseases like addiction, it is essential to understand the full complexity of dopamine neurons and the striatal systems they target. A growing number of studies are uncovering details of the heterogeneity in dopamine neuron subpopulations. Here, we review that work to synthesize current understanding of dopamine system heterogeneity across three levels, anatomical organization, functions in behavior, and modes of action, wherein we focus on signaling profiles and local mechanisms for modulation of dopamine release. Together, these studies reveal new and emerging dimensions of the striatal dopamine system, informing its contribution to dynamic motivational and decision-making processes.

Habitual Preference for the Nondrug Reward in a Drug Choice Setting

For adaptive and efficient decision making, it must be possible to select between habitual alternative courses of action. However, research in rodents suggests that, even in the context of simple decision-making, choice behavior remains goal-directed. In contrast, we recently found that during discrete trial choice between cocaine and water, water-restricted rats preferred water and this preference was habitual and inflexible (i.e., resistant to water devaluation by satiation). Here we sought to test the reproducibility and generality of this surprising finding by assessing habitual control of preference for saccharin over cocaine in non-restricted rats. Specifically, after the acquisition of preference for saccharin, saccharin was devalued and concurrent responding for both options was measured under extinction. As expected, rats responded more for saccharin than for cocaine during extinction, but this difference was unaffected by saccharin devaluation. Together with our previous research, this result indicates that preference for nondrug alternatives over cocaine is under habitual control, even under conditions that normally support goal-directed control of choice between nondrug options. The possible reasons for this difference are discussed.

Acute Stress Enhances Associative Learning via Dopamine Signaling in the Ventral Lateral Striatum

Acute stress transiently increases vigilance, enhancing the detection of salient stimuli in one's environment. This increased perceptual sensitivity is thought to promote the association of rewarding outcomes with relevant cues. The mesolimbic dopamine system is critical for learning cue-reward associations. Dopamine levels in the ventral striatum are elevated following exposure to stress. Together, this suggests that the mesolimbic dopamine system could mediate the influence of acute stress on cue-reward learning. To address this possibility, we examined how a single stressful experience influenced learning in an appetitive pavlovian conditioning task. Male rats underwent an episode of restraint prior to the first conditioning session. This acute stress treatment augmented conditioned responding in subsequent sessions. Voltammetry recordings of mesolimbic dopamine levels demonstrated that acute stress selectively increased reward-evoked dopamine release in the ventral lateral striatum (VLS), but not in the ventral medial striatum. Antagonizing dopamine receptors in the VLS blocked the stress-induced enhancement of conditioned responding. Collectively, these findings illustrate that stress engages dopamine signaling in the VLS to facilitate appetitive learning.SIGNIFICANCE STATEMENT Acute stress influences learning about aversive and rewarding outcomes. Dopamine neurons are sensitive to stress and critical for reward learning. However, it is unclear whether stress regulates reward learning via dopamine signaling. Using fast-scan cyclic voltammetry as rats underwent pavlovian conditioning, we demonstrate that a single stressful experience increases reward-evoked dopamine release in the ventral lateral striatum. This enhanced dopamine signal accompanies a long-lasting increase in conditioned behavioral responding. These findings highlight that the ventral lateral striatum is a node for mediating the effect of stress on reward processing.

Amygdala-Midbrain Connections Modulate Appetitive and Aversive Learning

The central amygdala (CeA) orchestrates adaptive responses to emotional events. While CeA substrates for defensive behaviors have been studied extensively, CeA circuits for appetitive behaviors and their relationship to threat-responsive circuits remain poorly defined. Here, we demonstrate that the CeA sends robust inhibitory projections to the lateral substantia nigra (SNL) that contribute to appetitive and aversive learning in mice. CeA→SNL neural responses to appetitive and aversive stimuli were modulated by expectation and magnitude consistent with a population-level salience signal, which was required for Pavlovian conditioned reward-seeking and defensive behaviors. CeA→SNL terminal activation elicited reinforcement when linked to voluntary actions but failed to support Pavlovian associations that rely on incentive value signals. Consistent with a disinhibitory mechanism, CeA inputs preferentially target SNL GABA neurons, and CeA→SNL and SNL dopamine neurons respond similarly to salient stimuli. Collectively, our results suggest that amygdala-nigra interactions represent a previously unappreciated mechanism for influencing emotional behaviors.

The Neurobiology of Impulsive Decision-Making and Reinforcement Learning in Nonhuman Animals

Impulsive decisions are those that favor immediate over delayed rewards, involve the acceptance of undue risk or uncertainty, or fail to adapt to environmental changes. Pathological levels of impulsive decision-making have been observed in individuals with mental illness, but there may be substantial heterogeneity in the processes that drive impulsive choices. Understanding this behavioral heterogeneity may be critical for understanding associated diverseness in the neural mechanisms that give rise to impulsivity. The application of reinforcement learning algorithms in the deconstruction of impulsive decision-making phenotypes can help bridge the gap between biology and behavior and provide insights into the biobehavioral heterogeneity of impulsive choice. This chapter will review the literature on the neurobiological mechanisms of impulsive decision-making in nonhuman animals; specifically, the role of the amine neuromodulatory systems (dopamine, serotonin, norepinephrine, and acetylcholine) in impulsive decision-making and reinforcement learning processes is discussed. Ultimately, the integration of reinforcement learning algorithms with sophisticated behavioral and neuroscience techniques may be critical for advancing the understanding of the neurochemical basis of impulsive decision-making.

Causal evidence supporting the proposal that dopamine transients function as temporal difference prediction errors

Reward-evoked dopamine transients are well established as prediction errors. However, the central tenet of temporal difference accounts-that similar transients evoked by reward-predictive cues also function as errors-remains untested. In the present communication we addressed this by showing that optogenetically shunting dopamine activity at the start of a reward-predicting cue prevents second-order conditioning without affecting blocking. These results indicate that cue-evoked transients function as temporal-difference prediction errors rather than reward predictions.

Dopamine transients do not act as model-free prediction errors during associative learning

Dopamine neurons are proposed to signal the reward prediction error in model-free reinforcement learning algorithms. This term represents the unpredicted or 'excess' value of the rewarding event, value that is then added to the intrinsic value of any antecedent cues, contexts or events. To support this proposal, proponents cite evidence that artificially-induced dopamine transients cause lasting changes in behavior. Yet these studies do not generally assess learning under conditions where an endogenous prediction error would occur. Here, to address this, we conducted three experiments where we optogenetically activated dopamine neurons while rats were learning associative relationships, both with and without reward. In each experiment, the antecedent cues failed to acquire value and instead entered into associations with the later events, whether valueless cues or valued rewards. These results show that in learning situations appropriate for the appearance of a prediction error, dopamine transients support associative, rather than model-free, learning.

Striatal and hippocampal contributions to flexible navigation in rats and humans

The hippocampus has been firmly established as playing a crucial role in flexible navigation. Recent evidence suggests that dorsal striatum may also play an important role in such goal-directed behaviour in both rodents and humans. Across recent studies, activity in the caudate nucleus has been linked to forward planning and adaptation to changes in the environment. In particular, several human neuroimaging studies have found the caudate nucleus tracks information traditionally associated with that by the hippocampus. In this brief review, we examine this evidence and argue the dorsal striatum encodes the transition structure of the environment during flexible, goal-directed behaviour. We highlight that future research should explore the following: (1) Investigate neural responses during spatial navigation via a biophysically plausible framework explained by reinforcement learning models and (2) Observe the interaction between cortical areas and both the dorsal striatum and hippocampus during flexible navigation.

Dopamine neuron ensembles signal the content of sensory prediction errors

Dopamine neurons respond to errors in predicting value-neutral sensory information. These data, combined with causal evidence that dopamine transients support sensory-based associative learning, suggest that the dopamine system signals a multidimensional prediction error. Yet such complexity is not evident in the activity of individual neurons or population averages. How then do downstream areas know what to learn in response to these signals? One possibility is that information about content is contained in the pattern of firing across many dopamine neurons. Consistent with this, here we show that the pattern of firing across a small group of dopamine neurons recorded in rats signals the identity of a mis-predicted sensory event. Further, this same information is reflected in the BOLD response elicited by sensory prediction errors in human midbrain. These data provide evidence that ensembles of dopamine neurons provide highly specific teaching signals, opening new possibilities for how this system might contribute to learning.

In vivo functional diversity of midbrain dopamine neurons within identified axonal projections

Functional diversity of midbrain dopamine (DA) neurons ranges across multiple scales, from differences in intrinsic properties and connectivity to selective task engagement in behaving animals. Distinct in vitro biophysical features of DA neurons have been associated with different axonal projection targets. However, it is unknown how this translates to different firing patterns of projection-defined DA subpopulations in the intact brain. We combined retrograde tracing with single-unit recording and labelling in mouse brain to create an in vivo functional topography of the midbrain DA system. We identified differences in burst firing among DA neurons projecting to dorsolateral striatum. Bursting also differentiated DA neurons in the medial substantia nigra (SN) projecting either to dorsal or ventral striatum. We found differences in mean firing rates and pause durations among ventral tegmental area (VTA) DA neurons projecting to lateral or medial shell of nucleus accumbens. Our data establishes a high-resolution functional in vivo landscape of midbrain DA neurons.

Occasion setters attain incentive motivational value: implications for contextual influences on reward-seeking

The context in which reward-paired cues are encountered can resolve ambiguity and set the occasion for appropriate reward-seeking. The psychological processes by which contexts regulate reward-seeking remain unclear as contexts are diffuse and difficult to isolate from other stimuli. To overcome this, we modeled a context as a phasic and discrete event—an occasion setter (OS)—which allowed for control over its presentation and influence on cue-driven reward-seeking. This allowed us to directly assess how OSs regulate the predictive and motivational significance of Pavlovian cues. Male rats (n = 50) were trained in a Pavlovian procedure where the presentation of an ambiguous conditioned stimulus (CS) was reinforced only if preceded by an occasion setting cue. We assessed the motivational value of the OS and CS alone or in combination using tests of conditioned reinforcement. Rats enhanced conditioned approach to the reward port during the CS when it was preceded by the OS. When allowed the opportunity, rats responded more to obtain presentations of the CS in combination with the OS than the CS alone. Critically, rats also worked to obtain presentations of the OS alone more than the CS alone, and this was resistant to manipulations of the value of the OS. We conclude that occasion setting can act via incentive motivational mechanisms and that, apart from resolving predictive information about ambiguous reward-paired cues, OSs themselves generate states of appetitive motivation that can facilitate reward-seeking.

Synchronicity: The Role of Midbrain Dopamine in Whole-Brain Coordination

Midbrain dopamine seems to play an outsized role in motivated behavior and learning. Widely associated with mediating reward-related behavior, decision making, and learning, dopamine continues to generate controversies in the field. While many studies and theories focus on what dopamine cells encode, the question of how the midbrain derives the information it encodes is poorly understood and comparatively less addressed. Recent anatomical studies suggest greater diversity and complexity of afferent inputs than previously appreciated, requiring rethinking of prior models. Here, we elaborate a hypothesis that construes midbrain dopamine as implementing a Bayesian selector in which individual dopamine cells sample afferent activity across distributed brain substrates, comprising evidence to be evaluated on the extent to which stimuli in the on-going sensorimotor stream organizes distributed, parallel processing, reflecting implicit value. To effectively generate a temporally resolved phasic signal, a population of dopamine cells must exhibit synchronous activity. We argue that synchronous activity across a population of dopamine cells signals consensus across distributed afferent substrates, invigorating responding to recognized opportunities and facilitating further learning. In framing our hypothesis, we shift from the question of how value is computed to the broader question of how the brain achieves coordination across distributed, parallel processing. We posit the midbrain is part of an “axis of agency” in which the prefrontal cortex (PFC), basal ganglia (BGS), and midbrain form an axis mediating control, coordination, and consensus, respectively.

Multiple Dopamine Systems: Weal and Woe of Dopamine

The ability to predict future outcomes increases the fitness of the animal. Decades of research have shown that dopamine neurons broadcast reward prediction error (RPE) signals-the discrepancy between actual and predicted reward-to drive learning to predict future outcomes. Recent studies have begun to show, however, that dopamine neurons are more diverse than previously thought. In this review, we will summarize a series of our studies that have shown unique properties of dopamine neurons projecting to the posterior "tail" of the striatum (TS) in terms of anatomy, activity, and function. Specifically, TS-projecting dopamine neurons are activated by a subset of negative events including threats from a novel object, send prediction errors for external threats, and reinforce avoidance behaviors. These results indicate that there are at least two axes of dopamine-mediated reinforcement learning in the brain-one learning from canonical RPEs and another learning from threat prediction errors. We argue that the existence of multiple learning systems is an adaptive strategy that makes possible each system optimized for its own needs. The compartmental organization in the mammalian striatum resembles that of a dopamine-recipient area in insects (mushroom body), pointing to a principle of dopamine function conserved across phyla.