Distributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons

Slowly evolving dopaminergic activity modulates the moment-to-moment probability of reward-related self-timed movements

Clues from human movement disorders have long suggested that the neurotransmitter dopamine plays a role in motor control, but how the endogenous dopaminergic system influences movement is unknown. Here we examined the relationship between dopaminergic signaling and the timing of reward-related movements in mice. Animals were trained to initiate licking after a self-timed interval following a start-timing cue; reward was delivered in response to movements initiated after a criterion time. The movement time was variable from trial-to-trial, as expected from previous studies. Surprisingly, dopaminergic signals ramped-up over seconds between the start-timing cue and the self-timed movement, with variable dynamics that predicted the movement/reward time on single trials. Steeply rising signals preceded early lick-initiation, whereas slowly rising signals preceded later initiation. Higher baseline signals also predicted earlier self-timed movements. Optogenetic activation of dopamine neurons during self-timing did not trigger immediate movements, but rather caused systematic early-shifting of movement initiation, whereas inhibition caused late-shifting, as if modulating the probability of movement. Consistent with this view, the dynamics of the endogenous dopaminergic signals quantitatively predicted the moment-by-moment probability of movement initiation on single trials. We propose that ramping dopaminergic signals, likely encoding dynamic reward expectation, can modulate the decision of when to move.

Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors

Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning, signaling discrepancies between actual versus expected reward predicted by the cues.^1-5 While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. The vast majority of previous studies examining dopamine responses primarily have used discrete reward-predictive stimuli,^1-15 whether Pavlovian conditioned stimuli for which no action is required to earn reward or explicit discriminative stimuli that essentially instruct an animal how and when to respond for reward. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single and sequential instrumental actions, in behavioral contexts with minimal overt changes in the animal's external environment. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal's own action, relative to non-contingent passive stimulation. This dopamine suppression generalized to food rewards was specific to the reinforced action, was temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. These findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.

The rostromedial tegmental (RMTg) "brake" on dopamine and behavior: A decade of progress but also much unfinished work

Between 2005 and 2009, several research groups identified a strikingly dense inhibitory input to midbrain dopamine neurons arising from a previously uncharted region posterior to the ventral tegmental area (VTA). This region is now denoted as either the rostromedial tegmental nucleus (RMTg) or the "tail of the VTA" (tVTA), and is recognized to express distinct genetic markers, encode negative "prediction errors" (inverse to dopamine neurons), and play critical roles in behavioral inhibition and punishment learning. RMTg neurons are also influenced by many categories of abused drugs, and may drive some aversive responses to such drugs, particularly cocaine and alcohol. However, despite much progress, many important questions remain about RMTg molecular/genetic properties, diversity of projection targets, and applications to addiction, depression, and other neuropsychiatric disorders. This article is part of the special Issue on 'Neurocircuitry Modulating Drug and Alcohol Abuse'.

The Aversion Function of the Limbic Dopaminergic Neurons and Their Roles in Functional Neurological Disorders

The Freudian theory of conversion suggested that the major symptoms of functional neurological disorders (FNDs) are due to internal conflicts at motivation, especially at the sex drive or libido. FND patients might behave properly at rewarding situations, but they do not know how to behave at aversive situations. Sex drive is the major source of dopamine (DA) release in the limbic area; however, the neural mechanism involved in FND is not clear. Dopaminergic (DAergic) neurons have been shown to play a key role in processing motivation-related information. Recently, DAergic neurons are found to be involved in reward-related prediction error, as well as the prediction of aversive information. Therefore, it is suggested that DA might change the rewarding reactions to aversive reactions at internal conflicts of FND. So DAergic neurons in the limbic areas might induce two major motivational functions: reward and aversion at internal conflicts. This article reviewed the recent advances on studies about DAergic neurons involved in aversive stimulus processing at internal conflicts and summarizes several neural pathways, including four limbic system brain regions, which are involved in the processing of aversion. Then the article discussed the vital function of these neural circuits in addictive behavior, depression treatment, and FNDs. In all, this review provided a prospect for future research on the aversion function of limbic system DA neurons and the therapy of FNDs.

Avian pallial circuits and cognition: A comparison to mammals

Cognitive functions are similar in birds and mammals. So, are therefore pallial cellular circuits and neuronal computations also alike? In search of answers, we move in from bird's pallial connectomes, to cortex-like sensory canonical circuits and connections, to forebrain micro-circuitries and finally to the avian "prefrontal" area. This voyage from macro- to micro-scale networks and areas reveals that both birds and mammals evolved similar neural and computational properties in either convergent or parallel manner, based upon circuitries inherited from common ancestry. Thus, these two vertebrate classes evolved separately within 315 million years with highly similar pallial architectures that produce comparable cognitive functions.

The Role of the Lateral Habenula in Inhibitory Learning from Reward Omission

The lateral habenula (LHb) is a phylogenetically primitive brain structure that plays a key role in learning to inhibit distinct responses to specific stimuli. This structure is activated by primary aversive stimuli, cues predicting an imminent aversive event, unexpected reward omissions, and cues associated with the omission of an expected reward. The most widely described physiological effect of LHb activation is acutely suppressing midbrain dopaminergic signaling. However, recent studies have identified multiple means by which the LHb promotes this effect as well as other mechanisms of action. These findings reveal the complex nature of LHb circuitry. The present paper reviews the role of this structure in learning from reward omission. We approach this topic from the perspective of computational models of behavioral change that account for inhibitory learning to frame key findings. Such findings are drawn from recent behavioral neuroscience studies that use novel brain imaging, stimulation, ablation, and reversible inactivation techniques. Further research and conceptual work are needed to clarify the nature of the mechanisms related to updating motivated behavior in which the LHb is involved. As yet, there is little understanding of whether such mechanisms are parallel or complementary to the well-known modulatory function of the more recently evolved prefrontal cortex.

Convergence Circuit Mapping: Genetic Approaches From Structure to Function

Understanding the complex neural circuits that underpin brain function and behavior has been a long-standing goal of neuroscience. Yet this is no small feat considering the interconnectedness of neurons and other cell types, both within and across brain regions. In this review, we describe recent advances in mouse molecular genetic engineering that can be used to integrate information on brain activity and structure at regional, cellular, and subcellular levels. The convergence of structural inputs can be mapped throughout the brain in a cell type-specific manner by antero- and retrograde viral systems expressing various fluorescent proteins and genetic switches. Furthermore, neural activity can be manipulated using opto- and chemo-genetic tools to interrogate the functional significance of this input convergence. Monitoring neuronal activity is obtained with precise spatiotemporal resolution using genetically encoded sensors for calcium changes and specific neurotransmitters. Combining these genetically engineered mapping tools is a compelling approach for unraveling the structural and functional brain architecture of complex behaviors and malfunctioned states of neurological disorders.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Call for a more balanced approach to understanding orbital frontal cortex function

Orbital frontal cortex (OFC) research has historically emphasized the function of this associative cortical area within top-down theoretical frameworks. This approach has largely focused on mapping OFC activity onto human-defined psychological or cognitive constructs and has often led to OFC circuitry bearing the weight of entire theoretical frameworks. New techniques and tools developed in the last decade have made it possible to revisit long-standing basic science questions in neuroscience and answer them with increasing sophistication. We can now study and specify the genetic, molecular, cellular, and circuit architecture of a brain region in much greater detail, which allows us to piece together how they contribute to emergent circuit functions. For instance, adopting such systematic and unbiased bottom-up approaches to elucidating the function of the visual system has paved the way to building a greater understanding of the spectrum of its computational capabilities. In the same vein, we argue that OFC research would benefit from a more balanced approach that also places focus on novel bottom-up investigations into OFC's computational capabilities. Furthermore, we believe that the knowledge gained by employing a more bottom-up approach to investigating OFC function will ultimately allow us to look at OFC's dysfunction in disease through a more nuanced biological lens. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Dopamine, Updated: Reward Prediction Error and Beyond

Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine's role in movement, motivation, and goal-directed planning.

A Causal Role for the Pedunculopontine Nucleus in Human Instrumental Learning

A critical mechanism for maximizing reward is instrumental learning. In standard instrumental learning models, action values are updated on the basis of reward prediction errors (RPEs), defined as the discrepancy between expectations and outcomes. A wealth of evidence across species and experimental techniques has established that RPEs are signaled by midbrain dopamine neurons. However, the way dopamine neurons receive information about reward outcomes remains poorly understood. Recent animal studies suggest that the pedunculopontine nucleus (PPN), a small brainstem structure considered as a locomotor center, is sensitive to reward and sends excitatory projection to dopaminergic nuclei. Here, we examined the hypothesis that the PPN could contribute to reward learning in humans. To this aim, we leveraged a clinical protocol that assessed the therapeutic impact of PPN deep-brain stimulation (DBS) in three patients with Parkinson disease. PPN local field potentials (LFPs), recorded while patients performed an instrumental learning task, showed a specific response to reward outcomes in a low-frequency (alpha-beta) band. Moreover, PPN DBS selectively improved learning from rewards but not from punishments, a pattern that is typically observed following dopaminergic treatment. Computational analyses indicated that the effect of PPN DBS on instrumental learning was best captured by an increase in subjective reward sensitivity. Taken together, these results support a causal role for PPN-mediated reward signals in human instrumental learning.

Subcortical encoding of agent-relevant associative signals for adaptive social behavior in the macaque

Primates are group-living creatures that constantly face the challenges posed by complex social demands. To date, the cortical mechanisms underlying social information processing have been the major focus of attention. However, emerging evidence suggests that subcortical regions also mediate the collection and processing of information from other agents. Here, we review the literature supporting the hypothesis that behavioral variables important for decision-making, i.e., stimulus, action, and outcome, are associated with agent information (self and other) in subcortical regions, such as the amygdala, striatum, lateral hypothalamus, and dopaminergic midbrain nuclei. Such self-relevant and other-relevant associative signals are then integrated into a social utility signal, presumably at the level of midbrain dopamine neurons. This social utility signal allows decision makers to organize their optimal behavior in accordance with social demands. Determining how self-relevant and other-relevant signals might be altered in psychiatric and neurodevelopmental disorders will be fundamental to better understand how social behaviors are dysregulated in disease conditions.

Dopamine and relapse to drug seeking

The actions of dopamine are essential to relapse to drug seeking but we still lack a precise understanding of how dopamine achieves these effects. Here we review recent advances from animal models in understanding how dopamine controls relapse to drug seeking. These advances have been enabled by important developments in understanding the basic neurochemical, molecular, anatomical, physiological and functional properties of the major dopamine pathways in the mammalian brain. The literature shows that although different forms of relapse to seeking different drugs of abuse each depend on dopamine, there are distinct dopamine mechanisms for relapse. Different circuit-level mechanisms, different populations of dopamine neurons and different activity profiles within these dopamine neurons, are important for driving different forms of relapse. This diversity highlights the need to better understand when, where and how dopamine contributes to relapse behaviours.

Precision weighting of cortical unsigned prediction error signals benefits learning, is mediated by dopamine, and is impaired in psychosis

Recent theories of cortical function construe the brain as performing hierarchical Bayesian inference. According to these theories, the precision of prediction errors plays a key role in learning and decision-making, is controlled by dopamine and contributes to the pathogenesis of psychosis. To test these hypotheses, we studied learning with variable outcome-precision in healthy individuals after dopaminergic modulation with a placebo, a dopamine receptor agonist bromocriptine or a dopamine receptor antagonist sulpiride (dopamine study n = 59) and in patients with early psychosis (psychosis study n = 74: 20 participants with first-episode psychosis, 30 healthy controls and 24 participants with at-risk mental state attenuated psychotic symptoms). Behavioural computational modelling indicated that precision weighting of prediction errors benefits learning in health and is impaired in psychosis. FMRI revealed coding of unsigned prediction errors, which signal surprise, relative to their precision in superior frontal cortex (replicated across studies, combined n = 133), which was perturbed by dopaminergic modulation, impaired in psychosis and associated with task performance and schizotypy (schizotypy correlation in 86 healthy volunteers). In contrast to our previous work, we did not observe significant precision-weighting of signed prediction errors, which signal valence, in the midbrain and ventral striatum in the healthy controls (or patients) in the psychosis study. We conclude that healthy people, but not patients with first-episode psychosis, take into account the precision of the environment when updating beliefs. Precision weighting of cortical prediction error signals is a key mechanism through which dopamine modulates inference and contributes to the pathogenesis of psychosis.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

In vivo patch-clamp recordings reveal distinct subthreshold signatures and threshold dynamics of midbrain dopamine neurons

The in vivo firing patterns of ventral midbrain dopamine neurons are controlled by afferent and intrinsic activity to generate sensory cue and prediction error signals that are essential for reward-based learning. Given the absence of in vivo intracellular recordings during the last three decades, the subthreshold membrane potential events that cause changes in dopamine neuron firing patterns remain unknown. To address this, we established in vivo whole-cell recordings and obtained over 100 spontaneously active, immunocytochemically-defined midbrain dopamine neurons in isoflurane-anaesthetized adult mice. We identified a repertoire of subthreshold membrane potential signatures associated with distinct in vivo firing patterns. Dopamine neuron activity in vivo deviated from single-spike pacemaking by phasic increases in firing rate via two qualitatively distinct biophysical mechanisms: 1) a prolonged hyperpolarization preceding rebound bursts, accompanied by a hyperpolarizing shift in action potential threshold; and 2) a transient depolarization leading to high-frequency plateau bursts, associated with a depolarizing shift in action potential threshold. Our findings define a mechanistic framework for the biophysical implementation of dopamine neuron firing patterns in the intact brain.

Actor-critic reinforcement learning in the songbird

It feels rewarding to ace your opponent on match point. Here, we propose common mechanisms underlie reward and performance learning. First, when a singing bird unexpectedly hits the right note, its dopamine (DA) neurons are activated as when a thirsty monkey receives an unexpected juice reward. Second, these DA signals reinforce vocal variations much as they reinforce stimulus-response associations. Third, limbic inputs to DA neurons signal the predicted quality of song syllables much like they signal the predicted reward value of a place or a stimulus during foraging. Finally, songbirds may solve difficult problems in reinforcement learning - such as credit assignment and catastrophic forgetting - with node perturbation and consolidation of reinforced vocal patterns in motor cortical circuits. Consolidation occurs downstream of a canonical 'actor-critic' circuit motif that learns to maximize performance quality in essentially the same way it learns to maximize reward: by computing and learning from prediction errors.

Distributional Reinforcement Learning in the Brain

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Rethinking brain-wide interactions through multi-region 'network of networks' models

The neural control of behavior is distributed across many functionally and anatomically distinct brain regions even in small nervous systems. While classical neuroscience models treated these regions as a set of hierarchically isolated nodes, the brain comprises a recurrently interconnected network in which each region is intimately modulated by many others. Uncovering these interactions is now possible through experimental techniques that access large neural populations from many brain regions simultaneously. Harnessing these large-scale datasets, however, requires new theoretical approaches. Here, we review recent work to understand brain-wide interactions using multi-region 'network of networks' models and discuss how they can guide future experiments. We also emphasize the importance of multi-region recordings, and posit that studying individual components in isolation will be insufficient to understand the neural basis of behavior.

Reward activity in ventral pallidum tracks satiety-sensitive preference and drives choice behavior

A key function of the nervous system is producing adaptive behavior across changing conditions, like physiological state. Although states like thirst and hunger are known to impact decision-making, the neurobiology of this phenomenon has been studied minimally. Here, we tracked evolving preference for sucrose and water as rats proceeded from a thirsty to sated state. As rats shifted from water choices to sucrose choices across the session, the activity of a majority of neurons in the ventral pallidum, a region crucial for reward-related behaviors, closely matched the evolving behavioral preference. The timing of this signal followed the pattern of a reward prediction error, occurring at the cue or the reward depending on when reward identity was revealed. Additionally, optogenetic stimulation of ventral pallidum neurons at the time of reward was able to reverse behavioral preference. Our results suggest that ventral pallidum neurons guide reward-related decisions across changing physiological states.

Functional Dissection of Basal Ganglia Inhibitory Inputs onto Substantia Nigra Dopaminergic Neurons

Substantia nigra (SNc) dopaminergic neurons respond to aversive stimuli with inhibitory pauses in firing followed by transient rebound activation. We tested integration of inhibitory synaptic inputs onto SNc neurons from genetically defined populations in dorsal striatum (striosome and matrix) and external globus pallidus (GPe; parvalbumin- and Lhx6-positive), and examined their contribution to pause-rebound firing. Activation of striosome projections, which target "dendron bouquets" in the pars reticulata (SNr), consistently quiets firing and relief from striosome inhibition triggers rebound activity. Striosomal inhibitory postsynaptic currents (IPSCs) display a prominent GABA-B receptor-mediated component that strengthens the impact of SNr dendrite synapses on somatic excitability and enables rebounding. By contrast, GPe projections activate GABA-A receptors on the soma and proximal dendrites but do not result in rebounding. Lastly, optical mapping shows that dorsal striatum selectively inhibits the ventral population of SNc neurons, which are intrinsically capable of rebounding. Therefore, we define a distinct striatonigral circuit for generating dopamine rebound.

Context-Dependent Multiplexing by Individual VTA Dopamine Neurons

Dopamine (DA) neurons of the VTA track cues and rewards to generate a reward prediction error signal during Pavlovian conditioning. Here we explored how these neurons respond to a self-paced, operant task in freely moving mice. The animal could trigger a reward-predicting cue by remaining in a specific location of an operant box for a brief time before moving to a spout for reward collection. VTA DA neurons were identified using DAT-Cre male mice that carried an optrode with minimal impact on the behavioral task. In vivo single-unit recordings revealed transient fast spiking responses to the cue and reward in correct trials, while for incorrect ones the activity paused, reflecting positive and negative error signals of a reward prediction. In parallel, a majority of VTA DA neurons simultaneously encoded multiple actions (e.g., movement velocity, acceleration, distance to goal, and licking) in sustained slow firing modulation. Applying a GLM, we show that such multiplexed encoding of rewarding and motor variables by individual DA neurons was only apparent while the mouse was engaged in the task. Downstream targets may exploit such goal-directed multiplexing of VTA DA neurons to adjust actions to optimize the task's outcome.SIGNIFICANCE STATEMENT VTA DA neurons code for multiple functions, including the reward prediction error but also motivation and locomotion. Here we show that about half of the recorded VTA DA neurons perform multiplexing: they exploit the phasic and tonic activity modes to encode, respectively, the cue/reward responses and motor parameters, most prominently when the mouse engages in a self-paced operand task. VTA non-DA neurons, by contrast, encode motor parameters regardless of task engagement.

Trial-by-trial dynamics of reward prediction error-associated signals during extinction learning and renewal

Reward prediction errors (RPEs) have been suggested to drive associative learning processes, but their precise temporal dynamics at the single-neuron level remain elusive. Here, we studied the neural correlates of RPEs, focusing on their trial-by-trial dynamics during an operant extinction learning paradigm. Within a single behavioral session, pigeons went through acquisition, extinction and renewal - the context-dependent response recovery after extinction. We recorded single units from the avian prefrontal cortex analogue, the nidopallium caudolaterale (NCL) and found that the omission of reward during extinction led to a peak of population activity that moved backwards in time as trials progressed. The chronological order of these signal changes during the progress of learning was indicative of temporal shifts of RPE signals that started during reward omission and then moved backwards to the presentation of the conditioned stimulus. Switches from operant choices to avoidance behavior (and vice versa) coincided with changes in population activity during the animals' decision-making. On the single unit level, we found more diverse patterns where some neurons' activity correlated with RPE signals whereas others correlated with the absolute value during the outcome period. Finally, we demonstrated that mere sensory contextual changes during the renewal test were sufficient to elicit signals likely associated with RPEs. Thus, RPEs are truly expectancy-driven since they can be elicited by changes in reward expectation, without an actual change in the quality or quantity of reward.

The Mesolimbic Dopamine Activity Signatures of Relapse to Alcohol-Seeking

The mesolimbic dopamine system comprises distinct compartments supporting different functions in learning and motivation. Less well understood is how complex addiction-related behaviors emerge from activity patterns across these compartments. Here we show how different forms of relapse to alcohol-seeking in male rats are assembled from activity across the VTA and the nucleus accumbens. First, we used chemogenetic approaches to show a causal role for VTA TH neurons in two forms of relapse to alcohol-seeking: renewal (context-induced reinstatement) and reacquisition. Then, using gCaMP fiber photometry of VTA TH neurons, we identified medial and lateral VTA TH neuron activity profiles during self-administration, renewal, and reacquisition. Next, we used optogenetic inhibition of VTA TH neurons to show distinct causal roles for VTA subregions in distinct forms of relapse. We then used dLight fiber photometry to measure dopamine binding across the ventral striatum (medial accumbens shell, accumbens core, lateral accumbens shell) and showed complex and heterogeneous profiles of dopamine binding during self-administration and relapse. Finally, we used representational similarity analysis to identify mesolimbic dopamine signatures of self-administration, extinction, and relapse. Our results show that signatures of relapse can be identified from heterogeneous activity profiles across the mesolimbic dopamine system and that these signatures are unique for different forms of relapse.SIGNIFICANCE STATEMENT It is axiomatic that the actions of dopamine are critical to drug addiction. Yet how relapse to drug-seeking is assembled from activity across the mesolimbic dopamine system is poorly understood. Here we show how relapse to alcohol-seeking relates to activity in specific VTA and accumbens compartments, how these change for different forms of relapse, and how relapse-associated activity relates to activity during self-administration and extinction. We report the mesolimbic dopamine activity signatures for relapse and show that these signatures are unique for different forms of relapse.

A quantitative reward prediction error signal in the ventral pallidum

The nervous system is hypothesized to compute reward prediction errors (RPEs) to promote adaptive behavior. Correlates of RPEs have been observed in the midbrain dopamine system, but the extent to which RPE signals exist in other reward-processing regions is less well understood. In the present study, we quantified outcome history-based RPE signals in the ventral pallidum (VP), a basal ganglia region functionally linked to reward-seeking behavior. We trained rats to respond to reward-predicting cues, and we fit computational models to predict the firing rates of individual neurons at the time of reward delivery. We found that a subset of VP neurons encoded RPEs and did so more robustly than the nucleus accumbens, an input to the VP. VP RPEs predicted changes in task engagement, and optogenetic manipulation of the VP during reward delivery bidirectionally altered rats' subsequent reward-seeking behavior. Our data suggest a pivotal role for the VP in computing teaching signals that influence adaptive reward seeking.

Anatomic resolution of neurotransmitter-specific projections to the VTA reveals diversity of GABAergic inputs

The ventral tegmental area (VTA) is important for reward processing and motivation. The anatomic organization of neurotransmitter-specific inputs to the VTA remains poorly resolved. In the present study, we mapped the major neurotransmitter projections to the VTA through cell-type-specific retrograde and anterograde tracing. We found that glutamatergic inputs arose from a variety of sources and displayed some connectivity biases toward specific VTA cell types. The sources of GABAergic projections were more widespread, displayed a high degree of differential innervation of subregions in the VTA and were largely biased toward synaptic contact with local GABA neurons. Inactivation of GABA release from the two major sources, locally derived versus distally derived, revealed distinct roles for these projections in behavioral regulation. Optogenetic manipulation of individual distal GABAergic inputs also revealed differential behavioral effects. These results demonstrate that GABAergic projections to the VTA are a major contributor to the regulation and diversification of the structure.

Phasic dopamine reinforces distinct striatal stimulus encoding in the olfactory tubercle driving dopaminergic reward prediction

The learning of stimulus-outcome associations allows for predictions about the environment. Ventral striatum and dopaminergic midbrain neurons form a larger network for generating reward prediction signals from sensory cues. Yet, the network plasticity mechanisms to generate predictive signals in these distributed circuits have not been entirely clarified. Also, direct evidence of the underlying interregional assembly formation and information transfer is still missing. Here we show that phasic dopamine is sufficient to reinforce the distinctness of stimulus representations in the ventral striatum even in the absence of reward. Upon such reinforcement, striatal stimulus encoding gives rise to interregional assemblies that drive dopaminergic neurons during stimulus-outcome learning. These assemblies dynamically encode the predicted reward value of conditioned stimuli. Together, our data reveal that ventral striatal and midbrain reward networks form a reinforcing loop to generate reward prediction coding.

It is not entirely understood how network plasticity produces the coding of predicted value during stimulus-outcome learning. Here, the authors reveal a reinforcing loop in distributed limbic circuits, transforming sensory stimuli into reward prediction coding broadcasted by dopamine neurons to the brain.

Beyond the Average View of Dopamine

Dopamine (DA) responses are synonymous with the 'reward prediction error' of reinforcement learning (RL), and are thought to update neural estimates of expected value. A recent study by Dabney et al. enriches this picture, demonstrating that DA neurons track variability in rewards, providing a readout of risk in the brain.

Signal dynamics of midbrain dopamine neurons during economic decision-making in monkeys

When we make economic choices, the brain first evaluates available options and then decides whether to choose them. Midbrain dopamine neurons are known to reinforce economic choices through their signal evoked by outcomes after decisions are made. However, although critical internal processing is executed while decisions are being made, little is known about the role of dopamine neurons during this period. We found that dopamine neurons exhibited dynamically changing signals related to the internal processing while rhesus monkeys were making decisions. These neurons encoded the value of an option immediately after it was offered and then gradually changed their activity to represent the animal's upcoming choice. Similar dynamics were observed in the orbitofrontal cortex, a center for economic decision-making, but the value-to-choice signal transition was completed earlier in dopamine neurons. Our findings suggest that dopamine neurons are a key component of the neural network that makes choices from values during ongoing decision-making processes.

Dopamine neurons do not constitute an obligatory stage in the final common path for the evaluation and pursuit of brain stimulation reward

The neurobiological study of reward was launched by the discovery of intracranial self-stimulation (ICSS). Subsequent investigation of this phenomenon provided the initial link between reward-seeking behavior and dopaminergic neurotransmission. We re-evaluated this relationship by psychophysical, pharmacological, optogenetic, and computational means. In rats working for direct, optical activation of midbrain dopamine neurons, we varied the strength and opportunity cost of the stimulation and measured time allocation, the proportion of trial time devoted to reward pursuit. We found that the dependence of time allocation on the strength and cost of stimulation was similar formally to that observed when electrical stimulation of the medial forebrain bundle served as the reward. When the stimulation is strong and cheap, the rats devote almost all their time to reward pursuit; time allocation falls off as stimulation strength is decreased and/or its opportunity cost is increased. A 3D plot of time allocation versus stimulation strength and cost produces a surface resembling the corner of a plateau (the “reward mountain”). We show that dopamine-transporter blockade shifts the mountain along both the strength and cost axes in rats working for optical activation of midbrain dopamine neurons. In contrast, the same drug shifted the mountain uniquely along the opportunity-cost axis when rats worked for electrical MFB stimulation in a prior study. Dopamine neurons are an obligatory stage in the dominant model of ICSS, which positions them at a key nexus in the final common path for reward seeking. This model fails to provide a cogent account for the differential effect of dopamine transporter blockade on the reward mountain. Instead, we propose that midbrain dopamine neurons and neurons with non-dopaminergic, MFB axons constitute parallel limbs of brain-reward circuitry that ultimately converge on the final-common path for the evaluation and pursuit of rewards.

Heterogeneity in striatal dopamine circuits: Form and function in dynamic reward seeking

The striatal dopamine system has long been studied in the context of reward learning, motivation, and movement. Given the prominent role dopamine plays in a variety of adaptive behavioral states, as well as diseases like addiction, it is essential to understand the full complexity of dopamine neurons and the striatal systems they target. A growing number of studies are uncovering details of the heterogeneity in dopamine neuron subpopulations. Here, we review that work to synthesize current understanding of dopamine system heterogeneity across three levels, anatomical organization, functions in behavior, and modes of action, wherein we focus on signaling profiles and local mechanisms for modulation of dopamine release. Together, these studies reveal new and emerging dimensions of the striatal dopamine system, informing its contribution to dynamic motivational and decision-making processes.

Cholinergic midbrain afferents modulate striatal circuits and shape encoding of action strategies

Assimilation of novel strategies into a consolidated action repertoire is a crucial function for behavioral adaptation and cognitive flexibility. Acetylcholine in the striatum plays a pivotal role in such adaptation, and its release has been causally associated with the activity of cholinergic interneurons. Here we show that the midbrain, a previously unknown source of acetylcholine in the striatum, is a major contributor to cholinergic transmission in the striatal complex. Neurons of the pedunculopontine and laterodorsal tegmental nuclei synapse with striatal cholinergic interneurons and give rise to excitatory responses. Furthermore, they produce uniform inhibition of spiny projection neurons. Inhibition of acetylcholine release from midbrain terminals in the striatum impairs the association of contingencies and the formation of habits in an instrumental task, and mimics the effects observed following inhibition of acetylcholine release from striatal cholinergic interneurons. These results suggest the existence of two hierarchically-organized modes of cholinergic transmission in the striatum, where cholinergic interneurons are modulated by cholinergic neurons of the midbrain.

Recurrent architecture for adaptive regulation of learning in the insect brain

Dopaminergic neurons (DANs) drive learning across the animal kingdom, but the upstream circuits that regulate their activity and thereby learning remain poorly understood. We provide a synaptic-resolution connectome of the circuitry upstream of all DANs in a learning center, the mushroom body of Drosophila larva. We discover afferent sensory pathways and a large population of neurons that provide feedback from mushroom body output neurons and link distinct memory systems (aversive and appetitive). We combine this with functional studies of DANs and their presynaptic partners and with comprehensive circuit modeling. We find that DANs compare convergent feedback from aversive and appetitive systems, which enables the computation of integrated predictions that may improve future learning. Computational modeling reveals that the discovered feedback motifs increase model flexibility and performance on learning tasks. Our study provides the most detailed view to date of biological circuit motifs that support associative learning.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

Temporally restricted dopaminergic control of reward-conditioned movements

Midbrain dopamine (DA) neurons encode both reward- and movement-related events and are implicated in disorders of reward processing as well as movement. Consequently, disentangling the contribution of DA neurons in reinforcing versus generating movements is challenging and has led to lasting controversy. In this study, we dissociated these functions by parametrically varying the timing of optogenetic manipulations in a Pavlovian conditioning task and examining the influence on anticipatory licking before reward delivery. Inhibiting both ventral tegmental area and substantia nigra pars compacta DA neurons in the post-reward period had a significantly greater behavioral effect than inhibition in the pre-reward period of the task. Furthermore, the contribution of DA neurons to behavior decreased linearly as a function of elapsed time after reward. Together, the results indicate a temporally restricted role of DA neurons primarily related to reinforcing stimulus-reward associations and suggest that directly generating movements is a comparatively less important function.

Opposing Contributions of GABAergic and Glutamatergic Ventral Pallidal Neurons to Motivational Behaviors

The ventral pallidum (VP) is critical for invigorating reward seeking and is also involved in punishment avoidance, but how it contributes to such opposing behavioral actions remains unclear. Here, we show that GABAergic and glutamatergic VP neurons selectively control behavior in opposing motivational contexts. In vivo recording combined with optogenetics in mice revealed that these two populations oppositely encode positive and negative motivational value, are differentially modulated by animal's internal state, and determine the behavioral response during motivational conflict. Furthermore, GABAergic VP neurons are essential for movements toward reward in a positive motivational context but suppress movements in an aversive context. In contrast, glutamatergic VP neurons are essential for movements to avoid a threat but suppress movements in an appetitive context. Our results indicate that GABAergic and glutamatergic VP neurons encode the drive for approach and avoidance, respectively, with the balance between their activities determining the type of motivational behavior.

Multiplex Neural Circuit Tracing With G-Deleted Rabies Viral Vectors

Neural circuits interconnect to organize large-scale networks that generate perception, cognition, memory, and behavior. Information in the nervous system is processed both through parallel, independent circuits and through intermixing circuits. Analyzing the interaction between circuits is particularly indispensable for elucidating how the brain functions. Monosynaptic circuit tracing with glycoprotein (G) gene-deleted rabies viral vectors (RVΔG) comprises a powerful approach for studying the structure and function of neural circuits. Pseudotyping of RVΔG with the foreign envelope EnvA permits expression of transgenes such as fluorescent proteins, genetically-encoded sensors, or optogenetic tools in cells expressing TVA, a cognate receptor for EnvA. Trans-complementation with rabies virus glycoproteins (RV-G) enables trans-synaptic labeling of input neurons directly connected to the starter neurons expressing both TVA and RV-G. However, it remains challenging to simultaneously map neuronal connections from multiple cell populations and their interactions between intermixing circuits solely with the EnvA/TVA-mediated RV tracing system in a single animal. To overcome this limitation, here, we multiplexed RVΔG circuit tracing by optimizing distinct viral envelopes (oEnvX) and their corresponding receptors (oTVX). Based on the EnvB/TVB and EnvE/DR46-TVB systems derived from the avian sarcoma leukosis virus (ASLV), we developed optimized TVB receptors with lower or higher affinity (oTVB-L or oTVB-H) and the chimeric envelope oEnvB, as well as an optimized TVE receptor with higher affinity (oTVE-H) and its chimeric envelope oEnvE. We demonstrated independence of RVΔG infection between the oEnvA/oTVA, oEnvB/oTVB, and oEnvE/oTVE systems and in vivo proof-of-concept for multiplex circuit tracing from two distinct classes of layer 5 neurons targeting either other cortical or subcortical areas. We also successfully labeled common input of the lateral geniculate nucleus to both cortico-cortical layer 5 neurons and inhibitory neurons of the mouse V1 with multiplex RVΔG tracing. These oEnvA/oTVA, oEnvB/oTVB, and oEnvE/oTVE systems allow for differential labeling of distinct circuits to uncover the mechanisms underlying parallel processing through independent circuits and integrated processing through interaction between circuits in the brain.

Phasic dopamine responses to a food-predictive cue are suppressed by the glucagon-like peptide-1 receptor agonist Exendin-4

Phasic dopamine activity is evoked by reliable predictors of food reward and plays a role in cue-triggered, goal-directed behavior. While this important signal is modulated by physiological state (e.g. hunger, satiety), the mechanisms by which physiological state is integrated by dopamine neurons is only beginning to be elucidated. Activation of central receptors for glucagon-like peptide-1 (GLP-1R) via long-acting agonists (e.g., Exendin-4) suppresses food intake and food-directed motivated behavior, in part, through action in regions with dopamine cell bodies, terminals, and/or neural populations that directly target the mesolimbic dopamine system. However, the effects of GLP-1R activation on cue-evoked, phasic dopamine signaling remain unknown. Here, in vivo fiber photometry was used to capture real-time signaling dynamics selectively from dopamine neurons in the ventral tegmental area of male and female transgenic (tyrosine hydroxylase-Cre; TH:Cre+) rats trained to associate an audio cue with the brief availability of a sucrose solution. Cue presentation evoked a brief spike in dopamine activity. Administration of Exendin-4 (Ex4; 0, 0.05, 0.1 μg) to the lateral ventricle both dose-dependently suppressed sucrose-directed behaviors and the magnitude of cue-evoked dopamine activity. Moreover, the amplitude of cue evoked dopamine activity was significantly correlated with subsequent sucrose-directed behaviors. While female rats exhibited overall reduced dopamine responses to the sucrose-paired cue relative to males, there was no significant interaction with Ex4. Together, these findings support a role for central GLP-1Rs in modulating a form of dopamine signaling that influences approach behavior and provide a potential mechanism whereby GLP-1 suppresses food-directed behaviors.

Heterogeneity of dopamine release sites in health and degeneration

Despite comprising only ~ 0.001% of all neurons in the human brain, ventral midbrain dopamine neurons exert a profound influence on human behavior and cognition. As a neuromodulator, dopamine selectively inhibits or enhances synaptic signaling to coordinate neural output for action, attention, and affect. Humans invariably lose brain dopamine during aging, and this can be exacerbated in disease states such as Parkinson's Disease. Further, it is well established in multiple disease states that cell loss is selective for a subset of highly sensitive neurons within the nigrostriatal dopamine tract. Regional differences in dopamine tone are regulated pre-synaptically, with subcircuits of projecting dopamine neurons exhibiting distinct molecular and physiological signatures. Specifically, proteins at dopamine release sites that synthesize and package cytosolic dopamine, modulate its release and reuptake, and alter neuronal excitability show regional differences that provide linkages to the observed sensitivity to neurodegeneration. The aim of this review is to outline the major components of dopamine homeostasis at neurotransmitter release sites and describe the regional differences most relevant to understanding why some, but not all, dopamine neurons exhibit heightened vulnerability to neurodegeneration.

Redefining Noradrenergic Neuromodulation of Behavior: Impacts of a Modular Locus Coeruleus Architecture

The locus coeruleus (LC) is a seemingly singular and compact neuromodulatory nucleus that is a prominent component of disparate theories of brain function due to its broad noradrenergic projections throughout the CNS. As a diffuse neuromodulatory system, noradrenaline affects learning and decision making, control of sleep and wakefulness, sensory salience including pain, and the physiology of correlated forebrain activity (ensembles and networks) and brain hemodynamic responses. However, our understanding of the LC is undergoing a dramatic shift due to the application of state-of-the-art methods that reveal a nucleus of many modules that provide targeted neuromodulation. Here, we review the evidence supporting a modular LC based on multiple levels of observation (developmental, genetic, molecular, anatomical, and neurophysiological). We suggest that the concept of the LC as a singular nucleus and, alongside it, the role of the LC in diverse theories of brain function must be reconsidered.

Three Rostromedial Tegmental Afferents Drive Triply Dissociable Aspects of Punishment Learning and Aversive Valence Encoding

Persistence of reward seeking despite punishment or other negative consequences is a defining feature of mania and addiction, and numerous brain regions have been implicated in such punishment learning, but in disparate ways that are difficult to reconcile. We now show that the ability of an aversive punisher to inhibit reward seeking depends on coordinated activity of three distinct afferents to the rostromedial tegmental nucleus (RMTg) arising from cortex, brainstem, and habenula that drive triply dissociable RMTg responses to aversive cues, outcomes, and prediction errors, respectively. These three pathways drive correspondingly dissociable aspects of punishment learning. The RMTg in turn drives negative, but not positive, valence encoding patterns in the ventral tegmental area (VTA). Hence, punishment learning involves pathways and functions that are highly distinct, yet tightly coordinated.

In vivo functional diversity of midbrain dopamine neurons within identified axonal projections

Functional diversity of midbrain dopamine (DA) neurons ranges across multiple scales, from differences in intrinsic properties and connectivity to selective task engagement in behaving animals. Distinct in vitro biophysical features of DA neurons have been associated with different axonal projection targets. However, it is unknown how this translates to different firing patterns of projection-defined DA subpopulations in the intact brain. We combined retrograde tracing with single-unit recording and labelling in mouse brain to create an in vivo functional topography of the midbrain DA system. We identified differences in burst firing among DA neurons projecting to dorsolateral striatum. Bursting also differentiated DA neurons in the medial substantia nigra (SN) projecting either to dorsal or ventral striatum. We found differences in mean firing rates and pause durations among ventral tegmental area (VTA) DA neurons projecting to lateral or medial shell of nucleus accumbens. Our data establishes a high-resolution functional in vivo landscape of midbrain DA neurons.

Expectancy-Related Changes in Dopaminergic Error Signals Are Impaired by Cocaine Self-Administration

Addiction is a disorder of behavioral control and learning. While this may reflect pre-existing propensities, drug use also clearly contributes by causing changes in outcome processing in prefrontal and striatal regions. This altered processing is associated with behavioral deficits, including changes in learning. These areas provide critical input to midbrain dopamine neurons regarding expected outcomes, suggesting that effects on learning may result from changes in dopaminergic error signaling. Here, we show that dopamine neurons recorded in rats that had self-administered cocaine failed to suppress firing on omission of an expected reward and exhibited lower amplitude and imprecisely timed increases in firing to an unexpected reward. Learning also appeared to have less of an effect on reward-evoked and cue-evoked firing in the cocaine-experienced rats. Overall, the changes are consistent with reduced fidelity of input regarding the expected outcomes, such as their size, timing, and overall value, because of cocaine use.

Subcortical Substrates of Explore-Exploit Decisions in Primates

The explore-exploit dilemma refers to the challenge of deciding when to forego immediate rewards and explore new opportunities that could lead to greater rewards in the future. While motivational neural circuits facilitate learning based on past choices and outcomes, it is unclear whether they also support computations relevant for deciding when to explore. We recorded neural activity in the amygdala and ventral striatum of rhesus macaques as they solved a task that required them to balance novelty-driven exploration with exploitation of what they had already learned. Using a partially observable Markov decision process (POMDP) model to quantify explore-exploit trade-offs, we identified that the ventral striatum and amygdala differ in how they represent the immediate value of exploitative choices and the future value of exploratory choices. These findings show that subcortical motivational circuits are important in guiding explore-exploit decisions.

Dichotomy between motor and cognitive functions of midbrain cholinergic neurons

Cholinergic neurons of the pedunculopontine nucleus (PPN) are interconnected with all the basal ganglia structures, as well as with motor centers in the brainstem and medulla. Recent theories put into question whether PPN cholinergic neurons form part of a locomotor region that directly regulates the motor output, and rather suggest a modulatory role in adaptive behavior involving both motor and cognitive functions. In support of this, experimental studies in animals suggest that cholinergic neurons reinforce actions by signaling reward prediction and shape adaptations in behavior during changes of environmental contingencies. This is further supported by clinical studies proposing that decreased cholinergic transmission originated in the PPN is associated with impaired sensorimotor integration and perseverant behavior, giving rise to some of the symptoms observed in Parkinson's disease and progressive supranuclear palsy. Altogether, the evidence suggests that cholinergic neurons of the PPN, mainly through their interactions with the basal ganglia, have a leading role in action control.

A Paranigral VTA Nociceptin Circuit that Constrains Motivation for Reward

Nociceptin and its receptor are widely distributed throughout the brain in regions associated with reward behavior, yet how and when they act is unknown. Here, we dissected the role of a nociceptin peptide circuit in reward seeking. We generated a prepronociceptin (Pnoc)-Cre mouse line that revealed a unique subpopulation of paranigral ventral tegmental area (pnVTA) neurons enriched in prepronociceptin. Fiber photometry recordings during progressive ratio operant behavior revealed pnVTAPnoc neurons become most active when mice stop seeking natural rewards. Selective pnVTAPnoc neuron ablation, inhibition, and conditional VTA nociceptin receptor (NOPR) deletion increased operant responding, revealing that the pnVTAPnoc nucleus and VTA NOPR signaling are necessary for regulating reward motivation. Additionally, optogenetic and chemogenetic activation of this pnVTAPnoc nucleus caused avoidance and decreased motivation for rewards. These findings provide insight into neuromodulatory circuits that regulate motivated behaviors through identification of a previously unknown neuropeptide-containing pnVTA nucleus that limits motivation for rewards.