Opposite initialization to novel cues in dopamine signaling in ventral and posterior striatum in mice

Interactions between ventrolateral prefrontal and anterior cingulate cortex during learning and behavioural change

Hypotheses and beliefs guide credit assignment - the process of determining which previous events or actions caused an outcome. Adaptive hypothesis formation and testing are crucial in uncertain and changing environments in which associations and meanings are volatile. Despite primates' abilities to form and test hypotheses, establishing what is causally responsible for the occurrence of particular outcomes remains a fundamental challenge for credit assignment and learning. Hypotheses about what surprises are due to stochasticity inherent in an environment as opposed to real, systematic changes are necessary for identifying the environment's predictive features, but are often hard to test. We review evidence that two highly interconnected frontal cortical regions, anterior cingulate cortex and ventrolateral prefrontal area 47/12o, provide a biological substrate for linking two crucial components of hypothesis-formation and testing: the control of information seeking and credit assignment. Neuroimaging, targeted disruptions, and neurophysiological studies link an anterior cingulate - 47/12o circuit to generation of exploratory behaviour, non-instrumental information seeking, and interpretation of subsequent feedback in the service of credit assignment. Our observations support the idea that information seeking and credit assignment are linked at the level of neural circuits and explain why this circuit is important for ensuring behaviour is flexible and adaptive.

A primate temporal cortex-zona incerta pathway for novelty seeking

Primates interact with the world by exploring visual objects; they seek opportunities to view novel objects even when these have no extrinsic reward value. How the brain controls this novelty seeking is unknown. Here we show that novelty seeking in monkeys is regulated by the zona incerta (ZI). As monkeys made eye movements to familiar objects to trigger an opportunity to view novel objects, many ZI neurons were preferentially activated by predictions of novel objects before the gaze shift. Low-intensity ZI stimulation facilitated gaze shifts, whereas ZI inactivation reduced novelty seeking. ZI-dependent novelty seeking was not regulated by neurons in the lateral habenula or by many dopamine neurons in the substantia nigra, traditionally associated with reward seeking. But the anterior ventral medial temporal cortex, an area important for object vision and memory, was a prominent source of novelty predictions. These data uncover a functional pathway in the primate brain that regulates novelty seeking.

Smoking as a Common Modulator of Sensory Gating and Reward Learning in Individuals with Psychotic Disorders

Motivational and perceptual disturbances co-occur in psychosis and have been linked to aberrations in reward learning and sensory gating, respectively. Although traditionally studied independently, when viewed through a predictive coding framework, these processes can both be linked to dysfunction in striatal dopaminergic prediction error signaling. This study examined whether reward learning and sensory gating are correlated in individuals with psychotic disorders, and whether nicotine—a psychostimulant that amplifies phasic striatal dopamine firing—is a common modulator of these two processes. We recruited 183 patients with psychotic disorders (79 schizophrenia, 104 psychotic bipolar disorder) and 129 controls and assessed reward learning (behavioral probabilistic reward task), sensory gating (P50 event-related potential), and smoking history. Reward learning and sensory gating were correlated across the sample. Smoking influenced reward learning and sensory gating in both patient groups; however, the effects were in opposite directions. Specifically, smoking was associated with improved performance in individuals with schizophrenia but impaired performance in individuals with psychotic bipolar disorder. These findings suggest that reward learning and sensory gating are linked and modulated by smoking. However, disorder-specific associations with smoking suggest that nicotine may expose pathophysiological differences in the architecture and function of prediction error circuitry in these overlapping yet distinct psychotic disorders.

The Aversion Function of the Limbic Dopaminergic Neurons and Their Roles in Functional Neurological Disorders

The Freudian theory of conversion suggested that the major symptoms of functional neurological disorders (FNDs) are due to internal conflicts at motivation, especially at the sex drive or libido. FND patients might behave properly at rewarding situations, but they do not know how to behave at aversive situations. Sex drive is the major source of dopamine (DA) release in the limbic area; however, the neural mechanism involved in FND is not clear. Dopaminergic (DAergic) neurons have been shown to play a key role in processing motivation-related information. Recently, DAergic neurons are found to be involved in reward-related prediction error, as well as the prediction of aversive information. Therefore, it is suggested that DA might change the rewarding reactions to aversive reactions at internal conflicts of FND. So DAergic neurons in the limbic areas might induce two major motivational functions: reward and aversion at internal conflicts. This article reviewed the recent advances on studies about DAergic neurons involved in aversive stimulus processing at internal conflicts and summarizes several neural pathways, including four limbic system brain regions, which are involved in the processing of aversion. Then the article discussed the vital function of these neural circuits in addictive behavior, depression treatment, and FNDs. In all, this review provided a prospect for future research on the aversion function of limbic system DA neurons and the therapy of FNDs.

Dopamine facilitates associative memory encoding in the entorhinal cortex

Mounting evidence shows that dopamine in the striatum is critically involved in reward-based reinforcement learning^1,2. However, it remains unclear how dopamine reward signals influence the entorhinal-hippocampal circuit, another brain network that is crucial for learning and memory^3-5. Here, using cell-type-specific electrophysiological recording⁶, we show that dopamine signals from the ventral tegmental area and substantia nigra control the encoding of cue-reward association rules in layer 2a fan cells of the lateral entorhinal cortex (LEC). When mice learned novel olfactory cue-reward associations using a pre-learned association rule, spike representations of LEC fan cells grouped newly learned rewarded cues with a pre-learned rewarded cue, but separated them from a pre-learned unrewarded cue. Optogenetic inhibition of fan cells impaired the learning of new associations while sparing the retrieval of pre-learned memory. Using fibre photometry, we found that dopamine sends novelty-induced reward expectation signals to the LEC. Inhibition of LEC dopamine signals disrupted the associative encoding of fan cells and impaired learning performance. These results suggest that LEC fan cells represent a cognitive map of abstract task rules, and that LEC dopamine facilitates the incorporation of new memories into this map.

Cell Type-Specific Membrane Potential Changes in Dorsolateral Striatum Accompanying Reward-Based Sensorimotor Learning

The striatum integrates sensorimotor and motivational signals, likely playing a key role in reward-based learning of goal-directed behavior. However, cell type-specific mechanisms underlying reinforcement learning remain to be precisely determined. Here, we investigated changes in membrane potential dynamics of dorsolateral striatal neurons comparing naïve mice and expert mice trained to lick a reward spout in response to whisker deflection. We recorded from three distinct cell types: (i) direct pathway striatonigral neurons, which express type 1 dopamine receptors; (ii) indirect pathway striatopallidal neurons, which express type 2 dopamine receptors; and (iii) tonically active, putative cholinergic, striatal neurons. Task learning was accompanied by cell type-specific changes in the membrane potential dynamics evoked by the whisker deflection and licking in successfully-performed trials. Both striatonigral and striatopallidal types of striatal projection neurons showed enhanced task-related depolarization across learning. Striatonigral neurons showed a prominent increase in a short latency sensory-evoked depolarization in expert compared to naïve mice. In contrast, the putative cholinergic striatal neurons developed a hyperpolarizing response across learning, driving a pause in their firing. Our results reveal cell type-specific changes in striatal membrane potential dynamics across the learning of a simple goal-directed sensorimotor transformation, helpful for furthering the understanding of the various potential roles of different basal ganglia circuits.

Diverse midbrain dopaminergic neuron subtypes and implications for complex clinical symptoms of Parkinson's disease

Parkinson’s disease (PD), the most common degenerative movement disorder, is clinically manifested with various motor and non-motor symptoms. Degeneration of midbrain substantia nigra pas compacta (SNc) dopaminergic neurons (DANs) is generally attributed to the motor syndrome. The underlying neuronal mechanisms of non-motor syndrome are largely unexplored. Besides SNc, midbrain ventral tegmental area (VTA) DANs also produce and release dopamine and modulate movement, reward, motivation, and memory. Degeneration of VTA DANs also occurs in postmortem brains of PD patients, implying an involvement of VTA DANs in PD-associated non-motor symptoms. However, it remains to be established that there is a distinct segregation of different SNc and VTA DAN subtypes in regulating different motor and non-motor functions, and that different DAN subpopulations are differentially affected by normal ageing or PD. Traditionally, the distinction among different DAN subtypes was mainly based on the location of cell bodies and axon terminals. With the recent advance of single cell RNA sequencing technology, DANs can be readily classified based on unique gene expression profiles. A combination of specific anatomic and molecular markers shows great promise to facilitate the identification of DAN subpopulations corresponding to different behavior modules under normal and disease conditions. In this review, we first summarize the recent progress in characterizing genetically, anatomically, and functionally diverse midbrain DAN subtypes. Then, we provide perspectives on how the preclinical research on the connectivity and functionality of DAN subpopulations improves our current understanding of cell-type and circuit specific mechanisms of the disease, which could be critically informative for designing new mechanistic treatments.

Hypothalamic dopamine neurons motivate mating through persistent cAMP signalling

Transient neuromodulation can have long-lasting effects on neural circuits and motivational states^1-4. Here we examine the dopaminergic mechanisms that underlie mating drive and its persistence in male mice. Brief investigation of females primes a male's interest to mate for tens of minutes, whereas a single successful mating triggers satiety that gradually recovers over days⁵. We found that both processes are controlled by specialized anteroventral and preoptic periventricular (AVPV/PVpo) dopamine neurons in the hypothalamus. During the investigation of females, dopamine is transiently released in the medial preoptic area (MPOA)-an area that is critical for mating behaviours. Optogenetic stimulation of AVPV/PVpo dopamine axons in the MPOA recapitulates the priming effect of exposure to a female. Using optical and molecular methods for tracking and manipulating intracellular signalling, we show that this priming effect emerges from the accumulation of mating-related dopamine signals in the MPOA through the accrual of cyclic adenosine monophosphate levels and protein kinase A activity. Dopamine transients in the MPOA are abolished after a successful mating, which is likely to ensure abstinence. Consistent with this idea, the inhibition of AVPV/PVpo dopamine neurons selectively demotivates mating, whereas stimulating these neurons restores the motivation to mate after sexual satiety. We therefore conclude that the accumulation or suppression of signals from specialized dopamine neurons regulates mating behaviours across minutes and days.

Models of heterogeneous dopamine signaling in an insect learning and memory center

The Drosophila mushroom body exhibits dopamine dependent synaptic plasticity that underlies the acquisition of associative memories. Recordings of dopamine neurons in this system have identified signals related to external reinforcement such as reward and punishment. However, other factors including locomotion, novelty, reward expectation, and internal state have also recently been shown to modulate dopamine neurons. This heterogeneity is at odds with typical modeling approaches in which these neurons are assumed to encode a global, scalar error signal. How is dopamine dependent plasticity coordinated in the presence of such heterogeneity? We develop a modeling approach that infers a pattern of dopamine activity sufficient to solve defined behavioral tasks, given architectural constraints informed by knowledge of mushroom body circuitry. Model dopamine neurons exhibit diverse tuning to task parameters while nonetheless producing coherent learned behaviors. Notably, reward prediction error emerges as a mode of population activity distributed across these neurons. Our results provide a mechanistic framework that accounts for the heterogeneity of dopamine activity during learning and behavior.

Dopamine neurons across the animal kingdom are involved in the formation of associative memories. While numerous studies have recorded activity in these neurons related to external and predicted rewards, the diversity of these neurons’ activity and their tuning to non-reward-related quantities such as novelty, movement, and internal state have proved challenging to account for in traditional modeling approaches. Using a well-characterized model system for learning and memory, the mushroom body of Drosophila fruit flies, Jiang and Litwin-Kumar provide an account of the diversity of signals across dopamine neurons. They show that models optimized to solve tasks like those encountered by flies exhibit heterogeneous activity across dopamine neurons, but nonetheless this activity is sufficient for the system to solve the tasks. The models will be useful to generate testable hypotheses about dopamine neuron activity across different experimental conditions.

Basic principles of hydrogel-based tissue transformation technologies and their applications

Emerging tissue transformation technologies provide an unprecedented opportunity to investigate system-level molecular and anatomical features in situ. Hydrogel-based methods engineer physicochemical tissue properties to render intact organs optically transparent and size and shape adjustable while preserving biomolecules at their physiological locations. When combined with advanced molecular tools, labeling, and imaging techniques, tissue transformation enables three-dimensional (3D) mapping of molecules, cells, and their interrelationships at increasing speeds and resolutions. In this review, we discuss the basic engineering principles of tissue transformation and labeling techniques as well as their broad applications, current challenges, and future potential.

Integrating the Roles of Midbrain Dopamine Circuits in Behavior and Neuropsychiatric Disease

Dopamine (DA) is a behaviorally and clinically diverse neuromodulator that controls CNS function. DA plays major roles in many behaviors including locomotion, learning, habit formation, perception, and memory processing. Reflecting this, DA dysregulation produces a wide variety of cognitive symptoms seen in neuropsychiatric diseases such as Parkinson’s, Schizophrenia, addiction, and Alzheimer’s disease. Here, we review recent advances in the DA systems neuroscience field and explore the advancing hypothesis that DA’s behavioral function is linked to disease deficits in a neural circuit-dependent manner. We survey different brain areas including the basal ganglia’s dorsomedial/dorsolateral striatum, the ventral striatum, the auditory striatum, and the hippocampus in rodent models. Each of these regions have different reported functions and, correspondingly, DA’s reflecting role in each of these regions also has support for being different. We then focus on DA dysregulation states in Parkinson’s disease, addiction, and Alzheimer’s Disease, emphasizing how these afflictions are linked to different DA pathways. We draw upon ideas such as selective vulnerability and region-dependent physiology. These bodies of work suggest that different channels of DA may be dysregulated in different sets of disease. While these are great advances, the fine and definitive segregation of such pathways in behavior and disease remains to be seen. Future studies will be required to define DA’s necessity and contribution to the functional plasticity of different striatal regions.

Wave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment

Significant evidence supports the view that dopamine shapes learning by encoding reward prediction errors. However, it is unknown whether striatal targets receive tailored dopamine dynamics based on regional functional specialization. Here, we report wave-like spatiotemporal activity patterns in dopamine axons and release across the dorsal striatum. These waves switch between activational motifs and organize dopamine transients into localized clusters within functionally related striatal subregions. Notably, wave trajectories were tailored to task demands, propagating from dorsomedial to dorsolateral striatum when rewards are contingent on animal behavior and in the opponent direction when rewards are independent of behavioral responses. We propose a computational architecture in which striatal dopamine waves are sculpted by inference about agency and provide a mechanism to direct credit assignment to specialized striatal subregions. Supporting model predictions, dorsomedial dopamine activity during reward-pursuit signaled the extent of instrumental control and interacted with reward waves to predict future behavioral adjustments.

Interpreting the role of the striatum during multiple phases of motor learning

The synaptic pathways in the striatum are central to basal ganglia functions including motor control, learning and organization, action selection, acquisition of motor skills, cognitive function, and emotion. Here, we review the role of the striatum and its connections in motor learning and performance. The development of new techniques to record neuronal activity and animal models of motor disorders using neurotoxin, pharmacological, and genetic manipulations are revealing pathways that underlie motor performance and motor learning, as well as how they are altered by pathophysiological mechanisms. We discuss approaches that can be used to analyze complex motor skills, particularly in rodents, and identify specific questions central to understanding how striatal circuits mediate motor learning.

Positive and negative prediction error signals to violated expectations of face and place stimuli distinctively activate FFA and PPA

Surprising scenarios can have different behavioural and neuronal consequences depending on the violation of the expectation. On the one hand, previous research has shown that the omission of a visual stimulus results in a robust cortical response representing that missing stimulus, a so-called negative prediction error. On the other hand, a large amount of studies revealed positive prediction error signals, entailing an increased neural response that can be attributed to the experience of a surprising, unexpected stimulus. However, there has been no evidence, so far, regarding how and when these prediction error signals co-occur. Here, we argue that the omission of an expected stimulus can and often does coincide with the appearance of an unexpected one. Therefore, we investigated whether positive and negative prediction error signals evoked by unpredicted cross-category stimulus transitions would temporally coincide during a speeded forced-choice fMRI paradigm. Foremost, our findings provide evidence of a behavioural effect regarding the facilitation of responses linked to expected stimuli. In addition, we obtained evidence for negative prediction error signals as seen in differential activation of FFA and PPA during unexpected place and face trials, respectively. Lastly, a psychophysiological interaction analysis revealed evidence for positive prediction error signals represented by context-dependent functional coupling between the right IFG and FFA or PPA, respectively, implicating a network that updates the internal representation after the appearance of an unexpected stimulus through involvement of this frontal area. The current results are consistent with a predictive coding account of cognition and underline the importance of considering the potential dual nature of expectation violations. Furthermore, our results put forward that positive and negative prediction error signalling can be directly linked to regions associated with the processing of different stimulus categories.

Bi-directional encoding of context-based odors and behavioral states by the nucleus of the lateral olfactory tract

The nucleus of the lateral olfactory tract (NLOT) is not only a part of the olfactory cortex that receives olfactory sensory inputs but also a part of the cortical amygdala, which regulates motivational behaviors. To examine how neural activity of the NLOT is modulated by decision-making processes that occur during various states of learned goal-directed behaviors, we recorded NLOT spike activities of mice performing odor-guided go/no-go tasks to obtain a water reward. We observed that several NLOT neurons exhibited sharp go-cue excitation and persistent no-go-cue suppression responses triggered by an odor onset. The bidirectional cue encoding introduced NLOT population response dynamics and provided a high odor decoding accuracy before executing cue-odor-evoked behaviors. The go-cue responsive neurons were also activated in the reward drinking state, indicating context-based odor-outcome associations. These findings suggest that NLOT neurons play an important role in the translation from context-based odor information to appropriate behavior.

•

We recorded NLOT spike activities in the odor-guided goal-directed behaviors

•

NLOT neurons were classified into five response types in the odor-sampling epoch

•

Many NLOT neurons exhibited go-cue excitation and no-go-cue suppression responses

•

The bidirectional responsive neurons were also activated in the reward drinking

Striatal dopamine mediates hallucination-like perception in mice

Hallucinations, a central symptom of psychotic disorders, are attributed to excessive dopamine in the brain. However, the neural circuit mechanisms by which dopamine produces hallucinations remain elusive, largely because hallucinations have been challenging to study in model organisms. We developed a task to quantify hallucination-like perception in mice. Hallucination-like percepts, defined as high-confidence false detections, increased after hallucination-related manipulations in mice and correlated with self-reported hallucinations in humans. Hallucination-like percepts were preceded by elevated striatal dopamine levels, could be induced by optogenetic stimulation of mesostriatal dopamine neurons, and could be reversed by the antipsychotic drug haloperidol. These findings reveal a causal role for dopamine-dependent striatal circuits in hallucination-like perception and open new avenues to develop circuit-based treatments for psychotic disorders.

What is dopamine doing in model-based reinforcement learning?

Experiments have implicated dopamine in model-based reinforcement learning (RL). These findings are unexpected as dopamine is thought to encode a reward prediction error (RPE), which is the key teaching signal in model-free RL. Here we examine two possible accounts for dopamine's involvement in model-based RL: the first that dopamine neurons carry a prediction error used to update a type of predictive state representation called a successor representation, the second that two well established aspects of dopaminergic activity, RPEs and surprise signals, can together explain dopamine's involvement in model-based RL.

Dopamine, Updated: Reward Prediction Error and Beyond

Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine's role in movement, motivation, and goal-directed planning.

Dopamine: The Neuromodulator of Long-Term Synaptic Plasticity, Reward and Movement Control

Dopamine (DA) is a key neurotransmitter involved in multiple physiological functions including motor control, modulation of affective and emotional states, reward mechanisms, reinforcement of behavior, and selected higher cognitive functions. Dysfunction in dopaminergic transmission is recognized as a core alteration in several devastating neurological and psychiatric disorders, including Parkinson's disease (PD), schizophrenia, bipolar disorder, attention deficit hyperactivity disorder (ADHD) and addiction. Here we will discuss the current insights on the role of DA in motor control and reward learning mechanisms and its involvement in the modulation of synaptic dynamics through different pathways. In particular, we will consider the role of DA as neuromodulator of two forms of synaptic plasticity, known as long-term potentiation (LTP) and long-term depression (LTD) in several cortical and subcortical areas. Finally, we will delineate how the effect of DA on dendritic spines places this molecule at the interface between the motor and the cognitive systems. Specifically, we will be focusing on PD, vascular dementia, and schizophrenia.

Corticostriatal control of defense behavior in mice induced by auditory looming cues

Animals exhibit innate defense behaviors in response to approaching threats cued by the dynamics of sensory inputs of various modalities. The underlying neural circuits have been mostly studied in the visual system, but remain unclear for other modalities. Here, by utilizing sounds with increasing (vs. decreasing) loudness to mimic looming (vs. receding) objects, we find that looming sounds elicit stereotypical sequential defensive reactions: freezing followed by flight. Both behaviors require the activity of auditory cortex, in particular the sustained type of responses, but are differentially mediated by corticostriatal projections primarily innervating D2 neurons in the tail of the striatum and corticocollicular projections to the superior colliculus, respectively. The behavioral transition from freezing to flight can be attributed to the differential temporal dynamics of the striatal and collicular neurons in their responses to looming sound stimuli. Our results reveal an essential role of the striatum in the innate defense control.

Dorsal Raphe Dopamine Neurons Signal Motivational Salience Dependent on Internal State, Expectation, and Behavioral Context

The ability to recognize motivationally salient events and adaptively respond to them is critical for survival. Here, we tested whether dopamine (DA) neurons in the dorsal raphe nucleus (DRN) contribute to this process in both male and female mice. Population recordings of DRN^DA neurons during associative learning tasks showed that their activity dynamically tracks the motivational salience, developing excitation to both reward-paired and shock-paired cues. The DRN^DA response to reward-predicting cues was diminished after satiety, suggesting modulation by internal states. DRN^DA activity was also greater for unexpected outcomes than for expected outcomes. Two-photon imaging of DRN^DA neurons demonstrated that the majority of individual neurons developed activation to reward-predicting cues and reward but not to shock-predicting cues, which was surprising and qualitatively distinct from the population results. Performing the same fear learning procedures in freely-moving and head-fixed groups revealed that head-fixation itself abolished the neural response to aversive cues, indicating its modulation by behavioral context. Overall, these results suggest that DRN^DA neurons encode motivational salience, dependent on internal and external factors.SIGNIFICANCE STATEMENT Dopamine (DA) contributes to motivational control, composed of at least two functional cell types, one signaling for motivational value and another for motivational salience. Here, we demonstrate that DA neurons in the dorsal raphe nucleus (DRN) encode the motivational salience in associative learning tasks. Neural responses were dynamic and modulated by the animal's internal state. The majority of single-cells developed responses to reward or paired cues, but not to shock-predicting cues. Additional experiments with freely-moving and head-fixed mice showed that head-fixation abolished the development of cue responses during fear learning. This work provides further characterization on the functional roles of overlooked DRN^DA populations and an example that neural responses can be altered by head-fixation, which is commonly used in neuroscience.

Whole-Brain Mapping of Direct Inputs to Dopamine D1 and D2 Receptor-Expressing Medium Spiny Neurons in the Posterior Dorsomedial Striatum

The posterior dorsomedial striatum (pDMS) is mainly composed of medium spiny neurons (MSNs) expressing either dopamine D1 receptors (D1Rs) or D2Rs. Activation of these two MSN types produces opposing effects on addictive behaviors. However, it remains unclear whether pDMS D1-MSNs or D2-MSNs receive afferent inputs from different brain regions or whether the extrastriatal afferents express distinct dopamine receptors. To assess whether these afferents also contained D1Rs or D2Rs, we generated double transgenic mice, in which D1R-expressing and D2R-expressing neurons were fluorescently labeled. We used rabies virus-mediated retrograde tracing in these mice to perform whole-brain mapping of direct inputs to D1-MSNs or D2-MSNs in the pDMS. We found that D1-MSNs preferentially received inputs from the secondary motor, secondary visual, and cingulate cortices, whereas D2-MSNs received inputs from the primary motor and primary sensory cortices, and the thalamus. We also discovered that the bed nucleus of the stria terminalis (BNST) and the central nucleus of the amygdala (CeA) contained abundant D2R-expressing, but few D1R-expressing, neurons in a triple transgenic mouse model. Remarkably, although limited D1R or D2R expression was observed in extrastriatal neurons that projected to D1-MSNs or D2-MSNs, we found that cortical structures preferentially contained D1R-expressing neurons that projected to D1-MSNs or D2-MSNs, while the thalamus, substantia nigra pars compacta (SNc), and BNST had more D2R-expressing cells that projected to D2-MSNs. Taken together, these findings provide a foundation for future understanding of the pDMS circuit and its role in action selection and reward-based behaviors.

Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task

Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.

A Unified Framework for Dopamine Signals across Timescales

Rapid phasic activity of midbrain dopamine neurons is thought to signal reward prediction errors (RPEs), resembling temporal difference errors used in machine learning. However, recent studies describing slowly increasing dopamine signals have instead proposed that they represent state values and arise independent from somatic spiking activity. Here we developed experimental paradigms using virtual reality that disambiguate RPEs from values. We examined dopamine circuit activity at various stages, including somatic spiking, calcium signals at somata and axons, and striatal dopamine concentrations. Our results demonstrate that ramping dopamine signals are consistent with RPEs rather than value, and this ramping is observed at all stages examined. Ramping dopamine signals can be driven by a dynamic stimulus that indicates a gradual approach to a reward. We provide a unified computational understanding of rapid phasic and slowly ramping dopamine signals: dopamine neurons perform a derivative-like computation over values on a moment-by-moment basis.

Evolution of in vivo dopamine monitoring techniques

The brain dopamine system is central to numerous behavioral processes, including movement, learning, and motivation. Accordingly, disruptions of this neural system underlie numerous neurological and psychiatric disorders. Current understanding of how dopamine neurotransmission contributes to behavior and its dysfunction has been driven by technological advancements that permit spatiotemporally-defined measurements of dopaminergic signaling in behaving animals. In this review, we will discuss the evolution of in vivo neural monitoring technologies for measuring dopamine neuron function. We focus on the dopamine system for two reasons: (1) the central role of dopamine neurotransmission in normal behavior and disease, and (2) dopamine neuron measurements have long been at the forefront of in vivo neural monitoring technologies. We will provide a brief overview of standard techniques for monitoring dopamine function, including electrophysiology, microdialysis, and voltammetry. Then, we will discuss recent advancements in optical technologies using genetically-encoded fluorescent proteins (GEFPs), including a critical evaluation of their advantages and limitations.

Imaging Neurotransmitter and Neuromodulator Dynamics In Vivo with Genetically Encoded Indicators

The actions of neuromodulation are thought to mediate the ability of the mammalian brain to dynamically adjust its functional state in response to changes in the environment. Altered neurotransmitter (NT) and neuromodulator (NM) signaling is central to the pathogenesis or treatment of many human neurological and psychiatric disorders, including Parkinson's disease, schizophrenia, depression, and addiction. To reveal the precise mechanisms by which these neurochemicals regulate healthy and diseased neural circuitry, one needs to measure their spatiotemporal dynamics in the living brain with great precision. Here, we discuss recent development, optimization, and applications of optical approaches to measure the spatial and temporal profiles of NT and NM release in the brain using genetically encoded sensors for in vivo studies.

The Tail of the Striatum: From Anatomy to Connectivity and Function

The dorsal striatum, the largest subcortical structure of the basal ganglia, is critical in controlling motor, procedural, and reinforcement-based behaviors. Although in mammals the striatum extends widely along the rostro-caudal axis, current knowledge and derived theories about its anatomo-functional organization largely rely on results obtained from studies of its rostral sectors, leading to potentially oversimplified working models of the striatum as a whole. Recent findings indicate that the extreme caudal part of the striatum, also referred to as the tail of striatum (TS), represents an additional functional domain. Here, we provide an overview of past and recent studies revealing that the TS displays a heterogeneous cell-type-specific organization, and a unique input-output connectivity, which poises the TS as an integrator of sensory processing.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Midbrain circuits of novelty processing

Novelty triggers an increase in orienting behavior that is critical to evaluate the potential salience of unknown events. As novelty becomes familiar upon repeated encounters, this increase in response rapidly habituates as a form of behavioral adaptation underlying goal-directed behaviors. Many neurodevelopmental, psychiatric and neurodegenerative disorders are associated with abnormal responses to novelty and/or familiarity, although the neuronal circuits and cellular/molecular mechanisms underlying these natural behaviors in the healthy brain are largely unknown, as is the maladaptive processes that occur to induce impairment of novelty signaling in diseased brains. In rodents, the development of cutting-edge tools that allow for measurements of real time activity dynamics in selectively identified neuronal ensembles by gene expression signatures is beginning to provide advances in understanding the neural bases of the novelty response. Accumulating evidence indicate that midbrain circuits, the majority of which linked to dopamine transmission, promote exploratory assessments and guide approach/avoidance behaviors to different types of novelty via specific projection sites. The present review article focuses on midbrain circuit analysis relevant to novelty processing and habituation with familiarity.

Maternal dietary imbalance between omega-6 and omega-3 fatty acids triggers the offspring's overeating in mice

The increasing prevalence of obesity and its effects on our society warrant intensifying basic animal research for understanding why habitual intake of highly palatable foods has increased due to recent global environmental changes. Here, we report that pregnant mice that consume a diet high in omega-6 (n-6) polyunsaturated fatty acids (PUFAs) and low in omega-3 (n-3) PUFAs (an n-6^high/n-3^low diet), whose n-6/n-3 ratio is approximately 120, induces hedonic consumption in the offspring by upregulating the midbrain dopaminergic system. We found that exposure to the n-6^high/n-3^low diet specifically increases the consumption of palatable foods via increased mesolimbic dopamine release. In addition, neurodevelopmental analyses revealed that this induced hedonic consumption is programmed during embryogenesis, as dopaminergic neurogenesis is increased during in utero access to the n-6^high/n-3^low diet. Our findings reveal that maternal consumption of PUFAs can have long-lasting effects on the offspring’s pattern for consuming highly palatable foods.

Sakayori et al. show that feeding pregnant mice with a diet high in omega-6 polyunsaturated fatty acids (PUFAs) and low in omega-3 PUFAs triggers hedonic consumption in the offspring by increasing its dopaminergic neurogenesis. This study suggests that maternal consumption of diets with unbalanced PUFAs contributes to the offspring’s overconsumption of foods.

Trial-by-trial dynamics of reward prediction error-associated signals during extinction learning and renewal

Reward prediction errors (RPEs) have been suggested to drive associative learning processes, but their precise temporal dynamics at the single-neuron level remain elusive. Here, we studied the neural correlates of RPEs, focusing on their trial-by-trial dynamics during an operant extinction learning paradigm. Within a single behavioral session, pigeons went through acquisition, extinction and renewal - the context-dependent response recovery after extinction. We recorded single units from the avian prefrontal cortex analogue, the nidopallium caudolaterale (NCL) and found that the omission of reward during extinction led to a peak of population activity that moved backwards in time as trials progressed. The chronological order of these signal changes during the progress of learning was indicative of temporal shifts of RPE signals that started during reward omission and then moved backwards to the presentation of the conditioned stimulus. Switches from operant choices to avoidance behavior (and vice versa) coincided with changes in population activity during the animals' decision-making. On the single unit level, we found more diverse patterns where some neurons' activity correlated with RPE signals whereas others correlated with the absolute value during the outcome period. Finally, we demonstrated that mere sensory contextual changes during the renewal test were sufficient to elicit signals likely associated with RPEs. Thus, RPEs are truly expectancy-driven since they can be elicited by changes in reward expectation, without an actual change in the quality or quantity of reward.

Dysregulation of Midbrain Dopamine System and the Pathophysiology of Schizophrenia

Dysregulation of the dopamine system is central to many models of the pathophysiology of psychosis in schizophrenia. However, emerging evidence suggests that this dysregulation is driven by the disruption of upstream circuits that provide afferent control of midbrain dopamine neurons. Furthermore, stress can profoundly disrupt this regulatory circuit, particularly when it is presented at critical vulnerable prepubertal time points. This review will discuss the dopamine system and the circuits that regulate it, focusing on the hippocampus, medial prefrontal cortex, thalamic nuclei, and medial septum, and the impact of stress. A greater understanding of the regulation of the dopamine system and its disruption in schizophrenia may provide a more complete neurobiological framework to interpret clinical findings and develop novel treatments.

Distinct signals in medial and lateral VTA dopamine neurons modulate fear extinction at different times

Dopamine (DA) neurons are thought to encode reward prediction error (RPE), in addition to other signals, such as salience. While RPE is known to support learning, the role of salience in learning remains less clear. To address this, we recorded and manipulated VTA DA neurons in mice during fear extinction. We applied deep learning to classify mouse freezing behavior, eliminating the need for human scoring. Our fiber photometry recordings showed DA neurons in medial and lateral VTA have distinct activity profiles during fear extinction: medial VTA activity more closely reflected RPE, while lateral VTA activity more closely reflected a salience-like signal. Optogenetic inhibition of DA neurons in either region slowed fear extinction, with the relevant time period for inhibition differing across regions. Our results indicate salience-like signals can have similar downstream consequences to RPE-like signals, although with different temporal dependencies.

Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Noradrenergic But Not Dopaminergic Neurons Signal Task State Changes and Predict Reengagement After a Failure

The two catecholamines, noradrenaline and dopamine, have been shown to play comparable roles in behavior. Both noradrenergic and dopaminergic neurons respond to cues predicting reward availability and novelty. However, even though both are thought to be involved in motivating actions, their roles in motivation have seldom been directly compared. We therefore examined the activity of putative noradrenergic neurons in the locus coeruleus and putative midbrain dopaminergic neurons in monkeys cued to perform effortful actions for rewards. The activity in both regions correlated with engagement with a presented option. By contrast, only noradrenaline neurons were also (i) predictive of engagement in a subsequent trial following a failure to engage and (ii) more strongly activated in nonrepeated trials, when cues indicated a new task condition. This suggests that while both catecholaminergic neurons are involved in promoting action, noradrenergic neurons are sensitive to task state changes, and their influence on behavior extends beyond the immediately rewarded action.

Amygdala-Midbrain Connections Modulate Appetitive and Aversive Learning

The central amygdala (CeA) orchestrates adaptive responses to emotional events. While CeA substrates for defensive behaviors have been studied extensively, CeA circuits for appetitive behaviors and their relationship to threat-responsive circuits remain poorly defined. Here, we demonstrate that the CeA sends robust inhibitory projections to the lateral substantia nigra (SNL) that contribute to appetitive and aversive learning in mice. CeA→SNL neural responses to appetitive and aversive stimuli were modulated by expectation and magnitude consistent with a population-level salience signal, which was required for Pavlovian conditioned reward-seeking and defensive behaviors. CeA→SNL terminal activation elicited reinforcement when linked to voluntary actions but failed to support Pavlovian associations that rely on incentive value signals. Consistent with a disinhibitory mechanism, CeA inputs preferentially target SNL GABA neurons, and CeA→SNL and SNL dopamine neurons respond similarly to salient stimuli. Collectively, our results suggest that amygdala-nigra interactions represent a previously unappreciated mechanism for influencing emotional behaviors.

Dopamine D2 receptors in discrimination learning and spine enlargement

Dopamine D2 receptors (D2Rs) are densely expressed in the striatum and have been linked to neuropsychiatric disorders such as schizophrenia^1,2. High-affinity binding of dopamine suggests that D2Rs detect transient reductions in dopamine concentration (the dopamine dip) during punishment learning^3-5. However, the nature and cellular basis of D2R-dependent behaviour are unclear. Here we show that tone reward conditioning induces marked stimulus generalization in a manner that depends on dopamine D1 receptors (D1Rs) in the nucleus accumbens (NAc) of mice, and that discrimination learning refines the conditioning using a dopamine dip. In NAc slices, a narrow dopamine dip (as short as 0.4 s) was detected by D2Rs to disinhibit adenosine A_2A receptor (A_2AR)-mediated enlargement of dendritic spines in D2R-expressing spiny projection neurons (D2-SPNs). Plasticity-related signalling by Ca²⁺/calmodulin-dependent protein kinase II and A_2ARs in the NAc was required for discrimination learning. By contrast, extinction learning did not involve dopamine dips or D2-SPNs. Treatment with methamphetamine, which dysregulates dopamine signalling, impaired discrimination learning and spine enlargement, and these impairments were reversed by a D2R antagonist. Our data show that D2Rs refine the generalized reward learning mediated by D1Rs.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

Aversion hot spots in the dopamine system

Through the development of optogenetics and other viral vector-based technologies, our view of the dopamine system has substantially advanced over the last decade. In particular, progress has been made in the reclassification of dopamine neurons based on subtypes displaying specific projections, which are associated with different features at the anatomical, molecular and behavioral level. Together, these discoveries have raised the possibility that individual groups of dopamine cells make a unique contribution to the processing of reward and aversion. Here, we review recent studies that have identified non-canonical dopamine pathways that are excited in response to aversive stimuli, including dopamine projections to the ventromedial shell of the nucleus accumbens, prefrontal cortex, tail of the striatum, and amygdala.

An integrative framework for perceptual disturbances in psychosis

Perceptual disturbances in psychosis, such as auditory verbal hallucinations, are associated with increased baseline activity in the associative auditory cortex and increased dopamine transmission in the associative striatum. Perceptual disturbances are also associated with perceptual biases that suggest increased reliance on prior expectations. We review theoretical models of perceptual inference and key supporting physiological evidence, as well as the anatomy of associative cortico-striatal loops that may be relevant to auditory perceptual inference. Integrating recent findings, we outline a working framework that bridges neurobiology and the phenomenology of perceptual disturbances via theoretical models of perceptual inference.

Cue-Evoked Dopamine Promotes Conditioned Responding during Learning

Dopamine neurons mediate the association of conditioned stimuli (CS) with reward (unconditioned stimuli, US) by signaling the discrepancy between predicted and actual reward during the US. Some theoretical models suggest that learning is also influenced by the salience or associability of the CS. A hallmark of CS associability models is that they can explain latent inhibition, i.e., the observation that novel CS are more effectively learned than familiar CS. Novel CS are known to activate dopamine neurons, but whether those responses affect associative learning has not been investigated. Here, we used fiber photometry to characterize dopamine responses to inconsequential familiar and novel stimuli. Using bidirectional optogenetic modulation during conditioning, we then show that CS-evoked dopamine promotes conditioned responses. This suggests that Pavlovian conditioning is influenced by CS dopamine, in addition to US reward prediction errors. Accordingly, the absence of dopamine responses to familiar CS might explain their slower learning in latent inhibition.

Dopamine and glutamate in schizophrenia: biology, symptoms and treatment

Glutamate and dopamine systems play distinct roles in terms of neuronal signalling, yet both have been proposed to contribute significantly to the pathophysiology of schizophrenia. In this paper we assess research that has implicated both systems in the aetiology of this disorder. We examine evidence from post-mortem, preclinical, pharmacological and in vivo neuroimaging studies. Pharmacological and preclinical studies implicate both systems, and in vivo imaging of the dopamine system has consistently identified elevated striatal dopamine synthesis and release capacity in schizophrenia. Imaging of the glutamate system and other aspects of research on the dopamine system have produced less consistent findings, potentially due to methodological limitations and the heterogeneity of the disorder. Converging evidence indicates that genetic and environmental risk factors for schizophrenia underlie disruption of glutamatergic and dopaminergic function. However, while genetic influences may directly underlie glutamatergic dysfunction, few genetic risk variants directly implicate the dopamine system, indicating that aberrant dopamine signalling is likely to be predominantly due to other factors. We discuss the neural circuits through which the two systems interact, and how their disruption may cause psychotic symptoms. We also discuss mechanisms through which existing treatments operate, and how recent research has highlighted opportunities for the development of novel pharmacological therapies. Finally, we consider outstanding questions for the field, including what remains unknown regarding the nature of glutamate and dopamine function in schizophrenia, and what needs to be achieved to make progress in developing new treatments.

Transient and sustained effects of dopamine and serotonin signaling in motivation-related behavior

Pharmacological studies of antidepressants and atypical antipsychotics have suggested a role of dopamine and serotonin signaling in depression. However, depressive symptoms and treatment effects are difficult to explain based simply on brain-wide decrease or increase in the concentrations of these molecules. Recent animal studies using advanced neuronal manipulation and observation techniques have revealed detailed dopamine and serotonin dynamics that regulate diverse aspects of motivation-related behavior. Dopamine and serotonin transiently modulate moment-to-moment behavior at timescales ranging from sub-second to minutes and also produce persistent effects, such as reward-related learning and stress responses that last longer than several days. Transient and sustained effects often exhibit specific roles depending on the projection sites, where distinct synaptic and cellular mechanisms are required to process the neurotransmitters for each transient and sustained timescale. Therefore, it appears that specific aspects of motivation-related behavior are regulated by distinct synaptic and cellular mechanisms in specific brain regions that underlie the transient and sustained effects of dopamine and serotonin signaling. Recent clinical studies have implied that subjects with depressive symptoms show impaired transient and sustained signaling functions; moreover, they exhibit heterogeneity in depressive symptoms and neuronal dysfunction. Depressive symptoms may be explained by the dysfunction of each transient and sustained signaling mechanism, and distinct patterns of impairment in the relevant mechanisms may explain the heterogeneity of symptoms. Thus, detailed understanding of dopamine and serotonin signaling may provide new insight into depressive symptoms.

Schizophrenia-An Overview

IMPORTANCE

Schizophrenia is a common, severe mental illness that most clinicians will encounter regularly during their practice. This report provides an overview of the clinical characteristics, epidemiology, genetics, neuroscience, and psychopharmacology of schizophrenia to provide a basis to understand the disorder and its treatment. This educational review is integrated with a clinical case to highlight how recent research findings can inform clinical understanding.

OBSERVATIONS

The first theme considered is the role of early-life environmental and genetic risk factors in altering neurodevelopmental trajectories to predispose an individual to the disorder and leading to the development of prodromal symptoms. The second theme is the role of cortical excitatory-inhibitory imbalance in the development of the cognitive and negative symptoms of the disorder. The third theme considers the role of psychosocial stressors, psychological factors, and subcortical dopamine dysfunction in the onset of the positive symptoms of the disorder. The final theme considers the mechanisms underlying treatment for schizophrenia and common adverse effects of treatment.

CONCLUSIONS AND RELEVANCE

Schizophrenia has a complex presentation with a multifactorial cause. Nevertheless, advances in neuroscience have identified roles for key circuits, particularly involving frontal, temporal, and mesostriatal brain regions, in the development of positive, negative, and cognitive symptoms. Current pharmacological treatments operate using the same mechanism, blockade of dopamine D2 receptor, which contribute to their adverse effects. However, the circuit mechanisms discussed herein identify novel potential treatment targets that may be of particular benefit in symptom domains not well served by existing medications.

Imaging neuromodulators with high spatiotemporal resolution using genetically encoded indicators

Multiple aspects of neural activity, from neuronal firing to neuromodulator release and signaling, underlie brain function and ultimately shape animal behavior. The recently developed and constantly growing toolbox of genetically encoded sensors for neural activity, including calcium, voltage, neurotransmitter and neuromodulator sensors, allows precise measurement of these signaling events with high spatial and temporal resolution. Here, we describe the engineering, characterization and application of our recently developed dLight1, a suite of genetically encoded dopamine (DA) sensors based on human inert DA receptors. dLight1 offers high molecular specificity, requisite affinity and kinetics and great sensitivity for measuring DA release in vivo. The detailed workflow described in this protocol can be used to systematically characterize and validate dLight1 in increasingly intact biological systems, from cultured cells to acute brain slices to behaving mice. For tool developers, we focus on characterizing five distinct properties of dLight1: dynamic range, affinity, molecular specificity, kinetics and interaction with endogenous signaling; for end users, we provide comprehensive step-by-step instructions for how to leverage fiber photometry and two-photon imaging to measure dLight1 transients in vivo. The instructions provided in this protocol are designed to help laboratory personnel with a broad range of experience (at the graduate or post-graduate level) to develop and utilize novel neuromodulator sensors in vivo, by using dLight1 as a benchmark.

Believing in dopamine

Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.