Diversity and homogeneity in responses of midbrain dopamine neurons

Muscarinic receptor activation preferentially inhibits rebound in vulnerable dopaminergic neurons

Dopaminergic subpopulations of the substantia nigra pars compacta (SNc) differentially degenerate in Parkinson's disease and are characterized by unique electrophysiological properties. The vulnerable population expresses a T-type calcium channel-mediated afterdepolarization (ADP) and shows rebound activity upon release from inhibition, whereas the resilient population does not have an ADP and is slower to fire after hyperpolarization. This rebound activity can trigger dopamine release in the striatum, an important component of basal ganglia function. Using whole-cell patch clamp electrophysiology on ex vivo slices from adult mice of both sexes, we find that muscarinic activation with the non-selective muscarinic agonist Oxotremorine inhibits rebound activity more strongly in vulnerable vs resilient SNc neurons. Here, we show that this effect depends on the direct activation of muscarinic receptors on the SNc dopaminergic neurons. Through a series of pharmacological and transgenic knock-out experiments, we tested whether the muscarinic inhibition of rebound was mediated through the canonical rebound-related ion channels: T-type calcium channels, hyperpolarization-activated cation channels (HCN), and A-type potassium channels. We find that muscarinic receptor activation inhibits HCN-mediated current (Ih) in vulnerable SNc neurons, but that Ih activity is not necessary for the muscarinic inhibition of rebound activity. Similarly, we find that Oxotremorine inhibits rebound activity independently of T-type calcium channels and A-type potassium channels. Together these findings reveal new principles governing acetylcholine and dopamine interactions, showing that muscarinic receptors directly affect SNc rebound activity in the midbrain at the somatodendritic level and differentially modify information processing in distinct SNc subpopulations.

Persistent enhancement of basolateral amygdala-dorsomedial striatum synapses causes compulsive-like behaviors in mice

Compulsive behaviors are observed in a range of psychiatric disorders, however the neural substrates underlying the behaviors are not clearly defined. Here we show that the basolateral amygdala-dorsomedial striatum (BLA-DMS) circuit activation leads to the manifestation of compulsive-like behaviors. We revealed that the BLA neurons projecting to the DMS, mainly onto dopamine D1 receptor-expressing neurons, largely overlap with the neuronal population that responds to aversive predator stress, a widely used anxiogenic stressor. Specific optogenetic activation of the BLA-DMS circuit induced a strong anxiety response followed by compulsive grooming. Furthermore, we developed a mouse model for compulsivity displaying a wide spectrum of compulsive-like behaviors by chronically activating the BLA-DMS circuit. In these mice, persistent molecular changes at the BLA-DMS synapses observed were causally related to the compulsive-like phenotypes. Together, our study demonstrates the involvement of the BLA-DMS circuit in the emergence of enduring compulsive-like behaviors via its persistent synaptic changes.

Contributions of the Basal Ganglia to Visual Perceptual Decisions

The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.

Anger, agency, risk and action: a neurobehavioral model with proof-of-concept in healthy young adults

Introduction

Anger can engender action by individuals and groups. It is thus important to understand anger's behavioral phenotypes and their underlying neural substrates. Here, we introduce a construct we term agentic anger, a negatively valenced internal state that motivates action to achieve risky goals. We evaluate our neurobehavioral model via testable hypotheses in two proof-of-concept studies.

Study 1 Methods

Study 1 used the Incentive Balloon Analogue Risk Task in a within-subjects, repeated measures design in 39 healthy volunteers to evaluate: (a) impact of blockade of reward on agentic anger, assessed by self-reports of negative activation (NA), (b) impact of achievement of reward on exuberance, assessed by self-reports of positive activation (PA), (c) the interrelationship of these valenced states, and (d) their relationship with personality.

Study 1 Results

Task-induced NA was positively correlated with task-induced PA, risk-taking on the task and trait Social Potency (SP), a measure of trait agency and reward sensitivity on the Multidimensional Personality Questionnaire Brief-Form.

Study 2 Methods

Study 2 assessed functional MRI response to stakes for risk-taking in healthy volunteers receiving 20 mg d-amphetamine in a double-blinded, placebo-controlled crossover design (N = 10 males), providing preliminary information on ventral striatal response to risky rewards during catecholamine activation.

Study 2 Results

Trait SP and task-induced PA were strongly positively related to catecholamine-facilitated BOLD response in the right nucleus accumbens, a brain region where DA prediction error signal shapes action value and selection. Participants' task-induced NA was strongly positively related with trait SP and task-induced PA, replicating the findings of Study 1.

Discussion

Together these results inform the phenomenology and neurobiology of agentic anger, which recruits incentive motivational circuitry and motivates personal action in response to goals that entail risk (defined as exposure to uncertainty, obstacles, potential harm, loss and/or financial, emotional, bodily, or moral peril). Neural mechanisms of agency, anger, exuberance, and risk-taking are discussed, with implications for personal and group action, decision-making, social justice, and behavior change.

Dendritic involvement in inhibition and disinhibition of vulnerable dopaminergic neurons in healthy and pathological conditions

Dopaminergic neurons in the substantia nigra pars compacta (SNc) differentially degenerate in Parkinson's Disease, with the ventral region degenerating more severely than the dorsal region. Compared with the dorsal neurons, the ventral neurons in the SNc have distinct dendritic morphology, electrophysiological characteristics, and circuit connections with the basal ganglia. These characteristics shape information processing in the ventral SNc and structure the balance of inhibition and disinhibition in the striatonigral circuitry. In this paper, I review foundational studies and recent work comparing the circuitry of the ventral and dorsal SNc neurons and discuss how loss of the ventral neurons early in Parkinson's Disease could affect the overall balance of inhibition and disinhibition of dopamine signals.

Asymmetric and adaptive reward coding via normalized reinforcement learning

Learning is widely modeled in psychology, neuroscience, and computer science by prediction error-guided reinforcement learning (RL) algorithms. While standard RL assumes linear reward functions, reward-related neural activity is a saturating, nonlinear function of reward; however, the computational and behavioral implications of nonlinear RL are unknown. Here, we show that nonlinear RL incorporating the canonical divisive normalization computation introduces an intrinsic and tunable asymmetry in prediction error coding. At the behavioral level, this asymmetry explains empirical variability in risk preferences typically attributed to asymmetric learning rates. At the neural level, diversity in asymmetries provides a computational mechanism for recently proposed theories of distributional RL, allowing the brain to learn the full probability distribution of future rewards. This behavioral and computational flexibility argues for an incorporation of biologically valid value functions in computational models of learning and decision-making.

Reinforcement learning models are widely used to characterize reward-driven learning in biological and computational agents. Standard reinforcement learning models use linear value functions, despite strong empirical evidence that biological value representations are nonlinear functions of external rewards. Here, we examine the properties of a biologically-based nonlinear reinforcement learning algorithm employing the canonical divisive normalization function, a neural computation commonly found in sensory, cognitive, and reward coding. We show that this normalized reinforcement learning algorithm implements a simple but powerful control of how reward learning reflects relative gains and losses. This property explains diverse behavioral and neural phenomena, and suggests the importance of using biologically valid value functions in computational models of learning and decision-making.

The Aversion Function of the Limbic Dopaminergic Neurons and Their Roles in Functional Neurological Disorders

The Freudian theory of conversion suggested that the major symptoms of functional neurological disorders (FNDs) are due to internal conflicts at motivation, especially at the sex drive or libido. FND patients might behave properly at rewarding situations, but they do not know how to behave at aversive situations. Sex drive is the major source of dopamine (DA) release in the limbic area; however, the neural mechanism involved in FND is not clear. Dopaminergic (DAergic) neurons have been shown to play a key role in processing motivation-related information. Recently, DAergic neurons are found to be involved in reward-related prediction error, as well as the prediction of aversive information. Therefore, it is suggested that DA might change the rewarding reactions to aversive reactions at internal conflicts of FND. So DAergic neurons in the limbic areas might induce two major motivational functions: reward and aversion at internal conflicts. This article reviewed the recent advances on studies about DAergic neurons involved in aversive stimulus processing at internal conflicts and summarizes several neural pathways, including four limbic system brain regions, which are involved in the processing of aversion. Then the article discussed the vital function of these neural circuits in addictive behavior, depression treatment, and FNDs. In all, this review provided a prospect for future research on the aversion function of limbic system DA neurons and the therapy of FNDs.

Involvement of Midbrain Dopamine Neuron Activity in Negative Reinforcement Learning in Mice

The activity of the midbrain dopamine system reflects the valence of environmental events and modulates various brain structures to modify an organism's behavior. A series of recent studies reported that the direct and indirect pathways in the striatum are critical for instrumental learning, but the dynamic changes in dopamine neuron activity that occur during negative reinforcement learning are still largely unclear. In the present study, by using a negative reinforcement learning paradigm employing foot shocks as aversive stimuli, bidirectional changes in substantia nigra pars compacta (SNc) dopamine neuron activity in the learning and habituation phases were observed. The results showed that in the learning phase, before mice had mastered the skill of escaping foot shocks, the presence of foot shocks induced a transient reduction in the activity of SNc dopamine neurons; however, in the habituation phase, in which the learned skill was automated, it induced a transient increase. Microinjection of a dopamine D1 receptor (D1R) or D2 receptor (D2R) antagonist into the dorsomedial striatum (DMS) significantly impaired learning behavior, suggesting that the modulatory effects of dopamine on both the direct and indirect pathways are required. Moreover, during the learning phase, excitatory synaptic transmission to DMS D2R-expressing medium spiny neurons (D2-MSNs) was potentiated. However, upon completion of the learning and habituation phases, the synapses onto D1R-expressing medium spiny neurons (D1-MSNs) were potentiated, and those onto D2-MSNs were restored to normal levels. The bidirectional changes in both SNc dopamine neuron activity and DMS synaptic plasticity might be the critical neural correlates for negative reinforcement learning.

Inferring spikes from calcium imaging in dopamine neurons

Calcium imaging has led to discoveries about neural correlates of behavior in subcortical neurons, including dopamine (DA) neurons. However, spike inference methods have not been tested in most populations of subcortical neurons. To address this gap, we simultaneously performed calcium imaging and electrophysiology in DA neurons in brain slices and applied a recently developed spike inference algorithm to the GCaMP fluorescence. This revealed that individual spikes can be inferred accurately in this population. Next, we inferred spikes in vivo from calcium imaging from these neurons during Pavlovian conditioning, as well as during navigation in virtual reality. In both cases, we quantitatively recapitulated previous in vivo electrophysiological observations. Our work provides a validated approach to infer spikes from calcium imaging in DA neurons and implies that aspects of both tonic and phasic spike patterns can be recovered.

Distracting stimuli evoke ventral tegmental area responses in rats during ongoing saccharin consumption

Disruptions in attention, salience and increased distractibility are implicated in multiple psychiatric conditions. The ventral tegmental area (VTA) is a potential site for converging information about external stimuli and internal states to be integrated and guide adaptive behaviours. Given the dual role of dopamine signals in both driving ongoing behaviours (e.g., feeding) and monitoring salient environmental stimuli, understanding the interaction between these functions is crucial. Here, we investigate VTA neuronal activity during distraction from ongoing feeding. We developed a task to assess distraction exploiting self-paced licking in rats. Rats trained to lick for saccharin were given a distraction test, in which three consecutive licks within 1 s triggered a random distractor (e.g. light and tone stimulus). On each trial they were quantified as distracted or not based on the length of their pauses in licking behaviour. We expressed GCaMP6s in VTA neurons and used fibre photometry to record calcium fluctuations during this task as a proxy for neuronal activity. Distractor stimuli caused rats to interrupt their consumption of saccharin, a behavioural effect which quickly habituated with repeat testing. VTA neural activity showed consistent increases to distractor presentations and, furthermore, these responses were greater on distracted trials compared to non-distracted trials. Interestingly, neural responses show a slower habituation than behaviour with consistent VTA responses seen to distractors even after they are no longer distracting. These data highlight the complex role of the VTA in maintaining ongoing appetitive and consummatory behaviours while also monitoring the environment for salient stimuli.

Distributional Reinforcement Learning in the Brain

Learning about rewards and punishments is critical for survival. Classical studies have demonstrated an impressive correspondence between the firing of dopamine neurons in the mammalian midbrain and the reward prediction errors of reinforcement learning algorithms, which express the difference between actual reward and predicted mean reward. However, it may be advantageous to learn not only the mean but also the complete distribution of potential rewards. Recent advances in machine learning have revealed a biologically plausible set of algorithms for reconstructing this reward distribution from experience. Here, we review the mathematical foundations of these algorithms as well as initial evidence for their neurobiological implementation. We conclude by highlighting outstanding questions regarding the circuit computation and behavioral readout of these distributional codes.

Functional Dissection of Basal Ganglia Inhibitory Inputs onto Substantia Nigra Dopaminergic Neurons

Substantia nigra (SNc) dopaminergic neurons respond to aversive stimuli with inhibitory pauses in firing followed by transient rebound activation. We tested integration of inhibitory synaptic inputs onto SNc neurons from genetically defined populations in dorsal striatum (striosome and matrix) and external globus pallidus (GPe; parvalbumin- and Lhx6-positive), and examined their contribution to pause-rebound firing. Activation of striosome projections, which target "dendron bouquets" in the pars reticulata (SNr), consistently quiets firing and relief from striosome inhibition triggers rebound activity. Striosomal inhibitory postsynaptic currents (IPSCs) display a prominent GABA-B receptor-mediated component that strengthens the impact of SNr dendrite synapses on somatic excitability and enables rebounding. By contrast, GPe projections activate GABA-A receptors on the soma and proximal dendrites but do not result in rebounding. Lastly, optical mapping shows that dorsal striatum selectively inhibits the ventral population of SNc neurons, which are intrinsically capable of rebounding. Therefore, we define a distinct striatonigral circuit for generating dopamine rebound.

Occasion setters determine responses of putative DA neurons to discriminative stimuli

Midbrain dopamine (DA) neurons are involved in the processing of rewards and reward-predicting stimuli, possibly analogous to reinforcement learning reward prediction errors. Here we studied the activity of putative DA neurons (n = 37) recorded in the ventral tegmental area of rats (n = 6) performing a behavioural task involving occasion setting. In this task an occasion setter (OS) indicated that the relationship between a discriminative stimulus (DS) and reinforcement is in effect, so that reinforcement of bar pressing occurred only after the OS (tone or houselight) was followed by the DS (houselight or tone). We found that responses of putative DA cells to the DS were enhanced when preceded by the OS, as were behavioural responses to obtain rewards. Surprisingly though, we did not find a homogeneous increase in the mean activity of the population of putative DA neurons to the OS, contrary to predictions of standard temporal-difference models of DA neurons. Instead, putative DA neurons exhibited a heterogeneous response on a single unit level, so that some units increased and others decreased their activity as a response to the OS. Similarly, putative non-DA cells did not show a homogeneous response to the DS on a population level, but also had heterogeneous responses on a single unit level. The heterogeneity in the responses of neurons in the ventral tegmental area may reflect how DA neurons encode context and point to local differences in DA signalling.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

An Integrative Way for Studying Neural Basis of Basic Emotions With fMRI

How emotions are represented in the nervous system is a crucial unsolved problem in the affective neuroscience. Many studies are striving to find the localization of basic emotions in the brain but failed. Thus, many psychologists suspect the specific neural loci for basic emotions, but instead, some proposed that there are specific neural structures for the core affects, such as arousal and hedonic value. The reason for this widespread difference might be that basic emotions used previously can be further divided into more “basic” emotions. Here we review brain imaging data and neuropsychological data, and try to address this question with an integrative model. In this model, we argue that basic emotions are not contrary to the dimensional studies of emotions (core affects). We propose that basic emotion should locate on the axis in the dimensions of emotion, and only represent one typical core affect (arousal or valence). Therefore, we propose four basic emotions: joy-on positive axis of hedonic dimension, sadness-on negative axis of hedonic dimension, fear, and anger-on the top of vertical dimensions. This new model about basic emotions and construction model of emotions is promising to improve and reformulate neurobiological models of basic emotions.

Shared Behavioral and Neurocircuitry Disruptions in Drug Addiction, Obesity, and Binge Eating Disorder: Focus on Group I mGluRs in the Mesolimbic Dopamine Pathway

Accumulated data from clinical and preclinical studies suggest that, in drug addiction and states of overeating, such as obesity and binge eating disorder (BED), there is an imbalance in circuits that are critical for motivation, reward saliency, executive function, and self-control. Central to these pathologies and the extensive topic of this Review are the aberrations in dopamine (DA) and glutamate (Glu) within the mesolimbic pathway. Group I metabotropic glutamate receptors (mGlus) are highly expressed in the mesolimbic pathway and are poised in key positions to modulate disruptions in synaptic plasticity and neurotransmitter release observed in drug addiction, obesity, and BED. The use of allosteric modulators of group I mGlus has been studied in drug addiction, as they offer several advantages over traditional orthosteric agents. However, they have yet to be studied in obesity or BED. With the substantial overlap between the neurocircuitry involved in drug addiction and eating disorders, group I mGlus may also provide novel targets for obesity and BED.

Schizotypal Traits are Linked to Dopamine-Induced Striato-Cortical Decoupling: A Randomized Double-Blind Placebo-Controlled Study

The dopamine hypothesis of schizophrenia implies that alterations in the dopamine system cause functional abnormalities in the brain that may converge to aberrant salience attribution and eventually lead to psychosis. Indeed, widespread brain disconnectivity across the psychotic spectrum has been revealed by resting-state functional magnetic resonance imaging (rs-fMRI). However, the dopaminergic involvement in intrinsic functional connectivity (iFC) and its putative relationship to the development of psychotic spectrum disorders remains partly unclear-in particular at the low-end of the psychosis continuum. Therefore, we investigated dopamine-induced changes in striatal iFC and their modulation by psychometrically assessed schizotypy. Our randomized, double-blind placebo-controlled study design included 54 healthy, right-handed male participants. Each participant was assessed with the Schizotypal Personality Questionnaire (SPQ) and underwent 10 minutes of rs-fMRI scanning. Participants then received either a placebo or 200 mg of L-DOPA, a dopamine precursor. We analyzed iFC of 6 striatal seeds that are known to evoke modulation of dopamine-related networks. The main effect of L-DOPA was a significant functional decoupling from the right ventral caudate to both occipital fusiform gyri. This dopamine-induced decoupling emerged primarily in participants with low SPQ scores, while participants with high positive SPQ scores showed decoupling indifferently of the L-DOPA challenge. Taken together, these findings demonstrate that schizotypal traits may be the result of dopamine-induced striato-occipital decoupling.

Financial gain- and loss-related BOLD signals in the human ventral tegmental area and substantia nigra pars compacta

Neurons in the ventral tegmental area (VTA) and substantia nigra pars compacta (SNC) play central roles in reward-related behaviours. Nonhuman animal studies suggest that these neurons also process aversive events. However, our understanding of how the human VTA and SNC responds to such events is limited and has been hindered by the technical challenge of using functional magnetic resonance imaging (fMRI) to investigate a small structure where the signal is particularly vulnerable to physiological noise. Here we show, using methods optimized specifically for the midbrain (including high-resolution imaging, a novel registration protocol, and physiological noise modelling), a BOLD (blood-oxygen-level dependent) signal to both financial gain and loss in the VTA and SNC, along with a response to nil outcomes that are better or worse than expected in the VTA. Taken together, these findings suggest that the human VTA and SNC are involved in the processing of both appetitive and aversive financial outcomes in humans.

What Is the Relationship between Dopamine and Effort?

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remains poorly understood, both at the behavioral and neurophysiological levels. This is important because unwillingness to overcome effort to gain reward is a common feature of many neuropsychiatric and neurological disorders. A recent surge in interest in the neurobiological basis of effort has led to seemingly conflicting results regarding the role of dopamine. We argue here that, upon closer examination, there is actually striking consensus across studies: dopamine primarily codes for future reward but is less sensitive to anticipated effort cost. This strong association between dopamine and the incentive effects of rewards places dopamine in a key position to promote reward-directed action.

The timing of action determines reward prediction signals in identified midbrain dopamine neurons

Animals adapt behavior in response to informative sensory cues using multiple brain circuits. The activity of midbrain dopamine (mDA) neurons is thought to convey a critical teaching signal: reward prediction error (RPE). Although RPE signals are thought to be essential to learning, little is known about the dynamic changes in mDA neuron activity as animals learn about novel sensory cues and appetitive rewards. Here we describe a large dataset of cell-attached recordings of identified dopaminergic neurons as naïve mice learned a novel cue-reward association. During learning mDA neuron activity results from summation of sensory cue-related and movement initiation-related response components. These components are both a function of reward expectation yet dissociable. Learning produces an increasingly precise coordination of action initiation following sensory cues that results in apparent RPE correlates. Our data thus provide new insights into circuit mechanisms underlying a critical computation in a highly conserved learning circuit.

Brain responses to different types of salience in antipsychotic naïve first episode psychosis: An fMRI study

Abnormal salience processing has been suggested to contribute to the formation of positive psychotic symptoms in schizophrenia and related conditions. Previous research utilising reward learning or anticipation paradigms has demonstrated cortical and subcortical abnormalities in people with psychosis, specifically in the prefrontal cortex, the dopaminergic midbrain and the striatum. In these paradigms, reward prediction errors attribute motivational salience to stimuli. However, little is known about possible abnormalities across different forms of salience processing in psychosis patients, and whether any such abnormalities involve the dopaminergic midbrain. The aim of our study was, therefore, to investigate possible alterations in psychosis in neural activity in response to various forms of salience: novelty, negative emotion, targetness (task-driven salience) and rareness/deviance. We studied 14 antipsychotic naïve participants with first episode psychosis, and 37 healthy volunteers. During fMRI scanning, participants performed a visual oddball task containing these four forms of salience. Psychosis patients showed abnormally reduced signalling in the substantia nigra/ventral tegmental area (SN/VTA) for novelty, negative emotional salience and targetness; reduced striatal and occipital (lingual gyrus) signalling to novelty and negative emotional salience, reduced signalling in the amygdala, anterior cingulate cortex and parahippocamal gyrus to negative emotional salience, and reduced cerebellar signalling to novelty and negative emotional salience. Our results indicate alterations of several forms of salience processing in patients with psychosis in the midbrain SN/VTA, with additional subcortical and cortical regions also showing alterations in salience signalling, the exact pattern of alterations depending on the form of salience in question.

Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules

Most elementary behaviors such as moving the arm to grasp an object or walking into the next room to explore a museum evolve on the time scale of seconds; in contrast, neuronal action potentials occur on the time scale of a few milliseconds. Learning rules of the brain must therefore bridge the gap between these two different time scales. Modern theories of synaptic plasticity have postulated that the co-activation of pre- and postsynaptic neurons sets a flag at the synapse, called an eligibility trace, that leads to a weight change only if an additional factor is present while the flag is set. This third factor, signaling reward, punishment, surprise, or novelty, could be implemented by the phasic activity of neuromodulators or specific neuronal inputs signaling special events. While the theoretical framework has been developed over the last decades, experimental evidence in support of eligibility traces on the time scale of seconds has been collected only during the last few years. Here we review, in the context of three-factor rules of synaptic plasticity, four key experiments that support the role of synaptic eligibility traces in combination with a third factor as a biological implementation of neoHebbian three-factor learning rules.

Reinforcement signaling of punishment versus relief in fruit flies

Painful events establish opponent memories: cues that precede pain are remembered negatively, whereas cues that follow pain, thus coinciding with relief are recalled positively. How do individual reinforcement-signaling neurons contribute to this “timing-dependent valence-reversal?” We addressed this question using an optogenetic approach in the fruit fly. Two types of fly dopaminergic neuron, each comprising just one paired cell, indeed established learned avoidance of odors that preceded their photostimulation during training, and learned approach to odors that followed the photostimulation. This is in striking parallel to punishment versus relief memories reinforced by a real noxious event. For only one of these neuron types, both effects were strong enough for further analyses. Notably, interfering with dopamine biosynthesis in these neurons partially impaired the punishing effect, but not the relieving after-effect of their photostimulation. We discuss how this finding constraints existing computational models of punishment versus relief memories and introduce a new model, which also incorporates findings from mammals. Furthermore, whether using dopaminergic neuron photostimulation or a real noxious event, more prolonged punishment led to stronger relief. This parametric feature of relief may also apply to other animals and may explain particular aspects of related behavioral dysfunction in humans.

Novelty-Sensitive Dopaminergic Neurons in the Human Substantia Nigra Predict Success of Declarative Memory Formation

The encoding of information into long-term declarative memory is facilitated by dopamine. This process depends on hippocampal novelty signals, but it remains unknown how midbrain dopaminergic neurons are modulated by declarative-memory-based information. We recorded individual substantia nigra (SN) neurons and cortical field potentials in human patients performing a recognition memory task. We found that 25% of SN neurons were modulated by stimulus novelty. Extracellular waveform shape and anatomical location indicated that these memory-selective neurons were putatively dopaminergic. The responses of memory-selective neurons appeared 527 ms after stimulus onset, changed after a single trial, and were indicative of recognition accuracy. SN neurons phase locked to frontal cortical theta-frequency oscillations, and the extent of this coordination predicted successful memory formation. These data reveal that dopaminergic neurons in the human SN are modulated by memory signals and demonstrate a progression of information flow in the hippocampal-basal ganglia-frontal cortex loop for memory encoding.

The biological and behavioral computations that influence dopamine responses

Phasic dopamine responses demonstrate remarkable simplicity; they code for the differences between received and predicted reward values. Yet this simplicity belies the subtle complexity of the psychological, computational, and contextual factors that influence this signal. Advances in behavioral paradigms and models, in monkeys and rodents, have demonstrated that phasic dopamine responses reflect numerous behavioral computations and factors including choice, subjective value, confidence, and context. The application of optogenetics has provided evidence that dopamine reward prediction error responses cause value learning. Furthermore, studies using advanced circuit tracing techniques have begun to uncover the biological network implementation of the reward learning algorithm. The purpose of this review is to summarize the recent advances in dopamine neurophysiology and synthesize an updated account of the behavioral function of dopamine signals.

Reward and value coding by dopamine neurons in non-human primates

Rewards are fundamental to everyday life. They confer pleasure, support learning, and mediate decisions. Dopamine-releasing neurons in the midbrain are critical for reward processing. These neurons receive input from more than 30 brain areas and send widespread projections to the basal ganglia and frontal cortex. Their phasic responses are tuned to rewards. Specifically, dopamine signals code reward prediction error, the difference between received and predicted rewards. Decades of research in awake, behaving non-human primates (NHP), have shown the importance of these neural signals for learning and decision making. In this review, we will provide an overview of the bedrock findings that support the reward prediction error hypothesis and examine evidence that this signal plays a role in learning and decision making. In addition, we will highlight some of the conceptual challenges in dopamine neurophysiology and identify future areas of research to address these challenges. Keeping with the theme of this special issue, we will focus on the role of NHP studies in understanding dopamine neurophysiology and make the argument that primate models are essential to this line of research.

Proximity of Substantia Nigra Microstimulation to Putative GABAergic Neurons Predicts Modulation of Human Reinforcement Learning

Neuronal firing in the substantia nigra (SN) immediately following reward is thought to play a crucial role in human reinforcement learning. As in Ramayya et al. (2014a) we applied microstimulation in the SN of patients undergoing deep brain stimulation (DBS) for the treatment of Parkinson's disease as they engaged in a two-alternative reinforcement learning task. We obtained microelectrode recordings to assess the proximity of the electrode tip to putative dopaminergic and GABAergic SN neurons and applied stimulation to assess the functional importance of these neuronal populations for learning. We found that the proximity of SN microstimulation to putative GABAergic neurons predicted the degree of stimulation-related changes in learning. These results extend previous work by supporting a specific role for SN GABA firing in reinforcement learning. Stimulation near these neurons appears to dampen the reinforcing effect of rewarding stimuli.

What can tiny mushrooms in fruit flies tell us about learning and memory?

Nervous systems have evolved to translate external stimuli into appropriate behavioral responses. In an ever-changing environment, flexible adjustment of behavioral choice by experience-dependent learning is essential for the animal's survival. Associative learning is a simple form of learning that is widely observed from worms to humans. To understand the whole process of learning, we need to know how sensory information is represented and transformed in the brain, how it is changed by experience, and how the changes are reflected on motor output. To tackle these questions, studying numerically simple invertebrate nervous systems has a great advantage. In this review, I will feature the Pavlovian olfactory learning in the fruit fly, Drosophila melanogaster. The mushroom body is a key brain area for the olfactory learning in this organism. Recently, comprehensive anatomical information and the genetic tool sets were made available for the mushroom body circuit. This greatly accelerated the physiological understanding of the learning process. One of the key findings was dopamine-induced long-term synaptic plasticity that can alter the representations of stimulus valence. I will mostly focus on the new studies within these few years and discuss what we can possibly learn about the vertebrate systems from this model organism.

Neural Circuitry of Reward Prediction Error

Dopamine neurons facilitate learning by calculating reward prediction error, or the difference between expected and actual reward. Despite two decades of research, it remains unclear how dopamine neurons make this calculation. Here we review studies that tackle this problem from a diverse set of approaches, from anatomy to electrophysiology to computational modeling and behavior. Several patterns emerge from this synthesis: that dopamine neurons themselves calculate reward prediction error, rather than inherit it passively from upstream regions; that they combine multiple separate and redundant inputs, which are themselves interconnected in a dense recurrent network; and that despite the complexity of inputs, the output from dopamine neurons is remarkably homogeneous and robust. The more we study this simple arithmetic computation, the knottier it appears to be, suggesting a daunting (but stimulating) path ahead for neuroscience more generally.

The phasic dopamine signal maturing: from reward via behavioural activation to formal economic utility

The phasic dopamine reward prediction error response is a major brain signal underlying learning, approach and decision making. This dopamine response consists of two components that reflect, initially, stimulus detection from physical impact and, subsequenttly, reward valuation; dopamine activations by punishers reflect physical impact rather than aversiveness. The dopamine reward signal is distinct from earlier reported and recently confirmed phasic changes with behavioural activation. Optogenetic activation of dopamine neurones in monkeys causes value learning and biases economic choices. The dopamine reward signal conforms to formal economic utility and thus constitutes a utility prediction error signal. In these combined ways, the dopamine reward prediction error signal constitutes a potential neuronal substrate for the crucial economic decision variable of utility.

Partial lesion of dopamine neurons of rat substantia nigra impairs conditioned place aversion but spares conditioned place preference

Midbrain dopamine neurons play critical roles in reward- and aversion-driven associative learning. However, it is not clear whether they do this by a common mechanism or by separate mechanisms that can be dissociated. In the present study we addressed this question by testing whether a partial lesion of the dopamine neurons of the rat SNc has comparable effects on conditioned place preference (CPP) learning and conditioned place aversion (CPA) learning. Partial lesions of dopamine neurons in the rat substantia nigra pars compacta (SNc) induced by bilateral intranigral infusion of 6-hydroxydopamine (6-OHDA, 3μg/side) or 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP, 200μg/side) impaired learning of conditioned place aversion (CPA) without affecting conditioned place preference (CPP) learning. Control experiments demonstrated that these lesions did not impair motor performance and did not alter the hedonic value of the sucrose and quinine. The number of dopamine neurons in the caudal part of the SNc positively correlated with the CPP scores of the 6-OHDA rats and negatively correlated with CPA scores of the SHAM rats. In addition, the CPA scores of the 6-OHDA rats positively correlated with the tissue content of striatal dopamine. Insomuch as reward-driven learning depends on an increase in dopamine release by nigral neurons, these findings show that this mechanism is functional even in rats with a partial lesion of the SNc. On the other hand, if aversion-driven learning depends on a reduction of extracellular dopamine in the striatum, the present study suggests that this mechanism is no longer functional after the partial SNc lesion.

Dopamine Inhibition Differentially Controls Excitability of Substantia Nigra Dopamine Neuron Subpopulations through T-Type Calcium Channels

While there is growing appreciation for diversity among ventral tegmental area dopamine neurons, much less is known regarding functional heterogeneity among the substantia nigra pars compacta (SNc) neurons. Here, we show that calbindin-positive dorsal tier and calbindin-negative ventral tier SNc dopaminergic neurons in mice comprise functionally distinct subpopulations distinguished by their dendritic calcium signaling, rebound excitation, and physiological responses to dopamine D2-receptor (D2) autoinhibition. While dopamine is known to inhibit action potential backpropagation, our experiments revealed an unexpected enhancement of excitatory responses and dendritic calcium signals in the presence of D2-receptor inhibition. Specifically, dopamine inhibition and direct hyperpolarization enabled the generation of low-threshold depolarizations that occurred in an all-or-none or graded manner, due to recruitment of T-type calcium channels. Interestingly, these effects occurred selectively in calbindin-negative dopaminergic neurons within the SNc. Thus, calbindin-positive and calbindin-negative SNc neurons differ substantially in their calcium channel composition and efficacy of excitatory inputs in the presence of dopamine inhibition.SIGNIFICANCE STATEMENT Substantia nigra dopaminergic neurons can be divided into two populations: the calbindin-negative ventral tier, which is vulnerable to neurodegeneration in Parkinson's disease, and the calbindin-positive dorsal tier, which is relatively resilient. Although tonic firing is similar in these subpopulations, we find that their responses to dopamine-mediated inhibition are strikingly different. During inhibition, calbindin-negative neurons exhibit increased sensitivity to excitatory inputs, which can then trigger large dendritic calcium transients due to strong expression of T-type calcium channels. Therefore, SNc neurons differ substantially in their calcium channel composition, which may contribute to their differential vulnerability. Furthermore, T-currents increase excitation efficacy onto calbindin-negative cells during dopamine inhibition, suggesting that shared inputs are differentially processed in subpopulations resulting in distinct downstream dopamine signals.

Enhanced Sensitivity to Hyperpolarizing Inhibition in Mesoaccumbal Relative to Nigrostriatal Dopamine Neuron Subpopulations

Midbrain dopamine neurons recorded in vivo pause their firing in response to reward omission and aversive stimuli. While the initiation of pauses typically involves synaptic or modulatory input, intrinsic membrane properties may also enhance or limit hyperpolarization, raising the question of how intrinsic conductances shape pauses in dopamine neurons. Using retrograde labeling and electrophysiological techniques combined with computational modeling, we examined the intrinsic conductances that shape pauses evoked by current injections and synaptic stimulation in subpopulations of dopamine neurons grouped according to their axonal projections to the nucleus accumbens or dorsal striatum in mice. Testing across a range of conditions and pulse durations, we found that mesoaccumbal and nigrostriatal neurons differ substantially in rebound properties with mesoaccumbal neurons displaying significantly longer delays to spiking following hyperpolarization. The underlying mechanism involves an inactivating potassium (I_A) current with decay time constants of up to 225 ms, and small-amplitude hyperpolarization-activated currents (I_H), characteristics that were most often observed in mesoaccumbal neurons. Pharmacological block of I_A completely abolished rebound delays and, importantly, shortened synaptically evoked inhibitory pauses, thereby demonstrating the involvement of A-type potassium channels in prolonging pauses evoked by GABAergic inhibition. Therefore, these results show that mesoaccumbal and nigrostriatal neurons display differential responses to hyperpolarizing inhibitory stimuli that favors a higher sensitivity to inhibition in mesoaccumbal neurons. These findings may explain, in part, observations from in vivo experiments that ventral tegmental area neurons tend to exhibit longer aversive pauses relative to SNc neurons.SIGNIFICANCE STATEMENT Our study examines rebound, postburst, and synaptically evoked inhibitory pauses in subpopulations of midbrain dopamine neurons. We show that pauses in dopamine neuron firing, evoked by either stimulation of GABAergic inputs or hyperpolarizing current injections, are enhanced by a subclass of potassium conductances that are recruited at voltages below spike threshold. Importantly, A-type potassium currents recorded in mesoaccumbal neurons displayed substantially slower inactivation kinetics, which, combined with weaker expression of hyperpolarization-activated currents, lengthened hyperpolarization-induced delays in spiking relative to nigrostriatal neurons. These results suggest that input integration differs among dopamine neurons favoring higher sensitivity to inhibition in mesoaccumbal neurons and may partially explain in vivo observations that ventral tegmental area neurons exhibit longer aversive pauses relative to SNc neurons.

Brain Computation Is Organized via Power-of-Two-Based Permutation Logic

There is considerable scientific interest in understanding how cell assemblies—the long-presumed computational motif—are organized so that the brain can generate intelligent cognition and flexible behavior. The Theory of Connectivity proposes that the origin of intelligence is rooted in a power-of-two-based permutation logic (N = 2ⁱ–1), producing specific-to-general cell-assembly architecture capable of generating specific perceptions and memories, as well as generalized knowledge and flexible actions. We show that this power-of-two-based permutation logic is widely used in cortical and subcortical circuits across animal species and is conserved for the processing of a variety of cognitive modalities including appetitive, emotional and social information. However, modulatory neurons, such as dopaminergic (DA) neurons, use a simpler logic despite their distinct subtypes. Interestingly, this specific-to-general permutation logic remained largely intact although NMDA receptors—the synaptic switch for learning and memory—were deleted throughout adulthood, suggesting that the logic is developmentally pre-configured. Moreover, this computational logic is implemented in the cortex via combining a random-connectivity strategy in superficial layers 2/3 with nonrandom organizations in deep layers 5/6. This randomness of layers 2/3 cliques—which preferentially encode specific and low-combinatorial features and project inter-cortically—is ideal for maximizing cross-modality novel pattern-extraction, pattern-discrimination and pattern-categorization using sparse code, consequently explaining why it requires hippocampal offline-consolidation. In contrast, the nonrandomness in layers 5/6—which consists of few specific cliques but a higher portion of more general cliques projecting mostly to subcortical systems—is ideal for feedback-control of motivation, emotion, consciousness and behaviors. These observations suggest that the brain’s basic computational algorithm is indeed organized by the power-of-two-based permutation logic. This simple mathematical logic can account for brain computation across the entire evolutionary spectrum, ranging from the simplest neural networks to the most complex.

Dopamine Neuron-Specific Optogenetic Stimulation in Rhesus Macaques

Optogenetic studies in mice have revealed new relationships between well-defined neurons and brain functions. However, there are currently no means to achieve the same cell-type specificity in monkeys, which possess an expanded behavioral repertoire and closer anatomical homology to humans. Here, we present a resource for cell-type-specific channelrhodopsin expression in Rhesus monkeys and apply this technique to modulate dopamine activity and monkey choice behavior. These data show that two viral vectors label dopamine neurons with greater than 95% specificity. Infected neurons were activated by light pulses, indicating functional expression. The addition of optical stimulation to reward outcomes promoted the learning of reward-predicting stimuli at the neuronal and behavioral level. Together, these results demonstrate the feasibility of effective and selective stimulation of dopamine neurons in non-human primates and a resource that could be applied to other cell types in the monkey brain.

•

Cell-type-specific promoter drives Cre-dependent ChR2 expression in monkey

•

Optogenetically activated neurons had dopamine-like features and reward responses

•

Dopamine neurons respond strongly to cues predicting optical stimulation

•

Monkeys choose predicted optogenetic stimulation over no predicted stimulation

Distributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons

Dopamine neurons encode the difference between actual and predicted reward, or reward prediction error (RPE). Although many models have been proposed to account for this computation, it has been difficult to test these models experimentally. Here we established an awake electrophysiological recording system, combined with rabies virus and optogenetic cell-type identification, to characterize the firing patterns of monosynaptic inputs to dopamine neurons while mice performed classical conditioning tasks. We found that each variable required to compute RPE, including actual and predicted reward, was distributed in input neurons in multiple brain areas. Further, many input neurons across brain areas signaled combinations of these variables. These results demonstrate that even simple arithmetic computations such as RPE are not localized in specific brain areas but, rather, distributed across multiple nodes in a brain-wide network. Our systematic method to examine both activity and connectivity revealed unexpected redundancy for a simple computation in the brain.

Resting-state functional connectivity of the human habenula in healthy individuals: Associations with subclinical depression

INTRODUCTION

The habenula (Hb) is postulated to play a critical role in reward and aversion processing across species, including humans, and has been increasingly implicated in depression. However, technical constraints have limited in vivo investigation of the human Hb, and its function remains poorly characterized. We sought to overcome these challenges by examining the whole-brain resting-state functional connectivity of the Hb and its possible relationship to depressive symptomatology using the high-resolution WU-Minn Human Connectome Project (HCP) dataset.

METHODS

Anatomical and resting-state functional MRI data from 50 healthy subjects with low or high subclinical depression scores (n = 25 each) were analyzed. Using novel semi-automated segmentation and optimization techniques, we generated individual-specific Hb seeds and calculated whole-brain functional connectivity for the entire cohort and the contrast of high vs. low depression groups.

RESULTS

In the entire cohort, the Hb exhibited significant connectivity with key brainstem structures (i.e., ventral tegmental area, substantia nigra, pons) as well as the anterior and posterior cingulate cortices, precuneus, thalamus, and sensorimotor cortex. Multiple regions showed differential Hb connectivity based on subclinical depression scores, including the amygdala, insula, and prefrontal, mid-cingulate, and entorhinal cortices.

CONCLUSIONS

Hb connectivity findings converged on areas associated with salience processing, sensorimotor systems, and the default mode network. We also detected substantial Hb-brainstem connectivity, consistent with prior histological and animal research. High and low subclinical depression groups exhibited differences in Hb connectivity with multiple regions previously linked to depression, suggesting the relationship between these structures as a potential target for future research and treatment. Hum Brain Mapp 37:2369-2384, 2016. © 2016 Wiley Periodicals, Inc.

Dopamine reward prediction-error signalling: a two-component response

Environmental stimuli and objects, including rewards, are often processed sequentially in the brain. Recent work suggests that the phasic dopamine reward prediction-error response follows a similar sequential pattern. An initial brief, unselective and highly sensitive increase in activity unspecifically detects a wide range of environmental stimuli, then quickly evolves into the main response component, which reflects subjective reward value and utility. This temporal evolution allows the dopamine reward prediction-error signal to optimally combine speed and accuracy.

Dopamine neurons share common response function for reward prediction error

Dopamine neurons are thought to signal reward prediction error, or the difference between actual and predicted reward. How dopamine neurons jointly encode this information, however, remains unclear. One possibility is that different neurons specialize in different aspects of prediction error; another is that each neuron calculates prediction error in the same way. We recorded from optogenetically-identified dopamine neurons in the lateral ventral tegmental area (VTA) while mice performed classical conditioning tasks. Our tasks allowed us to determine the full prediction error functions of dopamine neurons and compare them to each other. We found striking homogeneity among individual dopamine neurons: their responses to both unexpected and expected rewards followed the same function, just scaled up or down. As a result, we could describe both individual and population responses using just two parameters. Such uniformity ensures robust information coding, allowing each dopamine neuron to contribute fully to the prediction error signal.

Reward functions of the basal ganglia

Besides their fundamental movement function evidenced by Parkinsonian deficits, the basal ganglia are involved in processing closely linked non-motor, cognitive and reward information. This review describes the reward functions of three brain structures that are major components of the basal ganglia or are closely associated with the basal ganglia, namely midbrain dopamine neurons, pedunculopontine nucleus, and striatum (caudate nucleus, putamen, nucleus accumbens). Rewards are involved in learning (positive reinforcement), approach behavior, economic choices and positive emotions. The response of dopamine neurons to rewards consists of an early detection component and a subsequent reward component that reflects a prediction error in economic utility, but is unrelated to movement. Dopamine activations to non-rewarded or aversive stimuli reflect physical impact, but not punishment. Neurons in pedunculopontine nucleus project their axons to dopamine neurons and process sensory stimuli, movements and rewards and reward-predicting stimuli without coding outright reward prediction errors. Neurons in striatum, besides their pronounced movement relationships, process rewards irrespective of sensory and motor aspects, integrate reward information into movement activity, code the reward value of individual actions, change their reward-related activity during learning, and code own reward in social situations depending on whose action produces the reward. These data demonstrate a variety of well-characterized reward processes in specific basal ganglia nuclei consistent with an important function in non-motor aspects of motivated behavior.

Distinct Contributions of Ventromedial and Dorsolateral Subregions of the Human Substantia Nigra to Appetitive and Aversive Learning

The role of neurons in the substantia nigra (SN) and ventral tegmental area (VTA) of the midbrain in contributing to the elicitation of reward prediction errors during appetitive learning has been well established. Less is known about the differential contribution of these midbrain regions to appetitive versus aversive learning, especially in humans. Here we scanned human participants with high-resolution fMRI focused on the SN and VTA while they participated in a sequential Pavlovian conditioning paradigm involving an appetitive outcome (a pleasant juice), as well as an aversive outcome (an unpleasant bitter and salty flavor). We found a degree of regional specialization within the SN: Whereas a region of ventromedial SN correlated with a temporal difference reward prediction error during appetitive Pavlovian learning, a dorsolateral area correlated instead with an aversive expected value signal in response to the most distal cue, and to a reward prediction error in response to the most proximal cue to the aversive outcome. Furthermore, participants' affective reactions to both the appetitive and aversive conditioned stimuli more than 1 year after the fMRI experiment was conducted correlated with activation in the ventromedial and dorsolateral SN obtained during the experiment, respectively. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts.

SIGNIFICANCE STATEMENT

The role of the substantia nigra (SN) and ventral tegmental area (VTA) in appetitive learning is well established, but less is known about their contribution to aversive compared with appetitive learning, especially in humans. We used high-resolution fMRI to measure activity in the SN and VTA while participants underwent higher-order Pavlovian learning. We found a regional specialization within the SN: a ventromedial area was selectively engaged during appetitive learning, and a dorsolateral area during aversive learning. Activity in these areas predicted affective reactions to appetitive and aversive conditioned stimuli over 1 year later. These findings suggest that, whereas the human ventromedial SN contributes to long-term learning about rewards, the dorsolateral SN may be particularly important for long-term learning in aversive contexts.

Dopamine Rebound-Excitation Theory: Putting Brakes on PTSD

It is not uncommon for humans or animals to experience traumatic events in their lifetimes. However, the majority of individuals are resilient to long-term detrimental changes turning into anxiety and depression, such as post-traumatic stress disorder (PTSD). What underlying neural mechanism accounts for individual variability in stress resilience? Hyperactivity in fear circuits, such as the amygdalar system, is well-known to be the major pathophysiological basis for PTSD, much like a "stuck accelerator." Interestingly, increasing evidence demonstrates that dopamine (DA) - traditionally known for its role in motivation, reward prediction, and addiction - is also crucial in regulating fear learning and anxiety. Yet, how dopaminergic (DAergic) neurons control stress resilience is unclear, especially given that DAergic neurons have multiple subtypes with distinct temporal dynamics. Here, we propose the Rebound-Excitation Theory, which posits that DAergic neurons' rebound-excitation at the termination of fearful experiences serves as an important "brake" by providing intrinsic safety-signals to fear-processing neural circuits in a spatially and temporally controlled manner. We discuss how DAergic neuron rebound-excitation may be regulated by genetics and experiences, and how such physiological properties may be used as a brain-activity biomarker to predict and confer individual resilience to stress and anxiety.

Intact-Brain Analyses Reveal Distinct Information Carried by SNc Dopamine Subcircuits

Recent progress in understanding the diversity of midbrain dopamine neurons has highlighted the importance--and the challenges--of defining mammalian neuronal cell types. Although neurons may be best categorized using inclusive criteria spanning biophysical properties, wiring of inputs, wiring of outputs, and activity during behavior, linking all of these measurements to cell types within the intact brains of living mammals has been difficult. Here, using an array of intact-brain circuit interrogation tools, including CLARITY, COLM, optogenetics, viral tracing, and fiber photometry, we explore the diversity of dopamine neurons within the substantia nigra pars compacta (SNc). We identify two parallel nigrostriatal dopamine neuron subpopulations differing in biophysical properties, input wiring, output wiring to dorsomedial striatum (DMS) versus dorsolateral striatum (DLS), and natural activity patterns during free behavior. Our results reveal independently operating nigrostriatal information streams, with implications for understanding the logic of dopaminergic feedback circuits and the diversity of mammalian neuronal cell types.

Discrete coding of stimulus value, reward expectation, and reward prediction error in the dorsal striatum

To investigate how the striatum integrates sensory information with reward information for behavioral guidance, we recorded single-unit activity in the dorsal striatum of head-fixed rats participating in a probabilistic Pavlovian conditioning task with auditory conditioned stimuli (CSs) in which reward probability was fixed for each CS but parametrically varied across CSs. We found that the activity of many neurons was linearly correlated with the reward probability indicated by the CSs. The recorded neurons could be classified according to their firing patterns into functional subtypes coding reward probability in different forms such as stimulus value, reward expectation, and reward prediction error. These results suggest that several functional subgroups of dorsal striatal neurons represent different kinds of information formed through extensive prior exposure to CS-reward contingencies.

Components and characteristics of the dopamine reward utility signal

Rewards are defined by their behavioral functions in learning (positive reinforcement), approach behavior, economic choices, and emotions. Dopamine neurons respond to rewards with two components, similar to higher order sensory and cognitive neurons. The initial, rapid, unselective dopamine detection component reports all salient environmental events irrespective of their reward association. It is highly sensitive to factors related to reward and thus detects a maximal number of potential rewards. It also senses aversive stimuli but reports their physical impact rather than their aversiveness. The second response component processes reward value accurately and starts early enough to prevent confusion with unrewarded stimuli and objects. It codes reward value as a numeric, quantitative utility prediction error, consistent with formal concepts of economic decision theory. Thus, the dopamine reward signal is fast, highly sensitive and appropriate for driving and updating economic decisions.

Two spatiotemporally distinct value systems shape reward-based learning in the human brain

Avoiding repeated mistakes and learning to reinforce rewarding decisions is critical for human survival and adaptive actions. Yet, the neural underpinnings of the value systems that encode different decision-outcomes remain elusive. Here coupling single-trial electroencephalography with simultaneously acquired functional magnetic resonance imaging, we uncover the spatiotemporal dynamics of two separate but interacting value systems encoding decision-outcomes. Consistent with a role in regulating alertness and switching behaviours, an early system is activated only by negative outcomes and engages arousal-related and motor-preparatory brain structures. Consistent with a role in reward-based learning, a later system differentially suppresses or activates regions of the human reward network in response to negative and positive outcomes, respectively. Following negative outcomes, the early system interacts and downregulates the late system, through a thalamic interaction with the ventral striatum. Critically, the strength of this coupling predicts participants' switching behaviour and avoidance learning, directly implicating the thalamostriatal pathway in reward-based learning.