Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Glutamate inputs send prediction error of reward but not negative value of aversive stimuli to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs) but the mechanisms underlying RPE computation, particularly contributions of different neurotransmitters, remain poorly understood. Here we used a genetically-encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons. We found that glutamate inputs exhibit virtually all of the characteristics of RPE, rather than conveying a specific component of RPE computation such as reward or expectation. Notably, while glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli toward more positive responses, while excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning

How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.

What is resilience: an affiliative neuroscience approach

Resilience - a key topic in clinical science and practice - still lacks a clear conceptualization that integrates its evolutionary and human-specific features, refrains from exclusive focus on fear physiology, incorporates a developmental approach, and, most importantly, is not based on the negation (i.e., absence of symptoms following trauma). Building on the initial condition of mammals, whose brain matures in the context of the mother's body and caregiving behavior, we argue that systems and processes that participate in tuning the brain to the social ecology and adapting to its hardships mark the construct of resilience. These include the oxytocin system, the affiliative brain, and biobehavioral synchrony, all characterized by great flexibility across phylogenesis and ontogenesis. Three core features of resilience are outlined: plasticity, sociality and meaning. Mechanisms of sociality by which coordinated action supports diversity, endurance and adaptation are described across animal evolution. Humans' biobehavioral synchrony matures from maternal attuned behavior in the postpartum to adult-adult relationships of empathy, perspective-taking and intimacy, and extends from the mother-child relationship to other affiliative bonds throughout life, charting a fundamental trajectory in the development of resilience. Findings from three high-risk cohorts, each tapping a distinct disruption to maternal-infant bonding (prematurity, maternal depression, and early life stress/trauma), and followed from birth to adolescence/young adulthood, demonstrate how components of the neurobiology of affiliation confer resilience and uniquely shape the social brain.

Distributed and Mixed Information in Monosynaptic Inputs to Dopamine Neurons

Dopamine neurons encode the difference between actual and predicted reward, or reward prediction error (RPE). Although many models have been proposed to account for this computation, it has been difficult to test these models experimentally. Here we established an awake electrophysiological recording system, combined with rabies virus and optogenetic cell-type identification, to characterize the firing patterns of monosynaptic inputs to dopamine neurons while mice performed classical conditioning tasks. We found that each variable required to compute RPE, including actual and predicted reward, was distributed in input neurons in multiple brain areas. Further, many input neurons across brain areas signaled combinations of these variables. These results demonstrate that even simple arithmetic computations such as RPE are not localized in specific brain areas but, rather, distributed across multiple nodes in a brain-wide network. Our systematic method to examine both activity and connectivity revealed unexpected redundancy for a simple computation in the brain.

Complex Multiplexing of Reward-Cue- and Licking-Movement-Related Activity in Single Midline Thalamus Neurons

Midline thalamus is implicated in linking visceral and exteroceptive sensory information with behavior. However, whether neuronal activity is modulated with temporal precision by cues and actions in real time is unknown. Using single-neuron recording and a Pavlovian visual-cue/liquid-reward association task in rats, we discovered phasic responses to sensory cues, appropriately timed to modify information processing in output targets, as well as tonic modulations within and between trials that were differentially reward modulated, which may have distinct arousal functions. Many of the cue-responsive neurons also responded to repetitive licks, consistent with sensorimotor integration. Further, some lick-related neurons were activated only by the first rewarded lick and only if that lick were also part of a conditioned response sequence initiated earlier, consistent with binding action decisions to their ensuing outcome. This rich repertoire of responses provides electrophysiological evidence for midline thalamus as a site of complex information integration for reward-mediated behavior.

SIGNIFICANCE STATEMENT

Disparate brain circuits are involved in sensation, movement, and reward information. These must interact in order for the relationships between cues, actions, and outcomes to be learned. We found that responses of single neurons in midline thalamus to sensory cues are increased when associated with reward. This output may amplify similar signals generated in parallel by the dopamine system. In addition, some neurons coded a three-factor decision in which the neuron fired only if there was a movement, if it was the first one after the reward becoming available, and if it was part of a sequence triggered in response to a preceding cue. These data highlight midline thalamus as an important node integrating multiple types of information for linking sensation, actions, and rewards.

Dopamine Prediction Errors in Reward Learning and Addiction: From Theory to Neural Circuitry

Midbrain dopamine (DA) neurons are proposed to signal reward prediction error (RPE), a fundamental parameter in associative learning models. This RPE hypothesis provides a compelling theoretical framework for understanding DA function in reward learning and addiction. New studies support a causal role for DA-mediated RPE activity in promoting learning about natural reward; however, this question has not been explicitly tested in the context of drug addiction. In this review, we integrate theoretical models with experimental findings on the activity of DA systems, and on the causal role of specific neuronal projections and cell types, to provide a circuit-based framework for probing DA-RPE function in addiction. By examining error-encoding DA neurons in the neural network in which they are embedded, hypotheses regarding circuit-level adaptations that possibly contribute to pathological error signaling and addiction can be formulated and tested.

Resting-State Functional Connectivity of the Locus Coeruleus in Humans: In Comparison with the Ventral Tegmental Area/Substantia Nigra Pars Compacta and the Effects of Age

The locus coeruleus (LC) provides the primary noradrenergic inputs to the cerebral cortex. Despite numerous animal studies documenting the functions of the LC, research in humans is hampered by the small volume of this midbrain nucleus. Here, we took advantage of a probabilistic template, explored the cerebral functional connectivity of the LC with resting-state fMRI data of 250 healthy adults, and verified the findings by accounting for physiological noise in another data set. In addition, we contrasted connectivities of the LC and the ventral tegmental area/substantia nigra pars compacta. The results highlighted both shared and distinct connectivity of these 2 midbrain structures, as well as an opposite pattern of connectivity to bilateral amygdala, pulvinar, and right anterior insula. Additionally, LC connectivity to the fronto-parietal cortex and the cerebellum increases with age and connectivity to the visual cortex decreases with age. These findings may facilitate studies of the role of the LC in arousal, saliency responses and cognitive motor control and in the behavioral and cognitive manifestations during healthy and disordered aging. Although the first to demonstrate whole-brain LC connectivity, these findings need to be confirmed with high-resolution imaging.

Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning

There are two prevailing notions regarding the involvement of the corticobasal ganglia system in value‐based learning: (i) the direct and indirect pathways of the basal ganglia are crucial for appetitive and aversive learning, respectively, and (ii) the activity of midbrain dopamine neurons represents reward‐prediction error. Although (ii) constitutes a critical assumption of (i), it remains elusive how (ii) holds given (i), with the basal‐ganglia influence on the dopamine neurons. Here we present a computational neural‐circuit model that potentially resolves this issue. Based on the latest analyses of the heterogeneous corticostriatal neurons and connections, our model posits that the direct and indirect pathways, respectively, represent the values of upcoming and previous actions, and up‐regulate and down‐regulate the dopamine neurons via the basal‐ganglia output nuclei. This explains how the difference between the upcoming and previous values, which constitutes the core of reward‐prediction error, is calculated. Simultaneously, it predicts that blockade of the direct/indirect pathway causes a negative/positive shift of reward‐prediction error and thereby impairs learning from positive/negative error, i.e. appetitive/aversive learning. Through simulation of reward‐reversal learning and punishment‐avoidance learning, we show that our model could indeed account for the experimentally observed features that are suggested to support notion (i) and could also provide predictions on neural activity. We also present a behavioral prediction of our model, through simulation of inter‐temporal choice, on how the balance between the two pathways relates to the subject's time preference. These results indicate that our model, incorporating the heterogeneity of the cortical influence on the basal ganglia, is expected to provide a closed‐circuit mechanistic understanding of appetitive/aversive learning.

Reinforcement learning in depression: A review of computational research

Despite being considered primarily a mood disorder, major depressive disorder (MDD) is characterized by cognitive and decision making deficits. Recent research has employed computational models of reinforcement learning (RL) to address these deficits. The computational approach has the advantage in making explicit predictions about learning and behavior, specifying the process parameters of RL, differentiating between model-free and model-based RL, and the computational model-based functional magnetic resonance imaging and electroencephalography. With these merits there has been an emerging field of computational psychiatry and here we review specific studies that focused on MDD. Considerable evidence suggests that MDD is associated with impaired brain signals of reward prediction error and expected value ('wanting'), decreased reward sensitivity ('liking') and/or learning (be it model-free or model-based), etc., although the causality remains unclear. These parameters may serve as valuable intermediate phenotypes of MDD, linking general clinical symptoms to underlying molecular dysfunctions. We believe future computational research at clinical, systems, and cellular/molecular/genetic levels will propel us toward a better understanding of the disease.

Effects of fictive reward on rat's choice behavior

Choices of humans and non-human primates are influenced by both actually experienced and fictive outcomes. To test whether this is also the case in rodents, we examined rat's choice behavior in a binary choice task in which variable magnitudes of actual and fictive rewards were delivered. We found that the animal's choice was significantly influenced by the magnitudes of both actual and fictive rewards in the previous trial. A model-based analysis revealed, however, that the effect of fictive reward was more transient and influenced mostly the choice in the next trial, whereas the effect of actual reward was more sustained, consistent with incremental learning of action values. Our results suggest that the capacity to modify future choices based on fictive outcomes might be shared by many different animal species, but fictive outcomes are less effective than actual outcomes in the incremental value learning system.

Applications of Modern HCIs in Adaptive Mobile Learning

Our paper shows how the evolution of HCI devices progresses as a mobile learning tool. Mobile devices provide interesting applications for cognitive infocommunication. Our principal objective is to assist in developing educational games on these devices. Working with different educational institutes, we designed a flexible biofeedback-controlled self-rewarding framework. Several promising approaches and methods are proposed outside the box of educational games in this paper. The attention of players is regulated by changing rewards. We show both how educational games can be improved and how adaptive entertainment games may be developed in the near future.

Distinct dopaminergic control of the direct and indirect pathways in reward-based and avoidance learning behaviors

The nucleus accumbens (NAc) plays a pivotal role in reward and aversive learning and learning flexibility. Outputs of the NAc are transmitted through two parallel routes termed the direct and indirect pathways and controlled by the dopamine (DA) neurotransmitter. To explore how reward-based and avoidance learning is controlled in the NAc of the mouse, we developed the reversible neurotransmission-blocking (RNB) technique, in which transmission of each pathway could be selectively and reversibly blocked by the pathway-specific expression of transmission-blocking tetanus toxin and the asymmetric RNB technique, in which one side of the NAc was blocked by the RNB technique and the other intact side was pharmacologically manipulated by a transmitter agonist or antagonist. Our studies demonstrated that the activation of D1 receptors in the direct pathway and the inactivation of D2 receptors in the indirect pathway are key determinants that distinctly control reward-based and avoidance learning, respectively. The D2 receptor inactivation is also critical for flexibility of reward learning. Furthermore, reward and aversive learning is regulated by a set of common downstream receptors and signaling cascades, all of which are involved in the induction of long-term potentiation at cortico-accumbens synapses of the two pathways. In this article, we review our studies that specify the regulatory mechanisms of each pathway in learning behavior and propose a mechanistic model to explain how dynamic DA modulation promotes selection of actions that achieve reward-seeking outcomes and avoid aversive ones. The biological significance of the network organization consisting of two parallel transmission pathways is also discussed from the point of effective and prompt selection of neural outcomes in the neural network.

Behavioral flexibility is increased by optogenetic inhibition of neurons in the nucleus accumbens shell during specific time segments

Behavioral flexibility is vital for survival in an environment of changing contingencies. The nucleus accumbens may play an important role in behavioral flexibility, representing learned stimulus–reward associations in neural activity during response selection and learning from results. To investigate the role of nucleus accumbens neural activity in behavioral flexibility, we used light-activated halorhodopsin to inhibit nucleus accumbens shell neurons during specific time segments of a bar-pressing task requiring a win–stay/lose–shift strategy. We found that optogenetic inhibition during action selection in the time segment preceding a lever press had no effect on performance. However, inhibition occurring in the time segment during feedback of results—whether rewards or nonrewards—reduced the errors that occurred after a change in contingency. Our results demonstrate critical time segments during which nucleus accumbens shell neurons integrate feedback into subsequent responses. Inhibiting nucleus accumbens shell neurons in these time segments, during reinforced performance or after a change in contingencies, increases lose–shift behavior. We propose that the activity of nucleus shell accumbens shell neurons in these time segments plays a key role in integrating knowledge of results into subsequent behavior, as well as in modulating lose–shift behavior when contingencies change.

Heads for learning, tails for memory: reward, reinforcement and a role of dopamine in determining behavioral relevance across multiple timescales

Dopamine has long been tightly associated with aspects of reinforcement learning and motivation in simple situations where there are a limited number of stimuli to guide behavior and constrained range of outcomes. In naturalistic situations, however, there are many potential cues and foraging strategies that could be adopted, and it is critical that animals determine what might be behaviorally relevant in such complex environments. This requires not only detecting discrepancies with what they have recently experienced, but also identifying similarities with past experiences stored in memory. Here, we review what role dopamine might play in determining how and when to learn about the world, and how to develop choice policies appropriate to the situation faced. We discuss evidence that dopamine is shaped by motivation and memory and in turn shapes reward-based memory formation. In particular, we suggest that hippocampal-striatal-dopamine networks may interact to determine how surprising the world is and to either inhibit or promote actions at time of behavioral uncertainty.

Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior

Humans and animals take actions quickly when they expect that the actions lead to reward, reflecting their motivation. Injection of dopamine receptor antagonists into the striatum has been shown to slow such reward-seeking behavior, suggesting that dopamine is involved in the control of motivational processes. Meanwhile, neurophysiological studies have revealed that phasic response of dopamine neurons appears to represent reward prediction error, indicating that dopamine plays central roles in reinforcement learning. However, previous attempts to elucidate the mechanisms of these dopaminergic controls have not fully explained how the motivational and learning aspects are related and whether they can be understood by the way the activity of dopamine neurons itself is controlled by their upstream circuitries. To address this issue, we constructed a closed-circuit model of the corticobasal ganglia system based on recent findings regarding intracortical and corticostriatal circuit architectures. Simulations show that the model could reproduce the observed distinct motivational effects of D1- and D2-type dopamine receptor antagonists. Simultaneously, our model successfully explains the dopaminergic representation of reward prediction error as observed in behaving animals during learning tasks and could also explain distinct choice biases induced by optogenetic stimulation of the D1 and D2 receptor-expressing striatal neurons. These results indicate that the suggested roles of dopamine in motivational control and reinforcement learning can be understood in a unified manner through a notion that the indirect pathway of the basal ganglia represents the value of states/actions at a previous time point, an empirically driven key assumption of our model.

Direct-pathway striatal neurons regulate the retention of decision-making strategies

The dorsal striatum has been implicated in reward-based decision making, but the role played by specific striatal circuits in these processes is essentially unknown. Using cell phenotype-specific viral vectors to express engineered G-protein-coupled DREADD (designer receptors exclusively activated by designer drugs) receptors, we enhanced Gi/o- or Gs-protein-mediated signaling selectively in direct-pathway (striatonigral) neurons of the dorsomedial striatum in Long-Evans rats during discrete periods of training of a high versus low reward-discrimination task. Surprisingly, these perturbations had no impact on reward preference, task performance, or improvement of performance during training. However, we found that transiently increasing Gi/o signaling during training significantly impaired the retention of task strategies used to maximize reward obtainment during subsequent preference testing, whereas increasing Gs signaling produced the opposite effect and significantly enhanced the encoding of a high-reward preference in this decision-making task. Thus, the fact that the endurance of this improved performance was significantly altered over time-long after these neurons were manipulated-indicates that it is under bidirectional control of canonical G-protein-mediated signaling in striatonigral neurons during training. These data demonstrate that cAMP-dependent signaling in direct-pathway neurons play a well-defined role in reward-related behavior; that is, they modulate the plasticity required for the retention of task-specific information that is used to improve performance on future renditions of the task.

Beyond simple reinforcement learning: the computational neurobiology of reward-learning and valuation

Neural computational accounts of reward-learning have been dominated by the hypothesis that dopamine neurons behave like a reward-prediction error and thus facilitate reinforcement learning in striatal target neurons. While this framework is consistent with a lot of behavioral and neural evidence, this theory fails to account for a number of behavioral and neurobiological observations. In this special issue of EJN we feature a combination of theoretical and experimental papers highlighting some of the explanatory challenges faced by simple reinforcement-learning models and describing some of the ways in which the framework is being extended in order to address these challenges.

Temporal prediction errors modulate cingulate-insular coupling

Prediction error (i.e., the difference between the expected and the actual event's outcome) mediates adaptive behavior. Activity in the anterior mid-cingulate cortex (aMCC) and in the anterior insula (aINS) is associated with the commission of prediction errors under uncertainty. We propose a dynamic causal model of effective connectivity (i.e., neuronal coupling) between the aMCC, the aINS, and the striatum in which the task context drives activity in the aINS and the temporal prediction errors modulate extrinsic cingulate-insular connections. With functional magnetic resonance imaging, we scanned 15 participants when they performed a temporal prediction task. They observed visual animations and predicted when a stationary ball began moving after being contacted by another moving ball. To induced uncertainty-driven prediction errors, we introduced spatial gaps and temporal delays between the balls. Classical and Bayesian fMRI analyses provided evidence to support that the aMCC-aINS system along with the striatum not only responds when humans predict whether a dynamic event occurs but also when it occurs. Our results reveal that the insula is the entry port of a three-region pathway involved in the processing of temporal predictions. Moreover, prediction errors rather than attentional demands, task difficulty, or task duration exert an influence in the aMCC-aINS system. Prediction errors debilitate the effect of the aMCC on the aINS. Finally, our computational model provides a way forward to characterize the physiological parallel of temporal prediction errors elicited in dynamic tasks.

The role of uncertainty and reward on eye movements in a virtual driving task

Eye movements during natural tasks are well coordinated with ongoing task demands and many variables could influence gaze strategies. Sprague and Ballard (2003) proposed a gaze-scheduling model that uses a utility-weighted uncertainty metric to prioritize fixations on task-relevant objects and predicted that human gaze should be influenced by both reward structure and task-relevant uncertainties. To test this conjecture, we tracked the eye movements of participants in a simulated driving task where uncertainty and implicit reward (via task priority) were varied. Participants were instructed to simultaneously perform a Follow Task where they followed a lead car at a specific distance and a Speed Task where they drove at an exact speed. We varied implicit reward by instructing the participants to emphasize one task over the other and varied uncertainty in the Speed Task with the presence or absence of uniform noise added to the car's velocity. Subjects' gaze data were classified for the image content near fixation and segmented into looks. Gaze measures, including look proportion, duration and interlook interval, showed that drivers more closely monitor the speedometer if it had a high level of uncertainty, but only if it was also associated with high task priority or implicit reward. The interaction observed appears to be an example of a simple mechanism whereby the reduction of visual uncertainty is gated by behavioral relevance. This lends qualitative support for the primary variables controlling gaze allocation proposed in the Sprague and Ballard model.

Reinforcement learning: computing the temporal difference of values via distinct corticostriatal pathways

Midbrain dopamine neurons supposedly encode reward prediction error, but how error signals are computed remains elusive. Here, we propose a mechanism based on recent findings regarding corticostriatal circuits. Specifically, we propose that two distinct subpopulations of corticostriatal neurons differentially represent the animal's current and previous states/actions through unidirectional connectivity from one subpopulation to the other and strong recurrent excitation that exists only within the recipient subpopulation. These corticostriatal subpopulations selectively connect to the direct and indirect pathways of the basal ganglia, such that the temporal difference between the values of current and previous states/actions--the core of the error signal--can be computed. Our hypothesis suggests a unified view of basal ganglia functions and has important clinical implications.