Neural Circuitry of Reward Prediction Error

Consistency of Medical Data Using Intelligent Neuron Faster R-CNN Algorithm for Smart Health Care Application

The purpose of this study is to increase interest in health as human life is extended in modern society. Hence, many people in hospitals produce much medical data (EMR, PACS, OCS, EHR, MRI, X-ray) after treatment. Medical data are stored as structured and unstructured data. However, many medical data are causing errors, omissions and mistakes in the process of reading. This behavior is very important in dealing with human life and sometimes leads to medical accidents due to physician errors. Therefore, this research is conducted through the CNN intelligent agent cloud architecture to verify errors in reading existing medical image data. To reduce the error rule when reading medical image data, a faster R-CNN intelligent agent cloud architecture is proposed. It shows the result of increasing errors of existing error reading by more than 1.4 times (140%). In particular, it is an algorithm that analyses data stored by actual existing medical data through Conv feature map using deep ConvNet and ROI Projection. The data were verified using about 120,000 databases. It uses data to examine human lungs. In addition, the experimental environment established an environment that can handle GPU's high performance and NVIDIA SLI multi-OS and multiple Quadro GPUs were used. In this experiment, the verification data composition was verified and randomly extracted from about 120,000 medical records and the similarity compared to the original data were measured by comparing about 40% of the extracted images. Finally, we want to reduce and verify the error rate of medical data reading.

Cell-Type-Specific Outcome Representation in the Primary Motor Cortex

Adaptive movements are critical for animal survival. To guide future actions, the brain monitors various outcomes, including achievement of movement and appetitive goals. The nature of these outcome signals and their neuronal and network realization in the motor cortex (M1), which directs skilled movements, is largely unknown. Using a dexterity task, calcium imaging, optogenetic perturbations, and behavioral manipulations, we studied outcome signals in the murine forelimb M1. We found two populations of layer 2-3 neurons, termed success- and failure-related neurons, that develop with training, and report end results of trials. In these neurons, prolonged responses were recorded after success or failure trials independent of reward and kinematics. In addition, the initial state of layer 5 pyramidal tract neurons contained a memory trace of the previous trial's outcome. Intertrial cortical activity was needed to learn new task requirements. These M1 layer-specific performance outcome signals may support reinforcement motor learning of skilled behavior.

Direct dopamine terminal regulation by local striatal microcircuitry

Regulation of axonal dopamine release by local microcircuitry is at the hub of several biological processes that govern the timing and magnitude of signaling events in reward-related brain regions. An important characteristic of dopamine release from axon terminals in the striatum is that it is rapidly modulated by local regulatory mechanisms. These processes can occur via homosynaptic mechanisms-such as presynaptic dopamine autoreceptors and dopamine transporters - as well heterosynaptic mechanisms such as retrograde signaling from postsynaptic cholinergic and dynorphin systems, among others. Additionally, modulation of dopamine release via diffusible messengers, such as nitric oxide and hydrogen peroxide, allows for various metabolic factors to quickly and efficiently regulate dopamine release and subsequent signaling. Here we review how these mechanisms work in concert to influence the timing and magnitude of striatal dopamine signaling, independent of action potential activity at the level of dopaminergic cell bodies in the midbrain, thereby providing a parallel pathway by which dopamine can be modulated. Understanding the complexities of local regulation of dopamine signaling is required for building comprehensive frameworks of how activity throughout the dopamine system is integrated to drive signaling and control behavior.

Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys

Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.

Recurrent architecture for adaptive regulation of learning in the insect brain

Dopaminergic neurons (DANs) drive learning across the animal kingdom, but the upstream circuits that regulate their activity and thereby learning remain poorly understood. We provide a synaptic-resolution connectome of the circuitry upstream of all DANs in a learning center, the mushroom body of Drosophila larva. We discover afferent sensory pathways and a large population of neurons that provide feedback from mushroom body output neurons and link distinct memory systems (aversive and appetitive). We combine this with functional studies of DANs and their presynaptic partners and with comprehensive circuit modeling. We find that DANs compare convergent feedback from aversive and appetitive systems, which enables the computation of integrated predictions that may improve future learning. Computational modeling reveals that the discovered feedback motifs increase model flexibility and performance on learning tasks. Our study provides the most detailed view to date of biological circuit motifs that support associative learning.

Amphetamine disrupts haemodynamic correlates of prediction errors in nucleus accumbens and orbitofrontal cortex

In an uncertain world, the ability to predict and update the relationships between environmental cues and outcomes is a fundamental element of adaptive behaviour. This type of learning is typically thought to depend on prediction error, the difference between expected and experienced events and in the reward domain that has been closely linked to mesolimbic dopamine. There is also increasing behavioural and neuroimaging evidence that disruption to this process may be a cross-diagnostic feature of several neuropsychiatric and neurological disorders in which dopamine is dysregulated. However, the precise relationship between haemodynamic measures, dopamine and reward-guided learning remains unclear. To help address this issue, we used a translational technique, oxygen amperometry, to record haemodynamic signals in the nucleus accumbens (NAc) and orbitofrontal cortex (OFC), while freely moving rats performed a probabilistic Pavlovian learning task. Using a model-based analysis approach to account for individual variations in learning, we found that the oxygen signal in the NAc correlated with a reward prediction error, whereas in the OFC it correlated with an unsigned prediction error or salience signal. Furthermore, an acute dose of amphetamine, creating a hyperdopaminergic state, disrupted rats' ability to discriminate between cues associated with either a high or a low probability of reward and concomitantly corrupted prediction error signalling. These results demonstrate parallel but distinct prediction error signals in NAc and OFC during learning, both of which are affected by psychostimulant administration. Furthermore, they establish the viability of tracking and manipulating haemodynamic signatures of reward-guided learning observed in human fMRI studies by using a proxy signal for BOLD in a freely behaving rodent.

Dopaminergic activity and exercise behavior in anorexia nervosa

Driven exercise (i.e., the tendency to exercise in excess to influence weight/shape or regulate emotion) is difficult to manage in the context of anorexia nervosa, and is associated with poorer treatment outcomes, and psychological and medical severity. Driven exercise is observed in a considerable number of those diagnosed with anorexia nervosa; however, to date, this hallmark symptom remains poorly understood. Dopamine signaling is implicated in motivating and maintaining appetitive behavior among patients with eating disorders; but, much less is known about the role of dopamine signaling specific to the symptom of driven exercise. An improved understanding of this biobehavioral mechanism may inform the etiology of driven exercise in anorexia nervosa, with the potential to impact future research and treatment efforts. This review describes the role that dopamine serves in maintaining symptoms in the context of anorexia nervosa, and synthesizes current relevant evidence on exercise in AN and related dopaminergic activity. Throughout, theoretical implications are discussed, along with critical directions for future research.

A distributional code for value in dopamine-based reinforcement learning

Since its introduction, the reward prediction error theory of dopamine has explained a wealth of empirical phenomena, providing a unifying framework for understanding the representation of reward and value in the brain^1-3. According to the now canonical theory, reward predictions are represented as a single scalar quantity, which supports learning about the expectation, or mean, of stochastic outcomes. Here we propose an account of dopamine-based reinforcement learning inspired by recent artificial intelligence research on distributional reinforcement learning^4-6. We hypothesized that the brain represents possible future rewards not as a single mean, but instead as a probability distribution, effectively representing multiple future outcomes simultaneously and in parallel. This idea implies a set of empirical predictions, which we tested using single-unit recordings from mouse ventral tegmental area. Our findings provide strong evidence for a neural realization of distributional reinforcement learning.

Dopamine D1 and muscarinic acetylcholine receptors in dorsal striatum are required for high speed running

Dopamine (DA) signaling in the basal ganglia plays important roles in motor control. Motor deficiencies were previously reported in dopamine receptor D1 (D1R) and D2 (D2R) knockout mice. While these results indicate the involvement of DA receptors in motor execution, the null knockout (KO) mouse lacks the specificity necessary to determine when and where in the brain D1R and D2R function in motor execution. To address these questions, we restricted the loss of function temporally and spatially by using D1R conditional knockdown (cKD) mice and mice injected with antagonists against DA receptors directly into the dorsal striatum. In addition, we address the DA and acetylcholine (ACh) balance hypothesis by using antagonists against ACh receptors. We tested the motor ability of the mice with a newly devised task named the accelerating step-wheel. In this task, the maximum running speed was measured in a situation where the wheel rotation speed was gradually accelerated in one trial. We found significant decreases in the maximum running speed of D1R cKD mice and the mice injected with the antagonist against D1R or muscarinic ACh receptor. These results indicated that D1R and muscarinic ACh receptor in the dorsal striatum play pivotal roles in the execution of walking/running.

Believing in dopamine

Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.

Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico-basal ganglia systems

This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the “landscape” (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.

Three Rostromedial Tegmental Afferents Drive Triply Dissociable Aspects of Punishment Learning and Aversive Valence Encoding

Persistence of reward seeking despite punishment or other negative consequences is a defining feature of mania and addiction, and numerous brain regions have been implicated in such punishment learning, but in disparate ways that are difficult to reconcile. We now show that the ability of an aversive punisher to inhibit reward seeking depends on coordinated activity of three distinct afferents to the rostromedial tegmental nucleus (RMTg) arising from cortex, brainstem, and habenula that drive triply dissociable RMTg responses to aversive cues, outcomes, and prediction errors, respectively. These three pathways drive correspondingly dissociable aspects of punishment learning. The RMTg in turn drives negative, but not positive, valence encoding patterns in the ventral tegmental area (VTA). Hence, punishment learning involves pathways and functions that are highly distinct, yet tightly coordinated.

In vivo functional diversity of midbrain dopamine neurons within identified axonal projections

Functional diversity of midbrain dopamine (DA) neurons ranges across multiple scales, from differences in intrinsic properties and connectivity to selective task engagement in behaving animals. Distinct in vitro biophysical features of DA neurons have been associated with different axonal projection targets. However, it is unknown how this translates to different firing patterns of projection-defined DA subpopulations in the intact brain. We combined retrograde tracing with single-unit recording and labelling in mouse brain to create an in vivo functional topography of the midbrain DA system. We identified differences in burst firing among DA neurons projecting to dorsolateral striatum. Bursting also differentiated DA neurons in the medial substantia nigra (SN) projecting either to dorsal or ventral striatum. We found differences in mean firing rates and pause durations among ventral tegmental area (VTA) DA neurons projecting to lateral or medial shell of nucleus accumbens. Our data establishes a high-resolution functional in vivo landscape of midbrain DA neurons.

GABAAR α2-activated neuroimmune signal controls binge drinking and impulsivity through regulation of the CCL2/CX3CL1 balance

BACKGROUND AND PURPOSE

Toll-like receptors (TLRs) are a family of innate immune system receptors that respond to pathogen-derived and tissue damage-related ligands and are increasingly recognized for their impact on homeostasis and its dysregulation in the nervous system. TLR signaling participates in brain injury and addiction, but its role in the alcohol-seeking behavior, which initiates alcohol drinking, is still poorly understood. In this review, we discuss our findings designed to elucidate the potential contribution of the activated TLR4 signal located in neurons, on impulsivity and the predisposition to initiate alcohol drinking (binge drinking).

RESULTS

Our findings indicate that the TLR4 signal is innately activated in neurons from alcohol-preferring subjects, identifying a genetic contribution to the regulation of impulsivity and the alcohol-seeking propensity. Signal activation is through the non-canonical, previously unknown, binding of TLR4 to the α2 subunit of the γ-aminobutyric 2 acid A receptor (GABAAR α2). Activation is sustained by the stress hormone corticotrophin-releasing factor (CRF) and additional still poorly recognized ligand/scaffold proteins. Focus is on the effect of TLR4 signal activation on the balance between pro- and anti-inflammatory chemokines [chemokine (C-C motif) ligand 2 (CCL2)/chemokine (C-X3-C motif) ligand 1 (CX3CL1)] and its effect on binge drinking.

CONCLUSION

The results are discussed within the context of current findings on the distinct activation and functions of TLR signals located in neurons, as opposed to immune cells. They indicate that the balance between pro- and anti-inflammatory TLR4 signaling plays a major role in binge drinking. These findings have major impact on future basic and translational research, including the development of potential therapeutic and preventative strategies.

Neuromodulation in circuits of aversive emotional learning

Emotional learning and memory are functionally and dysfunctionally regulated by the neuromodulatory state of the brain. While the role of excitatory and inhibitory neural circuits mediating emotional learning and its control have been the focus of much research, we are only now beginning to understand the more diffuse role of neuromodulation in these processes. Recent experimental studies of the acetylcholine, noradrenaline and dopamine systems in fear learning and extinction of fear responding provide surprising answers to key questions in neuromodulation. One area of research has revealed how modular organization, coupled with context-dependent coding modes, allows for flexible brain-wide or targeted neuromodulation. Other work has shown how these neuromodulators act in downstream targets to enhance signal-to-noise ratios and gain, as well as to bind distributed circuits through neuronal oscillations. These studies elucidate how different neuromodulatory systems regulate aversive emotional processing and reveal fundamental principles of neuromodulatory function.

Characterization of glutamatergic VTA neural population responses to aversive and rewarding conditioning in freely-moving mice

The Ventral Tegmental Area (VTA) is a midbrain structure known to integrate aversive and rewarding stimuli, but little is known about the role of VTA glutamatergic (VGluT2) neurons in these functions. Direct activation of VGluT2 soma evokes rewarding behaviors, while activation of their downstream projections evokes aversive behaviors. To facilitate our understanding of these conflicting properties, we recorded calcium signals from VTA^VGluT2+ neurons using fiber photometry in VGluT2-cre mice to investigate how this population was recruited by aversive and rewarding stimulation, both during unconditioned and conditioned protocols. Our results revealed that, as a population, VTA^VGluT2+ neurons responded similarly to unconditioned-aversive and unconditioned-rewarding stimulation. During aversive and rewarding conditioning, the CS-evoked responses gradually increased across trials whilst the US-evoked response remained stable. Retrieval 24 h after conditioning, during which mice received only CS presentation, resulted in VTA^VGluT2+ neurons strongly responding to CS presentation and to the expected-US but only for aversive conditioning. To help understand these differences based on VTA^VGluT2+ neuronal networks, the inputs and outputs of VTA^VGluT2+ neurons were investigated using Cholera Toxin B (CTB) and rabies virus. Based on our results, we propose that the divergent VTA^VGluT2+ neuronal responses to aversion and reward conditioning may be partly due to the existence of VTA^VGluT2+ subpopulations that are characterized by their connectivity.

The basal ganglia and the cerebellum: nodes in an integrated network

The basal ganglia and the cerebellum are considered to be distinct subcortical systems that perform unique functional operations. The outputs of the basal ganglia and the cerebellum influence many of the same cortical areas but do so by projecting to distinct thalamic nuclei. As a consequence, the two subcortical systems were thought to be independent and to communicate only at the level of the cerebral cortex. Here, we review recent data showing that the basal ganglia and the cerebellum are interconnected at the subcortical level. The subthalamic nucleus in the basal ganglia is the source of a dense disynaptic projection to the cerebellar cortex. Similarly, the dentate nucleus in the cerebellum is the source of a dense disynaptic projection to the striatum. These observations lead to a new functional perspective that the basal ganglia, the cerebellum and the cerebral cortex form an integrated network. This network is topographically organized so that the motor, cognitive and affective territories of each node in the network are interconnected. This perspective explains how synaptic modifications or abnormal activity at one node can have network-wide effects. A future challenge is to define how the unique learning mechanisms at each network node interact to improve performance.

Discrete Evaluative and Premotor Circuits Enable Vocal Learning in Songbirds

Virtuosic motor performance requires the ability to evaluate and modify individual gestures within a complex motor sequence. Where and how the evaluative and premotor circuits operate within the brain to enable such temporally precise learning is poorly understood. Songbirds can learn to modify individual syllables within their complex vocal sequences, providing a system for elucidating the underlying evaluative and premotor circuits. We combined behavioral and optogenetic methods to identify 2 afferents to the ventral tegmental area (VTA) that serve evaluative roles in syllable-specific learning and to establish that downstream cortico-basal ganglia circuits serve a learning role that is only premotor. Furthermore, song performance-contingent optogenetic stimulation of either VTA afferent was sufficient to drive syllable-specific learning, and these learning effects were of opposite valence. Finally, functional, anatomical, and molecular studies support the idea that these evaluative afferents bidirectionally modulate VTA dopamine neurons to enable temporally precise vocal learning.

Spiraling Connectivity of NAc-VTA Circuitry

How do nucleus accumbens (NAc) subdivisions shape information flow into distinct ventral tegmental area (VTA) subcircuits? Yang et al. (2018) provide insightful answers to this question by expanding our knowledge about the circuit architecture and function of reciprocal connectivity between NAc and VTA.

Identification of midbrain dopamine neurons using features from spontaneous spike activity patterns. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2019;2019:2990-2993. [PMID: 31946517 DOI: 10.1109/embc.2019.8857574] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Although dopamine neurons are considered essential in various brain functions, their specific roles are under debate, partially due to the difficulty to identify dopamine neurons among surrounding neurons deep in the brain based only on the extracellularly recorded electric activities. Thus, a handy method to identify dopamine and non-dopamine neurons based on the spontaneous activity patterns is desired. Here we tried to discriminate optogenetically-identified dopamine neurons from other types of neurons and obtained 86.0% success.

Corollary Discharge for Action and Cognition

In motor systems, a copy of the movement command known as corollary discharge is broadcast to other regions of the brain to warn them of the impending movement. The premise of this review is that the concept of corollary discharge may generalize in revealing ways to the brain's cognitive systems. An oculomotor pathway from the brain stem to frontal cortex provides a well-established example of how corollary discharge is instantiated for sensorimotor processing. Building on causal evidence from inactivation of the pathway, we motivate forward models as a tool for understanding the contributions of corollary discharge to perception and movement. Finally, we extend the definition of corollary discharge to account for signals that may be used for cognitive forward models of decision making. This framework may provide new insights into signals and circuits that contribute to sequential decision processes, the breakdown of which may account for some symptoms of psychiatric disorders.

Synchronicity: The Role of Midbrain Dopamine in Whole-Brain Coordination

Midbrain dopamine seems to play an outsized role in motivated behavior and learning. Widely associated with mediating reward-related behavior, decision making, and learning, dopamine continues to generate controversies in the field. While many studies and theories focus on what dopamine cells encode, the question of how the midbrain derives the information it encodes is poorly understood and comparatively less addressed. Recent anatomical studies suggest greater diversity and complexity of afferent inputs than previously appreciated, requiring rethinking of prior models. Here, we elaborate a hypothesis that construes midbrain dopamine as implementing a Bayesian selector in which individual dopamine cells sample afferent activity across distributed brain substrates, comprising evidence to be evaluated on the extent to which stimuli in the on-going sensorimotor stream organizes distributed, parallel processing, reflecting implicit value. To effectively generate a temporally resolved phasic signal, a population of dopamine cells must exhibit synchronous activity. We argue that synchronous activity across a population of dopamine cells signals consensus across distributed afferent substrates, invigorating responding to recognized opportunities and facilitating further learning. In framing our hypothesis, we shift from the question of how value is computed to the broader question of how the brain achieves coordination across distributed, parallel processing. We posit the midbrain is part of an “axis of agency” in which the prefrontal cortex (PFC), basal ganglia (BGS), and midbrain form an axis mediating control, coordination, and consensus, respectively.

Dopamine blockade impairs the exploration-exploitation trade-off in rats

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

Lateral Habenular Burst Firing as a Target of the Rapid Antidepressant Effects of Ketamine

The revolutionary discovery of the rapid antidepressant ketamine has been a milestone in psychiatry field in the last half century. Unlike conventional antidepressants that often take weeks to months to show efficacy, ketamine causes rapid antidepressant effects, emerging as early as within 1h after administration. However, how ketamine improves mood symptoms so quickly has remained elusive. Here, we first introduce the historical background of ketamine as a rapid antidepressant. We then discuss current hypotheses underlying ketamine's rapid antidepressant effects, with a focus on our latest discovery that ketamine silences NMDAR-dependent burst firing in the 'anti-reward center', the lateral habenula. While ketamine may act on many brain regions, we argue that its rapid antidepressant effects are critically dependent on ketamine's action in the lateral habenula, with this brain region acting as a primary site of action (or one among a few primary nodes). This molecular-, cellular-, and circuit-based mechanism advances our understanding of the etiology of depression and suggests a new conceptual framework for the rapid antidepressant effects of ketamine.

What, If, and When to Move: Basal Ganglia Circuits and Self-Paced Action Initiation

Deciding what to do and when to move is vital to our survival. Clinical and fundamental studies have identified basal ganglia circuits as critical for this process. The main input nucleus of the basal ganglia, the striatum, receives inputs from frontal, sensory, and motor cortices and interconnected thalamic areas that provide information about potential goals, context, and actions and directly or indirectly modulates basal ganglia outputs. The striatum also receives dopaminergic inputs that can signal reward prediction errors and also behavioral transitions and movement initiation. Here we review studies and models of how direct and indirect pathways can modulate basal ganglia outputs to facilitate movement initiation, and we discuss the role of cortical and dopaminergic inputs to the striatum in determining what to do and if and when to do it. Complex but exciting scenarios emerge that shed new light on how basal ganglia circuits modulate self-paced movement initiation.

Central Processing of Itch in the Midbrain Reward Center

Itch is an aversive sensation that evokes a desire to scratch. Paradoxically, scratching the itch also produces a hedonic experience. The specific brain circuits processing these different aspects of itch, however, remain elusive. Here, we report that GABAergic (GABA) and dopaminergic (DA) neurons in the ventral tegmental area (VTA) are activated with different temporal patterns during acute and chronic itch. DA neuron activation lags behind GABA neurons and is dependent on scratching of the itchy site. Optogenetic manipulations of VTA GABA neurons rapidly modulated scratching behaviors through encoding itch-associated aversion. In contrast, optogenetic manipulations of VTA DA neurons revealed their roles in sustaining recurrent scratching episodes through signaling scratching-induced reward. A similar dichotomy exists for the role of VTA in chronic itch. These findings advance understanding of circuit mechanisms of the unstoppable itch-scratch cycles and shed important insights into chronic itch therapy.

How Reward and Aversion Shape Motivation and Decision Making: A Computational Account

Processing rewarding and aversive signals lies at the core of many adaptive behaviors, including value-based decision making. The brain circuits processing these signals are widespread and include the prefrontal cortex, amygdala and striatum, and their dopaminergic innervation. In this review, we integrate historic findings on the behavioral and neural mechanisms of value-based decision making with recent, groundbreaking work in this area. On the basis of this integrated view, we discuss a neuroeconomic framework of value-based decision making, use this to explain the motivation to pursue rewards and how motivation relates to the costs and benefits associated with different courses of action. As such, we consider substance addiction and overeating as states of altered value-based decision making, in which the expectation of reward chronically outweighs the costs associated with substance use and food consumption, respectively. Together, this review aims to provide a concise and accessible overview of important literature on the neural mechanisms of behavioral adaptation to reward and aversion and how these mediate motivated behaviors.

Multiple Dopamine Systems: Weal and Woe of Dopamine

The ability to predict future outcomes increases the fitness of the animal. Decades of research have shown that dopamine neurons broadcast reward prediction error (RPE) signals-the discrepancy between actual and predicted reward-to drive learning to predict future outcomes. Recent studies have begun to show, however, that dopamine neurons are more diverse than previously thought. In this review, we will summarize a series of our studies that have shown unique properties of dopamine neurons projecting to the posterior "tail" of the striatum (TS) in terms of anatomy, activity, and function. Specifically, TS-projecting dopamine neurons are activated by a subset of negative events including threats from a novel object, send prediction errors for external threats, and reinforce avoidance behaviors. These results indicate that there are at least two axes of dopamine-mediated reinforcement learning in the brain-one learning from canonical RPEs and another learning from threat prediction errors. We argue that the existence of multiple learning systems is an adaptive strategy that makes possible each system optimized for its own needs. The compartmental organization in the mammalian striatum resembles that of a dopamine-recipient area in insects (mushroom body), pointing to a principle of dopamine function conserved across phyla.

Neural Substrates of Drosophila Larval Anemotaxis

Animals use sensory information to move toward more favorable conditions. Drosophila larvae can move up or down gradients of odors (chemotax), light (phototax), and temperature (thermotax) by modulating the probability, direction, and size of turns based on sensory input. Whether larvae can anemotax in gradients of mechanosensory cues is unknown. Further, although many of the sensory neurons that mediate taxis have been described, the central circuits are not well understood. Here, we used high-throughput, quantitative behavioral assays to demonstrate Drosophila larvae anemotax in gradients of wind speeds and to characterize the behavioral strategies involved. We found that larvae modulate the probability, direction, and size of turns to move away from higher wind speeds. This suggests that similar central decision-making mechanisms underlie taxis in somatosensory and other sensory modalities. By silencing the activity of single or very few neuron types in a behavioral screen, we found two sensory (chordotonal and multidendritic class III) and six nerve cord neuron types involved in anemotaxis. We reconstructed the identified neurons in an electron microscopy volume that spans the entire larval nervous system and found they received direct input from the mechanosensory neurons or from each other. In this way, we identified local interneurons and first- and second-order subesophageal zone (SEZ) and brain projection neurons. Finally, silencing a dopaminergic brain neuron type impairs anemotaxis. These findings suggest that anemotaxis involves both nerve cord and brain circuits. The candidate neurons and circuitry identified in our study provide a basis for future detailed mechanistic understanding of the circuit principles of anemotaxis.

Dopamine and Cognitive Control in Prefrontal Cortex

Cognitive control, the ability to orchestrate behavior in accord with our goals, depends on the prefrontal cortex. These cognitive functions are heavily influenced by the neuromodulator dopamine. We review here recent insights exploring the influence of dopamine on neuronal response properties in prefrontal cortex (PFC) during ongoing behaviors in primates. This review suggests three major computational roles of dopamine in cognitive control: (i) gating sensory input, (ii) maintaining and manipulating working memory contents, and (iii) relaying motor commands. For each of these roles, we propose a neuronal microcircuit based on known mechanisms of action of dopamine in PFC, which are corroborated by computational network models. This conceptual approach accounts for the various roles of dopamine in prefrontal executive functioning.

Minimal Circuit Model of Reward Prediction Error Computations and Effects of Nicotinic Modulations

Dopamine (DA) neurons in the ventral tegmental area (VTA) are thought to encode reward prediction errors (RPE) by comparing actual and expected rewards. In recent years, much work has been done to identify how the brain uses and computes this signal. While several lines of evidence suggest the interplay of the DA and the inhibitory interneurons in the VTA implements the RPE computation, it still remains unclear how the DA neurons learn key quantities, for example the amplitude and the timing of primary rewards during conditioning tasks. Furthermore, endogenous acetylcholine and exogenous nicotine, also likely affect these computations by acting on both VTA DA and GABA (γ -aminobutyric acid) neurons via nicotinic-acetylcholine receptors (nAChRs). To explore the potential circuit-level mechanisms for RPE computations during classical-conditioning tasks, we developed a minimal computational model of the VTA circuitry. The model was designed to account for several reward-related properties of VTA afferents and recent findings on VTA GABA neuron dynamics during conditioning. With our minimal model, we showed that the RPE can be learned by a two-speed process computing reward timing and magnitude. By including models of nAChR-mediated currents in the VTA DA-GABA circuit, we showed that nicotine should reduce the acetylcholine action on the VTA GABA neurons by receptor desensitization and potentially boost DA responses to reward-related signals in a non-trivial manner. Together, our results delineate the mechanisms by which RPE are computed in the brain, and suggest a hypothesis on nicotine-mediated effects on reward-related perception and decision-making.

Neuropharmacology of the mesolimbic system and associated circuits on social hierarchies

Most socially living species are organized hierarchically, primarily based on individual differences in social dominance. Dominant individuals typically gain privileged access to important resources, such as food, mating partners and territories, whereas submissive conspecifics are often devoid of such benefits. The benefits associated with a high social status provide a strong incentive to become dominant. Importantly, motivational- and reward-related processes are regulated, to a large extent, by the mesolimbic system. Consequently, several studies point to a key role for the mesolimbic system in social hierarchy formation. This review summarizes the growing body of literature that implicates the mesolimbic system, and associated neural circuits, on social hierarchies. In particular, we discuss the neurochemical and pharmacological studies that have highlighted the contributions of the mesolimbic system and associated circuits including dopamine signaling through the D1 or D2 receptors, GABAergic neurotransmission, the androgen receptor system, and mitochondria and bioenergetics. Given that low social status has been linked to the emergence of anxiety- and depressive-like disorders, a greater understanding of the neurochemistry underlying social dominance could be of tremendous benefit for the development of pharmacological treatments to dysfunctions in social behaviors. This article is part of the Special Issue entitled 'The neuropharmacology of social behavior: from bench to bedside'.

Feature-specific prediction errors and surprise across macaque fronto-striatal circuits

To adjust expectations efficiently, prediction errors need to be associated with the precise features that gave rise to the unexpected outcome, but this credit assignment may be problematic if stimuli differ on multiple dimensions and it is ambiguous which feature dimension caused the outcome. Here, we report a potential solution: neurons in four recorded areas of the anterior fronto-striatal networks encode prediction errors that are specific to feature values of different dimensions of attended multidimensional stimuli. The most ubiquitous prediction error occurred for the reward-relevant dimension. Feature-specific prediction error signals a) emerge on average shortly after non-specific prediction error signals, b) arise earliest in the anterior cingulate cortex and later in dorsolateral prefrontal cortex, caudate and ventral striatum, and c) contribute to feature-based stimulus selection after learning. Thus, a widely-distributed feature-specific eligibility trace may be used to update synaptic weights for improved feature-based attention.

In order to adjust expectations efficiently, prediction errors need to be associated with the features that gave rise to the unexpected outcome. Here, the authors show that neurons in anterior fronto-striatal networks encode prediction errors that are specific to feature values of different stimulus dimensions.

FDISCO: Advanced solvent-based clearing method for imaging whole organs

Various optical clearing methods have emerged as powerful tools for deep biological imaging. Organic solvent-based clearing methods, such as three-dimensional imaging of solvent-cleared organs (3DISCO), present the advantages of high clearing efficiency and size reduction for panoptic imaging of large samples such as whole organs and even whole bodies. However, 3DISCO results in a rapid quenching of endogenous fluorescence, which has impeded its application. Here, we propose an advanced method named FDISCO to overcome this limitation. FDISCO can effectively preserve the fluorescence of various fluorescent probes and can achieve a long storage time of months while retaining potent clearing capability. We used FDISCO for high-resolution imaging and reconstruction of neuronal and vascular networks. Moreover, FDISCO is compatible with labeling by multiple viruses and enables fine visualization of neurons with weak fluorescence labeling in the whole brain. FDISCO represents an effective alternative to the three-dimensional mapping of whole organs and can be extensively used in biomedical studies.

A Neural Circuit Mechanism for Encoding Aversive Stimuli in the Mesolimbic Dopamine System

Ventral tegmental area (VTA) dopamine (DA) neurons play a central role in mediating motivated behaviors, but the circuitry through which they signal positive and negative motivational stimuli is incompletely understood. Using in vivo fiber photometry, we simultaneously recorded activity in DA terminals in different nucleus accumbens (NAc) subnuclei during an aversive and reward conditioning task. We find that DA terminals in the ventral NAc medial shell (vNAcMed) are excited by unexpected aversive outcomes and to cues that predict them, whereas DA terminals in other NAc subregions are persistently depressed. Excitation to reward-predictive cues dominated in the NAc lateral shell and was largely absent in the vNAcMed. Moreover, we demonstrate that glutamatergic (VGLUT2-expressing) neurons in the lateral hypothalamus represent a key afferent input for providing information about aversive outcomes to vNAcMed-projecting DA neurons. Collectively, we reveal the distinct functional contributions of separate mesolimbic DA subsystems and their afferent pathways underlying motivated behaviors. VIDEO ABSTRACT.

The timing of action determines reward prediction signals in identified midbrain dopamine neurons

Animals adapt behavior in response to informative sensory cues using multiple brain circuits. The activity of midbrain dopamine (mDA) neurons is thought to convey a critical teaching signal: reward prediction error (RPE). Although RPE signals are thought to be essential to learning, little is known about the dynamic changes in mDA neuron activity as animals learn about novel sensory cues and appetitive rewards. Here we describe a large dataset of cell-attached recordings of identified dopaminergic neurons as naïve mice learned a novel cue-reward association. During learning mDA neuron activity results from summation of sensory cue-related and movement initiation-related response components. These components are both a function of reward expectation yet dissociable. Learning produces an increasingly precise coordination of action initiation following sensory cues that results in apparent RPE correlates. Our data thus provide new insights into circuit mechanisms underlying a critical computation in a highly conserved learning circuit.

Learning what to approach

Most decisions share a common goal: maximize reward and minimize punishment. Achieving this goal requires learning which choices are likely to lead to favorable outcomes. Dopamine is essential for this process, enabling learning by signaling the difference between what we expect to get and what we actually get. Although all animals appear to use this dopamine prediction error circuit, some do so more than others, and this neural heterogeneity correlates with individual variability in behavior. In this issue of PLOS Biology, Lee and colleagues show that manipulating a simple task parameter can bias the animals’ behavioral strategy and modulate dopamine release, implying that how we learn is just as flexible as what we learn.

Pavlovian-instrumental interactions in active avoidance: The bark of neutral trials

In active avoidance tasks, subjects have to learn to execute particular actions in order to avoid an aversive stimulus, such as a shock. Such paradigms pose a number of psychological and neural enigmas, and so have attracted substantial computational interest. However, the ratio of conjecture to confirmation remains high. Here, we perform a theoretical inquiry into a recent experiment by Gentry, Lee, and Roesch ('Phasic dopamine release in the rat nucleus accumbens predicts approach and avoidance performance', Nat. Commun., 7:13154) who measured phasic dopamine concentrations in the nucleus accumbens core of rats whilst they avoided shocks, acquired food, or acted to gain no programmed outcome. These last, neutral, trials turned out to be a perfect probe for the workings of avoidance, partly because of the substantial differences between subjects and sessions revealed in the experiment. We suggest a way to interpret this probe, gaining support for opponency-, safety-, and Pavlovian-influenced treatments of avoidance.

Nucleus Accumbens Dopamine Signaling Regulates Sexual Preference for Females in Male Mice

Sexual preference for the opposite sex is a fundamental behavior underlying reproductive success, but the neural mechanisms remain unclear. Here, we examined the role of dopamine signaling in the nucleus accumbens core (NAcc) in governing chemosensory-mediated preference for females in TrpC2^-/- and wild-type male mice. TrpC2^-/- males, deficient in VNO-mediated signaling, do not display mating or olfactory preference toward females. We found that, during social interaction with females, TrpC2^-/- males do not show increased NAcc dopamine levels, observed in wild-type males. Optogenetic stimulation of VTA-NAcc dopaminergic neurons in TrpC2^-/- males during exposure to a female promoted preference response to female pheromones and elevated copulatory behavior toward females. Additionally, we found that signaling through the D1 receptor in the NAcc is necessary for the olfactory preference for female-soiled bedding. Our study establishes a critical role for the mesolimbic dopaminergic system in governing pheromone-mediated responses and mate choice in male mice.

Nutritive, Post-ingestive Signals Are the Primary Regulators of AgRP Neuron Activity

The brain regulates food intake by processing sensory cues and peripheral physiological signals, but the neural basis of this integration remains unclear. Hypothalamic, agouti-related protein (AgRP)-expressing neurons are critical regulators of food intake. AgRP neuron activity is high during hunger and is rapidly reduced by the sight and smell of food. Here, we reveal two distinct components of AgRP neuron activity regulation: a rapid but transient sensory-driven signal and a slower, sustained calorie-dependent signal. We discovered that nutrients are necessary and sufficient for sustained reductions in AgRP neuron activity and that activity reductions are proportional to the calories obtained. This change in activity is recapitulated by exogenous administration of gut-derived satiation signals. Furthermore, we showed that the nutritive value of food trains sensory systems-in a single trial-to drive rapid, anticipatory AgRP neuron activity inhibition. Together, these data demonstrate that nutrients are the primary regulators of AgRP neuron activity.

A dopaminergic switch for fear to safety transitions

Overcoming aversive emotional memories requires neural systems that detect when fear responses are no longer appropriate so that they can be extinguished. The midbrain ventral tegmental area (VTA) dopamine system has been implicated in reward and more broadly in signaling when a better-than-expected outcome has occurred. This suggests that it may be important in guiding fear to safety transitions. We report that when an expected aversive outcome does not occur, activity in midbrain dopamine neurons is necessary to extinguish behavioral fear responses and engage molecular signaling events in extinction learning circuits. Furthermore, a specific dopamine projection to the nucleus accumbens medial shell is partially responsible for this effect. In contrast, a separate dopamine projection to the medial prefrontal cortex opposes extinction learning. This demonstrates a novel function for the canonical VTA-dopamine reward system and reveals opposing behavioral roles for different dopamine neuron projections in fear extinction learning.

Fear memories are overcome only when it is ascertained that fearful responses are not appropriate. Here the authors demonstrate that activity in dopamine neurons is necessary to extinguish fear responses and two distinct dopamine neuron projections exert opposing effects on extinction learning.

Belief state representation in the dopamine system

Learning to predict future outcomes is critical for driving appropriate behaviors. Reinforcement learning (RL) models have successfully accounted for such learning, relying on reward prediction errors (RPEs) signaled by midbrain dopamine neurons. It has been proposed that when sensory data provide only ambiguous information about which state an animal is in, it can predict reward based on a set of probabilities assigned to hypothetical states (called the belief state). Here we examine how dopamine RPEs and subsequent learning are regulated under state uncertainty. Mice are first trained in a task with two potential states defined by different reward amounts. During testing, intermediate-sized rewards are given in rare trials. Dopamine activity is a non-monotonic function of reward size, consistent with RL models operating on belief states. Furthermore, the magnitude of dopamine responses quantitatively predicts changes in behavior. These results establish the critical role of state inference in RL.

Identity prediction errors in the human midbrain update reward-identity expectations in the orbitofrontal cortex

There is general consensus that dopaminergic midbrain neurons signal reward prediction errors, computed as the difference between expected and received reward value. However, recent work in rodents shows that these neurons also respond to errors related to inferred value and sensory features, indicating an expanded role for dopamine beyond learning cached values. Here we utilize a transreinforcer reversal learning task and functional magnetic resonance imaging (fMRI) to test whether prediction error signals in the human midbrain are evoked when the expected identity of an appetitive food odor reward is violated, while leaving value matched. We found that midbrain fMRI responses to identity and value errors are correlated, suggesting a common neural origin for these error signals. Moreover, changes in reward-identity expectations, encoded in the orbitofrontal cortex (OFC), are directly related to midbrain activity, demonstrating that identity-based error signals in the midbrain support the formation of outcome identity expectations in OFC.

Responses in the dopaminergic midbrain are known to signal prediction errors for reward value. Here, the authors show that the human midbrain also encodes errors in predicted reward identity, and that these signals update expectations of reward identity in the orbitofrontal cortex.

Firing of Putative Dopamine Neurons in Ventral Tegmental Area Is Modulated by Probability of Success during Performance of a Stop-Change Task

Response inhibition, the ability to refrain from unwanted actions, is an essential component of complex behavior and is often impaired across numerous neuropsychiatric disorders such as addiction, attention-deficit hyperactivity disorder (ADHD), schizophrenia, and obsessive-compulsive disorder. Accordingly, much research has been devoted to characterizing brain regions responsible for the regulation of response inhibition. The stop-signal task, a task in which animals are required to inhibit a prepotent response in the presence of a STOP cue, is one of the most well-studied tasks of response inhibition. While pharmacological evidence suggests that dopamine (DA) contributes to the regulation of response inhibition, what is exactly encoded by DA neurons during performance of response inhibition tasks is unknown. To address this issue, we recorded from single units in the ventral tegmental area (VTA), while rats performed a stop-change task. We found that putative DA neurons fired less and higher to cues and reward on STOP trials relative to GO trials, respectively, and that firing was reduced during errors. These results suggest that DA neurons in VTA encode the uncertainty associated with the probability of obtaining reward on difficult trials instead of the saliency associated with STOP cues or the need to resolve conflict between competing responses during response inhibition.