1
|
Nazlı İ, Ferrari A, Huber-Huber C, de Lange FP. Forward and backward blocking in statistical learning. PLoS One 2024; 19:e0306797. [PMID: 39102398 PMCID: PMC11299817 DOI: 10.1371/journal.pone.0306797] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Accepted: 06/24/2024] [Indexed: 08/07/2024] Open
Abstract
Prediction errors have a prominent role in many forms of learning. For example, in reinforcement learning, agents learn by updating the association between states and outcomes as a function of the prediction error elicited by the event. One paradigm often used to study error-driven learning is blocking. In forward blocking, participants are first presented with stimulus A, followed by outcome X (A→X). In the second phase, A and B are presented together, followed by X (AB→X). Here, A→X blocks the formation of B→X, given that X is already fully predicted by A. In backward blocking, the order of phases is reversed. Here, the association between B and X that is formed during the first learning phase of AB→X is weakened when participants learn exclusively A→X in the second phase. The present study asked the question whether forward and backward blocking occur during visual statistical learning, i.e., the incidental learning of the statistical structure of the environment. In a series of studies, using both forward and backward blocking, we observed statistical learning of temporal associations among pairs of images. While we found no forward blocking, we observed backward blocking, thereby suggesting a retrospective revaluation process in statistical learning and supporting a functional similarity between statistical learning and reinforcement learning.
Collapse
Affiliation(s)
- İlayda Nazlı
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Ambra Ferrari
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Max Plank Institute for Psycholinguistics, Nijmegen, The Netherlands
| | - Christoph Huber-Huber
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Rovereto, Italy
| | - Floris P. de Lange
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
2
|
Engel L, Wolff AR, Blake M, Collins VL, Sinha S, Saunders BT. Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning. Curr Biol 2024; 34:3086-3101.e4. [PMID: 38925117 PMCID: PMC11279555 DOI: 10.1016/j.cub.2024.05.069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2024] [Revised: 04/25/2024] [Accepted: 05/29/2024] [Indexed: 06/28/2024]
Abstract
Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward rewards. Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) are crucial for this process, via engagement of a reciprocally connected network with their striatal targets. Critically, it remains unknown how dopamine neuron activity itself engages dopamine signals throughout the striatum, across learning. Here, we investigated how optogenetic Pavlovian cue conditioning of VTA or SNc dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. Although VTA stimulation-evoked robust dopamine release in NAc core, shell, and DMS, predictive cues preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS. Despite the development of vigorous movement late in training, conditioned dopamine signals did not emerge in the DLS, even during Pavlovian conditioning with SNc dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show a broad dissociation in the fundamental prediction and reward-related information generated by VTA and SNc dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale adaptations across the striatal network emerge during learning to coordinate behavior.
Collapse
Affiliation(s)
- Liv Engel
- Department of Neuroscience, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA; Medical Discovery Team on Addiction, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA
| | - Amy R Wolff
- Department of Neuroscience, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA; Medical Discovery Team on Addiction, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA
| | - Madelyn Blake
- Department of Neuroscience, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA
| | - Val L Collins
- Department of Neuroscience, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA; Medical Discovery Team on Addiction, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA
| | - Sonal Sinha
- Krieger School of Arts & Sciences, Johns Hopkins University, 3400 N. Charles St, Baltimore, MD 21218, USA
| | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA; Medical Discovery Team on Addiction, University of Minnesota, 2001 6th St SE, Minneapolis, MN 55455, USA.
| |
Collapse
|
3
|
Millard SJ, Hoang IB, Sherwood S, Taira M, Reyes V, Greer Z, O'Connor SL, Wassum KM, James MH, Barker DJ, Sharpe MJ. Cognitive representations of intracranial self-stimulation of midbrain dopamine neurons depend on stimulation frequency. Nat Neurosci 2024; 27:1253-1259. [PMID: 38741021 PMCID: PMC11239488 DOI: 10.1038/s41593-024-01643-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2022] [Accepted: 04/05/2024] [Indexed: 05/16/2024]
Abstract
Dopamine neurons in the ventral tegmental area support intracranial self-stimulation (ICSS), yet the cognitive representations underlying this phenomenon remain unclear. Here, 20-Hz stimulation of dopamine neurons, which approximates a physiologically relevant prediction error, was not sufficient to support ICSS beyond a continuously reinforced schedule and did not endow cues with a general or specific value. However, 50-Hz stimulation of dopamine neurons was sufficient to drive robust ICSS and was represented as a specific reward to motivate behavior. The frequency dependence of this effect is due to the rate (not the number) of action potentials produced by dopamine neurons, which differently modulates dopamine release downstream.
Collapse
Affiliation(s)
- Samuel J Millard
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Ivy B Hoang
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Savannah Sherwood
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Masakazu Taira
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
- Department of Psychology, University of Sydney, Camperdown, New South Wales, Australia
| | - Vanessa Reyes
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Zara Greer
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Shayna L O'Connor
- Department of Psychiatry, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
- Brain Health Institute, Rutgers Biomedical Health Sciences, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
- Department of Psychology, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
| | - Kate M Wassum
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Morgan H James
- Department of Psychiatry, Robert Wood Johnson Medical School, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
- Brain Health Institute, Rutgers Biomedical Health Sciences, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
| | - David J Barker
- Brain Health Institute, Rutgers Biomedical Health Sciences, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
- Department of Psychology, Rutgers, The State University of New Jersey, Piscataway, NJ, USA
| | - Melissa J Sharpe
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA.
- Department of Psychology, University of Sydney, Camperdown, New South Wales, Australia.
| |
Collapse
|
4
|
McGovern DJ, Phillips A, Ly A, Prévost ED, Ward L, Siletti K, Kim YS, Fenno LE, Ramakrishnan C, Deisseroth K, Ford CP, Root DH. Salience signaling and stimulus scaling of ventral tegmental area glutamate neuron subtypes. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.12.598688. [PMID: 38915564 PMCID: PMC11195246 DOI: 10.1101/2024.06.12.598688] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/26/2024]
Abstract
Ventral tegmental area (VTA) glutamatergic neurons participate in reward, aversion, drug-seeking, and stress. Subsets of VTA VGluT2+ neurons are capable of co-transmitting glutamate and GABA (VGluT2+VGaT+ neurons), transmitting glutamate without GABA (VGluT2+VGaT- neurons), or co-transmitting glutamate and dopamine (VGluT2+TH+ neurons), but whether these molecularly distinct subpopulations show behavior-related differences is not wholly understood. We identified that neuronal activity of each VGluT2+ subpopulation is sensitive to reward value but signaled this in different ways. The phasic maximum activity of VGluT2+VGaT+ neurons increased with sucrose concentration, whereas VGluT2+VGaT- neurons increased maximum and sustained activity with sucrose concentration, and VGluT2+TH+ neurons increased sustained but not maximum activity with sucrose concentration. Additionally, VGluT2+ subpopulations signaled consummatory preferences in different ways. VGluT2+VGaT- neurons and VGluT2+TH+ neurons showed a signaling preference for a behaviorally-preferred fat reward over sucrose, but in temporally-distinct ways. In contrast, VGluT2+VGaT+ neurons uniquely signaled a less behaviorally-preferred sucrose reward compared with fat. Further experiments suggested that VGluT2+VGaT+ consummatory reward-related activity was related to sweetness, partially modulated by hunger state, and not dependent on caloric content or behavioral preference. All VGluT2+ subtypes increased neuronal activity following aversive stimuli but VGluT2+VGaT+ neurons uniquely scaled their magnitude and sustained activity with footshock intensity. Optogenetic activation of VGluT2+VGaT+ neurons during low intensity footshock enhanced fear-related behavior without inducing place preference or aversion. We interpret these data such that VTA glutamatergic subpopulations signal different elements of rewarding and aversive experiences and highlight the unique role of VTA VGluT2+VGaT+ neurons in enhancing the salience of behavioral experiences.
Collapse
Affiliation(s)
- Dillon J. McGovern
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Alysabeth Phillips
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Annie Ly
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Emily D. Prévost
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Lucy Ward
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Kayla Siletti
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| | - Yoon Seok Kim
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA 94305, USA
| | - Lief E. Fenno
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA 94305, USA
- Current address: Department of Neuroscience, Dell Medical School, The University of Texas at Austin 78712
| | - Charu Ramakrishnan
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA 94305, USA
| | - Karl Deisseroth
- Department of Bioengineering, Stanford University, Stanford, CA 94305, USA
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA 94305, USA
- Howard Hughes Medical Institute, Stanford University, Stanford, CA 94305, USA
| | - Christopher P. Ford
- Department of Pharmacology, University of Colorado School of Medicine, Anschutz Medical Campus, Aurora, CO 80045
| | - David H. Root
- Department of Psychology and Neuroscience, University of Colorado Boulder, 2860 Wilderness Pl, Boulder, CO 80301
| |
Collapse
|
5
|
Garr E, Cheng Y, Jeong H, Brooke S, Castell L, Bal A, Magnard R, Namboodiri VMK, Janak PH. Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies. SCIENCE ADVANCES 2024; 10:eadn4203. [PMID: 38809978 PMCID: PMC11135394 DOI: 10.1126/sciadv.adn4203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 04/23/2024] [Indexed: 05/31/2024]
Abstract
Learning causal relationships relies on understanding how often one event precedes another. To investigate how dopamine neuron activity and neurotransmitter release change when a retrospective relationship is degraded for a specific pair of events, we used outcome-selective Pavlovian contingency degradation in rats. Conditioned responding was attenuated for the cue-reward contingency that was degraded, as was dopamine neuron activity in the midbrain and dopamine release in the ventral striatum in response to the cue and subsequent reward. Contingency degradation also abolished the trial-by-trial history dependence of the dopamine responses at the time of trial outcome. This profile of changes in cue- and reward-evoked responding is not easily explained by a standard reinforcement learning model. An alternative model based on learning causal relationships was better able to capture dopamine responses during contingency degradation, as well as conditioned behavior following optogenetic manipulations of dopamine during noncontingent rewards. Our results suggest that mesostriatal dopamine encodes the contingencies between meaningful events during learning.
Collapse
Affiliation(s)
- Eric Garr
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yifeng Cheng
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Huijeong Jeong
- Department of Neurology, University of California, San Francisco, CA 94158, USA
| | - Sara Brooke
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
| | - Laia Castell
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Aneesh Bal
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Robin Magnard
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Vijay Mohan K. Namboodiri
- Department of Neurology, University of California, San Francisco, CA 94158, USA
- Weill Institute for Neurosciences, Kavli Institute for Fundamental Neuroscience, Center for Integrative Neuroscience, University of California, San Francisco, CA 94158, USA
| | - Patricia H. Janak
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
- Kavli Neuroscience Discovery Institute, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| |
Collapse
|
6
|
Maitra R, Lemmers-Jansen ILJ, Vooren M, Vanes L, Szentgyorgyi T, Crisp C, Mouchlianitis E, Shergill SS. Understanding the mechanisms underlying cognitive control in psychosis. Psychol Med 2024:1-10. [PMID: 38780379 DOI: 10.1017/s0033291724001119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 05/25/2024]
Abstract
BACKGROUND Cognitive control (CC) involves a top-down mechanism to flexibly respond to complex stimuli and is impaired in schizophrenia. METHODS This study investigated the impact of increasing complexity of CC processing in 140 subjects with psychosis and 39 healthy adults, with assessments of behavioral performance, neural regions of interest and symptom severity. RESULTS The lowest level of CC (Stroop task) was impaired in all patients; the intermediate level of CC (Faces task) with explicit emotional information was most impaired in patients with first episode psychosis. Patients showed activation of distinct neural CC and reward networks, but iterative learning based on the higher-order of CC during the trust game, was most impaired in chronic schizophrenia. Subjects with first episode psychosis, and patients with lower symptom load, demonstrate flexibility of the CC network to facilitate learning, which appeared compromised in the more chronic stages of schizophrenia. CONCLUSION These data suggest optimal windows for opportunities to introduce therapeutic interventions to improve CC.
Collapse
Affiliation(s)
- R Maitra
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- Tavistock and Portman NHS Foundation Trust, London, UK
| | - I L J Lemmers-Jansen
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- Department of Clinical, Neuro and Developmental Psychology, Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Institute for Brain and Behavior Amsterdam (iBBA), Amsterdam, Netherlands
| | - M Vooren
- Faculty of Behavioural and Movement Sciences, Educational Studies, Section Methods and Statistics, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- LEARN! Research Institute, Vrije Universiteit Amsterdam, Amsterdam, Netherlands
- Amsterdam Center for Learning Analytics (ACLA), Vrije Universiteit Amsterdam, Amsterdam, Netherlands
| | - Lucy Vanes
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Timea Szentgyorgyi
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
| | - Charlotte Crisp
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- School of Psychological Science, University of Bristol, Bristol, UK
| | - Elias Mouchlianitis
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- Department of Psychology, University of East London, London, UK
| | - S S Shergill
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK
- Kent and Medway Medical School, University of Kent, Canterbury, UK
| |
Collapse
|
7
|
Bornhoft KN, Prohofsky J, O'Neal TJ, Wolff AR, Saunders BT. Valence ambiguity dynamically shapes striatal dopamine heterogeneity. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.17.594692. [PMID: 38798567 PMCID: PMC11118546 DOI: 10.1101/2024.05.17.594692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/29/2024]
Abstract
Adaptive decision making relies on dynamic updating of learned associations where environmental cues come to predict positive and negatively valenced stimuli, such as food or threat. Flexible cue-guided behaviors depend on a network of brain systems, including dopamine signaling in the striatum, which is critical for learning and maintenance of conditioned behaviors. Critically, it remains unclear how dopamine signaling encodes multi-valent, dynamic learning contexts, where positive and negative associations must be rapidly disambiguated. To understand this, we employed a Pavlovian discrimination paradigm, where cues predicting positive and negative outcomes were intermingled during conditioning sessions, and their meaning was serially reversed across training. We found that rats readily distinguished these cues, and updated their behavior rapidly upon valence reversal. Using fiber photometry, we recorded dopamine signaling in three major striatal subregions -,the dorsolateral striatum (DLS), the nucleus accumbens core, and the nucleus accumbens medial shell - and found heterogeneous responses to positive and negative conditioned cues and their predicted outcomes. Valence ambiguity introduced by cue reversal reshaped striatal dopamine on different timelines: nucleus accumbens core and shell signals updated more readily than those in the DLS. Together, these results suggest that striatal dopamine flexibly encodes multi-valent learning contexts, and these signals are dynamically modulated by changing contingencies to resolve ambiguity about the meaning of environmental cues.
Collapse
|
8
|
Imtiaz Z, Kato A, Kopell BH, Qasim SE, Davis AN, Martinez LN, Heflin M, Kulkarni K, Morsi A, Gu X, Saez I. Human Substantia Nigra Neurons Encode Reward Expectations. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.10.593406. [PMID: 38766086 PMCID: PMC11100806 DOI: 10.1101/2024.05.10.593406] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2024]
Abstract
Dopamine (DA) signals originating from substantia nigra (SN) neurons are centrally involved in the regulation of motor and reward processing. DA signals behaviorally relevant events where reward outcomes differ from expectations (reward prediction errors, RPEs). RPEs play a crucial role in learning optimal courses of action and in determining response vigor when an agent expects rewards. Nevertheless, how reward expectations, crucial for RPE calculations, are conveyed to and represented in the dopaminergic system is not fully understood, especially in the human brain where the activity of DA neurons is difficult to study. One possibility, suggested by evidence from animal models, is that DA neurons explicitly encode reward expectations. Alternatively, they may receive RPE information directly from upstream brain regions. To address whether SN neuron activity directly reflects reward expectation information, we directly examined the encoding of reward expectation signals in human putative DA neurons by performing single-unit recordings from the SN of patients undergoing neurosurgery. Patients played a two-armed bandit decision-making task in which they attempted to maximize reward. We show that neuronal firing rates (FR) of putative DA neurons during the reward expectation period explicitly encode reward expectations. First, activity in these neurons was modulated by previous trial outcomes, such that FR were greater after positive outcomes than after neutral or negative outcome trials. Second, this increase in FR was associated with shorter reaction times, consistent with an invigorating effect of DA neuron activity during expectation. These results suggest that human DA neurons explicitly encode reward expectations, providing a neurophysiological substrate for a signal critical for reward learning.
Collapse
Affiliation(s)
- Zarghona Imtiaz
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ayaka Kato
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Brian H. Kopell
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Salman E. Qasim
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Arianna Neal Davis
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Lizbeth Nunez Martinez
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Matt Heflin
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Kaustubh Kulkarni
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Amr Morsi
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Xiaosi Gu
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ignacio Saez
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
9
|
Sias AC, Jafar Y, Goodpaster CM, Ramírez-Armenta K, Wrenn TM, Griffin NK, Patel K, Lamparelli AC, Sharpe MJ, Wassum KM. Dopamine projections to the basolateral amygdala drive the encoding of identity-specific reward memories. Nat Neurosci 2024; 27:728-736. [PMID: 38396258 PMCID: PMC11110430 DOI: 10.1038/s41593-024-01586-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2022] [Accepted: 01/24/2024] [Indexed: 02/25/2024]
Abstract
To make adaptive decisions, we build an internal model of the associative relationships in an environment and use it to make predictions and inferences about specific available outcomes. Detailed, identity-specific cue-reward memories are a core feature of such cognitive maps. Here we used fiber photometry, cell-type and pathway-specific optogenetic manipulation, Pavlovian cue-reward conditioning and decision-making tests in male and female rats, to reveal that ventral tegmental area dopamine (VTADA) projections to the basolateral amygdala (BLA) drive the encoding of identity-specific cue-reward memories. Dopamine is released in the BLA during cue-reward pairing; VTADA→BLA activity is necessary and sufficient to link the identifying features of a reward to a predictive cue but does not assign general incentive properties to the cue or mediate reinforcement. These data reveal a dopaminergic pathway for the learning that supports adaptive decision-making and help explain how VTADA neurons achieve their emerging multifaceted role in learning.
Collapse
Affiliation(s)
- Ana C Sias
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Yousif Jafar
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Caitlin M Goodpaster
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | | | - Tyler M Wrenn
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Nicholas K Griffin
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | - Keshav Patel
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
| | | | - Melissa J Sharpe
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA
- Brain Research Institute, University of California, Los Angeles, Los Angeles, CA, USA
- Integrative Center for Learning and Memory, University of California, Los Angeles, Los Angeles, CA, USA
- Integrative Center for Addictive Disorders, University of California, Los Angeles, Los Angeles, CA, USA
- Department of Psychology, University of Sydney, Sydney, New South Wales, Australia
| | - Kate M Wassum
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA.
- Brain Research Institute, University of California, Los Angeles, Los Angeles, CA, USA.
- Integrative Center for Learning and Memory, University of California, Los Angeles, Los Angeles, CA, USA.
- Integrative Center for Addictive Disorders, University of California, Los Angeles, Los Angeles, CA, USA.
| |
Collapse
|
10
|
Seiler JL, Zhuang X, Nelson AB, Lerner TN. Dopamine across timescales and cell types: Relevance for phenotypes in Parkinson's disease progression. Exp Neurol 2024; 374:114693. [PMID: 38242300 DOI: 10.1016/j.expneurol.2024.114693] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 01/09/2024] [Accepted: 01/15/2024] [Indexed: 01/21/2024]
Abstract
Dopamine neurons in the substantia nigra pars compacta (SNc) synthesize and release dopamine, a critical neurotransmitter for movement and learning. SNc dopamine neurons degenerate in Parkinson's Disease (PD), causing a host of motor and non-motor symptoms. Here, we review recent conceptual advances in our basic understanding of the dopamine system - including our rapidly advancing knowledge of dopamine neuron heterogeneity - with special attention to their importance for understanding PD. In PD patients, dopamine neuron degeneration progresses from lateral SNc to medial SNc, suggesting clinically relevant heterogeneity in dopamine neurons. With technical advances in dopamine system interrogation, we can understand the relevance of this heterogeneity for PD progression and harness it to develop new treatments.
Collapse
Affiliation(s)
- Jillian L Seiler
- Department of Neuroscience, Northwestern University Feinberg School of Medicine, Chicago, IL, USA; Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Xiaowen Zhuang
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA; Weill Institute for Neuroscience, University of California San Francisco, San Francisco, CA, USA; Kavli Institute for Fundamental Neuroscience, University of California San Francisco, San Francisco, CA, USA; Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Alexandra B Nelson
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA; Neuroscience Graduate Program, University of California San Francisco, San Francisco, CA, USA; Weill Institute for Neuroscience, University of California San Francisco, San Francisco, CA, USA; Kavli Institute for Fundamental Neuroscience, University of California San Francisco, San Francisco, CA, USA; Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA.
| | - Talia N Lerner
- Department of Neuroscience, Northwestern University Feinberg School of Medicine, Chicago, IL, USA; Northwestern University Interdepartmental Neuroscience Program (NUIN), Evanston, IL, USA; Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA.
| |
Collapse
|
11
|
Bernklau TW, Righetti B, Mehrke LS, Jacob SN. Striatal dopamine signals reflect perceived cue-action-outcome associations in mice. Nat Neurosci 2024; 27:747-757. [PMID: 38291283 PMCID: PMC11001585 DOI: 10.1038/s41593-023-01567-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Accepted: 12/21/2023] [Indexed: 02/01/2024]
Abstract
Striatal dopamine drives associative learning by acting as a teaching signal. Much work has focused on simple learning paradigms, including Pavlovian and instrumental learning. However, higher cognition requires that animals generate internal concepts of their environment, where sensory stimuli, actions and outcomes become flexibly associated. Here, we performed fiber photometry dopamine measurements across the striatum of male mice as they learned cue-action-outcome associations based on implicit and changing task rules. Reinforcement learning models of the behavioral and dopamine data showed that rule changes lead to adjustments of learned cue-action-outcome associations. After rule changes, mice discarded learned associations and reset outcome expectations. Cue- and outcome-triggered dopamine signals became uncoupled and dependent on the adopted behavioral strategy. As mice learned the new association, coupling between cue- and outcome-triggered dopamine signals and task performance re-emerged. Our results suggest that dopaminergic reward prediction errors reflect an agent's perceived locus of control.
Collapse
Affiliation(s)
- Tobias W Bernklau
- Translational Neurotechnology Laboratory, Department of Neurosurgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Graduate School of Systemic Neurosciences, Ludwig-Maximilians-University Munich, Munich, Germany
| | - Beatrice Righetti
- Translational Neurotechnology Laboratory, Department of Neurosurgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Leonie S Mehrke
- Translational Neurotechnology Laboratory, Department of Neurosurgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Simon N Jacob
- Translational Neurotechnology Laboratory, Department of Neurosurgery, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany.
| |
Collapse
|
12
|
Engel L, Wolff AR, Blake M, Collins VL, Sinha S, Saunders BT. Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.07.01.547331. [PMID: 38585717 PMCID: PMC10996462 DOI: 10.1101/2023.07.01.547331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/09/2024]
Abstract
Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward acquisition of rewards. Dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNC) are crucial for this process. Dopamine neurons are embedded in a reciprocally connected network with their striatal targets, the functional organization of which remains poorly understood. Here, we investigated how learning during optogenetic Pavlovian cue conditioning of VTA or SNC dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent dopamine biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. While VTA stimulation evoked robust dopamine release in NAc core, shell, and DMS, cues predictive of this activation preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Corresponding negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS, and not the shell. Despite development of vigorous movement late in training, conditioned dopamine signals did not similarly emerge in the DLS, even during Pavlovian conditioning with SNC dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show broad dissociation in the fundamental prediction and reward-related information generated by different dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale plasticity across the striatal network emerges during Pavlovian learning to coordinate behavior.
Collapse
Affiliation(s)
- Liv Engel
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
- Current Address: Department of Psychology, University of Toronto
| | - Amy R Wolff
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| | - Madelyn Blake
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| | - Val L Collins
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| | | | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| |
Collapse
|
13
|
Liu Q, Zhao Y, Attanti S, Voss JL, Schoenbaum G, Kahnt T. Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks. Nat Commun 2024; 15:1704. [PMID: 38402210 PMCID: PMC10894191 DOI: 10.1038/s41467-024-45880-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 02/06/2024] [Indexed: 02/26/2024] Open
Abstract
Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.
Collapse
Affiliation(s)
- Qingfang Liu
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Yao Zhao
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Sumedha Attanti
- Mayo Clinic Alix School of Medicine, Scottsdale, AZ, 85259, USA
| | - Joel L Voss
- Department of Neurology, The University of Chicago, Chicago, IL, 60611, USA
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Thorsten Kahnt
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA.
| |
Collapse
|
14
|
Wolff AR, Saunders BT. Sensory Cues Potentiate VTA Dopamine Mediated Reinforcement. eNeuro 2024; 11:ENEURO.0421-23.2024. [PMID: 38238080 PMCID: PMC10875637 DOI: 10.1523/eneuro.0421-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/16/2023] [Revised: 12/19/2023] [Accepted: 01/04/2024] [Indexed: 01/25/2024] Open
Abstract
Sensory cues are critical for shaping decisions and invigorating actions during reward seeking. Dopamine neurons in the ventral tegmental area (VTA) are central in this process, supporting associative learning in Pavlovian and instrumental settings. Studies of intracranial self-stimulation (ICSS) behavior, which show that animals will work hard to receive stimulation of dopamine neurons, support the notion that dopamine transmits a reward or value signal to support learning. Recent studies have begun to question this, however, emphasizing dopamine's value-free functions, leaving its contribution to behavioral reinforcement somewhat muddled. Here, we investigated the role of sensory stimuli in dopamine-mediated reinforcement, using an optogenetic ICSS paradigm in tyrosine hydroxylase (TH)-Cre rats. We find that while VTA dopamine neuron activation in the absence of explicit external cues is sufficient to maintain robust self-stimulation, the presence of cues dramatically potentiates ICSS behavior. Our results support a framework where dopamine can have some base value as a reinforcer, but the impact of this signal is modulated heavily by the sensory learning context.
Collapse
Affiliation(s)
- Amy R Wolff
- Department of Neuroscience, University of Minnesota, Minneapolis 55455, Minnesota
- Medical Discovery Team on Addiction, University of Minnesota, Minneapolis 55455, Minnesota
| | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota, Minneapolis 55455, Minnesota
- Medical Discovery Team on Addiction, University of Minnesota, Minneapolis 55455, Minnesota
| |
Collapse
|
15
|
Pool ER, Pauli WM, Cross L, O'Doherty JP. Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans. Nat Commun 2023; 14:8057. [PMID: 38052792 PMCID: PMC10697955 DOI: 10.1038/s41467-023-43747-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Accepted: 11/17/2023] [Indexed: 12/07/2023] Open
Abstract
We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.
Collapse
Affiliation(s)
- Eva R Pool
- Swiss Center for Affective Sciences, University of Geneva, Geneva, Switzerland.
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| | - Wolfgang M Pauli
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, USA
| | - Logan Cross
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Department of Computer Science, Stanford University, Palo Alto, CA, USA
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
16
|
van der Merwe R, Nadel J, Copes-Finke D, Pawelko S, Scott J, Ghanem M, Fox M, Morehouse C, McLaughlin R, Maddox C, Albert-Lyons R, Malaki G, Groce V, Turocy A, Aggadi N, Jin X, Howard C. Characterization of striatal dopamine projections across striatal subregions in behavioral flexibility. Eur J Neurosci 2023; 58:4466-4486. [PMID: 36617434 PMCID: PMC10329096 DOI: 10.1111/ejn.15910] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2021] [Revised: 12/13/2022] [Accepted: 12/30/2022] [Indexed: 01/10/2023]
Abstract
Behavioural flexibility is key to survival in a dynamic environmentWhile flexible, goal-directed behaviours are initially dependent on dorsomedial striatum, they become dependent on lateral striatum as behaviours become inflexible. Similarly, lesions of dopamine terminals in lateral striatum disrupt the development of inflexible habits. This work suggests that dopamine release in lateral striatum may drive inflexible behaviours, though few studies have investigated a causative role of subpopulations of striatal dopamine terminals in reversal learning, a measure of flexibility. Here, we performed two optogenetic experiments to activate dopamine terminals in dorsomedial (DMS), dorsolateral (DLS) or ventral (nucleus accumbens [NAc]) striatum in DAT-Cre mice that expressed channelrhodopsin-2 via viral injection (Experiment I) or through transgenic breeding with an Ai32 reporter line (Experiment II) to determine how specific dopamine subpopulations impact reversal learning. Mice performed a reversal task in which they self-stimulated DMS, DLS, or NAc dopamine terminals by pressing one of two levers before action-outcome lever contingencies were reversed. Largely consistent with presumed ventromedial/lateral striatal function, we found that mice self-stimulating medial dopamine terminals reversed lever preference following contingency reversal, while mice self-stimulating NAc showed parial flexibility, and DLS self-stimulation resulted in impaired reversal. Impairments in DLS mice were characterized by more regressive errors and reliance on lose-stay strategies following reversal, as well as reduced within-session learning, suggesting reward insensitivity and overreliance on previously learned actions. This study supports a model of striatal function in which DMS and ventral dopamine facilitate goal-directed responding, and DLS dopamine supports more inflexible responding.
Collapse
Affiliation(s)
- R.K. van der Merwe
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - J.A. Nadel
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
- Northwestern University Interdepartmental Neuroscience Program (NUIN), Evanston, IL, USA
| | - D. Copes-Finke
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - S. Pawelko
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - J.S. Scott
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - M. Ghanem
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - M. Fox
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - C. Morehouse
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - R. McLaughlin
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - C. Maddox
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - R. Albert-Lyons
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - G. Malaki
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - V. Groce
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - A. Turocy
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - N. Aggadi
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| | - X. Jin
- Center for Motor Control and Disease, Key Laboratory of Brain Functional Genomics, East China Normal University, Shanghai 200062, China
- NYU–ECNU Institute of Brain and Cognitive Science, New York University Shanghai, Shanghai 200062, China
| | - C.D. Howard
- Neuroscience Department, Oberlin College, 173 Lorain St., Oberlin, OH, USA
| |
Collapse
|
17
|
Fraser KM, Collins VL, Wolff AR, Ottenheimer DJ, Bornhoft KN, Pat F, Chen BJ, Janak PH, Saunders BT. Contexts facilitate dynamic value encoding in the mesolimbic dopamine system. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.05.565687. [PMID: 37961363 PMCID: PMC10635154 DOI: 10.1101/2023.11.05.565687] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.
Collapse
Affiliation(s)
- Kurt M Fraser
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | | | - Amy R Wolff
- Department of Neuroscience, University of Minnesota
| | | | | | - Fiona Pat
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | - Bridget J Chen
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | - Patricia H Janak
- Department of Psychological and Brain Sciences, Johns Hopkins University
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University
| | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| |
Collapse
|
18
|
Pereira AR, Alemi M, Cerqueira-Nunes M, Monteiro C, Galhardo V, Cardoso-Cruz H. Dynamics of Lateral Habenula-Ventral Tegmental Area Microcircuit on Pain-Related Cognitive Dysfunctions. Neurol Int 2023; 15:1303-1319. [PMID: 37987455 PMCID: PMC10660716 DOI: 10.3390/neurolint15040082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Revised: 10/20/2023] [Accepted: 10/25/2023] [Indexed: 11/22/2023] Open
Abstract
Chronic pain is a health problem that affects the ability to work and perform other activities, and it generally worsens over time. Understanding the complex pain interaction with brain circuits could help predict which patients are at risk of developing central dysfunctions. Increasing evidence from preclinical and clinical studies suggests that aberrant activity of the lateral habenula (LHb) is associated with depressive symptoms characterized by excessive negative focus, leading to high-level cognitive dysfunctions. The primary output region of the LHb is the ventral tegmental area (VTA), through a bidirectional connection. Recently, there has been growing interest in the complex interactions between the LHb and VTA, particularly regarding their crucial roles in behavior regulation and their potential involvement in the pathological impact of chronic pain on cognitive functions. In this review, we briefly discuss the structural and functional roles of the LHb-VTA microcircuit and their impact on cognition and mood disorders in order to support future studies addressing brain plasticity during chronic pain conditions.
Collapse
Affiliation(s)
- Ana Raquel Pereira
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| | - Mobina Alemi
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| | - Mariana Cerqueira-Nunes
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
- Programa Doutoral em Neurociências, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| | - Clara Monteiro
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| | - Vasco Galhardo
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| | - Helder Cardoso-Cruz
- Instituto de Investigação e Inovação em Saúde—Pain Neurobiology Group, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal; (A.R.P.); (M.A.); (M.C.-N.); (C.M.); (V.G.)
- Instituto de Biologia Molecular e Celular, Universidade do Porto, Rua Alfredo Allen 208, 4200-135 Porto, Portugal
- Departamento de Biomedicina—Unidade de Biologia Experimental, Faculdade de Medicina, Universidade do Porto, Rua Doutor Plácido da Costa, 4200-450 Porto, Portugal
| |
Collapse
|
19
|
Wolff AR, Saunders BT. Sensory cues potentiate VTA dopamine mediated reinforcement. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.10.18.562986. [PMID: 37904916 PMCID: PMC10614908 DOI: 10.1101/2023.10.18.562986] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/01/2023]
Abstract
Sensory cues are critical for shaping decisions and invigorating actions during reward seeking. Dopamine neurons in the ventral tegmental area (VTA) are critical in this process, supporting associative learning in Pavlovian and instrumental settings. Studies of intracranial self stimulation (ICSS) behavior, which show that animals will work hard to receive stimulation of dopamine neurons, support the notion that dopamine transmits a reward or value signal to support learning. Recent studies have begun to question this, however, emphasizing dopamine's value-free functions, leaving its contribution to behavioral reinforcement somewhat muddled. Here, we investigated the role of sensory stimuli in dopamine-mediated reinforcement, using an optogenetic ICSS paradigm in tyrosine hydroxylase (TH)-cre rats. We find that while VTA dopamine neuron activation in the absence of any external cueing stimulus is sufficient to maintain robust self stimulation, the presence of cues dramatically potentiates ICSS behavior. Our results support a framework where dopamine can have some base value as a reinforcer, but the impact of this signal is modulated heavily by the sensory learning context.
Collapse
Affiliation(s)
- Amy R Wolff
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| |
Collapse
|
20
|
Duffer K, Gillis ZS, Morrison SE. Excitatory and Inhibitory Signaling in the Nucleus Accumbens Encode Different Aspects of a Pavlovian Cue in Sign Tracking and Goal Tracking Rats. eNeuro 2023; 10:ENEURO.0196-23.2023. [PMID: 37643864 PMCID: PMC10488220 DOI: 10.1523/eneuro.0196-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Revised: 07/26/2023] [Accepted: 08/08/2023] [Indexed: 08/31/2023] Open
Abstract
When a Pavlovian cue is presented separately from its associated reward, some animals will acquire a sign tracking (ST) response - approach and/or interaction with the cue - while others will acquire a goal tracking response - approach to the site of reward. We have previously shown that cue-evoked excitations in the nucleus accumbens (NAc) encode the vigor of both behaviors; in contrast, reward-related responses diverge over the course of training, possibly reflecting neurochemical differences between sign tracker and goal tracker individuals. However, a substantial subset of neurons in the NAc exhibit inhibitory, rather than excitatory, cue-evoked responses, and the evolution of their signaling during Pavlovian conditioning remains unknown. Using single-neuron recordings in behaving rats, we show that NAc neurons with cue-evoked inhibitions have distinct coding properties from neurons with cue-evoked excitations. Cue-evoked inhibitions become more numerous over the course of training and, like excitations, may encode the vigor of sign tracking and goal tracking behavior. However, the responses of cue-inhibited neurons do not evolve differently between sign tracker and goal tracker individuals. Moreover, cue-evoked inhibitions, unlike excitations, are insensitive to extinction of the cue-reward relationship. Finally, we show that cue-evoked excitations are greatly diminished by reward devaluation, while inhibitory cue responses are virtually unaffected. Overall, these findings converge with existing evidence that cue-excited neurons in NAc, but not cue-inhibited neurons, are profoundly sensitive to the same behavior variations that are often associated with changes in dopamine release.
Collapse
Affiliation(s)
- Kyle Duffer
- Department of Neuroscience, University of Pittsburgh, Pittsburgh, PA 15260
| | - Zachary S Gillis
- Department of Neuroscience, University of Pittsburgh, Pittsburgh, PA 15260
| | - Sara E Morrison
- Department of Neuroscience, University of Pittsburgh, Pittsburgh, PA 15260
| |
Collapse
|
21
|
Bakhurin KI, Hughes RN, Jiang Q, Hossain M, Gutkin B, Fallon IP, Yin H. Force tuning explains changes in phasic dopamine signaling during stimulus-reward learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.04.23.537994. [PMID: 37162997 PMCID: PMC10168281 DOI: 10.1101/2023.04.23.537994] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/11/2023]
Abstract
According to a popular hypothesis, phasic dopamine (DA) activity encodes a reward prediction error (RPE) necessary for reinforcement learning. However, recent work showed that DA neurons are necessary for performance rather than learning. One limitation of previous work on phasic DA signaling and RPE is the limited behavioral measures. Here, we measured subtle force exertion while recording and manipulating DA activity in the ventral tegmental area (VTA) during stimulus-reward learning. We found two major populations of DA neurons that increased firing before forward and backward force exertion. Force tuning is the same regardless of learning, reward predictability, or outcome valence. Changes in the pattern of force exertion can explain results traditionally used to support the RPE hypothesis, such as modulation by reward magnitude, probability, and unpredicted reward delivery or omission. Thus VTA DA neurons are not used to signal RPE but to regulate force exertion during motivated behavior.
Collapse
|
22
|
Fraser KM, Pribut HJ, Janak PH, Keiflin R. From Prediction to Action: Dissociable Roles of Ventral Tegmental Area and Substantia Nigra Dopamine Neurons in Instrumental Reinforcement. J Neurosci 2023; 43:3895-3908. [PMID: 37185097 PMCID: PMC10217998 DOI: 10.1523/jneurosci.0028-23.2023] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2023] [Revised: 04/06/2023] [Accepted: 04/14/2023] [Indexed: 05/17/2023] Open
Abstract
Reward seeking requires the coordination of motor programs to achieve goals. Midbrain dopamine neurons are critical for reinforcement, and their activation is sufficient for learning about cues, actions, and outcomes. Here we examine in detail the mechanisms underlying the ability of ventral tegmental area (VTA) and substantia nigra (SNc) dopamine neurons to support instrumental learning. By exploiting numerous behavioral tasks in combination with time-limited optogenetic manipulations in male and female rats, we reveal that VTA and SNc dopamine neurons generate reinforcement through separable psychological processes. VTA dopamine neurons imbue actions and their associated cues with motivational value that allows flexible and persistent pursuit, whereas SNc dopamine neurons support time-limited, precise, action-specific learning that is nonscalable and inflexible. This architecture is reminiscent of actor-critic reinforcement learning models with VTA and SNc instructing the critic and actor, respectively. Our findings indicate that heterogeneous dopamine systems support unique forms of instrumental learning that ultimately result in disparate reward-seeking strategies.SIGNIFICANCE STATEMENT Dopamine neurons in the midbrain are essential for learning, motivation, and movement. Here we describe in detail the ability of VTA and SNc dopamine neurons to generate instrumental reinforcement, a process where an agent learns about actions they can emit to earn reward. While rats will avidly work and learn to respond for activation of VTA and SNc dopamine neurons, we find that only VTA dopamine neurons imbue actions and their associated cues with motivational value that spur continued pursuit of reward. Our data support a hypothesis that VTA and SNc dopamine neurons engage distinct psychological processes that have consequences for our understanding of these neurons in health and disease.
Collapse
Affiliation(s)
- Kurt M Fraser
- Department of Psychological & Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, Maryland 21218
| | - Heather J Pribut
- Department of Psychological & Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, Maryland 21218
| | - Patricia H Janak
- Department of Psychological & Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, Maryland 21218
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, Maryland 21205
- Kavli Neuroscience Discovery Institute, Johns Hopkins University School of Medicine, Baltimore, Maryland 21205
| | - Ronald Keiflin
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, Santa Barbara, California 93106
- Neuroscience Research Institute, University of California Santa Barbara, Santa Barbara, California, 93106
| |
Collapse
|
23
|
Urrutia Desmaison JD, Sala RW, Ayyaz A, Nondhalee P, Popa D, Léna C. Cerebellar control of fear learning via the cerebellar nuclei-Multiple pathways, multiple mechanisms? Front Syst Neurosci 2023; 17:1176668. [PMID: 37229350 PMCID: PMC10203220 DOI: 10.3389/fnsys.2023.1176668] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 04/17/2023] [Indexed: 05/27/2023] Open
Abstract
Fear learning is mediated by a large network of brain structures and the understanding of their roles and interactions is constantly progressing. There is a multitude of anatomical and behavioral evidence on the interconnection of the cerebellar nuclei to other structures in the fear network. Regarding the cerebellar nuclei, we focus on the coupling of the cerebellar fastigial nucleus to the fear network and the relation of the cerebellar dentate nucleus to the ventral tegmental area. Many of the fear network structures that receive direct projections from the cerebellar nuclei are playing a role in fear expression or in fear learning and fear extinction learning. We propose that the cerebellum, via its projections to the limbic system, acts as a modulator of fear learning and extinction learning, using prediction-error signaling and regulation of fear related thalamo-cortical oscillations.
Collapse
|
24
|
Takahashi YK, Stalnaker TA, Mueller LE, Harootonian SK, Langdon AJ, Schoenbaum G. Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model. Nat Neurosci 2023; 26:830-839. [PMID: 37081296 PMCID: PMC10646487 DOI: 10.1038/s41593-023-01310-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 03/16/2023] [Indexed: 04/22/2023]
Abstract
Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.
Collapse
Affiliation(s)
- Yuji K Takahashi
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| | - Thomas A Stalnaker
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | - Lauren E Mueller
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | | | - Angela J Langdon
- Intramural Research Program, National Institute of Mental Health, Bethesda, MD, USA.
| | - Geoffrey Schoenbaum
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| |
Collapse
|
25
|
Júnior JBL, Carvalho Crespo LGS, Samuels RI, Coimbra NC, Carey RJ, Carrera MP. Morphine and dopamine: Low dose apomorphine can prevent both the induction and expression of morphine locomotor sensitization and conditioning. Behav Brain Res 2023; 448:114434. [PMID: 37100351 DOI: 10.1016/j.bbr.2023.114434] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 04/06/2023] [Accepted: 04/08/2023] [Indexed: 04/28/2023]
Abstract
The disinhibition of dopamine neurons in the VTA by morphine is considered an important contributor to the reward potency of morphine. In this report, three experiments were conducted in which a low dose of apomorphine (0.05mg/kg) was used as a pretreatment to reduce dopamine activity. Locomotor hyperactivity was used as the behavioral response to morphine (10.0mg/kg). In the first experiment, five treatments with morphine induced the development of locomotor and conditioned hyperactivity that were prevented by apomorphine given 10min prior to morphine. Apomorphine before either vehicle or morphine induced equivalent reductions in locomotion. In the second experiment, the apomorphine pretreatment was initiated after induction of a conditioned hyperactivity and the apomorphine prevented the expression of the conditioning. To assess the effects of the apomorphine on VTA and the nucleus accumbens, ERK measurements were carried out after the induction of locomotor and conditioned hyperactivity. Increased ERK activation was found and these effects were prevented by the apomorphine in both experiments. A third experiment was conducted to assess the effects of acute morphine on ERK before locomotor stimulation was induced by morphine. Acute morphine did not increase locomotion, but a robust ERK response was produced indicating that the morphine induced ERK activation was not secondary to locomotor stimulation. The ERK activation was again prevented by the apomorphine pretreatment. We suggest that contiguity between the ongoing behavioral activity and the morphine activation of the dopamine reward system incentivizes and potentiates the ongoing behavior generating equivalent behavioral sensitization and conditioned effects.
Collapse
Affiliation(s)
- Joaquim Barbosa Leite Júnior
- Behavioral Pharmacology Group, Laboratory of Animal Morphology and Pathology, Universidade Estadual do Norte Fluminense Darcy Ribeiro, Avenida Alberto Lamego, 2000, Campos dos Goytacazes, 28013-602, RJ, Brazil
| | - Luiz Gustavo Soares Carvalho Crespo
- Behavioral Pharmacology Group, Laboratory of Animal Morphology and Pathology, Universidade Estadual do Norte Fluminense Darcy Ribeiro, Avenida Alberto Lamego, 2000, Campos dos Goytacazes, 28013-602, RJ, Brazil
| | - Richard Ian Samuels
- Department of Entomology and Plant Pathology, Universidade Estadual do Norte Fluminense Darcy Ribeiro, Campos dos Goytacazes, RJ, Brazil
| | - Norberto Cysne Coimbra
- Department of Pharmacology, Ribeirão Preto Medical School of the University of São Paulo, SP, Brazil
| | - Robert J Carey
- Department of Psychiatry, SUNY Upstate Medical University, 800 Irving Avenue, Syracuse, NY 13210, USA
| | - Marinete Pinheiro Carrera
- Behavioral Pharmacology Group, Laboratory of Animal Morphology and Pathology, Universidade Estadual do Norte Fluminense Darcy Ribeiro, Avenida Alberto Lamego, 2000, Campos dos Goytacazes, 28013-602, RJ, Brazil.
| |
Collapse
|
26
|
Pool ER, Pauli WM, Cross L, O'Doherty JP. Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.01.26.525637. [PMID: 36747799 PMCID: PMC9901183 DOI: 10.1101/2023.01.26.525637] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/30/2023]
Abstract
Pavlovian learning depends on multiple and parallel associations leading to distinct classes of conditioned responses that vary in their flexibility following changes in the value of an associated outcome. Here, we aimed to differentiate brain areas involved in learning and encoding associations that are sensitive to changes in the value of an outcome from those that are not sensitive to such changes. To address this question, we combined a Pavlovian learning task with outcome devaluation, eye-tracking and functional magnetic resonance imaging. We used computational modeling to identify brain regions involved in learning stimulus-reward associations and stimulus-stimulus associations, by testing for brain areas correlating with reward-prediction errors and state-prediction errors, respectively. We found that, contrary to theoretical predictions about reward prediction errors being exclusively model-free, voxels correlating with reward prediction errors in the ventral striatum and subgenual anterior cingulate cortex were sensitive to devaluation. On the other hand, brain areas correlating with state prediction errors were found to be devaluation insensitive. In a supplementary analysis, we distinguished brain regions encoding predictions about outcome taste identity from those involved in encoding predictions about its expected spatial location. A subset of regions involved in taste identity predictions were devaluation sensitive while those involved in encoding predictions about spatial location were devaluation insensitive. These findings provide insights into the role of multiple associative mechanisms in the brain in mediating Pavlovian conditioned behavior - illustrating how distinct neural pathways can in parallel produce both devaluation sensitive and devaluation insensitive behaviors.
Collapse
|
27
|
Moin Afshar N, Cinotti F, Martin D, Khamassi M, Calu DJ, Taylor JR, Groman SM. Reward-Mediated, Model-Free Reinforcement-Learning Mechanisms in Pavlovian and Instrumental Tasks Are Related. J Neurosci 2023; 43:458-471. [PMID: 36216504 PMCID: PMC9864557 DOI: 10.1523/jneurosci.1113-22.2022] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2022] [Revised: 10/03/2022] [Accepted: 10/06/2022] [Indexed: 01/25/2023] Open
Abstract
Model-free and model-based computations are argued to distinctly update action values that guide decision-making processes. It is not known, however, if these model-free and model-based reinforcement learning mechanisms recruited in operationally based instrumental tasks parallel those engaged by pavlovian-based behavioral procedures. Recently, computational work has suggested that individual differences in the attribution of incentive salience to reward predictive cues, that is, sign- and goal-tracking behaviors, are also governed by variations in model-free and model-based value representations that guide behavior. Moreover, it is not appreciated if these systems that are characterized computationally using model-free and model-based algorithms are conserved across tasks for individual animals. In the current study, we used a within-subject design to assess sign-tracking and goal-tracking behaviors using a pavlovian conditioned approach task and then characterized behavior using an instrumental multistage decision-making (MSDM) task in male rats. We hypothesized that both pavlovian and instrumental learning processes may be driven by common reinforcement-learning mechanisms. Our data confirm that sign-tracking behavior was associated with greater reward-mediated, model-free reinforcement learning and that it was also linked to model-free reinforcement learning in the MSDM task. Computational analyses revealed that pavlovian model-free updating was correlated with model-free reinforcement learning in the MSDM task. These data provide key insights into the computational mechanisms mediating associative learning that could have important implications for normal and abnormal states.SIGNIFICANCE STATEMENT Model-free and model-based computations that guide instrumental decision-making processes may also be recruited in pavlovian-based behavioral procedures. Here, we used a within-subject design to test the hypothesis that both pavlovian and instrumental learning processes were driven by common reinforcement-learning mechanisms. Sign-tracking and goal-tracking behaviors were assessed in rats using a pavlovian conditioned approach task, and then instrumental behavior was characterized using an MSDM task. We report that sign-tracking behavior was associated with greater model-free, but not model-based, learning in the MSDM task. These data suggest that pavlovian and instrumental behaviors may be driven by conserved reinforcement-learning mechanisms.
Collapse
Affiliation(s)
- Neema Moin Afshar
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
| | - François Cinotti
- Department of Experimental Psychology, University of Oxford, Oxford OX2 6GG, United Kingdom
| | - David Martin
- Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Mehdi Khamassi
- Institute of Intelligent Systems and Robotics, Centre National de la Recherche Scientifique, Sorbonne University, 75005 Paris, France
| | - Donna J Calu
- Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Jane R Taylor
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Psychology, Yale University, New Haven, Connecticut 06520
| | - Stephanie M Groman
- Department of Psychiatry, Yale School of Medicine, New Haven, Connecticut 06511
- Department of Neuroscience, University of Minnesota Medical School, Minneapolis, Minnesota 55455
- Department of Psychology, University of Minnesota, Minneapolis, Minnesota 55455
| |
Collapse
|
28
|
Analysis of individual differences in neurofeedback training illuminates successful self-regulation of the dopaminergic midbrain. Commun Biol 2022; 5:845. [PMID: 35986202 PMCID: PMC9391365 DOI: 10.1038/s42003-022-03756-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Accepted: 07/21/2022] [Indexed: 11/27/2022] Open
Abstract
The dopaminergic midbrain is associated with reinforcement learning, motivation and decision-making – functions often disturbed in neuropsychiatric disorders. Previous research has shown that dopaminergic midbrain activity can be endogenously modulated via neurofeedback. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examine whether the activation of particular brain regions associates with successful regulation transfer when feedback is no longer available. Moreover, to elucidate mechanisms underlying effective self-regulation, we study the relation of successful transfer with learning (temporal difference coding) outside the midbrain during neurofeedback training and with individual reward sensitivity in a monetary incentive delay (MID) task. Fifty-nine participants underwent neurofeedback training either in standard (Study 1 N = 15, Study 2 N = 28) or control feedback group (Study 1, N = 16). We find that successful self-regulation is associated with prefrontal reward sensitivity in the MID task (N = 25), with a decreasing relation between prefrontal activity and midbrain learning signals during neurofeedback training and with increased activity within cognitive control areas during transfer. The association between midbrain self-regulation and prefrontal temporal difference and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings provide insights in the control of midbrain activity and may facilitate individually tailoring neurofeedback training. Analysis of real-time fMRI data from 59 participants undergoing neurofeedback training suggests that reinforcement learning contributes to successful self-regulation in the dopaminergic midbrain.
Collapse
|
29
|
de Jong JW, Fraser KM, Lammel S. Mesoaccumbal Dopamine Heterogeneity: What Do Dopamine Firing and Release Have to Do with It? Annu Rev Neurosci 2022; 45:109-129. [PMID: 35226827 PMCID: PMC9271543 DOI: 10.1146/annurev-neuro-110920-011929] [Citation(s) in RCA: 26] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Ventral tegmental area (VTA) dopamine (DA) neurons are often thought to uniformly encode reward prediction errors. Conversely, DA release in the nucleus accumbens (NAc), the prominent projection target of these neurons, has been implicated in reinforcement learning, motivation, aversion, and incentive salience. This contrast between heterogeneous functions of DA release versus a homogeneous role for DA neuron activity raises numerous questions regarding how VTA DA activity translates into NAc DA release. Further complicating this issue is increasing evidence that distinct VTA DA projections into defined NAc subregions mediate diverse behavioral functions. Here, we evaluate evidence for heterogeneity within the mesoaccumbal DA system and argue that frameworks of DA function must incorporate the precise topographic organization of VTA DA neurons to clarify their contribution to health and disease.
Collapse
Affiliation(s)
- Johannes W de Jong
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, California, USA;
| | - Kurt M Fraser
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, California, USA;
| | - Stephan Lammel
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, California, USA;
| |
Collapse
|
30
|
Seitz BM, Hoang IB, DiFazio LE, Blaisdell AP, Sharpe MJ. Dopamine errors drive excitatory and inhibitory components of backward conditioning in an outcome-specific manner. Curr Biol 2022; 32:3210-3218.e3. [PMID: 35752165 DOI: 10.1016/j.cub.2022.06.035] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2022] [Revised: 04/29/2022] [Accepted: 06/13/2022] [Indexed: 01/06/2023]
Abstract
For over two decades, phasic activity in midbrain dopamine neurons was considered synonymous with the prediction error in temporal-difference reinforcement learning.1-4 Central to this proposal is the notion that reward-predictive stimuli become endowed with the scalar value of predicted rewards. When these cues are subsequently encountered, their predictive value is compared to the value of the actual reward received, allowing for the calculation of prediction errors.5,6 Phasic firing of dopamine neurons was proposed to reflect this computation,1,2 facilitating the backpropagation of value from the predicted reward to the reward-predictive stimulus, thus reducing future prediction errors. There are two critical assumptions of this proposal: (1) that dopamine errors can only facilitate learning about scalar value and not more complex features of predicted rewards, and (2) that the dopamine signal can only be involved in anticipatory cue-reward learning in which cues or actions precede rewards. Recent work7-15 has challenged the first assumption, demonstrating that phasic dopamine signals across species are involved in learning about more complex features of the predicted outcomes, in a manner that transcends this value computation. Here, we tested the validity of the second assumption. Specifically, we examined whether phasic midbrain dopamine activity would be necessary for backward conditioning-when a neutral cue reliably follows a rewarding outcome.16-20 Using a specific Pavlovian-to-instrumental transfer (PIT) procedure,21-23 we show rats learn both excitatory and inhibitory components of a backward association, and that this association entails knowledge of the specific identity of the reward and cue. We demonstrate that brief optogenetic inhibition of VTADA neurons timed to the transition between the reward and cue reduces both of these components of backward conditioning. These findings suggest VTADA neurons are capable of facilitating associations between contiguously occurring events, regardless of the content of those events. We conclude that these data may be in line with suggestions that the VTADA error acts as a universal teaching signal. This may provide insight into why dopamine function has been implicated in myriad psychological disorders that are characterized by very distinct reinforcement-learning deficits.
Collapse
Affiliation(s)
- Benjamin M Seitz
- Department of Psychology, University of California, Los Angeles, Portola Plaza, Los Angeles, CA 91602, USA
| | - Ivy B Hoang
- Department of Psychology, University of California, Los Angeles, Portola Plaza, Los Angeles, CA 91602, USA
| | - Lauren E DiFazio
- Department of Psychology, University of California, Los Angeles, Portola Plaza, Los Angeles, CA 91602, USA
| | - Aaron P Blaisdell
- Department of Psychology, University of California, Los Angeles, Portola Plaza, Los Angeles, CA 91602, USA
| | - Melissa J Sharpe
- Department of Psychology, University of California, Los Angeles, Portola Plaza, Los Angeles, CA 91602, USA.
| |
Collapse
|
31
|
Mitra S, Basu S, Singh O, Srivastava A, Singru PS. Calcium-binding proteins typify the dopaminergic neuronal subtypes in the ventral tegmental area of zebra finch, Taeniopygia guttata. J Comp Neurol 2022; 530:2562-2586. [PMID: 35715989 DOI: 10.1002/cne.25352] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2021] [Revised: 05/09/2022] [Accepted: 05/11/2022] [Indexed: 11/11/2022]
Abstract
Calcium-binding proteins (CBPs) regulate neuronal function in midbrain dopamine (DA)-ergic neurons in mammals by buffering and sensing the intracellular Ca2+ , and vesicular release. In birds, the equivalent set of neurons are important in song learning, directed singing, courtship, and energy balance, yet the status of CBPs in these neurons is unknown. Herein, for the first time, we probe the nature of CBPs, namely, Calbindin-, Calretinin-, Parvalbumin-, and Secretagogin-expressing DA neurons in the ventral tegmental area (VTA) and substantia nigra (SN) in the midbrain of zebra finch, Taeniopygia guttata. qRT-PCR analysis of ventral midbrain tissue fragment revealed higher Calbindin- and Calretinin-mRNA levels compared to Parvalbumin and Secretagogin. Application of immunofluorescence showed CBP-immunoreactive (-i) neurons in VTA (anterior [VTAa], mid [VTAm], caudal [VTAc]), SN (compacta [SNc], and reticulata [SNr]). Compared to VTAa, higher Calbindin- and Parvalbumin-immunoreactivity (-ir), and lower Calretinin-ir were observed in VTAm and VTAc. Secretagogin-ir was highly localized to VTAa. In SN, Calbindin- and Calretinin-ir were higher in SNc, SNr was Parvalbumin enriched, and Secretagogin-ir was not detected. Weak, moderate, and intense tyrosine hydroxylase (TH)-i VTA neurons were demarcated as subtypes 1, 2, and 3, respectively. While subtype 1 TH-i neurons were neither Calbindin- nor Calretinin-i, ∼80 and ∼65% subtype 2 and ∼30 and ∼45% subtype 3 TH-i neurons co-expressed Calbindin and Calretinin, respectively. All TH-i neuronal subtypes co-expressed Parvalbumin with reciprocal relationship with TH-ir. We suggest that the CBPs may determine VTA DA neuronal heterogeneity and differentially regulate their activity in T. guttata.
Collapse
Affiliation(s)
- Saptarsi Mitra
- School of Biological Sciences, National Institute of Science Education and Research (NISER), Bhubaneswar, India.,Homi Bhabha National Institute (HBNI), Mumbai, India
| | - Sumela Basu
- School of Biological Sciences, National Institute of Science Education and Research (NISER), Bhubaneswar, India.,Homi Bhabha National Institute (HBNI), Mumbai, India
| | - Omprakash Singh
- School of Biological Sciences, National Institute of Science Education and Research (NISER), Bhubaneswar, India.,Homi Bhabha National Institute (HBNI), Mumbai, India
| | - Abhinav Srivastava
- School of Biological Sciences, National Institute of Science Education and Research (NISER), Bhubaneswar, India.,Homi Bhabha National Institute (HBNI), Mumbai, India
| | - Praful S Singru
- School of Biological Sciences, National Institute of Science Education and Research (NISER), Bhubaneswar, India.,Homi Bhabha National Institute (HBNI), Mumbai, India
| |
Collapse
|
32
|
Millard SJ, Bearden CE, Karlsgodt KH, Sharpe MJ. The prediction-error hypothesis of schizophrenia: new data point to circuit-specific changes in dopamine activity. Neuropsychopharmacology 2022; 47:628-640. [PMID: 34588607 PMCID: PMC8782867 DOI: 10.1038/s41386-021-01188-y] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Revised: 08/23/2021] [Accepted: 09/07/2021] [Indexed: 02/07/2023]
Abstract
Schizophrenia is a severe psychiatric disorder affecting 21 million people worldwide. People with schizophrenia suffer from symptoms including psychosis and delusions, apathy, anhedonia, and cognitive deficits. Strikingly, schizophrenia is characterised by a learning paradox involving difficulties learning from rewarding events, whilst simultaneously 'overlearning' about irrelevant or neutral information. While dysfunction in dopaminergic signalling has long been linked to the pathophysiology of schizophrenia, a cohesive framework that accounts for this learning paradox remains elusive. Recently, there has been an explosion of new research investigating how dopamine contributes to reinforcement learning, which illustrates that midbrain dopamine contributes in complex ways to reinforcement learning, not previously envisioned. This new data brings new possibilities for how dopamine signalling contributes to the symptomatology of schizophrenia. Building on recent work, we present a new neural framework for how we might envision specific dopamine circuits contributing to this learning paradox in schizophrenia in the context of models of reinforcement learning. Further, we discuss avenues of preclinical research with the use of cutting-edge neuroscience techniques where aspects of this model may be tested. Ultimately, it is hoped that this review will spur to action more research utilising specific reinforcement learning paradigms in preclinical models of schizophrenia, to reconcile seemingly disparate symptomatology and develop more efficient therapeutics.
Collapse
Affiliation(s)
- Samuel J. Millard
- grid.19006.3e0000 0000 9632 6718Department of Psychology, University of California, Los Angeles, CA 90095 USA
| | - Carrie E. Bearden
- grid.19006.3e0000 0000 9632 6718Department of Psychology, University of California, Los Angeles, CA 90095 USA ,grid.19006.3e0000 0000 9632 6718Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA 90095 USA
| | - Katherine H. Karlsgodt
- grid.19006.3e0000 0000 9632 6718Department of Psychology, University of California, Los Angeles, CA 90095 USA ,grid.19006.3e0000 0000 9632 6718Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles, CA 90095 USA
| | - Melissa J. Sharpe
- grid.19006.3e0000 0000 9632 6718Department of Psychology, University of California, Los Angeles, CA 90095 USA
| |
Collapse
|
33
|
Favier M, Carcenac C, Savasta M, Carnicella S. Dopamine D3 Receptors: A Potential Target to Treat Motivational Deficits in Parkinson's Disease. Curr Top Behav Neurosci 2022; 60:109-132. [PMID: 35469394 DOI: 10.1007/7854_2022_316] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
Abstract
Parkinson's disease (PD), which is traditionally viewed as a motor disorder involving the degeneration of dopaminergic (DA) neurons, has recently been identified as a quintessential neuropsychiatric condition. Indeed, a plethora of non-motor symptoms may occur in PD, including apathy. Apathy can be defined as a lack of motivation or a deficit of goal-directed behaviors and results in a pathological decrease of self-initiated voluntary behavior. Apathy in PD appears to fluctuate with the DA state of the patients, suggesting a critical role of DA neurotransmission in the pathophysiology of this neuropsychiatric syndrome. Using a lesion-based approach, we developed a rodent model which exhibits specific alteration in the preparatory component of motivational processes, reminiscent to apathy in PD. We found a selective decrease of DA D3 receptors (D3R) expression in the dorsal striatum of lesioned rats. Next, we showed that inhibition of D3R neurotransmission in non-lesioned animals was sufficient to reproduce the motivational deficit observed in our model. Interestingly, we also found that pharmacologically targeting D3R efficiently reversed the motivational deficit induced by the lesion. Our findings, among other recent data, suggest a critical role of D3R in parkinsonian apathy and highlight this receptor as a promising target for treating motivational deficits.
Collapse
Affiliation(s)
- Mathieu Favier
- Douglas Mental Health University Institute, McGill University, Verdun, QC, Canada
| | - Carole Carcenac
- Inserm, U1216, Univ. Grenoble Alpes, Grenoble Institut Neurosciences, Grenoble, France
| | - Marc Savasta
- Inserm, Délégation régionale Provence-Alpes-Côte d'Azur et Corse, Marseille CEDEX 09, France
| | - Sebastien Carnicella
- Inserm, U1216, Univ. Grenoble Alpes, Grenoble Institut Neurosciences, Grenoble, France.
| |
Collapse
|
34
|
Hollon NG, Williams EW, Howard CD, Li H, Traut TI, Jin X. Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors. Curr Biol 2021; 31:5350-5363.e5. [PMID: 34637751 DOI: 10.1016/j.cub.2021.09.040] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2021] [Revised: 08/31/2021] [Accepted: 09/15/2021] [Indexed: 01/08/2023]
Abstract
Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning, signaling discrepancies between actual versus expected reward predicted by the cues.1-5 While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. The vast majority of previous studies examining dopamine responses primarily have used discrete reward-predictive stimuli,1-15 whether Pavlovian conditioned stimuli for which no action is required to earn reward or explicit discriminative stimuli that essentially instruct an animal how and when to respond for reward. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single and sequential instrumental actions, in behavioral contexts with minimal overt changes in the animal's external environment. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal's own action, relative to non-contingent passive stimulation. This dopamine suppression generalized to food rewards was specific to the reinforced action, was temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. These findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.
Collapse
Affiliation(s)
- Nick G Hollon
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA
| | - Elora W Williams
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA
| | - Christopher D Howard
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA
| | - Hao Li
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA
| | - Tavish I Traut
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA
| | - Xin Jin
- Molecular Neurobiology Laboratory, The Salk Institute for Biological Studies, La Jolla, CA 92037, USA; Center for Motor Control and Disease, Key Laboratory of Brain Functional Genomics, East China Normal University, Shanghai 200062, China; NYU-ECNU Institute of Brain and Cognitive Science, New York University Shanghai, Shanghai 200062, China.
| |
Collapse
|
35
|
K Namboodiri VM, Stuber GD. The learning of prospective and retrospective cognitive maps within neural circuits. Neuron 2021; 109:3552-3575. [PMID: 34678148 PMCID: PMC8809184 DOI: 10.1016/j.neuron.2021.09.034] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Revised: 08/26/2021] [Accepted: 09/16/2021] [Indexed: 11/18/2022]
Abstract
Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.
Collapse
Affiliation(s)
- Vijay Mohan K Namboodiri
- Department of Neurology, Center for Integrative Neuroscience, Kavli Institute for Fundamental Neuroscience, Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA.
| | - Garret D Stuber
- Center for the Neurobiology of Addiction, Pain, and Emotion, Department of Anesthesiology and Pain Medicine, Department of Pharmacology, Neuroscience Graduate Program, University of Washington, Seattle, WA 98195, USA.
| |
Collapse
|
36
|
Reverse-translational identification of a cerebellar satiation network. Nature 2021; 600:269-273. [PMID: 34789878 DOI: 10.1038/s41586-021-04143-5] [Citation(s) in RCA: 43] [Impact Index Per Article: 14.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2021] [Accepted: 10/14/2021] [Indexed: 11/08/2022]
Abstract
The brain is the seat of body weight homeostasis. However, our inability to control the increasing prevalence of obesity highlights a need to look beyond canonical feeding pathways to broaden our understanding of body weight control1-3. Here we used a reverse-translational approach to identify and anatomically, molecularly and functionally characterize a neural ensemble that promotes satiation. Unbiased, task-based functional magnetic resonance imaging revealed marked differences in cerebellar responses to food in people with a genetic disorder characterized by insatiable appetite. Transcriptomic analyses in mice revealed molecularly and topographically -distinct neurons in the anterior deep cerebellar nuclei (aDCN) that are activated by feeding or nutrient infusion in the gut. Selective activation of aDCN neurons substantially decreased food intake by reducing meal size without compensatory changes to metabolic rate. We found that aDCN activity terminates food intake by increasing striatal dopamine levels and attenuating the phasic dopamine response to subsequent food consumption. Our study defines a conserved satiation centre that may represent a novel therapeutic target for the management of excessive eating, and underscores the utility of a 'bedside-to-bench' approach for the identification of neural circuits that influence behaviour.
Collapse
|
37
|
Adolescent Dopamine Neurons Represent Reward Differently during Action and State Guided Learning. J Neurosci 2021; 41:9419-9430. [PMID: 34611024 DOI: 10.1523/jneurosci.1321-21.2021] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Revised: 09/08/2021] [Accepted: 09/16/2021] [Indexed: 12/25/2022] Open
Abstract
Neuronal underpinning of learning cause-and-effect associations in the adolescent brain remains poorly understood. Two fundamental forms of associative learning are Pavlovian (classical) conditioning, where a stimulus is followed by an outcome, and operant (instrumental) conditioning, where outcome is contingent on action execution. Both forms of learning, when associated with a rewarding outcome, rely on midbrain dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SN). We find that, in adolescent male rats, reward-guided associative learning is encoded differently by midbrain dopamine neurons in each conditioning paradigm. Whereas simultaneously recorded VTA and SN adult neurons have a similar phasic response to reward delivery during both forms of conditioning, adolescent neurons display a muted reward response during operant but a profoundly larger reward response during Pavlovian conditioning. These results suggest that adolescent neurons assign a different value to reward when it is not gated by action. The learning rate of adolescents and adults during both forms of conditioning was similar, supporting the notion that differences in reward response in each paradigm may be because of differences in motivation and independent of state versus action value learning. Static characteristics of dopamine neurons, such as dopamine cell number and size, were similar in the VTA and SN of both ages, but there were age-related differences in stimulated dopamine release and correlated spike activity, suggesting that differences in reward responsiveness by adolescent dopamine neurons are not because of differences in intrinsic properties of these neurons but engagement of different dopaminergic networks.SIGNIFICANCE STATEMENT Reckless behavior and impulsive decision-making by adolescents suggest that motivated behavioral states are encoded differently by the adolescent brain. Motivated behavior, which is dependent on the function of the dopamine system, follows learning of cause-and-effect associations in the environment. We find that dopamine neurons in adolescents encode reward differently depending on the cause-and-effect relationship of the means to receive that reward. Compared with adults, reward contingent on action led to a muted response, whereas reward that followed a cue but was not gated by action produced an augmented phasic response. These data demonstrate an age-related difference in dopamine neuron response to reward that is not uniform and is guided by processes that differentiate between state and action values.
Collapse
|
38
|
Kutlu MG, Zachry JE, Melugin PR, Cajigas SA, Chevee MF, Kelly SJ, Kutlu B, Tian L, Siciliano CA, Calipari ES. Dopamine release in the nucleus accumbens core signals perceived saliency. Curr Biol 2021; 31:4748-4761.e8. [PMID: 34529938 PMCID: PMC9084920 DOI: 10.1016/j.cub.2021.08.052] [Citation(s) in RCA: 81] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2020] [Revised: 07/15/2021] [Accepted: 08/18/2021] [Indexed: 12/01/2022]
Abstract
A large body of work has aimed to define the precise information encoded by dopaminergic projections innervating the nucleus accumbens (NAc). Prevailing models are based on reward prediction error (RPE) theory, in which dopamine updates associations between rewards and predictive cues by encoding perceived errors between predictions and outcomes. However, RPE cannot describe multiple phenomena to which dopamine is inextricably linked, such as behavior driven by aversive and neutral stimuli. We combined a series of behavioral tasks with direct, subsecond dopamine monitoring in the NAc of mice, machine learning, computational modeling, and optogenetic manipulations to describe behavior and related dopamine release patterns across multiple contingencies reinforced by differentially valenced outcomes. We show that dopamine release only conforms to RPE predictions in a subset of learning scenarios but fits valence-independent perceived saliency encoding across conditions. Here, we provide an extended, comprehensive framework for accumbal dopamine release in behavioral control.
Collapse
Affiliation(s)
- Munir Gunes Kutlu
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA
| | - Jennifer E Zachry
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA
| | - Patrick R Melugin
- Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN 37232, USA
| | - Stephanie A Cajigas
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA; Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN 37232, USA
| | - Maxime F Chevee
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA
| | - Shannon J Kelly
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA
| | - Banu Kutlu
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA; Libraries Strategic Technologies, Penn State University Libraries, University Park, PA 16802, USA
| | - Lin Tian
- Department of Biochemistry and Molecular Medicine, University of California, Davis, Sacramento, CA 95817, USA
| | - Cody A Siciliano
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA; Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN 37232, USA; Vanderbilt Center for Addiction Research, Vanderbilt University, Nashville, TN 37232, USA
| | - Erin S Calipari
- Department of Pharmacology, Vanderbilt University, Nashville, TN 37232, USA; Vanderbilt Brain Institute, Vanderbilt University, Nashville, TN 37232, USA; Vanderbilt Center for Addiction Research, Vanderbilt University, Nashville, TN 37232, USA; Department of Molecular Physiology and Biophysics, Vanderbilt University, Nashville, TN 37232, USA; Department of Psychiatry and Behavioral Sciences, Vanderbilt University, Nashville, TN 37232, USA.
| |
Collapse
|
39
|
Seitz BM, Blaisdell AP, Sharpe MJ. Higher-Order Conditioning and Dopamine: Charting a Path Forward. Front Behav Neurosci 2021; 15:745388. [PMID: 34671247 PMCID: PMC8520911 DOI: 10.3389/fnbeh.2021.745388] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2021] [Accepted: 09/06/2021] [Indexed: 11/16/2022] Open
Abstract
Higher-order conditioning involves learning causal links between multiple events, which then allows one to make novel inferences. For example, observing a correlation between two events (e.g., a neighbor wearing a particular sports jersey), later helps one make new predictions based on this knowledge (e.g., the neighbor's wife's favorite sports team). This type of learning is important because it allows one to benefit maximally from previous experiences and perform adaptively in complex environments where many things are ambiguous or uncertain. Two procedures in the lab are often used to probe this kind of learning, second-order conditioning (SOC) and sensory preconditioning (SPC). In second-order conditioning (SOC), we first teach subjects that there is a relationship between a stimulus and an outcome (e.g., a tone that predicts food). Then, an additional stimulus is taught to precede the predictive stimulus (e.g., a light leads to the food-predictive tone). In sensory preconditioning (SPC), this order of training is reversed. Specifically, the two neutral stimuli (i.e., light and tone) are first paired together and then the tone is paired separately with food. Interestingly, in both SPC and SOC, humans, rodents, and even insects, and other invertebrates will later predict that both the light and tone are likely to lead to food, even though they only experienced the tone directly paired with food. While these processes are procedurally similar, a wealth of research suggests they are associatively and neurobiologically distinct. However, midbrain dopamine, a neurotransmitter long thought to facilitate basic Pavlovian conditioning in a relatively simplistic manner, appears critical for both SOC and SPC. These findings suggest dopamine may contribute to learning in ways that transcend differences in associative and neurological structure. We discuss how research demonstrating that dopamine is critical to both SOC and SPC places it at the center of more complex forms of cognition (e.g., spatial navigation and causal reasoning). Further, we suggest that these more sophisticated learning procedures, coupled with recent advances in recording and manipulating dopamine neurons, represent a new path forward in understanding dopamine's contribution to learning and cognition.
Collapse
Affiliation(s)
- Benjamin M Seitz
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
| | - Aaron P Blaisdell
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
| | - Melissa J Sharpe
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA, United States
| |
Collapse
|
40
|
Hoang IB, Sharpe MJ. The basolateral amygdala and lateral hypothalamus bias learning towards motivationally significant events. Curr Opin Behav Sci 2021. [DOI: 10.1016/j.cobeha.2021.04.014] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
41
|
Mukhin VN, Borovets IR, Sizov VV, Pavlov KI, Klimenko VM. Differential Influence of Amyloid-β on the Kinetics of Dopamine Release in the Dorsal and Ventral Striatum of Rats. Neurotox Res 2021; 39:1285-1292. [PMID: 33991320 DOI: 10.1007/s12640-021-00371-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Revised: 04/22/2021] [Accepted: 05/04/2021] [Indexed: 12/25/2022]
Abstract
Dopaminergic dysfunction is a part of Alzheimer's disease pathology. The brain accumulation of amyloid-β of toxic form is a key link of the pathology, which, according to the literature, is also true for dopaminergic dysfunction. An increase in the amyloid-β level in the brain changes the maximum of the evoked dopamine release in the dorsal and ventral parts of the striatum of the experimental animals. Theoretically, this may be due to the change in the intensity of dopamine release from the nerve terminals or its reuptake. However, it has not been studied. To fill this gap, we examined the amyloid-β induced changes in the kinetics of the evoked dopamine release in the dorsal striatum and the nucleus accumbens core and shell. Amyloid-β solution (fragments 25-35) was injected into the ventricular system of the anesthetized male Wistar rats. Before and after injection, electrically evoked dopamine kinetics was registered with fast-scan cyclic voltammetry. The results had shown that the amount of dopamine release decreases in the dorsal striatum and increases in the nucleus accumbens shell. No changes were found in the intensity of dopamine reuptake.
Collapse
Affiliation(s)
- Valery N Mukhin
- Ivan P. Pavlov Department of Physiology, Federal State Budgetary Scientific Institution Institute of Experimental Medicine, Acad. Pavlov str., 12,, St. Petersburg, 197376, Russia.
| | - Ivan R Borovets
- Ivan P. Pavlov Department of Physiology, Federal State Budgetary Scientific Institution Institute of Experimental Medicine, Acad. Pavlov str., 12,, St. Petersburg, 197376, Russia
| | - Vadim V Sizov
- Ivan P. Pavlov Department of Physiology, Federal State Budgetary Scientific Institution Institute of Experimental Medicine, Acad. Pavlov str., 12,, St. Petersburg, 197376, Russia
| | - Konstantin I Pavlov
- Ivan P. Pavlov Department of Physiology, Federal State Budgetary Scientific Institution Institute of Experimental Medicine, Acad. Pavlov str., 12,, St. Petersburg, 197376, Russia
| | - Victor M Klimenko
- Ivan P. Pavlov Department of Physiology, Federal State Budgetary Scientific Institution Institute of Experimental Medicine, Acad. Pavlov str., 12,, St. Petersburg, 197376, Russia
| |
Collapse
|
42
|
Zubair M, Murris SR, Isa K, Onoe H, Koshimizu Y, Kobayashi K, Vanduffel W, Isa T. Divergent Whole Brain Projections from the Ventral Midbrain in Macaques. Cereb Cortex 2021; 31:2913-2931. [PMID: 33558867 PMCID: PMC8107798 DOI: 10.1093/cercor/bhaa399] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2020] [Revised: 11/09/2020] [Accepted: 12/10/2020] [Indexed: 12/18/2022] Open
Abstract
To understand the connectome of the axonal arborizations of dopaminergic midbrain neurons, we investigated the anterograde spread of highly sensitive viral tracers injected into the ventral tegmental area (VTA) and adjacent areas in 3 macaques. In 2 monkeys, injections were centered on the lateral VTA with some spread into the substantia nigra, while in one animal the injection targeted the medial VTA with partial spread into the ventro-medial thalamus. Double-labeling with antibodies against transduced fluorescent proteins (FPs) and tyrosine hydroxylase indicated that substantial portions of transduced midbrain neurons were dopaminergic. Interestingly, cortical terminals were found either homogeneously in molecular layer I, or more heterogeneously, sometimes forming patches, in the deeper laminae II-VI. In the animals with injections in lateral VTA, terminals were most dense in somatomotor cortex and the striatum. In contrast, when the medial VTA was transduced, dense terminals were found in dorsal prefrontal and temporal cortices, while projections to striatum were sparse. In all monkeys, orbitofrontal and occipito-parietal cortex received strong and weak innervation, respectively. Thus, the dopaminergic ventral midbrain sends heterogeneous projections throughout the brain. Furthermore, our results suggest the existence of subgroups in meso-dopaminergic neurons depending on their location in the primate ventral midbrain.
Collapse
Affiliation(s)
- Muhammad Zubair
- Laboratory of Neuro- and Psychophysiology, Department of Neurosciences, KU Leuven Medical School, Leuven 3000, Belgium
- Leuven Brain Institute, Leuven 3000, Belgium
- Department of Neuroscience, Graduate School of Medicine, Kyoto University, Kyoto 606-8501, Japan
| | - Sjoerd R Murris
- Laboratory of Neuro- and Psychophysiology, Department of Neurosciences, KU Leuven Medical School, Leuven 3000, Belgium
- Leuven Brain Institute, Leuven 3000, Belgium
| | - Kaoru Isa
- Department of Neuroscience, Graduate School of Medicine, Kyoto University, Kyoto 606-8501, Japan
| | - Hirotaka Onoe
- Human Brain Research Center, Graduate School of Medicine, Kyoto University, Kyoto 606-8507, Japan
| | - Yoshinori Koshimizu
- Department of Neuroscience, Graduate School of Medicine, Kyoto University, Kyoto 606-8501, Japan
| | - Kenta Kobayashi
- Section of Viral Vector Development, National Institute for Physiological Sciences, Okazaki, 444-8585, Japan
| | - Wim Vanduffel
- Laboratory of Neuro- and Psychophysiology, Department of Neurosciences, KU Leuven Medical School, Leuven 3000, Belgium
- Leuven Brain Institute, Leuven 3000, Belgium
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA 02129, USA
- Department of Radiology, Harvard Medical School, Boston, MA 02144, USA
| | - Tadashi Isa
- Department of Neuroscience, Graduate School of Medicine, Kyoto University, Kyoto 606-8501, Japan
- Human Brain Research Center, Graduate School of Medicine, Kyoto University, Kyoto 606-8507, Japan
- Institute for the Advanced Study of Human Biology (WPI-ASHBi), Kyoto University, Kyoto 606-8501, Japan
| |
Collapse
|
43
|
Starkweather CK, Uchida N. Dopamine signals as temporal difference errors: recent advances. Curr Opin Neurobiol 2021; 67:95-105. [PMID: 33186815 PMCID: PMC8107188 DOI: 10.1016/j.conb.2020.08.014] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2020] [Revised: 08/24/2020] [Accepted: 08/26/2020] [Indexed: 11/28/2022]
Abstract
In the brain, dopamine is thought to drive reward-based learning by signaling temporal difference reward prediction errors (TD errors), a 'teaching signal' used to train computers. Recent studies using optogenetic manipulations have provided multiple pieces of evidence supporting that phasic dopamine signals function as TD errors. Furthermore, novel experimental results have indicated that when the current state of the environment is uncertain, dopamine neurons compute TD errors using 'belief states' or a probability distribution over potential states. It remains unclear how belief states are computed but emerging evidence suggests involvement of the prefrontal cortex and the hippocampus. These results refine our understanding of the role of dopamine in learning and the algorithms by which dopamine functions in the brain.
Collapse
Affiliation(s)
- Clara Kwon Starkweather
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138, USA
| | - Naoshige Uchida
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA 02138, USA.
| |
Collapse
|
44
|
Lerner TN, Holloway AL, Seiler JL. Dopamine, Updated: Reward Prediction Error and Beyond. Curr Opin Neurobiol 2021; 67:123-130. [PMID: 33197709 PMCID: PMC8116345 DOI: 10.1016/j.conb.2020.10.012] [Citation(s) in RCA: 43] [Impact Index Per Article: 14.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2020] [Revised: 10/12/2020] [Accepted: 10/14/2020] [Indexed: 01/10/2023]
Abstract
Dopamine neurons have been intensely studied for their roles in reinforcement learning. A dominant theory of how these neurons contribute to learning is through the encoding of a reward prediction error (RPE) signal. Recent advances in dopamine research have added nuance to RPE theory by incorporating the ideas of sensory prediction error, distributional encoding, and belief states. Further nuance is likely to be added shortly by convergent lines of research on dopamine neuron diversity. Finally, a major challenge is to reconcile RPE theory with other current theories of dopamine function to account for dopamine's role in movement, motivation, and goal-directed planning.
Collapse
Affiliation(s)
- Talia N Lerner
- Feinberg School of Medicine and Department of Physiology, Northwestern University, Chicago, IL, USA; Northwestern University Interdepartmental Neuroscience Program, Chicago, IL, USA.
| | - Ashley L Holloway
- Feinberg School of Medicine and Department of Physiology, Northwestern University, Chicago, IL, USA; Northwestern University Interdepartmental Neuroscience Program, Chicago, IL, USA
| | - Jillian L Seiler
- Feinberg School of Medicine and Department of Physiology, Northwestern University, Chicago, IL, USA; Department of Psychology, University of Illinois at Chicago, Chicago, IL, USA
| |
Collapse
|
45
|
Speranza L, di Porzio U, Viggiano D, de Donato A, Volpicelli F. Dopamine: The Neuromodulator of Long-Term Synaptic Plasticity, Reward and Movement Control. Cells 2021; 10:735. [PMID: 33810328 PMCID: PMC8066851 DOI: 10.3390/cells10040735] [Citation(s) in RCA: 93] [Impact Index Per Article: 31.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 03/20/2021] [Accepted: 03/23/2021] [Indexed: 01/11/2023] Open
Abstract
Dopamine (DA) is a key neurotransmitter involved in multiple physiological functions including motor control, modulation of affective and emotional states, reward mechanisms, reinforcement of behavior, and selected higher cognitive functions. Dysfunction in dopaminergic transmission is recognized as a core alteration in several devastating neurological and psychiatric disorders, including Parkinson's disease (PD), schizophrenia, bipolar disorder, attention deficit hyperactivity disorder (ADHD) and addiction. Here we will discuss the current insights on the role of DA in motor control and reward learning mechanisms and its involvement in the modulation of synaptic dynamics through different pathways. In particular, we will consider the role of DA as neuromodulator of two forms of synaptic plasticity, known as long-term potentiation (LTP) and long-term depression (LTD) in several cortical and subcortical areas. Finally, we will delineate how the effect of DA on dendritic spines places this molecule at the interface between the motor and the cognitive systems. Specifically, we will be focusing on PD, vascular dementia, and schizophrenia.
Collapse
Affiliation(s)
- Luisa Speranza
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, 1300 Morris Park Avenue, Bronx, NY 10461, USA;
| | - Umberto di Porzio
- Institute of Genetics and Biophysics “Adriano Buzzati Traverso”, CNR, 80131 Naples, Italy
| | - Davide Viggiano
- Department of Translational Medical Sciences, Genetic Research Institute “Gaetano Salvatore”, University of Campania “L. Vanvitelli”, IT and Biogem S.c.a.r.l., 83031 Ariano Irpino, Italy; (D.V.); (A.d.D.)
| | - Antonio de Donato
- Department of Translational Medical Sciences, Genetic Research Institute “Gaetano Salvatore”, University of Campania “L. Vanvitelli”, IT and Biogem S.c.a.r.l., 83031 Ariano Irpino, Italy; (D.V.); (A.d.D.)
| | - Floriana Volpicelli
- Department of Pharmacy, School of Medicine and Surgery, University of Naples Federico II, 80131 Naples, Italy;
| |
Collapse
|
46
|
Inglis JB, Valentin VV, Ashby FG. Modulation of Dopamine for Adaptive Learning: A Neurocomputational Model. COMPUTATIONAL BRAIN & BEHAVIOR 2021; 4:34-52. [PMID: 34151186 PMCID: PMC8210637 DOI: 10.1007/s42113-020-00083-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
There have been many proposals that learning rates in the brain are adaptive, in the sense that they increase or decrease depending on environmental conditions. The majority of these models are abstract and make no attempt to describe the neural circuitry that implements the proposed computations. This article describes a biologically detailed computational model that overcomes this shortcoming. Specifically, we propose a neural circuit that implements adaptive learning rates by modulating the gain on the dopamine response to reward prediction errors, and we model activity within this circuit at the level of spiking neurons. The model generates a dopamine signal that depends on the size of the tonically active dopamine neuron population and the phasic spike rate. The model was tested successfully against results from two single-neuron recording studies and a fast-scan cyclic voltammetry study. We conclude by discussing the general applicability of the model to dopamine mediated tasks that transcend the experimental phenomena it was initially designed to address.
Collapse
Affiliation(s)
- Jeffrey B Inglis
- Interdepartmental Graduate Program in Dynamical Neuroscience, University of California, Santa Barbara
| | - Vivian V Valentin
- Department of Psychological & Brain Sciences, University of California, Santa Barbara
| | - F Gregory Ashby
- Department of Psychological & Brain Sciences, University of California, Santa Barbara
| |
Collapse
|
47
|
Abstract
IMPORTANCE The tools and insights of behavioral neuroscience grow apace, yet their clinical application is lagging. OBSERVATIONS This article suggests that associative learning theory may be the algorithmic bridge to connect a burgeoning understanding of the brain with the challenges to the mind with which all clinicians and researchers are concerned. CONCLUSIONS AND RELEVANCE Instead of giving up, talking past one another, or resting on the laurels of face validity, a consilient and collaborative approach is suggested: visiting laboratory meetings and clinical rounds and attempting to converse in the language of behavior and cognition to better understand and ultimately treat patients.
Collapse
Affiliation(s)
- Philip R. Corlett
- Clinical Neuroscience Research Unit, Department of Psychiatry, Yale University, School of Medicine, New Haven, Connecticut
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, Baltimore, Maryland
| |
Collapse
|
48
|
Tsutsui-Kimura I, Matsumoto H, Akiti K, Yamada MM, Uchida N, Watabe-Uchida M. Distinct temporal difference error signals in dopamine axons in three regions of the striatum in a decision-making task. eLife 2020; 9:e62390. [PMID: 33345774 PMCID: PMC7771962 DOI: 10.7554/elife.62390] [Citation(s) in RCA: 45] [Impact Index Per Article: 11.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2020] [Accepted: 12/18/2020] [Indexed: 12/24/2022] Open
Abstract
Different regions of the striatum regulate different types of behavior. However, how dopamine signals differ across striatal regions and how dopamine regulates different behaviors remain unclear. Here, we compared dopamine axon activity in the ventral, dorsomedial, and dorsolateral striatum, while mice performed a perceptual and value-based decision task. Surprisingly, dopamine axon activity was similar across all three areas. At a glance, the activity multiplexed different variables such as stimulus-associated values, confidence, and reward feedback at different phases of the task. Our modeling demonstrates, however, that these modulations can be inclusively explained by moment-by-moment changes in the expected reward, that is the temporal difference error. A major difference between areas was the overall activity level of reward responses: reward responses in dorsolateral striatum were positively shifted, lacking inhibitory responses to negative prediction errors. The differences in dopamine signals put specific constraints on the properties of behaviors controlled by dopamine in these regions.
Collapse
Affiliation(s)
- Iku Tsutsui-Kimura
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
| | - Hideyuki Matsumoto
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
- Department of Physiology, Osaka City University Graduate School of MedicineOsakaJapan
| | - Korleki Akiti
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
| | - Melissa M Yamada
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
| | - Mitsuko Watabe-Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard UniversityCambridgeUnited States
| |
Collapse
|
49
|
Augustin SM, Loewinger GC, O'Neal TJ, Kravitz AV, Lovinger DM. Dopamine D2 receptor signaling on iMSNs is required for initiation and vigor of learned actions. Neuropsychopharmacology 2020; 45:2087-2097. [PMID: 32811899 PMCID: PMC7547091 DOI: 10.1038/s41386-020-00799-1] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Revised: 07/30/2020] [Accepted: 08/05/2020] [Indexed: 12/15/2022]
Abstract
Striatal dopamine D2 receptors (D2Rs) are important for motor output. Selective deletion of D2Rs from indirect pathway-projecting medium spiny neurons (iMSNs) impairs locomotor activities in a task-specific manner. However, the role of D2Rs in the initiation of motor actions in reward seeking and taking is not fully understood, and there is little information about how receptors contribute under different task demands and with different outcome types. The iMSN-D2Rs modulate neuronal activity and synaptic transmission, exerting control on circuit functions that may play distinct roles in action learning and performance. Selective deletion of D2Rs on iMSNs resulted in slower action initiation and response rate in an instrumental conditioning task, but only when performance demand was increased. The iMSN-Drd2KO mice were also slower to initiate swimming in a T-maze procedural learning task but were unimpaired in cognitive function and behavioral flexibility. In contrast, in a Pavlovian discrimination learning task, iMSN-Drd2KO mice exhibited normal acquisition and extinction of rewarded responding. The iMSN-Drd2KO mice showed performance deficits at all phases of rotarod skill learning. These findings reveal that dopamine modulation through iMSN-D2Rs influences the ability to self-initiate actions, as well as the willingness and/or vigor with which these responses are performed. However, these receptors seem to have little influence on simple associative learning or on stimulus-driven responding. The loss of normal D2R roles may contribute to disorders in which impaired dopamine signaling leads to hypokinesia or impaired initiation of specific voluntary actions.
Collapse
Affiliation(s)
- Shana M Augustin
- Laboratory for Integrative Neuroscience, National Institute on Alcohol Abuse and Alcoholism, National Institutes of Health, Rockville, MD, 20852, USA
| | - Gabriel C Loewinger
- Laboratory for Integrative Neuroscience, National Institute on Alcohol Abuse and Alcoholism, National Institutes of Health, Rockville, MD, 20852, USA
- Department of Biostatistics, Harvard TH Chan School of Public Health, Boston, MA, 02115, USA
| | - Timothy J O'Neal
- National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, MD, 20892, USA
- Graduate Program in Neuroscience and Center for Neurobiology of Addiction, Pain, and Emotion, University of Washington, Seattle, Washington, 98195, USA
| | - Alexxai V Kravitz
- National Institute of Diabetes and Digestive and Kidney Diseases, National Institutes of Health, Bethesda, MD, 20892, USA
- Departments of Psychiatry, Anesthesiology, and Neuroscience, Washington University School of Medicine, St. Louis, Missouri, 63110, USA
| | - David M Lovinger
- Laboratory for Integrative Neuroscience, National Institute on Alcohol Abuse and Alcoholism, National Institutes of Health, Rockville, MD, 20852, USA.
| |
Collapse
|
50
|
Ma L, Tian MX, Sun QY, Liu NN, Dong JF, Feng K, Wu YK, Wang YX, Wang GY, Chen W, Xi JJ, Kang JH. Fetal growth restriction mice are more likely to exhibit depression-like behaviors due to stress-induced loss of dopaminergic neurons in the VTA. FASEB J 2020; 34:13257-13271. [PMID: 32860269 DOI: 10.1096/fj.202000534r] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2020] [Revised: 07/05/2020] [Accepted: 07/16/2020] [Indexed: 11/11/2022]
Abstract
Fetal growth restriction (FGR) is a severe perinatal complication that can increase risk for mental illness. To investigate the mechanism by which FGR mice develop mental illness in adulthood, we established the FGR mouse model and the FGR mice did not display obvious depression-like behaviors, but after environmental stress exposure, FGR mice were more likely to exhibit depression-like behaviors than control mice. Moreover, FGR mice had significantly fewer dopaminergic neurons in the ventral tegmental area but no difference in serotoninergic neurons in the dorsal raphe. RNA-seq analysis showed that the downregulated genes in the midbrain of FGR mice were associated with many mental diseases and were especially involved in the regulation of NMDA-selective glutamate receptor (NMDAR) activity. Furthermore, the NMDAR antagonist memantine can relieve the stress-induced depression-like behaviors of FGR mice. In summary, our findings provide a theoretical basis for future research and treatment of FGR-related depression.
Collapse
Affiliation(s)
- Li Ma
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Meng-Xue Tian
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China.,Institute of Translational Research, Tongji Hospital, School of Life Sciences and Technology, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, Tongji University, Shanghai, China
| | - Qiao-Yi Sun
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Na-Na Liu
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Jian-Feng Dong
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Ke Feng
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Yu-Kang Wu
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Yu-Xi Wang
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Gui-Ying Wang
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Wen Chen
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Jia-Jie Xi
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| | - Jiu-Hong Kang
- Clinical and Translational Research Center of Shanghai First Maternity and Infant Hospital, Shanghai Key Laboratory of Signaling and Disease Research, Collaborative Innovation Center for Brain Science, School of Life Sciences and Technology, Tongji University, Shanghai, China
| |
Collapse
|