1
|
Gershman SJ, Assad JA, Datta SR, Linderman SW, Sabatini BL, Uchida N, Wilbrecht L. Explaining dopamine through prediction errors and beyond. Nat Neurosci 2024:10.1038/s41593-024-01705-4. [PMID: 39054370 DOI: 10.1038/s41593-024-01705-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 06/13/2024] [Indexed: 07/27/2024]
Abstract
The most influential account of phasic dopamine holds that it reports reward prediction errors (RPEs). The RPE-based interpretation of dopamine signaling is, in its original form, probably too simple and fails to explain all the properties of phasic dopamine observed in behaving animals. This Perspective helps to resolve some of the conflicting interpretations of dopamine that currently exist in the literature. We focus on the following three empirical challenges to the RPE theory of dopamine: why does dopamine (1) ramp up as animals approach rewards, (2) respond to sensory and motor features and (3) influence action selection? We argue that the prediction error concept, once it has been suitably modified and generalized based on an analysis of each computational problem, answers each challenge. Nonetheless, there are a number of additional empirical findings that appear to demand fundamentally different theoretical explanations beyond encoding RPE. Therefore, looking forward, we discuss the prospects for a unifying theory that respects the diversity of dopamine signaling and function as well as the complex circuitry that both underlies and responds to dopaminergic transmission.
Collapse
Affiliation(s)
- Samuel J Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, USA.
- Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA, USA.
| | - John A Assad
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | | | - Scott W Linderman
- Department of Statistics and Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA
| | - Bernardo L Sabatini
- Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University, Cambridge, MA, USA
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
- Howard Hughes Medical Institute, Chevy Chase, MD, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Linda Wilbrecht
- Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| |
Collapse
|
2
|
Heer CM, Sheffield MEJ. Distinct catecholaminergic pathways projecting to hippocampal CA1 transmit contrasting signals during navigation in familiar and novel environments. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.11.29.569214. [PMID: 38076843 PMCID: PMC10705417 DOI: 10.1101/2023.11.29.569214] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/22/2023]
Abstract
Neuromodulatory inputs to the hippocampus play pivotal roles in modulating synaptic plasticity, shaping neuronal activity, and influencing learning and memory. Recently it has been shown that the main sources of catecholamines to the hippocampus, ventral tegmental area (VTA) and locus coeruleus (LC), may have overlapping release of neurotransmitters and effects on the hippocampus. Therefore, to dissect the impacts of both VTA and LC circuits on hippocampal function, a thorough examination of how these pathways might differentially operate during behavior and learning is necessary. We therefore utilized 2-photon microscopy to functionally image the activity of VTA and LC axons within the CA1 region of the dorsal hippocampus in head-fixed male mice navigating linear paths within virtual reality (VR) environments. We found that within familiar environments some VTA axons and the vast majority of LC axons showed a correlation with the animals' running speed. However, as mice approached previously learned rewarded locations, a large majority of VTA axons exhibited a gradual ramping-up of activity, peaking at the reward location. In contrast, LC axons displayed a pre-movement signal predictive of the animal's transition from immobility to movement. Interestingly, a marked divergence emerged following a switch from the familiar to novel VR environments. Many LC axons showed large increases in activity that remained elevated for over a minute, while the previously observed VTA axon ramping-to-reward dynamics disappeared during the same period. In conclusion, these findings highlight distinct roles of VTA and LC catecholaminergic inputs in the dorsal CA1 hippocampal region. These inputs encode unique information, with reward information in VTA inputs and novelty and kinematic information in LC inputs, likely contributing to differential modulation of hippocampal activity during behavior and learning.
Collapse
Affiliation(s)
- Chad M Heer
- The Department of Neurobiology, The University of Chicago, Chicago, IL, USA
| | - Mark E J Sheffield
- The Department of Neurobiology, The University of Chicago, Chicago, IL, USA
| |
Collapse
|
3
|
Labouesse MA, Wilhelm M, Kagiampaki Z, Yee AG, Denis R, Harada M, Gresch A, Marinescu AM, Otomo K, Curreli S, Serratosa Capdevila L, Zhou X, Cola RB, Ravotto L, Glück C, Cherepanov S, Weber B, Zhou X, Katner J, Svensson KA, Fellin T, Trudeau LE, Ford CP, Sych Y, Patriarchi T. A chemogenetic approach for dopamine imaging with tunable sensitivity. Nat Commun 2024; 15:5551. [PMID: 38956067 PMCID: PMC11219860 DOI: 10.1038/s41467-024-49442-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Accepted: 06/05/2024] [Indexed: 07/04/2024] Open
Abstract
Genetically-encoded dopamine (DA) sensors enable high-resolution imaging of DA release, but their ability to detect a wide range of extracellular DA levels, especially tonic versus phasic DA release, is limited by their intrinsic affinity. Here we show that a human-selective dopamine receptor positive allosteric modulator (PAM) can be used to boost sensor affinity on-demand. The PAM enhances DA detection sensitivity across experimental preparations (in vitro, ex vivo and in vivo) via one-photon or two-photon imaging. In vivo photometry-based detection of optogenetically-evoked DA release revealed that DETQ administration produces a stable 31 minutes window of potentiation without effects on animal behavior. The use of the PAM revealed region-specific and metabolic state-dependent differences in tonic DA levels and enhanced single-trial detection of behavior-evoked phasic DA release in cortex and striatum. Our chemogenetic strategy can potently and flexibly tune DA imaging sensitivity and reveal multi-modal (tonic/phasic) DA signaling across preparations and imaging approaches.
Collapse
Affiliation(s)
- Marie A Labouesse
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
- Neuroscience Center Zurich, University and ETH Zürich, Zürich, Switzerland
- Department of Health Sciences and Technology, ETH Zürich, Zürich, Switzerland
| | - Maria Wilhelm
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
- Institute for Neuroscience, ETH Zurich, Zurich, Switzerland
| | | | - Andrew G Yee
- Department of Pharmacology, University of Colorado School of Medicine, Aurora, CO, USA
| | - Raphaelle Denis
- Department of Pharmacology & Physiology, Faculty of Medicine, SNC and CIRCA Research groups, Université de Montréal, Montréal, QC, Canada
- Department of Neurosciences, Faculty of Medicine, SNC and CIRCA Research groups, Université de Montréal, Montréal, QC, Canada
| | - Masaya Harada
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | - Andrea Gresch
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | | | - Kanako Otomo
- Department of Health Sciences and Technology, ETH Zürich, Zürich, Switzerland
| | - Sebastiano Curreli
- Optical Approaches to Brain Function Laboratory, Istituto Italiano di Tecnologia, Genova, Italy
| | | | - Xuehan Zhou
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | - Reto B Cola
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | - Luca Ravotto
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | - Chaim Glück
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
| | - Stanislav Cherepanov
- Institute of Cellular and Integrative Neuroscience, University of Strasbourg, Strasbourg, France
| | - Bruno Weber
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland
- Neuroscience Center Zurich, University and ETH Zürich, Zürich, Switzerland
| | - Xin Zhou
- Eli Lilly and Company, Indianapolis, IN, USA
| | | | | | - Tommaso Fellin
- Optical Approaches to Brain Function Laboratory, Istituto Italiano di Tecnologia, Genova, Italy
| | - Louis-Eric Trudeau
- Department of Pharmacology & Physiology, Faculty of Medicine, SNC and CIRCA Research groups, Université de Montréal, Montréal, QC, Canada
- Department of Neurosciences, Faculty of Medicine, SNC and CIRCA Research groups, Université de Montréal, Montréal, QC, Canada
| | - Christopher P Ford
- Department of Pharmacology, University of Colorado School of Medicine, Aurora, CO, USA
| | - Yaroslav Sych
- Institute of Cellular and Integrative Neuroscience, University of Strasbourg, Strasbourg, France
| | - Tommaso Patriarchi
- Institute of Pharmacology and Toxicology, University of Zürich, Zürich, Switzerland.
- Neuroscience Center Zurich, University and ETH Zürich, Zürich, Switzerland.
| |
Collapse
|
4
|
Ohtake M, Abe K, Hasegawa M, Itokazu T, Selvakumar V, Matunis A, Stacy E, Froebrich E, Huynh N, Lee H, Kambe Y, Yamamoto T, Sato TK, Sato TR. Encoding of self-initiated actions in axon terminals of the mesocortical pathway. NEUROPHOTONICS 2024; 11:033408. [PMID: 38726349 PMCID: PMC11080647 DOI: 10.1117/1.nph.11.3.033408] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/05/2023] [Revised: 03/01/2024] [Accepted: 03/05/2024] [Indexed: 05/12/2024]
Abstract
Significance The initiation of goal-directed actions is a complex process involving the medial prefrontal cortex and dopaminergic inputs through the mesocortical pathway. However, it is unclear what information the mesocortical pathway conveys and how it impacts action initiation. In this study, we unveiled the indispensable role of mesocortical axon terminals in encoding the execution of movements in self-initiated actions. Aim To investigate the role of mesocortical axon terminals in encoding the execution of movements in self-initiated actions. Approach We designed a lever-press task in which mice internally determine the timing of the press, receiving a larger reward for longer waiting periods. Results Our study revealed that self-initiated actions depend on dopaminergic signaling mediated by D2 receptors, whereas sensory-triggered lever-press actions do not involve D2 signaling. Microprism-mediated two-photon calcium imaging further demonstrated ramping activity in mesocortical axon terminals approximately 0.5 s before the self-initiated lever press. Remarkably, the ramping patterns remained consistent whether the mice responded to cues immediately for a smaller reward or held their response for a larger reward. Conclusions We conclude that mesocortical dopamine axon terminals encode the timing of self-initiated actions, shedding light on a crucial aspect of the intricate neural mechanisms governing goal-directed behavior.
Collapse
Affiliation(s)
- Makoto Ohtake
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
- Yokohama City University, Department of Neurosurgery, Yokohama, Japan
| | - Kenta Abe
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
| | - Masashi Hasegawa
- Rutgers, The State University of New Jersey, Robert Wood Johnson Medical School, Center for Advanced Biotechnology and Medicine, Department of Neuroscience and Cell Biology, Piscataway, New Jersey, United States
| | - Takahide Itokazu
- Osaka University, Department of Neuro-Medical Science, Osaka, Japan
| | - Vihashini Selvakumar
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
| | - Ashley Matunis
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
- College of Charleston, Department of Biology, Charleston, South Carolina, United States
| | - Emma Stacy
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
- College of Charleston, Department of Biology, Charleston, South Carolina, United States
| | - Emily Froebrich
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
- College of Charleston, Department of Biology, Charleston, South Carolina, United States
| | - Nathan Huynh
- Kagoshima University, Department of Pharmacology, Kagoshima, Japan
| | - Haesuk Lee
- Kagoshima University, Department of Pharmacology, Kagoshima, Japan
| | - Yuki Kambe
- Kagoshima University, Department of Pharmacology, Kagoshima, Japan
| | - Tetsuya Yamamoto
- Yokohama City University, Department of Neurosurgery, Yokohama, Japan
| | - Tatsuo K. Sato
- Kagoshima University, Department of Pharmacology, Kagoshima, Japan
- FOREST, Japan Science and Technology Agency, Saitama, Japan
| | - Takashi R. Sato
- Medical University of South Carolina, Department of Neuroscience, Charleston, South Carolina, United States
| |
Collapse
|
5
|
Schütt HH, Kim D, Ma WJ. Reward prediction error neurons implement an efficient code for reward. Nat Neurosci 2024; 27:1333-1339. [PMID: 38898182 DOI: 10.1038/s41593-024-01671-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/03/2022] [Accepted: 04/29/2024] [Indexed: 06/21/2024]
Abstract
We use efficient coding principles borrowed from sensory neuroscience to derive the optimal neural population to encode a reward distribution. We show that the responses of dopaminergic reward prediction error neurons in mouse and macaque are similar to those of the efficient code in the following ways: the neurons have a broad distribution of midpoints covering the reward distribution; neurons with higher thresholds have higher gains, more convex tuning functions and lower slopes; and their slope is higher when the reward distribution is narrower. Furthermore, we derive learning rules that converge to the efficient code. The learning rule for the position of the neuron on the reward axis closely resembles distributional reinforcement learning. Thus, reward prediction error neuron responses may be optimized to broadcast an efficient reward signal, forming a connection between efficient coding and reinforcement learning, two of the most successful theories in computational neuroscience.
Collapse
Affiliation(s)
- Heiko H Schütt
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA.
- Department of Behavioural and Cognitive Sciences, Université du Luxembourg, Esch-Belval, Luxembourg.
| | - Dongjae Kim
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
- Department of AI-Based Convergence, Dankook University, Yongin, Republic of Korea
| | - Wei Ji Ma
- Center for Neural Science and Department of Psychology, New York University, New York, NY, USA
| |
Collapse
|
6
|
Song MR, Lee SW. Rethinking dopamine-guided action sequence learning. Eur J Neurosci 2024; 60:3447-3465. [PMID: 38798086 DOI: 10.1111/ejn.16426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 04/21/2024] [Accepted: 05/08/2024] [Indexed: 05/29/2024]
Abstract
As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.
Collapse
Affiliation(s)
- Minryung R Song
- Department of Brain and Cognitive Sciences, KAIST, Daejeon, South Korea
| | - Sang Wan Lee
- Department of Brain and Cognitive Sciences, KAIST, Daejeon, South Korea
- Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea
- KI for Health Science and Technology, KAIST, Daejeon, South Korea
- Center for Neuroscience-inspired AI, KAIST, Daejeon, South Korea
| |
Collapse
|
7
|
Kleinman MR, Foster DJ. Spatial localization of hippocampal replay requires dopamine signaling. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.04.597435. [PMID: 38895442 PMCID: PMC11185723 DOI: 10.1101/2024.06.04.597435] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/21/2024]
Abstract
Sequenced reactivations of hippocampal neurons called replays, concomitant with sharp-wave ripples in the local field potential, are critical for the consolidation of episodic memory, but whether replays depend on the brain's reward or novelty signals is unknown. Here we combined chemogenetic silencing of dopamine neurons in ventral tegmental area (VTA) and simultaneous electrophysiological recordings in dorsal hippocampal CA1, in freely behaving rats experiencing changes to reward magnitude and environmental novelty. Surprisingly, VTA silencing did not prevent ripple increases where reward was increased, but caused dramatic, aberrant ripple increases where reward was unchanged. These increases were associated with increased reverse-ordered replays. On familiar tracks this effect disappeared, and ripples tracked reward prediction error, indicating that non-VTA reward signals were sufficient to direct replay. Our results reveal a novel dependence of hippocampal replay on dopamine, and a role for a VTA-independent reward prediction error signal that is reliable only in familiar environments.
Collapse
Affiliation(s)
- Matthew R Kleinman
- Helen Wills Neuroscience Institute and Department of Psychology, University of California, Berkeley, CA 94720, USA
| | - David J Foster
- Helen Wills Neuroscience Institute and Department of Psychology, University of California, Berkeley, CA 94720, USA
- Lead contact
| |
Collapse
|
8
|
Garr E, Cheng Y, Jeong H, Brooke S, Castell L, Bal A, Magnard R, Namboodiri VMK, Janak PH. Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies. SCIENCE ADVANCES 2024; 10:eadn4203. [PMID: 38809978 PMCID: PMC11135394 DOI: 10.1126/sciadv.adn4203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 04/23/2024] [Indexed: 05/31/2024]
Abstract
Learning causal relationships relies on understanding how often one event precedes another. To investigate how dopamine neuron activity and neurotransmitter release change when a retrospective relationship is degraded for a specific pair of events, we used outcome-selective Pavlovian contingency degradation in rats. Conditioned responding was attenuated for the cue-reward contingency that was degraded, as was dopamine neuron activity in the midbrain and dopamine release in the ventral striatum in response to the cue and subsequent reward. Contingency degradation also abolished the trial-by-trial history dependence of the dopamine responses at the time of trial outcome. This profile of changes in cue- and reward-evoked responding is not easily explained by a standard reinforcement learning model. An alternative model based on learning causal relationships was better able to capture dopamine responses during contingency degradation, as well as conditioned behavior following optogenetic manipulations of dopamine during noncontingent rewards. Our results suggest that mesostriatal dopamine encodes the contingencies between meaningful events during learning.
Collapse
Affiliation(s)
- Eric Garr
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yifeng Cheng
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Huijeong Jeong
- Department of Neurology, University of California, San Francisco, CA 94158, USA
| | - Sara Brooke
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
| | - Laia Castell
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Aneesh Bal
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Robin Magnard
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Vijay Mohan K. Namboodiri
- Department of Neurology, University of California, San Francisco, CA 94158, USA
- Weill Institute for Neurosciences, Kavli Institute for Fundamental Neuroscience, Center for Integrative Neuroscience, University of California, San Francisco, CA 94158, USA
| | - Patricia H. Janak
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
- Kavli Neuroscience Discovery Institute, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| |
Collapse
|
9
|
Imtiaz Z, Kato A, Kopell BH, Qasim SE, Davis AN, Martinez LN, Heflin M, Kulkarni K, Morsi A, Gu X, Saez I. Human Substantia Nigra Neurons Encode Reward Expectations. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.10.593406. [PMID: 38766086 PMCID: PMC11100806 DOI: 10.1101/2024.05.10.593406] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/22/2024]
Abstract
Dopamine (DA) signals originating from substantia nigra (SN) neurons are centrally involved in the regulation of motor and reward processing. DA signals behaviorally relevant events where reward outcomes differ from expectations (reward prediction errors, RPEs). RPEs play a crucial role in learning optimal courses of action and in determining response vigor when an agent expects rewards. Nevertheless, how reward expectations, crucial for RPE calculations, are conveyed to and represented in the dopaminergic system is not fully understood, especially in the human brain where the activity of DA neurons is difficult to study. One possibility, suggested by evidence from animal models, is that DA neurons explicitly encode reward expectations. Alternatively, they may receive RPE information directly from upstream brain regions. To address whether SN neuron activity directly reflects reward expectation information, we directly examined the encoding of reward expectation signals in human putative DA neurons by performing single-unit recordings from the SN of patients undergoing neurosurgery. Patients played a two-armed bandit decision-making task in which they attempted to maximize reward. We show that neuronal firing rates (FR) of putative DA neurons during the reward expectation period explicitly encode reward expectations. First, activity in these neurons was modulated by previous trial outcomes, such that FR were greater after positive outcomes than after neutral or negative outcome trials. Second, this increase in FR was associated with shorter reaction times, consistent with an invigorating effect of DA neuron activity during expectation. These results suggest that human DA neurons explicitly encode reward expectations, providing a neurophysiological substrate for a signal critical for reward learning.
Collapse
Affiliation(s)
- Zarghona Imtiaz
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ayaka Kato
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Brian H. Kopell
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Salman E. Qasim
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Arianna Neal Davis
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Lizbeth Nunez Martinez
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Matt Heflin
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Kaustubh Kulkarni
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Amr Morsi
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Xiaosi Gu
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Ignacio Saez
- Nash Family Department of Neuroscience and the Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurosurgery, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| |
Collapse
|
10
|
Costa KM, Zhang Z, Zhuo Y, Li G, Li Y, Schoenbaum G. Dopamine and acetylcholine correlations in the nucleus accumbens depend on behavioral task states. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.05.03.592439. [PMID: 38746204 PMCID: PMC11092761 DOI: 10.1101/2024.05.03.592439] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Dopamine in the nucleus accumbens ramps up as animals approach desired goals. These ramps have received intense scrutiny because they seem to violate long-held hypotheses on dopamine function. Furthermore, it has been proposed that they are driven by local acetylcholine release, i.e., that they are mechanistically separate from dopamine signals related to reward prediction errors. Here, we tested this hypothesis by simultaneously recording accumbal dopamine and acetylcholine signals in rats executing a task involving motivated approach. Contrary to recent reports, we found that dopamine ramps were not coincidental with changes in acetylcholine. Instead, we found that acetylcholine could be positively, negatively, or uncorrelated with dopamine depending on whether the task phase was determined by a salient cue, reward prediction error, or active approach, respectively. Our results suggest that accumbal dopamine and acetylcholine are largely independent but may combine to engage different postsynaptic mechanisms depending on the behavioral task states.
Collapse
|
11
|
Floeder JR, Jeong H, Mohebi A, Namboodiri VMK. Mesolimbic dopamine ramps reflect environmental timescales. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.27.587103. [PMID: 38659749 PMCID: PMC11042231 DOI: 10.1101/2024.03.27.587103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.
Collapse
Affiliation(s)
- Joseph R Floeder
- Neuroscience Graduate Program, University of California, San Francisco, CA, USA
| | - Huijeong Jeong
- Department of Neurology, University of California, San Francisco, CA, USA
| | - Ali Mohebi
- Department of Neurology, University of California, San Francisco, CA, USA
| | - Vijay Mohan K Namboodiri
- Neuroscience Graduate Program, University of California, San Francisco, CA, USA
- Department of Neurology, University of California, San Francisco, CA, USA
- Weill Institute for Neurosciences, Kavli Institute for Fundamental Neuroscience, Center for Integrative Neuroscience, University of California, San Francisco, CA, USA
| |
Collapse
|
12
|
Mah A, Golden C, Constantinople C. Mesolimbic dopamine encodes reward prediction errors independent of learning rates. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.04.18.590090. [PMID: 38659861 PMCID: PMC11042285 DOI: 10.1101/2024.04.18.590090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
Biological accounts of reinforcement learning posit that dopamine encodes reward prediction errors (RPEs), which are multiplied by a learning rate to update state or action values. These values are thought to be represented in synaptic weights in the striatum, and updated by dopamine-dependent plasticity, suggesting that dopamine release might reflect the product of the learning rate and RPE. Here, we leveraged the fact that animals learn faster in volatile environments to characterize dopamine encoding of learning rates. We trained rats on a task with semi-observable states offering different rewards, and rats adjusted how quickly they initiated trials across states using RPEs. Computational modeling and behavioral analyses showed that learning rates were higher following state transitions, and scaled with trial-by-trial changes in beliefs about hidden states, approximating normative Bayesian strategies. Notably, dopamine release in the nucleus accumbens encoded RPEs independent of learning rates, suggesting that dopamine-independent mechanisms instantiate dynamic learning rates.
Collapse
Affiliation(s)
- Andrew Mah
- Center for Neural Science, New York University
| | | | | |
Collapse
|
13
|
Harada M, Capdevila LS, Wilhelm M, Burdakov D, Patriarchi T. Stimulation of VTA dopamine inputs to LH upregulates orexin neuronal activity in a DRD2-dependent manner. eLife 2024; 12:RP90158. [PMID: 38567902 PMCID: PMC10990487 DOI: 10.7554/elife.90158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/05/2024] Open
Abstract
Dopamine and orexins (hypocretins) play important roles in regulating reward-seeking behaviors. It is known that hypothalamic orexinergic neurons project to dopamine neurons in the ventral tegmental area (VTA), where they can stimulate dopaminergic neuronal activity. Although there are reciprocal connections between dopaminergic and orexinergic systems, whether and how dopamine regulates the activity of orexin neurons is currently not known. Here we implemented an opto-Pavlovian task in which mice learn to associate a sensory cue with optogenetic dopamine neuron stimulation to investigate the relationship between dopamine release and orexin neuron activity in the lateral hypothalamus (LH). We found that dopamine release can be evoked in LH upon optogenetic stimulation of VTA dopamine neurons and is also naturally evoked by cue presentation after opto-Pavlovian learning. Furthermore, orexin neuron activity could also be upregulated by local stimulation of dopaminergic terminals in the LH in a way that is partially dependent on dopamine D2 receptors (DRD2). Our results reveal previously unknown orexinergic coding of reward expectation and unveil an orexin-regulatory axis mediated by local dopamine inputs in the LH.
Collapse
Affiliation(s)
- Masaya Harada
- Institute of Pharmacology and Toxicology, University of ZürichZürichSwitzerland
| | | | - Maria Wilhelm
- Institute of Pharmacology and Toxicology, University of ZürichZürichSwitzerland
| | - Denis Burdakov
- Neuroscience Center Zürich, University and ETH ZürichZürichSwitzerland
- Department of Health Sciences and Technology, ETH ZürichZürichSwitzerland
| | - Tommaso Patriarchi
- Institute of Pharmacology and Toxicology, University of ZürichZürichSwitzerland
- Neuroscience Center Zürich, University and ETH ZürichZürichSwitzerland
| |
Collapse
|
14
|
Holly EN, Galanaugh J, Fuccillo MV. Local regulation of striatal dopamine: A diversity of circuit mechanisms for a diversity of behavioral functions? Curr Opin Neurobiol 2024; 85:102839. [PMID: 38309106 PMCID: PMC11066854 DOI: 10.1016/j.conb.2024.102839] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 01/02/2024] [Accepted: 01/03/2024] [Indexed: 02/05/2024]
Abstract
Striatal dopamine governs a wide range of behavioral functions, yet local dopamine concentrations can be dissociated from somatic activity. Here, we discuss how dopamine's diverse roles in behavior may be driven by local circuit mechanisms shaping dopamine release. We first look at historical and recent work demonstrating that striatal circuits interact with dopaminergic terminals to either initiate the release of dopamine or modulate the release of dopamine initiated by spiking in midbrain dopamine neurons, with particular attention to GABAergic and cholinergic local circuit mechanisms. Then we discuss some of the first in vivo studies of acetylcholine-dopamine interactions in striatum and broadly discuss necessary future work in understanding the roles of midbrain versus striatal dopamine regulation.
Collapse
Affiliation(s)
- Elizabeth N Holly
- Center for Molecular and Behavioral Neuroscience, Rutgers University, 197 University Ave, Newark, NJ 07102, USA. https://twitter.com/ENHolly
| | - Jamie Galanaugh
- Neuroscience Graduate Group, Perelman School of Medicine at the University of Pennsylvania, 415 Curie Blvd, Philadelphia, PA 19104, USA. https://twitter.com/jamie_galanaugh
| | - Marc V Fuccillo
- Department of Neuroscience, Perelman School of Medicine at the University of Pennsylvania, 415 Curie Blvd, Philadelphia, PA 19104, USA.
| |
Collapse
|
15
|
Zhuo Y, Luo B, Yi X, Dong H, Miao X, Wan J, Williams JT, Campbell MG, Cai R, Qian T, Li F, Weber SJ, Wang L, Li B, Wei Y, Li G, Wang H, Zheng Y, Zhao Y, Wolf ME, Zhu Y, Watabe-Uchida M, Li Y. Improved green and red GRAB sensors for monitoring dopaminergic activity in vivo. Nat Methods 2024; 21:680-691. [PMID: 38036855 PMCID: PMC11009088 DOI: 10.1038/s41592-023-02100-w] [Citation(s) in RCA: 16] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Accepted: 10/23/2023] [Indexed: 12/02/2023]
Abstract
Dopamine (DA) plays multiple roles in a wide range of physiological and pathological processes via a large network of dopaminergic projections. To dissect the spatiotemporal dynamics of DA release in both dense and sparsely innervated brain regions, we developed a series of green and red fluorescent G-protein-coupled receptor activation-based DA (GRABDA) sensors using a variety of DA receptor subtypes. These sensors have high sensitivity, selectivity and signal-to-noise ratio with subsecond response kinetics and the ability to detect a wide range of DA concentrations. We then used these sensors in mice to measure both optogenetically evoked and behaviorally relevant DA release while measuring neurochemical signaling in the nucleus accumbens, amygdala and cortex. Using these sensors, we also detected spatially resolved heterogeneous cortical DA release in mice performing various behaviors. These next-generation GRABDA sensors provide a robust set of tools for imaging dopaminergic activity under a variety of physiological and pathological conditions.
Collapse
Affiliation(s)
- Yizhou Zhuo
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Bin Luo
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Peking-Tsinghua Center for Life Sciences, New Cornerstone Science Laboratory, Academy for Advanced Interdisciplinary Studies, Beijing, China
| | - Xinyang Yi
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Hui Dong
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Peking-Tsinghua Center for Life Sciences, New Cornerstone Science Laboratory, Academy for Advanced Interdisciplinary Studies, Beijing, China
| | - Xiaolei Miao
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Department of Anesthesiology, Beijing Chaoyang Hospital, Capital Medical University, Beijing, China
| | - Jinxia Wan
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Peking-Tsinghua Center for Life Sciences, New Cornerstone Science Laboratory, Academy for Advanced Interdisciplinary Studies, Beijing, China
| | - John T Williams
- Vollum Institute, Oregon Health & Science University, Portland, OR, USA
| | - Malcolm G Campbell
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Ruyi Cai
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Tongrui Qian
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Fengling Li
- Shenzhen Key Laboratory of Drug Addiction, Shenzhen Neher Neural Plasticity Laboratory, the Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Sophia J Weber
- Department of Behavioral Neuroscience, Oregon Health & Science University, Portland, OR, USA
| | - Lei Wang
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Peking University-Tsinghua University-National Institute of Biological Sciences Joint Graduate Program, Peking University, Beijing, China
| | - Bozhi Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
- Department of Neurology, The First Medical Center, Chinese PLA General Hospital, Beijing, China
| | - Yu Wei
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Guochuan Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Huan Wang
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Yu Zheng
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Yulin Zhao
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China
| | - Marina E Wolf
- Department of Behavioral Neuroscience, Oregon Health & Science University, Portland, OR, USA
| | - Yingjie Zhu
- Shenzhen Key Laboratory of Drug Addiction, Shenzhen Neher Neural Plasticity Laboratory, the Brain Cognition and Brain Disease Institute, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Mitsuko Watabe-Uchida
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Yulong Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing, China.
- PKU-IDG/McGovern Institute for Brain Research, Beijing, China.
- Peking-Tsinghua Center for Life Sciences, New Cornerstone Science Laboratory, Academy for Advanced Interdisciplinary Studies, Beijing, China.
- Chinese Institute for Brain Research, Beijing, China.
- Institute of Molecular Physiology, Shenzhen Bay Laboratory, Shenzhen, China.
- National Biomedical Imaging Center, Peking University, Beijing, China.
| |
Collapse
|
16
|
Glykos V, Fujisawa S. Memory-specific encoding activities of the ventral tegmental area dopamine and GABA neurons. eLife 2024; 12:RP89743. [PMID: 38512339 PMCID: PMC10957172 DOI: 10.7554/elife.89743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/22/2024] Open
Abstract
Although the midbrain dopamine (DA) system plays a crucial role in higher cognitive functions, including updating and maintaining short-term memory, the encoding properties of the somatic spiking activity of ventral tegmental area (VTA) DA neurons for short-term memory computations have not yet been identified. Here, we probed and analyzed the activity of optogenetically identified DA and GABA neurons while mice engaged in short-term memory-dependent behavior in a T-maze task. Single-neuron analysis revealed that significant subpopulations of DA and GABA neurons responded differently between left and right trials in the memory delay. With a series of control behavioral tasks and regression analysis tools, we show that firing rate differences are linked to short-term memory-dependent decisions and cannot be explained by reward-related processes, motivated behavior, or motor-related activities. This evidence provides novel insights into the mnemonic encoding activities of midbrain DA and GABA neurons.
Collapse
Affiliation(s)
- Vasileios Glykos
- Laboratory for Systems Neurophysiology, RIKEN Center for Brain Science, Wako, Japan
- Synapse Biology Unit, Okinawa Institute of Science and Technology, Okinawa, Japan
| | - Shigeyoshi Fujisawa
- Laboratory for Systems Neurophysiology, RIKEN Center for Brain Science, Wako, Japan
| |
Collapse
|
17
|
Amo R, Uchida N, Watabe-Uchida M. Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons. Neuron 2024; 112:1001-1019.e6. [PMID: 38278147 PMCID: PMC10957320 DOI: 10.1016/j.neuron.2023.12.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2023] [Revised: 11/10/2023] [Accepted: 12/21/2023] [Indexed: 01/28/2024]
Abstract
Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.
Collapse
Affiliation(s)
- Ryunosuke Amo
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| | - Mitsuko Watabe-Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA.
| |
Collapse
|
18
|
Lin S, Fan CY, Wang HR, Li XF, Zeng JL, Lan PX, Li HX, Zhang B, Hu C, Xu J, Luo JH. Frontostriatal circuit dysfunction leads to cognitive inflexibility in neuroligin-3 R451C knockin mice. Mol Psychiatry 2024:10.1038/s41380-024-02505-9. [PMID: 38459194 DOI: 10.1038/s41380-024-02505-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/25/2023] [Revised: 02/24/2024] [Accepted: 02/28/2024] [Indexed: 03/10/2024]
Abstract
Cognitive and behavioral rigidity are observed in various psychiatric diseases, including in autism spectrum disorder (ASD). However, the underlying mechanism remains to be elucidated. In this study, we found that neuroligin-3 (NL3) R451C knockin mouse model of autism (KI mice) exhibited deficits in behavioral flexibility in choice selection tasks. Single-unit recording of medium spiny neuron (MSN) activity in the nucleus accumbens (NAc) revealed altered encoding of decision-related cue and impaired updating of choice anticipation in KI mice. Additionally, fiber photometry demonstrated significant disruption in dynamic mesolimbic dopamine (DA) signaling for reward prediction errors (RPEs), along with reduced activity in medial prefrontal cortex (mPFC) neurons projecting to the NAc in KI mice. Interestingly, NL3 re-expression in the mPFC, but not in the NAc, rescued the deficit of flexible behaviors and simultaneously restored NAc-MSN encoding, DA dynamics, and mPFC-NAc output in KI mice. Taken together, this study reveals the frontostriatal circuit dysfunction underlying cognitive inflexibility and establishes a critical role of the mPFC NL3 deficiency in this deficit in KI mice. Therefore, these findings provide new insights into the mechanisms of cognitive and behavioral inflexibility and potential intervention strategies.
Collapse
Affiliation(s)
- Shen Lin
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China.
- Fujian Provincial Institutes of Brain Disorders and Brain Sciences, First Affiliated Hospital, Fujian Medical University, Fuzhou, China.
| | - Cui-Ying Fan
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
| | - Hao-Ran Wang
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
- Nanhu Brain-Computer Interface Institute, Hangzhou, China
- Liangzhu Laboratory, MOE Frontier Science Center for Brain Science and Brain-Machine Integration, State Key Laboratory of Brain-machine Intelligence, Zhejiang University, Hangzhou, China
| | - Xiao-Fan Li
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
| | - Jia-Li Zeng
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
| | - Pei-Xuan Lan
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
| | - Hui-Xian Li
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China
| | - Bin Zhang
- Key Laboratory of Novel Targets and Drug Study for Neural Repair of Zhejiang Province, School of Medicine, Hangzhou City University, Hangzhou, China
| | - Chun Hu
- Institute for Brain Research and Rehabilitation, Key Laboratory of Brain Cognition and Education Sciences of Ministry of Education, South China Normal University, Guangzhou, China
| | - Junyu Xu
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China.
- Liangzhu Laboratory, MOE Frontier Science Center for Brain Science and Brain-Machine Integration, State Key Laboratory of Brain-machine Intelligence, Zhejiang University, Hangzhou, China.
- NHC and CAMS Key Laboratory of Medical Neurobiology, Zhejiang University, Hangzhou, China.
| | - Jian-Hong Luo
- Affiliated Mental Health Center & Hangzhou Seventh People's Hospital, School of Brain Science and Brain Medicine, Zhejiang University School of Medicine, Hangzhou, China.
- Nanhu Brain-Computer Interface Institute, Hangzhou, China.
- Liangzhu Laboratory, MOE Frontier Science Center for Brain Science and Brain-Machine Integration, State Key Laboratory of Brain-machine Intelligence, Zhejiang University, Hangzhou, China.
- NHC and CAMS Key Laboratory of Medical Neurobiology, Zhejiang University, Hangzhou, China.
| |
Collapse
|
19
|
Issa JB, Radvansky BA, Xuan F, Dombeck DA. Lateral entorhinal cortex subpopulations represent experiential epochs surrounding reward. Nat Neurosci 2024; 27:536-546. [PMID: 38272968 PMCID: PMC11097142 DOI: 10.1038/s41593-023-01557-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Accepted: 12/13/2023] [Indexed: 01/27/2024]
Abstract
During goal-directed navigation, 'what' information, describing the experiences occurring in periods surrounding a reward, can be combined with spatial 'where' information to guide behavior and form episodic memories. This integrative process likely occurs in the hippocampus, which receives spatial information from the medial entorhinal cortex; however, the source of the 'what' information is largely unknown. Here, we show that mouse lateral entorhinal cortex (LEC) represents key experiential epochs during reward-based navigation tasks. We discover separate populations of neurons that signal goal approach and goal departure and a third population signaling reward consumption. When reward location is moved, these populations immediately shift their respective representations of each experiential epoch relative to reward, while optogenetic inhibition of LEC disrupts learning the new reward location. Therefore, the LEC contains a stable code of experiential epochs surrounding and including reward consumption, providing reward-centric information to contextualize the spatial information carried by the medial entorhinal cortex.
Collapse
Affiliation(s)
- John B Issa
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
| | - Brad A Radvansky
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
| | - Feng Xuan
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
| | - Daniel A Dombeck
- Department of Neurobiology, Northwestern University, Evanston, IL, USA.
| |
Collapse
|
20
|
Eshel N, Touponse GC, Wang AR, Osterman AK, Shank AN, Groome AM, Taniguchi L, Cardozo Pinto DF, Tucciarone J, Bentzley BS, Malenka RC. Striatal dopamine integrates cost, benefit, and motivation. Neuron 2024; 112:500-514.e5. [PMID: 38016471 PMCID: PMC10922131 DOI: 10.1016/j.neuron.2023.10.038] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Revised: 07/06/2023] [Accepted: 10/26/2023] [Indexed: 11/30/2023]
Abstract
Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.
Collapse
Affiliation(s)
- Neir Eshel
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
| | - Gavin C Touponse
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Allan R Wang
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Amber K Osterman
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Amei N Shank
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Alexandra M Groome
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Lara Taniguchi
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Daniel F Cardozo Pinto
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Jason Tucciarone
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Brandon S Bentzley
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Robert C Malenka
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
| |
Collapse
|
21
|
Qian L, Burrell M, Hennig JA, Matias S, Murthy VN, Gershman SJ, Uchida N. The role of prospective contingency in the control of behavior and dopamine signals during associative learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.05.578961. [PMID: 38370735 PMCID: PMC10871210 DOI: 10.1101/2024.02.05.578961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.
Collapse
Affiliation(s)
- Lechen Qian
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- These authors contributed equally
| | - Mark Burrell
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- These authors contributed equally
| | - Jay A. Hennig
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Sara Matias
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| | - Venkatesh. N. Murthy
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| | - Samuel J. Gershman
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| |
Collapse
|
22
|
Jordan R. The locus coeruleus as a global model failure system. Trends Neurosci 2024; 47:92-105. [PMID: 38102059 DOI: 10.1016/j.tins.2023.11.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 09/27/2023] [Accepted: 11/20/2023] [Indexed: 12/17/2023]
Abstract
Predictive processing models posit that brains constantly attempt to predict their sensory inputs. Prediction errors signal when these predictions are incorrect and are thought to be instructive signals that drive corrective plasticity. Recent findings support the idea that the locus coeruleus (LC) - a brain-wide neuromodulatory system - signals several types of prediction error. I discuss how these findings support models proposing that the LC signals global model failures: instances where predictions about the world are strongly violated. Focusing on the cortex, I explore the utility of this signal in learning rate control, how the LC circuit may compute the signal, and how this view may aid our understanding of neurodivergence.
Collapse
Affiliation(s)
- Rebecca Jordan
- Simons Initiative for the Developing Brain, University of Edinburgh, 1 George Square, EH8 9JZ, Edinburgh, UK.
| |
Collapse
|
23
|
de Jong JW, Liang Y, Verharen JPH, Fraser KM, Lammel S. State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways. Nat Neurosci 2024; 27:309-318. [PMID: 38212586 DOI: 10.1038/s41593-023-01547-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2022] [Accepted: 12/07/2023] [Indexed: 01/13/2024]
Abstract
The nervous system uses fast- and slow-adapting sensory detectors in parallel to enable neuronal representations of external states and their temporal dynamics. It is unknown whether this dichotomy also applies to internal representations that have no direct correlation in the physical world. Here we find that two distinct dopamine (DA) neuron subtypes encode either a state or its rate-of-change. In mice performing a reward-seeking task, we found that the animal's behavioral state and rate-of-change were encoded by the sustained activity of DA neurons in medial ventral tegmental area (VTA) DA neurons and transient activity in lateral VTA DA neurons, respectively. The neural activity patterns of VTA DA cell bodies matched DA release patterns within anatomically defined mesoaccumbal pathways. Based on these results, we propose a model in which the DA system uses two parallel lines for proportional-differential encoding of a state variable and its temporal dynamics.
Collapse
Affiliation(s)
- Johannes W de Jong
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Yilan Liang
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Jeroen P H Verharen
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Kurt M Fraser
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
| | - Stephan Lammel
- Department of Molecular and Cell Biology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA.
| |
Collapse
|
24
|
Blanco-Pozo M, Akam T, Walton ME. Dopamine-independent effect of rewards on choices through hidden-state inference. Nat Neurosci 2024; 27:286-297. [PMID: 38216649 PMCID: PMC10849965 DOI: 10.1038/s41593-023-01542-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 12/01/2023] [Indexed: 01/14/2024]
Abstract
Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world's state, not the dopaminergic RPEs they produce.
Collapse
Affiliation(s)
- Marta Blanco-Pozo
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| | - Thomas Akam
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| | - Mark E Walton
- Department of Experimental Psychology, Oxford University, Oxford, UK.
- Wellcome Centre for Integrative Neuroimaging, Oxford University, Oxford, UK.
| |
Collapse
|
25
|
Tolooshams B, Matias S, Wu H, Temereanca S, Uchida N, Murthy VN, Masset P, Ba D. Interpretable deep learning for deconvolutional analysis of neural signals. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.05.574379. [PMID: 38260512 PMCID: PMC10802267 DOI: 10.1101/2024.01.05.574379] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/24/2024]
Abstract
The widespread adoption of deep learning to build models that capture the dynamics of neural populations is typically based on "black-box" approaches that lack an interpretable link between neural activity and function. Here, we propose to apply algorithm unrolling, a method for interpretable deep learning, to design the architecture of sparse deconvolutional neural networks and obtain a direct interpretation of network weights in relation to stimulus-driven single-neuron activity through a generative model. We characterize our method, referred to as deconvolutional unrolled neural learning (DUNL), and show its versatility by applying it to deconvolve single-trial local signals across multiple brain areas and recording modalities. To exemplify use cases of our decomposition method, we uncover multiplexed salience and reward prediction error signals from midbrain dopamine neurons in an unbiased manner, perform simultaneous event detection and characterization in somatosensory thalamus recordings, and characterize the responses of neurons in the piriform cortex. Our work leverages the advances in interpretable deep learning to gain a mechanistic understanding of neural dynamics.
Collapse
Affiliation(s)
- Bahareh Tolooshams
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge MA, 02138
- Computing + Mathematical Sciences, California Institute of Technology, Pasadena, CA, 91125
| | - Sara Matias
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- Department of Molecular and Cellular Biology, Harvard University, Cambridge MA, 02138
| | - Hao Wu
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- Department of Molecular and Cellular Biology, Harvard University, Cambridge MA, 02138
| | - Simona Temereanca
- Carney Institute for Brain Science, Brown University, Providence, RI, 02906
| | - Naoshige Uchida
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- Department of Molecular and Cellular Biology, Harvard University, Cambridge MA, 02138
| | - Venkatesh N. Murthy
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- Department of Molecular and Cellular Biology, Harvard University, Cambridge MA, 02138
| | - Paul Masset
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- Department of Molecular and Cellular Biology, Harvard University, Cambridge MA, 02138
- Department of Psychology, McGill University, Montréal QC, H3A 1G1
| | - Demba Ba
- Center for Brain Science, Harvard University, Cambridge MA, 02138
- John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge MA, 02138
- Kempner Institute for the Study of Natural & Artificial Intelligence, Harvard University, Cambridge MA, 02138
| |
Collapse
|
26
|
Beas S, Khan I, Gao C, Loewinger G, Macdonald E, Bashford A, Rodriguez-Gonzalez S, Pereira F, Penzo MA. Dissociable encoding of motivated behavior by parallel thalamo-striatal projections. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2023.07.07.548113. [PMID: 37781624 PMCID: PMC10541145 DOI: 10.1101/2023.07.07.548113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
The successful pursuit of goals requires the coordinated execution and termination of actions that lead to positive outcomes. This process is thought to rely on motivational states that are guided by internal drivers, such as hunger or fear. However, the mechanisms by which the brain tracks motivational states to shape instrumental actions are not fully understood. The paraventricular nucleus of the thalamus (PVT) is a midline thalamic nucleus that shapes motivated behaviors via its projections to the nucleus accumbens (NAc)1-8 and monitors internal state via interoceptive inputs from the hypothalamus and brainstem3,9-14. Recent studies indicate that the PVT can be subdivided into two major neuronal subpopulations, namely PVTD2(+) and PVTD2(-), which differ in genetic identity, functionality, and anatomical connectivity to other brain regions, including the NAc4,15,16. In this study, we used fiber photometry to investigate the in vivo dynamics of these two distinct PVT neuronal types in mice performing a reward foraging-like behavioral task. We discovered that PVTD2(+) and PVTD2(-) neurons encode the execution and termination of goal-oriented actions, respectively. Furthermore, activity in the PVTD2(+) neuronal population mirrored motivation parameters such as vigor and satiety. Similarly, PVTD2(-) neurons, also mirrored some of these parameters but to a much lesser extent. Importantly, these features were largely preserved when activity in PVT projections to the NAc was selectively assessed. Collectively, our results highlight the existence of two parallel thalamo-striatal projections that participate in the dynamic regulation of goal pursuits and provide insight into the mechanisms by which the brain tracks motivational states to shape instrumental actions.
Collapse
Affiliation(s)
- Sofia Beas
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
- Department of Neurobiology, University of Alabama at Birmingham, Birmingham, AL, USA
| | - Isbah Khan
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
| | - Claire Gao
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
| | - Gabriel Loewinger
- Machine Learning Team, National Institute of Mental Health, Bethesda, MD, USA
| | - Emma Macdonald
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
| | - Alison Bashford
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
| | | | - Francisco Pereira
- Machine Learning Team, National Institute of Mental Health, Bethesda, MD, USA
| | - Mario A. Penzo
- Unit on the Neurobiology of Affective Memory, National Institute of Mental Health, Bethesda, MD, USA
| |
Collapse
|
27
|
Golden CEM, Kaur D, Mah A, Martin AC, Levy DH, Yamaguchi T, Lin D, Aoki C, Constantinople CM. Estrogenic control of reward prediction errors and reinforcement learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.12.09.570945. [PMID: 38105956 PMCID: PMC10723450 DOI: 10.1101/2023.12.09.570945] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/19/2023]
Abstract
Gonadal hormones act throughout the brain 1 , and nearly all neuropsychiatric disorders vary in symptom severity with hormonal fluctuations over the reproductive cycle, gestation, and perimenopause 2-4 . Yet the mechanisms by which hormones influence mental and cognitive processes are unclear. Exogenous estrogenic hormones modulate dopamine signaling in the nucleus accumbens core (NAcc) 5,6 , which instantiates reward prediction errors (RPEs) for reinforcement learning 7-16 . Here we show that endogenous estrogenic hormones enhance RPEs and sensitivity to previous rewards by regulating expression of dopamine reuptake proteins in the NAcc. We trained rats to perform a temporal wagering task with different reward states; rats adjusted how quickly they initiated trials across states, balancing effort against expected rewards. Dopamine release in the NAcc reflected RPEs that predicted and causally in-fluenced subsequent initiation times. When fertile, females more quickly adjusted their initiation times to match reward states due to enhanced dopaminergic RPEs in the NAcc. Proteomics revealed reduced expression of dopamine transporters in fertile stages of the reproductive cycle. Finally, genetic suppression of midbrain estrogen receptors eliminated hormonal modulation of behavior. Estrogenic hormones therefore control the rate of reinforcement learning by regulating RPEs via dopamine reuptake, providing a mechanism by which hormones influence neural dynamics for motivation and learning.
Collapse
|
28
|
Hassall CD, Yan Y, Hunt LT. The neural correlates of continuous feedback processing. Psychophysiology 2023; 60:e14399. [PMID: 37485986 PMCID: PMC10851313 DOI: 10.1111/psyp.14399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Revised: 07/12/2023] [Accepted: 07/12/2023] [Indexed: 07/25/2023]
Abstract
Feedback processing is commonly studied by analyzing the brain's response to discrete rather than continuous events. Such studies have led to the hypothesis that rapid phasic midbrain dopaminergic activity tracks reward prediction errors (RPEs), the effects of which are measurable at the scalp via electroencephalography (EEG). Although studies using continuous feedback are sparse, recent animal work suggests that moment-to-moment changes in reward are tracked by slowly ramping midbrain dopaminergic activity. Some have argued that these ramping signals index state values rather than RPEs. Our goal here was to develop an EEG measure of continuous feedback processing in humans, then test whether its behavior could be accounted for by the RPE hypothesis. Participants completed a stimulus-response learning task in which a continuous reward cue gradually increased or decreased over time. A regression-based unmixing approach revealed EEG activity with a topography and time course consistent with the stimulus-preceding negativity (SPN), a scalp potential previously linked to reward anticipation and tonic dopamine release. Importantly, this reward-related activity depended on outcome expectancy: as predicted by the RPE hypothesis, activity for expected reward cues was reduced compared to unexpected reward cues. These results demonstrate the possibility of using human scalp-recorded potentials to track continuous feedback processing, and test candidate hypotheses of this activity.
Collapse
Affiliation(s)
- Cameron D. Hassall
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of PsychologyMacEwan UniversityEdmontonAlbertaCanada
| | - Yan Yan
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of PsychologyStanford UniversityStanfordCaliforniaUSA
| | - Laurence T. Hunt
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of Experimental PsychologyUniversity of OxfordOxfordUK
| |
Collapse
|
29
|
Antony JW, Van Dam J, Massey JR, Barnett AJ, Bennion KA. Long-term, multi-event surprise correlates with enhanced autobiographical memory. Nat Hum Behav 2023; 7:2152-2168. [PMID: 37322234 DOI: 10.1038/s41562-023-01631-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 05/16/2023] [Indexed: 06/17/2023]
Abstract
Neurobiological and psychological models of learning emphasize the importance of prediction errors (surprises) for memory formation. This relationship has been shown for individual momentary surprising events; however, it is less clear whether surprise that unfolds across multiple events and timescales is also linked with better memory of those events. We asked basketball fans about their most positive and negative autobiographical memories of individual plays, games and seasons, allowing surprise measurements spanning seconds, hours and months. We used advanced analytics on National Basketball Association play-by-play data and betting odds spanning 17 seasons, more than 22,000 games and more than 5.6 million plays to compute and align the estimated surprise value of each memory. We found that surprising events were associated with better recall of positive memories on the scale of seconds and months and negative memories across all three timescales. Game and season memories could not be explained by surprise at shorter timescales, suggesting that long-term, multi-event surprise correlates with memory. These results expand notions of surprise in models of learning and reinforce its relevance in real-world domains.
Collapse
Affiliation(s)
- James W Antony
- Department of Psychology and Child Development, California Polytechnic State University, San Luis Obispo, CA, USA.
| | - Jacob Van Dam
- Department of Psychology and Child Development, California Polytechnic State University, San Luis Obispo, CA, USA
| | - Jarett R Massey
- Department of Psychology and Child Development, California Polytechnic State University, San Luis Obispo, CA, USA
| | | | - Kelly A Bennion
- Department of Psychology and Child Development, California Polytechnic State University, San Luis Obispo, CA, USA
| |
Collapse
|
30
|
Pinto SR, Uchida N. Tonic dopamine and biases in value learning linked through a biologically inspired reinforcement learning model. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.10.566580. [PMID: 38014087 PMCID: PMC10680794 DOI: 10.1101/2023.11.10.566580] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
A hallmark of various psychiatric disorders is biased future predictions. Here we examined the mechanisms for biased value learning using reinforcement learning models incorporating recent findings on synaptic plasticity and opponent circuit mechanisms in the basal ganglia. We show that variations in tonic dopamine can alter the balance between learning from positive and negative reward prediction errors, leading to biased value predictions. This bias arises from the sigmoidal shapes of the dose-occupancy curves and distinct affinities of D1- and D2-type dopamine receptors: changes in tonic dopamine differentially alters the slope of the dose-occupancy curves of these receptors, thus sensitivities, at baseline dopamine concentrations. We show that this mechanism can explain biased value learning in both mice and humans and may also contribute to symptoms observed in psychiatric disorders. Our model provides a foundation for understanding the basal ganglia circuit and underscores the significance of tonic dopamine in modulating learning processes.
Collapse
Affiliation(s)
- Sandra Romero Pinto
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
- Program in Speech and Hearing Bioscience and Technology, Division of Medical Sciences, Harvard Medical School, Boston, MA 02115, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| |
Collapse
|
31
|
Ott T, Stein AM, Nieder A. Dopamine receptor activation regulates reward expectancy signals during cognitive control in primate prefrontal neurons. Nat Commun 2023; 14:7537. [PMID: 37985776 PMCID: PMC10661983 DOI: 10.1038/s41467-023-43271-6] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 11/06/2023] [Indexed: 11/22/2023] Open
Abstract
Dopamine neurons respond to reward-predicting cues but also modulate information processing in the prefrontal cortex essential for cognitive control. Whether dopamine controls reward expectation signals in prefrontal cortex that motivate cognitive control is unknown. We trained two male macaques on a working memory task while varying the reward size earned for successful task completion. We recorded neurons in lateral prefrontal cortex while simultaneously stimulating dopamine D1 receptor (D1R) or D2 receptor (D2R) families using micro-iontophoresis. We show that many neurons predict reward size throughout the trial. D1R stimulation showed mixed effects following reward cues but decreased reward expectancy coding during the memory delay. By contrast, D2R stimulation increased reward expectancy coding in multiple task periods, including cueing and memory periods. Stimulation of either dopamine receptors increased the neurons' selective responses to reward size upon reward delivery. The differential modulation of reward expectancy by dopamine receptors suggests that dopamine regulates reward expectancy necessary for successful cognitive control.
Collapse
Affiliation(s)
- Torben Ott
- Animal Physiology, Institute of Neurobiology, Auf der Morgenstelle 28, University of Tübingen, 72076, Tübingen, Germany.
- Bernstein Center for Computational Neuroscience and Institute of Biology, Humboldt-University of Berlin, 10099, Berlin, Germany.
| | - Anna Marlina Stein
- Animal Physiology, Institute of Neurobiology, Auf der Morgenstelle 28, University of Tübingen, 72076, Tübingen, Germany
| | - Andreas Nieder
- Animal Physiology, Institute of Neurobiology, Auf der Morgenstelle 28, University of Tübingen, 72076, Tübingen, Germany.
| |
Collapse
|
32
|
Masset P, Tano P, Kim HR, Malik AN, Pouget A, Uchida N. Multi-timescale reinforcement learning in the brain. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.12.566754. [PMID: 38014166 PMCID: PMC10680596 DOI: 10.1101/2023.11.12.566754] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
To thrive in complex environments, animals and artificial agents must learn to act adaptively to maximize fitness and rewards. Such adaptive behavior can be learned through reinforcement learning1, a class of algorithms that has been successful at training artificial agents2-6 and at characterizing the firing of dopamine neurons in the midbrain7-9. In classical reinforcement learning, agents discount future rewards exponentially according to a single time scale, controlled by the discount factor. Here, we explore the presence of multiple timescales in biological reinforcement learning. We first show that reinforcement agents learning at a multitude of timescales possess distinct computational benefits. Next, we report that dopamine neurons in mice performing two behavioral tasks encode reward prediction error with a diversity of discount time constants. Our model explains the heterogeneity of temporal discounting in both cue-evoked transient responses and slower timescale fluctuations known as dopamine ramps. Crucially, the measured discount factor of individual neurons is correlated across the two tasks suggesting that it is a cell-specific property. Together, our results provide a new paradigm to understand functional heterogeneity in dopamine neurons, a mechanistic basis for the empirical observation that humans and animals use non-exponential discounts in many situations10-14, and open new avenues for the design of more efficient reinforcement learning algorithms.
Collapse
Affiliation(s)
- Paul Masset
- Department of Molecular and Cellular Biology, Harvard University, USA
- Center for Brain Science, Harvard University, USA
| | - Pablo Tano
- Department of Basic Neuroscience, University of Geneva, Switzerland
| | - HyungGoo R. Kim
- Department of Molecular and Cellular Biology, Harvard University, USA
- Center for Brain Science, Harvard University, USA
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
- Center for Neuroscience Imaging Research, Institute for Basic Science (IBS), Suwon 16419, Republic of Korea
| | - Athar N. Malik
- Department of Molecular and Cellular Biology, Harvard University, USA
- Center for Brain Science, Harvard University, USA
- Department of Neurosurgery, Warren Alpert Medical School of Brown University, USA
- Norman Prince Neurosciences Institute, Rhode Island Hospital, USA
| | - Alexandre Pouget
- Department of Basic Neuroscience, University of Geneva, Switzerland
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, USA
- Center for Brain Science, Harvard University, USA
| |
Collapse
|
33
|
Qü AJ, Tai LH, Hall CD, Tu EM, Eckstein MK, Mishchanchuk K, Lin WC, Chase JB, MacAskill AF, Collins AG, Gershman SJ, Wilbrecht L. Nucleus accumbens dopamine release reflects Bayesian inference during instrumental learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.10.566306. [PMID: 38014354 PMCID: PMC10680647 DOI: 10.1101/2023.11.10.566306] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
Dopamine release in the nucleus accumbens has been hypothesized to signal reward prediction error, the difference between observed and predicted reward, suggesting a biological implementation for reinforcement learning. Rigorous tests of this hypothesis require assumptions about how the brain maps sensory signals to reward predictions, yet this mapping is still poorly understood. In particular, the mapping is non-trivial when sensory signals provide ambiguous information about the hidden state of the environment. Previous work using classical conditioning tasks has suggested that reward predictions are generated conditional on probabilistic beliefs about the hidden state, such that dopamine implicitly reflects these beliefs. Here we test this hypothesis in the context of an instrumental task (a two-armed bandit), where the hidden state switches repeatedly. We measured choice behavior and recorded dLight signals reflecting dopamine release in the nucleus accumbens core. Model comparison based on the behavioral data favored models that used Bayesian updating of probabilistic beliefs. These same models also quantitatively matched the dopamine measurements better than non-Bayesian alternatives. We conclude that probabilistic belief computation plays a fundamental role in instrumental performance and associated mesolimbic dopamine signaling.
Collapse
Affiliation(s)
- Albert J. Qü
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Center for Computational Biology, University of California, Berkeley, CA, 94720, USA
| | - Lung-Hao Tai
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Christopher D. Hall
- Sainsbury Wellcome Centre for Neural Circuits and Behaviour, University College London, London, W1T 4JG, UK
| | - Emilie M. Tu
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
| | | | - Karyna Mishchanchuk
- Department of Neuroscience, Physiology and Pharmacology, University College London, UK
| | - Wan Chen Lin
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Juliana B. Chase
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
| | - Andrew F. MacAskill
- Department of Neuroscience, Physiology and Pharmacology, University College London, UK
| | - Anne G.E. Collins
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| | - Samuel J. Gershman
- Department of Psychology and Center for Brain Science, Harvard University, Cambridge, MA, 02138, USA
- Center for Brains, Minds, and Machines, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA
| | - Linda Wilbrecht
- Department of Psychology, University of California, Berkeley, CA, 94720, USA
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA, 94720, USA
| |
Collapse
|
34
|
Amo R, Uchida N, Watabe-Uchida M. Glutamate inputs send prediction error of reward but not negative value of aversive stimuli to dopamine neurons. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.09.566472. [PMID: 37986868 PMCID: PMC10659341 DOI: 10.1101/2023.11.09.566472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/22/2023]
Abstract
Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs) but the mechanisms underlying RPE computation, particularly contributions of different neurotransmitters, remain poorly understood. Here we used a genetically-encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons. We found that glutamate inputs exhibit virtually all of the characteristics of RPE, rather than conveying a specific component of RPE computation such as reward or expectation. Notably, while glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli toward more positive responses, while excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.
Collapse
Affiliation(s)
- Ryunosuke Amo
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| | - Mitsuko Watabe-Uchida
- Department of Molecular and Cellular Biology, Center for Brain Science, Harvard University, Cambridge, MA 02138, USA
| |
Collapse
|
35
|
Stetsenko A, Koos T. Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system. Proc Natl Acad Sci U S A 2023; 120:e2309015120. [PMID: 37903252 PMCID: PMC10636325 DOI: 10.1073/pnas.2309015120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 09/29/2023] [Indexed: 11/01/2023] Open
Abstract
The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.
Collapse
Affiliation(s)
- Anya Stetsenko
- Center for Molecular and Behavioral Neuroscience, Rutgers University, Newark, NJ07102
| | - Tibor Koos
- Center for Molecular and Behavioral Neuroscience, Rutgers University, Newark, NJ07102
| |
Collapse
|
36
|
Krausz TA, Comrie AE, Kahn AE, Frank LM, Daw ND, Berke JD. Dual credit assignment processes underlie dopamine signals in a complex spatial environment. Neuron 2023; 111:3465-3478.e7. [PMID: 37611585 PMCID: PMC10841332 DOI: 10.1016/j.neuron.2023.07.017] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Revised: 06/23/2023] [Accepted: 07/25/2023] [Indexed: 08/25/2023]
Abstract
Animals frequently make decisions based on expectations of future reward ("values"). Values are updated by ongoing experience: places and choices that result in reward are assigned greater value. Yet, the specific algorithms used by the brain for such credit assignment remain unclear. We monitored accumbens dopamine as rats foraged for rewards in a complex, changing environment. We observed brief dopamine pulses both at reward receipt (scaling with prediction error) and at novel path opportunities. Dopamine also ramped up as rats ran toward reward ports, in proportion to the value at each location. By examining the evolution of these dopamine place-value signals, we found evidence for two distinct update processes: progressive propagation of value along taken paths, as in temporal difference learning, and inference of value throughout the maze, using internal models. Our results demonstrate that within rich, naturalistic environments dopamine conveys place values that are updated via multiple, complementary learning algorithms.
Collapse
Affiliation(s)
- Timothy A Krausz
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA
| | - Alison E Comrie
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA
| | - Ari E Kahn
- Department of Psychology, and Princeton Neuroscience Institute, Princeton University, Princeton, Princeton, NJ 08544, USA
| | - Loren M Frank
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA; Howard Hughes Medical Institute, Chevy Chase, MD 20815, USA; Department of Physiology, University of California, San Francisco, San Francisco, CA 94158, USA
| | - Nathaniel D Daw
- Department of Psychology, and Princeton Neuroscience Institute, Princeton University, Princeton, Princeton, NJ 08544, USA
| | - Joshua D Berke
- Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA; Kavli Institute for Fundamental Neuroscience, and Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA 94158, USA; Department of Neurology and Department of Psychiatry and Behavioral Science, University of California, San Francisco, San Francisco, CA 94158, USA.
| |
Collapse
|
37
|
Matityahu L, Gilin N, Sarpong GA, Atamna Y, Tiroshi L, Tritsch NX, Wickens JR, Goldberg JA. Acetylcholine waves and dopamine release in the striatum. Nat Commun 2023; 14:6852. [PMID: 37891198 PMCID: PMC10611775 DOI: 10.1038/s41467-023-42311-5] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Accepted: 10/06/2023] [Indexed: 10/29/2023] Open
Abstract
Striatal dopamine encodes reward, with recent work showing that dopamine release occurs in spatiotemporal waves. However, the mechanism of dopamine waves is unknown. Here we report that acetylcholine release in mouse striatum also exhibits wave activity, and that the spatial scale of striatal dopamine release is extended by nicotinic acetylcholine receptors. Based on these findings, and on our demonstration that single cholinergic interneurons can induce dopamine release, we hypothesized that the local reciprocal interaction between cholinergic interneurons and dopamine axons suffices to drive endogenous traveling waves. We show that the morphological and physiological properties of cholinergic interneuron - dopamine axon interactions can be modeled as a reaction-diffusion system that gives rise to traveling waves. Analytically-tractable versions of the model show that the structure and the nature of propagation of acetylcholine and dopamine traveling waves depend on their coupling, and that traveling waves can give rise to empirically observed correlations between these signals. Thus, our study provides evidence for striatal acetylcholine waves in vivo, and proposes a testable theoretical framework that predicts that the observed dopamine and acetylcholine waves are strongly coupled phenomena.
Collapse
Affiliation(s)
- Lior Matityahu
- Department of Medical Neurobiology, Institute of Medical Research Israel - Canada, The Faculty of Medicine, The Hebrew University of Jerusalem, 9112102, Jerusalem, Israel
| | - Naomi Gilin
- Department of Medical Neurobiology, Institute of Medical Research Israel - Canada, The Faculty of Medicine, The Hebrew University of Jerusalem, 9112102, Jerusalem, Israel
| | - Gideon A Sarpong
- Okinawa Institute of Science and Technology Graduate University, Okinawa, Japan
| | - Yara Atamna
- Department of Medical Neurobiology, Institute of Medical Research Israel - Canada, The Faculty of Medicine, The Hebrew University of Jerusalem, 9112102, Jerusalem, Israel
| | - Lior Tiroshi
- Department of Medical Neurobiology, Institute of Medical Research Israel - Canada, The Faculty of Medicine, The Hebrew University of Jerusalem, 9112102, Jerusalem, Israel
| | - Nicolas X Tritsch
- Neuroscience Institute, New York University Grossman School of Medicine, New York, NY, 10016, USA
| | - Jeffery R Wickens
- Okinawa Institute of Science and Technology Graduate University, Okinawa, Japan
| | - Joshua A Goldberg
- Department of Medical Neurobiology, Institute of Medical Research Israel - Canada, The Faculty of Medicine, The Hebrew University of Jerusalem, 9112102, Jerusalem, Israel.
| |
Collapse
|
38
|
Konishi M, Igarashi KM, Miura K. Biologically plausible local synaptic learning rules robustly implement deep supervised learning. Front Neurosci 2023; 17:1160899. [PMID: 37886676 PMCID: PMC10598703 DOI: 10.3389/fnins.2023.1160899] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2023] [Accepted: 08/31/2023] [Indexed: 10/28/2023] Open
Abstract
In deep neural networks, representational learning in the middle layer is essential for achieving efficient learning. However, the currently prevailing backpropagation learning rules (BP) are not necessarily biologically plausible and cannot be implemented in the brain in their current form. Therefore, to elucidate the learning rules used by the brain, it is critical to establish biologically plausible learning rules for practical memory tasks. For example, learning rules that result in a learning performance worse than that of animals observed in experimental studies may not be computations used in real brains and should be ruled out. Using numerical simulations, we developed biologically plausible learning rules to solve a task that replicates a laboratory experiment where mice learned to predict the correct reward amount. Although the extreme learning machine (ELM) and weight perturbation (WP) learning rules performed worse than the mice, the feedback alignment (FA) rule achieved a performance equal to that of BP. To obtain a more biologically plausible model, we developed a variant of FA, FA_Ex-100%, which implements direct dopamine inputs that provide error signals locally in the layer of focus, as found in the mouse entorhinal cortex. The performance of FA_Ex-100% was comparable to that of conventional BP. Finally, we tested whether FA_Ex-100% was robust against rule perturbations and biologically inevitable noise. FA_Ex-100% worked even when subjected to perturbations, presumably because it could calibrate the correct prediction error (e.g., dopaminergic signals) in the next step as a teaching signal if the perturbation created a deviation. These results suggest that simplified and biologically plausible learning rules, such as FA_Ex-100%, can robustly facilitate deep supervised learning when the error signal, possibly conveyed by dopaminergic neurons, is accurate.
Collapse
Affiliation(s)
- Masataka Konishi
- Department of Biosciences, School of Biological and Environmental Sciences, Kwansei Gakuin University, Sanda, Hyogo, Japan
| | - Kei M. Igarashi
- Department of Anatomy and Neurobiology, School of Medicine, University of California, Irvine, Irvine, CA, United States
| | - Keiji Miura
- Department of Biosciences, School of Biological and Environmental Sciences, Kwansei Gakuin University, Sanda, Hyogo, Japan
| |
Collapse
|
39
|
Issa JB, Radvansky BA, Xuan F, Dombeck DA. Lateral entorhinal cortex subpopulations represent experiential epochs surrounding reward. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.10.09.561557. [PMID: 37873482 PMCID: PMC10592707 DOI: 10.1101/2023.10.09.561557] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/25/2023]
Abstract
During goal-directed navigation, "what" information, which describes the experiences occurring in periods surrounding a reward, can be combined with spatial "where" information to guide behavior and form episodic memories1,2. This integrative process is thought to occur in the hippocampus3, which receives spatial information from the medial entorhinal cortex (MEC)4; however, the source of the "what" information and how it is represented is largely unknown. Here, by establishing a novel imaging method, we show that the lateral entorhinal cortex (LEC) of mice represents key experiential epochs during a reward-based navigation task. We discover a population of neurons that signals goal approach and a separate population of neurons that signals goal departure. A third population of neurons signals reward consumption. When reward location is moved, these populations immediately shift their respective representations of each experiential epoch relative to reward, while optogenetic inhibition of LEC disrupts learning of the new reward location. Together, these results indicate the LEC provides a stable code of experiential epochs surrounding and including reward consumption, providing reward-centric information to contextualize the spatial information carried by the MEC. Such parallel representations are well-suited for generating episodic memories of rewarding experiences and guiding flexible and efficient goal-directed navigation5-7.
Collapse
Affiliation(s)
- John B. Issa
- Department of Neurobiology, Northwestern University, Evanston, IL 60608, USA
| | - Brad A. Radvansky
- Department of Neurobiology, Northwestern University, Evanston, IL 60608, USA
| | - Feng Xuan
- Department of Neurobiology, Northwestern University, Evanston, IL 60608, USA
| | - Daniel A. Dombeck
- Department of Neurobiology, Northwestern University, Evanston, IL 60608, USA
| |
Collapse
|
40
|
Bech P, Crochet S, Dard R, Ghaderi P, Liu Y, Malekzadeh M, Petersen CCH, Pulin M, Renard A, Sourmpis C. Striatal Dopamine Signals and Reward Learning. FUNCTION 2023; 4:zqad056. [PMID: 37841525 PMCID: PMC10572094 DOI: 10.1093/function/zqad056] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 09/28/2023] [Accepted: 09/29/2023] [Indexed: 10/17/2023] Open
Abstract
We are constantly bombarded by sensory information and constantly making decisions on how to act. In order to optimally adapt behavior, we must judge which sequences of sensory inputs and actions lead to successful outcomes in specific circumstances. Neuronal circuits of the basal ganglia have been strongly implicated in action selection, as well as the learning and execution of goal-directed behaviors, with accumulating evidence supporting the hypothesis that midbrain dopamine neurons might encode a reward signal useful for learning. Here, we review evidence suggesting that midbrain dopaminergic neurons signal reward prediction error, driving synaptic plasticity in the striatum underlying learning. We focus on phasic increases in action potential firing of midbrain dopamine neurons in response to unexpected rewards. These dopamine neurons prominently innervate the dorsal and ventral striatum. In the striatum, the released dopamine binds to dopamine receptors, where it regulates the plasticity of glutamatergic synapses. The increase of striatal dopamine accompanying an unexpected reward activates dopamine type 1 receptors (D1Rs) initiating a signaling cascade that promotes long-term potentiation of recently active glutamatergic input onto striatonigral neurons. Sensorimotor-evoked glutamatergic input, which is active immediately before reward delivery will thus be strengthened onto neurons in the striatum expressing D1Rs. In turn, these neurons cause disinhibition of brainstem motor centers and disinhibition of the motor thalamus, thus promoting motor output to reinforce rewarded stimulus-action outcomes. Although many details of the hypothesis need further investigation, altogether, it seems likely that dopamine signals in the striatum might underlie important aspects of goal-directed reward-based learning.
Collapse
Affiliation(s)
- Pol Bech
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Sylvain Crochet
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Robin Dard
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Parviz Ghaderi
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Yanqi Liu
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Meriam Malekzadeh
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Carl C H Petersen
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Mauro Pulin
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Anthony Renard
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| | - Christos Sourmpis
- Laboratory of Sensory Processing, Brain Mind Institute, Faculty of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne CH-1015, Switzerland
| |
Collapse
|
41
|
Azcorra M, Gaertner Z, Davidson C, He Q, Kim H, Nagappan S, Hayes CK, Ramakrishnan C, Fenno L, Kim YS, Deisseroth K, Longnecker R, Awatramani R, Dombeck DA. Unique functional responses differentially map onto genetic subtypes of dopamine neurons. Nat Neurosci 2023; 26:1762-1774. [PMID: 37537242 PMCID: PMC10545540 DOI: 10.1038/s41593-023-01401-9] [Citation(s) in RCA: 23] [Impact Index Per Article: 23.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2022] [Accepted: 07/05/2023] [Indexed: 08/05/2023]
Abstract
Dopamine neurons are characterized by their response to unexpected rewards, but they also fire during movement and aversive stimuli. Dopamine neuron diversity has been observed based on molecular expression profiles; however, whether different functions map onto such genetic subtypes remains unclear. In this study, we established that three genetic dopamine neuron subtypes within the substantia nigra pars compacta, characterized by the expression of Slc17a6 (Vglut2), Calb1 and Anxa1, each have a unique set of responses to rewards, aversive stimuli and accelerations and decelerations, and these signaling patterns are highly correlated between somas and axons within subtypes. Remarkably, reward responses were almost entirely absent in the Anxa1+ subtype, which instead displayed acceleration-correlated signaling. Our findings establish a connection between functional and genetic dopamine neuron subtypes and demonstrate that molecular expression patterns can serve as a common framework to dissect dopaminergic functions.
Collapse
Affiliation(s)
- Maite Azcorra
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
- Department of Neurology, Northwestern University, Chicago, IL, USA
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Zachary Gaertner
- Department of Neurology, Northwestern University, Chicago, IL, USA
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Connor Davidson
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Qianzi He
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Hailey Kim
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
| | - Shivathmihai Nagappan
- Department of Neurobiology, Northwestern University, Evanston, IL, USA
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA
| | - Cooper K Hayes
- Department of Microbiology and Immunology, Northwestern University, Chicago, IL, USA
| | - Charu Ramakrishnan
- Department of Bioengineering, Stanford University School of Medicine, Stanford, CA, USA
| | - Lief Fenno
- Department of Bioengineering, Stanford University School of Medicine, Stanford, CA, USA
- Departments of Neuroscience & Psychiatry, The University of Texas at Austin, Austin, TX, USA
| | - Yoon Seok Kim
- Department of Bioengineering, Stanford University School of Medicine, Stanford, CA, USA
| | - Karl Deisseroth
- Department of Bioengineering, Stanford University School of Medicine, Stanford, CA, USA
| | - Richard Longnecker
- Department of Microbiology and Immunology, Northwestern University, Chicago, IL, USA
| | - Rajeshwar Awatramani
- Department of Neurology, Northwestern University, Chicago, IL, USA.
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA.
| | - Daniel A Dombeck
- Department of Neurobiology, Northwestern University, Evanston, IL, USA.
- Aligning Science Across Parkinson's (ASAP) Collaborative Research Network, Chevy Chase, MD, USA.
| |
Collapse
|
42
|
Lloyd K, Dayan P. Reframing dopamine: A controlled controller at the limbic-motor interface. PLoS Comput Biol 2023; 19:e1011569. [PMID: 37847681 PMCID: PMC10610519 DOI: 10.1371/journal.pcbi.1011569] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 10/27/2023] [Accepted: 10/03/2023] [Indexed: 10/19/2023] Open
Abstract
Pavlovian influences notoriously interfere with operant behaviour. Evidence suggests this interference sometimes coincides with the release of the neuromodulator dopamine in the nucleus accumbens. Suppressing such interference is one of the targets of cognitive control. Here, using the examples of active avoidance and omission behaviour, we examine the possibility that direct manipulation of the dopamine signal is an instrument of control itself. In particular, when instrumental and Pavlovian influences come into conflict, dopamine levels might be affected by the controlled deployment of a reframing mechanism that recasts the prospect of possible punishment as an opportunity to approach safety, and the prospect of future reward in terms of a possible loss of that reward. We operationalize this reframing mechanism and fit the resulting model to rodent behaviour from two paradigmatic experiments in which accumbens dopamine release was also measured. We show that in addition to matching animals' behaviour, the model predicts dopamine transients that capture some key features of observed dopamine release at the time of discriminative cues, supporting the idea that modulation of this neuromodulator is amongst the repertoire of cognitive control strategies.
Collapse
Affiliation(s)
- Kevin Lloyd
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
- University of Tübingen, Tübingen, Germany
| |
Collapse
|
43
|
Bukwich M, Campbell MG, Zoltowski D, Kingsbury L, Tomov MS, Stern J, Kim HR, Drugowitsch J, Linderman SW, Uchida N. Competitive integration of time and reward explains value-sensitive foraging decisions and frontal cortex ramping dynamics. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.05.556267. [PMID: 37732217 PMCID: PMC10508756 DOI: 10.1101/2023.09.05.556267] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/22/2023]
Abstract
The ability to make advantageous decisions is critical for animals to ensure their survival. Patch foraging is a natural decision-making process in which animals decide when to leave a patch of depleting resources to search for a new one. To study the algorithmic and neural basis of patch foraging behavior in a controlled laboratory setting, we developed a virtual foraging task for head-fixed mice. Mouse behavior could be explained by ramp-to-threshold models integrating time and rewards antagonistically. Accurate behavioral modeling required inclusion of a slowly varying "patience" variable, which modulated sensitivity to time. To investigate the neural basis of this decision-making process, we performed dense electrophysiological recordings with Neuropixels probes broadly throughout frontal cortex and underlying subcortical areas. We found that decision variables from the reward integrator model were represented in neural activity, most robustly in frontal cortical areas. Regression modeling followed by unsupervised clustering identified a subset of neurons with ramping activity. These neurons' firing rates ramped up gradually in single trials over long time scales (up to tens of seconds), were inhibited by rewards, and were better described as being generated by a continuous ramp rather than a discrete stepping process. Together, these results identify reward integration via a continuous ramping process in frontal cortex as a likely candidate for the mechanism by which the mammalian brain solves patch foraging problems.
Collapse
Affiliation(s)
- Michael Bukwich
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Current address: Sainsbury Wellcome Centre, University College London, London, W1T 4JG, UK
| | - Malcolm G Campbell
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - David Zoltowski
- Department of Statistics, Stanford University, Stanford, CA, 94305
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305
| | - Lyle Kingsbury
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - Momchil S Tomov
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Current address: Motional AD LLC, Boston, MA 02210
| | - Joshua Stern
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| | - HyungGoo R Kim
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon 16419, Republic of Korea
- Department of Biomedical Engineering, Sungkyunkwan University, Suwon 16419, Republic of Korea
| | - Jan Drugowitsch
- Department of Neurobiology, Harvard Medical School, Boston, MA, 02115
| | - Scott W Linderman
- Department of Statistics, Stanford University, Stanford, CA, 94305
- Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA 94305
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, 02138
- Center for Brain Science, Harvard University, Cambridge, MA, 02138
| |
Collapse
|
44
|
Zhuo Y, Luo B, Yi X, Dong H, Wan J, Cai R, Williams JT, Qian T, Campbell MG, Miao X, Li B, Wei Y, Li G, Wang H, Zheng Y, Watabe-Uchida M, Li Y. Improved dual-color GRAB sensors for monitoring dopaminergic activity in vivo. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.24.554559. [PMID: 37662187 PMCID: PMC10473776 DOI: 10.1101/2023.08.24.554559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
Dopamine (DA) plays multiple roles in a wide range of physiological and pathological processes via a vast network of dopaminergic projections. To fully dissect the spatiotemporal dynamics of DA release in both dense and sparsely innervated brain regions, we developed a series of green and red fluorescent GPCR activation-based DA (GRABDA) sensors using a variety of DA receptor subtypes. These sensors have high sensitivity, selectivity, and signal-to-noise properties with subsecond response kinetics and the ability to detect a wide range of DA concentrations. We then used these sensors in freely moving mice to measure both optogenetically evoked and behaviorally relevant DA release while measuring neurochemical signaling in the nucleus accumbens, amygdala, and cortex. Using these sensors, we also detected spatially resolved heterogeneous cortical DA release in mice performing various behaviors. These next-generation GRABDA sensors provide a robust set of tools for imaging dopaminergic activity under a variety of physiological and pathological conditions.
Collapse
Affiliation(s)
- Yizhou Zhuo
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
- These authors contributed equally
| | - Bin Luo
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing 100871, China
- These authors contributed equally
| | - Xinyang Yi
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Hui Dong
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing 100871, China
| | - Jinxia Wan
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing 100871, China
| | - Ruyi Cai
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - John T. Williams
- Vollum Institute, Oregon Health and Science University, Portland, OR 97239, USA
| | - Tongrui Qian
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Malcolm G. Campbell
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA
| | - Xiaolei Miao
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- Department of Anesthesiology, Beijing Chaoyang Hospital, Capital Medical University, Beijing 100020, China
| | - Bozhi Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- Department of Neurology, the First Medical Center, Chinese PLA General Hospital, Fuxing Road 28, Haidian District, Beijing 100853, China
| | - Yu Wei
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Guochuan Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Huan Wang
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Yu Zheng
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
| | - Mitsuko Watabe-Uchida
- Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, 16 Divinity Avenue, Cambridge, MA 02138, USA
| | - Yulong Li
- State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, Beijing 100871, China
- PKU-IDG/McGovern Institute for Brain Research, Beijing 100871, China
- Peking-Tsinghua Center for Life Sciences, Beijing 100871, China
- Chinese Institute for Brain Research, Beijing 102206, China
- Institute of Molecular Physiology, Shenzhen Bay Laboratory, Shenzhen, Guangdong 518055, China
- National Biomedical Imaging Center, Peking University, Beijing 100871, China
| |
Collapse
|
45
|
Juarez B, Kong MS, Jo YS, Elum JE, Yee JX, Ng-Evans S, Cline M, Hunker AC, Quinlan MA, Baird MA, Elerding AJ, Johnson M, Ban D, Mendez A, Goodwin NL, Soden ME, Zweifel LS. Temporal scaling of dopamine neuron firing and dopamine release by distinct ion channels shape behavior. SCIENCE ADVANCES 2023; 9:eadg8869. [PMID: 37566654 PMCID: PMC10421029 DOI: 10.1126/sciadv.adg8869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Accepted: 07/10/2023] [Indexed: 08/13/2023]
Abstract
Dopamine is broadly implicated in reinforcement learning, but how patterns of dopamine activity are generated is poorly resolved. Here, we demonstrate that two ion channels, Kv4.3 and BKCa1.1, regulate the pattern of dopamine neuron firing and dopamine release on different time scales to influence separate phases of reinforced behavior in mice. Inactivation of Kv4.3 in VTA dopamine neurons increases ex vivo pacemaker activity and excitability that is associated with increased in vivo firing rate and ramping dynamics before lever press in a learned instrumental paradigm. Loss of Kv4.3 enhances performance of the learned response and facilitates extinction. In contrast, loss of BKCa1.1 increases burst firing and phasic dopamine release that enhances learning of an instrumental response and enhances extinction burst lever pressing in early extinction that is associated with a greater change in activity between reinforced and unreinforced actions. These data demonstrate that disruption of intrinsic regulators of neuronal activity differentially affects dopamine dynamics during reinforcement and extinction learning.
Collapse
Affiliation(s)
- Barbara Juarez
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
- Department of Pharmacology, University of Washington, Seattle, WA, USA
| | - Mi-Seon Kong
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Yong S. Jo
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Jordan E. Elum
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Joshua X. Yee
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Scott Ng-Evans
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Marcella Cline
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Avery C. Hunker
- Department of Pharmacology, University of Washington, Seattle, WA, USA
| | - Meagan A. Quinlan
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Madison A. Baird
- Department of Pharmacology, University of Washington, Seattle, WA, USA
| | | | - Mia Johnson
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Derek Ban
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | - Adriana Mendez
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
| | | | - Marta E. Soden
- Department of Pharmacology, University of Washington, Seattle, WA, USA
| | - Larry S. Zweifel
- Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, WA, USA
- Department of Pharmacology, University of Washington, Seattle, WA, USA
| |
Collapse
|
46
|
Wärnberg E, Kumar A. Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia. Proc Natl Acad Sci U S A 2023; 120:e2221994120. [PMID: 37527344 PMCID: PMC10410740 DOI: 10.1073/pnas.2221994120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2022] [Accepted: 06/08/2023] [Indexed: 08/03/2023] Open
Abstract
It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, "three-factor" corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.
Collapse
Affiliation(s)
- Emil Wärnberg
- Department of Neuroscience, Karolinska Institutet, 171 77Stockholm, Sweden
- Division of Computational Science and Technology, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 114 28Stockholm, Sweden
| | - Arvind Kumar
- Division of Computational Science and Technology, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 114 28Stockholm, Sweden
| |
Collapse
|
47
|
Zheng Y, Li Y. Past, present, and future of tools for dopamine detection. Neuroscience 2023:S0306-4522(23)00295-6. [PMID: 37419404 DOI: 10.1016/j.neuroscience.2023.06.025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Revised: 06/26/2023] [Accepted: 06/29/2023] [Indexed: 07/09/2023]
Abstract
Dopamine (DA) is a critical neuromodulator involved in various brain functions. To understand how DA regulates neural circuits and behaviors in the physiological and pathological conditions, it is essential to have tools that enable the direct detection of DA dynamics in vivo. Recently, genetically encoded DA sensors based on G protein-coupled receptors revolutionized this field, as it allows us to track in vivo DA dynamic with unprecedented spatial-temporal resolution, high molecular specificity, and sub-second kinetics. In this review, we first summarize traditional DA detection methods. Then we focus on the development of genetically encoded DA sensors and feature its significance to understanding dopaminergic neuromodulation across diverse behaviors and species. Finally, we present our perspectives about the future direction of the next-generation DA sensors and extend their potential applications. Overall, this review offers a comprehensive perspective on the past, present, and future of DA detection tools, with important implications for the study of DA functions in health and disease.
Collapse
Affiliation(s)
- Yu Zheng
- Peking-Tsinghua Center for Life Sciences, 100871 Beijing, China
| | - Yulong Li
- Peking-Tsinghua Center for Life Sciences, 100871 Beijing, China; State Key Laboratory of Membrane Biology, Peking University School of Life Sciences, 100871 Beijing, China; PKU-IDG/McGovern Institute for Brain Research, 100871 Beijing, China; National Biomedical Imaging Center, Peking University, 100871 Beijing, China.
| |
Collapse
|
48
|
Sawicki J, Berner R, Loos SAM, Anvari M, Bader R, Barfuss W, Botta N, Brede N, Franović I, Gauthier DJ, Goldt S, Hajizadeh A, Hövel P, Karin O, Lorenz-Spreen P, Miehl C, Mölter J, Olmi S, Schöll E, Seif A, Tass PA, Volpe G, Yanchuk S, Kurths J. Perspectives on adaptive dynamical systems. CHAOS (WOODBURY, N.Y.) 2023; 33:071501. [PMID: 37486668 DOI: 10.1063/5.0147231] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/20/2023] [Accepted: 05/24/2023] [Indexed: 07/25/2023]
Abstract
Adaptivity is a dynamical feature that is omnipresent in nature, socio-economics, and technology. For example, adaptive couplings appear in various real-world systems, such as the power grid, social, and neural networks, and they form the backbone of closed-loop control strategies and machine learning algorithms. In this article, we provide an interdisciplinary perspective on adaptive systems. We reflect on the notion and terminology of adaptivity in different disciplines and discuss which role adaptivity plays for various fields. We highlight common open challenges and give perspectives on future research directions, looking to inspire interdisciplinary approaches.
Collapse
Affiliation(s)
- Jakub Sawicki
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Akademie Basel, Fachhochschule Nordwestschweiz FHNW, Leonhardsstrasse 6, 4009 Basel, Switzerland
| | - Rico Berner
- Department of Physics, Humboldt-Universität zu Berlin, Newtonstraße 15, 12489 Berlin, Germany
| | - Sarah A M Loos
- DAMTP, University of Cambridge, Wilberforce Road, Cambridge CB3 0WA, United Kingdom
| | - Mehrnaz Anvari
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Fraunhofer Institute for Algorithms and Scientific Computing, Schloss Birlinghoven, 53757 Sankt-Augustin, Germany
| | - Rolf Bader
- Institute of Systematic Musicology, University of Hamburg, Hamburg, Germany
| | - Wolfram Barfuss
- Transdisciplinary Research Area: Sustainable Futures, University of Bonn, 53113 Bonn, Germany
- Center for Development Research (ZEF), University of Bonn, 53113 Bonn, Germany
| | - Nicola Botta
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Department of Computer Science and Engineering, Chalmers University of Technology, 412 96 Göteborg, Sweden
| | - Nuria Brede
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Department of Computer Science, University of Potsdam, An der Bahn 2, 14476 Potsdam, Germany
| | - Igor Franović
- Scientific Computing Laboratory, Center for the Study of Complex Systems, Institute of Physics Belgrade, University of Belgrade, Pregrevica 118, 11080 Belgrade, Serbia
| | - Daniel J Gauthier
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
| | - Sebastian Goldt
- Department of Physics, International School of Advanced Studies (SISSA), Trieste, Italy
| | - Aida Hajizadeh
- Research Group Comparative Neuroscience, Leibniz Institute for Neurobiology, Magdeburg, Germany
| | - Philipp Hövel
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
| | - Omer Karin
- Department of Mathematics, Imperial College London, London SW7 2AZ, United Kingdom
| | - Philipp Lorenz-Spreen
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Lentzeallee 94, 14195 Berlin, Germany
| | - Christoph Miehl
- Akademie Basel, Fachhochschule Nordwestschweiz FHNW, Leonhardsstrasse 6, 4009 Basel, Switzerland
| | - Jan Mölter
- Department of Mathematics, School of Computation, Information and Technology, Technical University of Munich, Boltzmannstraße 3, 85748 Garching bei München, Germany
| | - Simona Olmi
- Akademie Basel, Fachhochschule Nordwestschweiz FHNW, Leonhardsstrasse 6, 4009 Basel, Switzerland
| | - Eckehard Schöll
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Akademie Basel, Fachhochschule Nordwestschweiz FHNW, Leonhardsstrasse 6, 4009 Basel, Switzerland
| | - Alireza Seif
- Pritzker School of Molecular Engineering, The University of Chicago, Chicago, Illinois 60637, USA
| | - Peter A Tass
- Department of Neurosurgery, Stanford University School of Medicine, Stanford, California 94304, USA
| | - Giovanni Volpe
- Department of Physics, University of Gothenburg, Gothenburg, Sweden
| | - Serhiy Yanchuk
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Department of Physics, Humboldt-Universität zu Berlin, Newtonstraße 15, 12489 Berlin, Germany
| | - Jürgen Kurths
- Potsdam Institute for Climate Impact Research, Telegrafenberg, 14473 Potsdam, Germany
- Department of Physics, Humboldt-Universität zu Berlin, Newtonstraße 15, 12489 Berlin, Germany
| |
Collapse
|
49
|
Enriquez-Traba J, Yarur-Castillo HE, Flores RJ, Weil T, Roy S, Usdin TB, LaGamma CT, Arenivar M, Wang H, Tsai VS, Moritz AE, Sibley DR, Moratalla R, Freyberg ZZ, Tejeda HA. Dissociable control of motivation and reinforcement by distinct ventral striatal dopamine receptors. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.06.27.546539. [PMID: 37425766 PMCID: PMC10327105 DOI: 10.1101/2023.06.27.546539] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/11/2023]
Abstract
Dopamine release in striatal circuits, including the nucleus accumbens (NAc), tracks separable features of reward such as motivation and reinforcement. However, the cellular and circuit mechanisms by which dopamine receptors transform dopamine release into distinct constructs of reward remain unclear. Here, we show that dopamine D3 receptor (D3R) signaling in the NAc drives motivated behavior by regulating local NAc microcircuits. Furthermore, D3Rs co-express with dopamine D1 receptors (D1Rs), which regulate reinforcement, but not motivation. Paralleling dissociable roles in reward function, we report non-overlapping physiological actions of D3R and D1R signaling in NAc neurons. Our results establish a novel cellular framework wherein dopamine signaling within the same NAc cell type is physiologically compartmentalized via actions on distinct dopamine receptors. This structural and functional organization provides neurons in a limbic circuit with the unique ability to orchestrate dissociable aspects of reward-related behaviors that are relevant to the etiology of neuropsychiatric disorders.
Collapse
|
50
|
Jordan R, Keller GB. The locus coeruleus broadcasts prediction errors across the cortex to promote sensorimotor plasticity. eLife 2023; 12:85111. [PMID: 37285281 DOI: 10.7554/elife.85111] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/09/2023] Open
Abstract
Prediction errors are differences between expected and actual sensory input and are thought to be key computational signals that drive learning related plasticity. One way that prediction errors could drive learning is by activating neuromodulatory systems to gate plasticity. The catecholaminergic locus coeruleus (LC) is a major neuromodulatory system involved in neuronal plasticity in the cortex. Using two-photon calcium imaging in mice exploring a virtual environment, we found that the activity of LC axons in the cortex correlated with the magnitude of unsigned visuomotor prediction errors. LC response profiles were similar in both motor and visual cortical areas, indicating that LC axons broadcast prediction errors throughout the dorsal cortex. While imaging calcium activity in layer 2/3 of the primary visual cortex, we found that optogenetic stimulation of LC axons facilitated learning of a stimulus-specific suppression of visual responses during locomotion. This plasticity - induced by minutes of LC stimulation - recapitulated the effect of visuomotor learning on a scale that is normally observed during visuomotor development across days. We conclude that prediction errors drive LC activity, and that LC activity facilitates sensorimotor plasticity in the cortex, consistent with a role in modulating learning rates.
Collapse
Affiliation(s)
- Rebecca Jordan
- Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
| | - Georg B Keller
- Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland
- Faculty of Sciences, University of Basel, Basel, Switzerland
| |
Collapse
|