1
|
Zhang Z, Takahashi YK, Montesinos-Cartegena M, Kahnt T, Langdon AJ, Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on hippocampus. Nat Commun 2024; 15:8911. [PMID: 39414794 DOI: 10.1038/s41467-024-53308-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Accepted: 10/07/2024] [Indexed: 10/18/2024] Open
Abstract
The orbitofrontal cortex (OFC) and hippocampus (HC) both contribute to the cognitive maps that support flexible behavior. Previously, we used the dopamine neurons to measure the functional role of OFC. We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which expected rewards were manipulated across blocks. We found that ipsilateral OFC lesions degraded dopaminergic prediction errors, consistent with reduced resolution of the task states. Here we have repeated this experiment in male rats with ipsilateral HC lesions. The results show HC also shapes the task states, however unlike OFC, which provides information local to the trial, the HC is necessary for estimating upper-level hidden states that distinguish blocks. The results contrast the roles of the OFC and HC in cognitive mapping and suggest that the dopamine neurons access rich information from distributed regions regarding the environment's structure, potentially enabling this teaching signal to support complex behaviors.
Collapse
Affiliation(s)
- Zhewei Zhang
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| | - Yuji K Takahashi
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | | | - Thorsten Kahnt
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | - Angela J Langdon
- Intramural Research Program, National Institute on Mental Health, Bethesda, MD, USA
| | - Geoffrey Schoenbaum
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| |
Collapse
|
2
|
Kim MJ, Gibson DJ, Hu D, Yoshida T, Hueske E, Matsushima A, Mahar A, Schofield CJ, Sompolpong P, Tran KT, Tian L, Graybiel AM. Dopamine release plateau and outcome signals in dorsal striatum contrast with classic reinforcement learning formulations. Nat Commun 2024; 15:8856. [PMID: 39402067 PMCID: PMC11473536 DOI: 10.1038/s41467-024-53176-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/26/2023] [Accepted: 10/03/2024] [Indexed: 10/17/2024] Open
Abstract
We recorded dopamine release signals in centromedial and centrolateral sectors of the striatum as mice learned consecutive versions of visual cue-outcome conditioning tasks. Dopamine release responses differed for the centromedial and centrolateral sites. In neither sector could these be accounted for by classic reinforcement learning alone as classically applied to the activity of nigral dopamine-containing neurons. Medially, cue responses ranged from initial sharp peaks to modulated plateau responses; outcome (reward) responses during cue conditioning were minimal or, initially, negative. At centrolateral sites, by contrast, strong, transient dopamine release responses occurred at both cue and outcome. Prolonged, plateau release responses to cues emerged in both regions when discriminative behavioral responses became required. At most sites, we found no evidence for a transition from outcome signaling to cue signaling, a hallmark of temporal difference reinforcement learning as applied to midbrain dopaminergic neuronal activity. These findings delineate a reshaping of striatal dopamine release activity during learning and suggest that current views of reward prediction error encoding need review to accommodate distinct learning-related spatial and temporal patterns of striatal dopamine release in the dorsal striatum.
Collapse
Affiliation(s)
- Min Jung Kim
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
- Advanced Imaging Research Center, University of Texas, Southwestern Medical Center, Dallas, TX, 75390, USA
| | - Daniel J Gibson
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Dan Hu
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Tomoko Yoshida
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Emily Hueske
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Ayano Matsushima
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Ara Mahar
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Cynthia J Schofield
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins University School of Medicine, Baltimore, MD, 21205, USA
| | - Patlapa Sompolpong
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
- Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY, 10029, USA
| | - Kathy T Tran
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA
| | - Lin Tian
- Max Planck Florida Institute for Neuroscience, Jupiter, FL, 33458, USA
| | - Ann M Graybiel
- McGovern Institute for Brain Research and Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, 43 Vassar St., Cambridge, MA, 02139, USA.
| |
Collapse
|
3
|
Shimbo A, Takahashi YK, Langdon AJ, Stalnaker TA, Schoenbaum G. Cracking and Packing Information about the Features of Expected Rewards in the Orbitofrontal Cortex. J Neurosci 2024; 44:e0714242024. [PMID: 39122558 PMCID: PMC11376335 DOI: 10.1523/jneurosci.0714-24.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2024] [Revised: 07/29/2024] [Accepted: 08/02/2024] [Indexed: 08/12/2024] Open
Abstract
The orbitofrontal cortex (OFC) is crucial for tracking various aspects of expected outcomes, thereby helping to guide choices and support learning. Our previous study showed that the effects of reward timing and size on the activity of single units in OFC were dissociable when these attributes were manipulated independently ( Roesch et al., 2006). However, in real-life decision-making scenarios, outcome features often change simultaneously, so here we investigated how OFC neurons in male rats integrate information about the timing and identity (flavor) of reward and respond to changes in these features, according to whether they were changed simultaneously or separately. We found that a substantial number of OFC neurons fired differentially to immediate versus delayed reward and to the different reward flavors. However, contrary to the previous study, selectivity for timing was strongly correlated with selectivity for identity. Taken together with the previous research, these results suggest that when reward features are correlated, OFC tends to "pack" them into unitary constructs, whereas when they are independent, OFC tends to "crack" them into separate constructs. Furthermore, we found that when both reward timing and flavor were changed, reward-responsive OFC neurons showed unique activity patterns preceding and during the omission of an expected reward. Interestingly, this OFC activity is similar and slightly preceded the ventral tegmental area dopamine (VTA DA) activity observed in a previous study ( Takahashi et al., 2023), consistent with the role of OFC in providing predictive information to VTA DA neurons.
Collapse
Affiliation(s)
- Akihiro Shimbo
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, Baltimore, Maryland 21224
| | - Yuji K Takahashi
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, Baltimore, Maryland 21224
| | - Angela J Langdon
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, Baltimore, Maryland 21224
| | - Thomas A Stalnaker
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, Baltimore, Maryland 21224
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, Baltimore, Maryland 21224
| |
Collapse
|
4
|
Skandalakis GP, Neudorfer C, Payne CA, Bond E, Tavakkoli AD, Barrios-Martinez J, Trutti AC, Koutsarnakis C, Coenen VA, Komaitis S, Hadjipanayis CG, Stranjalis G, Yeh FC, Banihashemi L, Hong J, Lozano AM, Kogan M, Horn A, Evans LT, Kalyvas A. Establishing connectivity through microdissections of midbrain stimulation-related neural circuits. Brain 2024; 147:3083-3098. [PMID: 38808482 PMCID: PMC11370807 DOI: 10.1093/brain/awae173] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 03/15/2024] [Accepted: 04/21/2024] [Indexed: 05/30/2024] Open
Abstract
Comprehensive understanding of the neural circuits involving the ventral tegmental area is essential for elucidating the anatomofunctional mechanisms governing human behaviour, in addition to the therapeutic and adverse effects of deep brain stimulation for neuropsychiatric diseases. Although the ventral tegmental area has been targeted successfully with deep brain stimulation for different neuropsychiatric diseases, the axonal connectivity of the region is not fully understood. Here, using fibre microdissections in human cadaveric hemispheres, population-based high-definition fibre tractography and previously reported deep brain stimulation hotspots, we find that the ventral tegmental area participates in an intricate network involving the serotonergic pontine nuclei, basal ganglia, limbic system, basal forebrain and prefrontal cortex, which is implicated in the treatment of obsessive-compulsive disorder, major depressive disorder, Alzheimer's disease, cluster headaches and aggressive behaviours.
Collapse
Affiliation(s)
- Georgios P Skandalakis
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
- Department of Neurosurgery, National and Kapodistrian University of Athens Medical School, Evangelismos General Hospital, Athens 10676, Greece
| | - Clemens Neudorfer
- Center for Brain Circuit Therapeutics Department of Neurology Brigham & Women’s Hospital, Harvard Medical School, Boston, MA 02115, USA
- MGH Neurosurgery & Center for Neurotechnology and Neurorecovery (CNTR) at MGH Neurology Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Department of Neurology, Charité—Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany
| | - Caitlin A Payne
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
| | - Evalina Bond
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
| | - Armin D Tavakkoli
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
| | | | - Anne C Trutti
- Integrative Model-Based Cognitive Neuroscience Research Unit, University of Amsterdam, Amsterdam 15926, The Netherlands
| | - Christos Koutsarnakis
- Department of Neurosurgery, National and Kapodistrian University of Athens Medical School, Evangelismos General Hospital, Athens 10676, Greece
| | - Volker A Coenen
- Department of Stereotactic and Functional Neurosurgery, Medical Center of the University of Freiburg, Freiburg 79106, Germany
- Medical Faculty of the University of Freiburg, Freiburg 79110, Germany
- Center for Deep Brain Stimulation, Medical Center of the University of Freiburg, Freiburg 79106, Germany
| | - Spyridon Komaitis
- Queens Medical Center, Nottingham University Hospitals NHS Foundation Trust, Nottingham NG7 2UH, UK
| | | | - George Stranjalis
- Department of Neurosurgery, National and Kapodistrian University of Athens Medical School, Evangelismos General Hospital, Athens 10676, Greece
| | - Fang-Cheng Yeh
- Department of Neurological Surgery, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Layla Banihashemi
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA 15213, USA
- Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA 15213, USA
| | - Jennifer Hong
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
| | - Andres M Lozano
- Division of Neurosurgery, University Health Network, University of Toronto, Toronto, ON M5T 1P5, Canada
| | - Michael Kogan
- Department of Neurosurgery, University of New Mexico School of Medicine, Albuquerque, NM 87106, USA
| | - Andreas Horn
- Center for Brain Circuit Therapeutics Department of Neurology Brigham & Women’s Hospital, Harvard Medical School, Boston, MA 02115, USA
- MGH Neurosurgery & Center for Neurotechnology and Neurorecovery (CNTR) at MGH Neurology Massachusetts General Hospital, Harvard Medical School, Boston, MA 02114, USA
- Movement Disorder and Neuromodulation Unit, Department of Neurology, Department of Neurology, Charité—Universitätsmedizin Berlin, Corporate member of Freie Universität Berlin and Humboldt-Universität zu Berlin, 10117 Berlin, Germany
| | - Linton T Evans
- Section of Neurosurgery, Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
| | - Aristotelis Kalyvas
- Division of Neurosurgery, University Health Network, University of Toronto, Toronto, ON M5T 1P5, Canada
| |
Collapse
|
5
|
Taira M, Millard SJ, Verghese A, DiFazio LE, Hoang IB, Jia R, Sias A, Wikenheiser A, Sharpe MJ. Dopamine Release in the Nucleus Accumbens Core Encodes the General Excitatory Components of Learning. J Neurosci 2024; 44:e0120242024. [PMID: 38969504 PMCID: PMC11358529 DOI: 10.1523/jneurosci.0120-24.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Revised: 06/18/2024] [Accepted: 06/20/2024] [Indexed: 07/07/2024] Open
Abstract
Dopamine release in the nucleus accumbens core (NAcC) is generally considered to be a proxy for phasic firing of the ventral tegmental area dopamine (VTADA) neurons. Thus, dopamine release in NAcC is hypothesized to reflect a unitary role in reward prediction error signaling. However, recent studies reveal more diverse roles of dopamine neurons, which support an emerging idea that dopamine regulates learning differently in distinct circuits. To understand whether the NAcC might regulate a unique component of learning, we recorded dopamine release in NAcC while male rats performed a backward conditioning task where a reward is followed by a neutral cue. We used this task because we can delineate different components of learning, which include sensory-specific inhibitory and general excitatory components. Furthermore, we have shown that VTADA neurons are necessary for both the specific and general components of backward associations. Here, we found that dopamine release in NAcC increased to the reward across learning while reducing to the cue that followed as it became more expected. This mirrors the dopamine prediction error signal seen during forward conditioning and cannot be accounted for temporal-difference reinforcement learning. Subsequent tests allowed us to dissociate these learning components and revealed that dopamine release in NAcC reflects the general excitatory component of backward associations, but not their sensory-specific component. These results emphasize the importance of examining distinct functions of different dopamine projections in reinforcement learning.
Collapse
Affiliation(s)
- Masakazu Taira
- Department of Psychology, University of Sydney, Camperdown, New South Wales 2006, Australia
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Samuel J Millard
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Anna Verghese
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Lauren E DiFazio
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Ivy B Hoang
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Ruiting Jia
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Ana Sias
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Andrew Wikenheiser
- Department of Psychology, University of California, Los Angeles 90095, California
| | - Melissa J Sharpe
- Department of Psychology, University of Sydney, Camperdown, New South Wales 2006, Australia
- Department of Psychology, University of California, Los Angeles 90095, California
| |
Collapse
|
6
|
Lehmann CM, Miller NE, Nair VS, Costa KM, Schoenbaum G, Moussawi K. Generalized cue reactivity in dopamine neurons after opioids. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.02.597025. [PMID: 38853878 PMCID: PMC11160774 DOI: 10.1101/2024.06.02.597025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2024]
Abstract
Cue reactivity is the maladaptive neurobiological and behavioral response upon exposure to drug cues and is a major driver of relapse. The leading hypothesis is that dopamine release by addictive drugs represents a persistently positive reward prediction error that causes runaway enhancement of dopamine responses to drug cues, leading to their pathological overvaluation compared to non-drug reward alternatives. However, this hypothesis has not been directly tested. Here we developed Pavlovian and operant procedures to measure firing responses, within the same dopamine neurons, to drug versus natural reward cues, which we found to be similarly enhanced compared to cues predicting natural rewards in drug-naïve controls. This enhancement was associated with increased behavioral reactivity to the drug cue, suggesting that dopamine release is still critical to cue reactivity, albeit not as previously hypothesized. These results challenge the prevailing hypothesis of cue reactivity, warranting new models of dopaminergic function in drug addiction, and provide critical insights into the neurobiology of cue reactivity with potential implications for relapse prevention.
Collapse
Affiliation(s)
- Collin M. Lehmann
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Nora E. Miller
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Varun S. Nair
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
| | - Kauê M. Costa
- Department of Psychology, University of Alabama at Birmingham; Birmingham, 35233, USA
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse, National Institutes of Health; Baltimore, 21224, USA
| | - Khaled Moussawi
- Department of Psychiatry, University of Pittsburgh; Pittsburgh, 15219, USA
- Department of Neurology, University of California San Francisco; San Francisco, 94158, USA
| |
Collapse
|
7
|
Garr E, Cheng Y, Jeong H, Brooke S, Castell L, Bal A, Magnard R, Namboodiri VMK, Janak PH. Mesostriatal dopamine is sensitive to changes in specific cue-reward contingencies. SCIENCE ADVANCES 2024; 10:eadn4203. [PMID: 38809978 PMCID: PMC11135394 DOI: 10.1126/sciadv.adn4203] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 04/23/2024] [Indexed: 05/31/2024]
Abstract
Learning causal relationships relies on understanding how often one event precedes another. To investigate how dopamine neuron activity and neurotransmitter release change when a retrospective relationship is degraded for a specific pair of events, we used outcome-selective Pavlovian contingency degradation in rats. Conditioned responding was attenuated for the cue-reward contingency that was degraded, as was dopamine neuron activity in the midbrain and dopamine release in the ventral striatum in response to the cue and subsequent reward. Contingency degradation also abolished the trial-by-trial history dependence of the dopamine responses at the time of trial outcome. This profile of changes in cue- and reward-evoked responding is not easily explained by a standard reinforcement learning model. An alternative model based on learning causal relationships was better able to capture dopamine responses during contingency degradation, as well as conditioned behavior following optogenetic manipulations of dopamine during noncontingent rewards. Our results suggest that mesostriatal dopamine encodes the contingencies between meaningful events during learning.
Collapse
Affiliation(s)
- Eric Garr
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Yifeng Cheng
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Huijeong Jeong
- Department of Neurology, University of California, San Francisco, CA 94158, USA
| | - Sara Brooke
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
| | - Laia Castell
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Aneesh Bal
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Robin Magnard
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Vijay Mohan K. Namboodiri
- Department of Neurology, University of California, San Francisco, CA 94158, USA
- Weill Institute for Neurosciences, Kavli Institute for Fundamental Neuroscience, Center for Integrative Neuroscience, University of California, San Francisco, CA 94158, USA
| | - Patricia H. Janak
- Department of Psychological & Brain Sciences, Krieger School of Arts & Sciences, Johns Hopkins University, Baltimore, MD 21218, USA
- Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Baltimore, MD 21205, USA
- Kavli Neuroscience Discovery Institute, Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| |
Collapse
|
8
|
Floeder JR, Jeong H, Mohebi A, Namboodiri VMK. Mesolimbic dopamine ramps reflect environmental timescales. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.03.27.587103. [PMID: 38659749 PMCID: PMC11042231 DOI: 10.1101/2024.03.27.587103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/26/2024]
Abstract
Mesolimbic dopamine activity occasionally exhibits ramping dynamics, reigniting debate on theories of dopamine signaling. This debate is ongoing partly because the experimental conditions under which dopamine ramps emerge remain poorly understood. Here, we show that during Pavlovian and instrumental conditioning, mesolimbic dopamine ramps are only observed when the inter-trial interval is short relative to the trial period. These results constrain theories of dopamine signaling and identify a critical variable determining the emergence of dopamine ramps.
Collapse
Affiliation(s)
- Joseph R Floeder
- Neuroscience Graduate Program, University of California, San Francisco, CA, USA
| | - Huijeong Jeong
- Department of Neurology, University of California, San Francisco, CA, USA
| | - Ali Mohebi
- Department of Neurology, University of California, San Francisco, CA, USA
| | - Vijay Mohan K Namboodiri
- Neuroscience Graduate Program, University of California, San Francisco, CA, USA
- Department of Neurology, University of California, San Francisco, CA, USA
- Weill Institute for Neurosciences, Kavli Institute for Fundamental Neuroscience, Center for Integrative Neuroscience, University of California, San Francisco, CA, USA
| |
Collapse
|
9
|
Qian L, Burrell M, Hennig JA, Matias S, Murthy VN, Gershman SJ, Uchida N. The role of prospective contingency in the control of behavior and dopamine signals during associative learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.05.578961. [PMID: 38370735 PMCID: PMC10871210 DOI: 10.1101/2024.02.05.578961] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.
Collapse
Affiliation(s)
- Lechen Qian
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- These authors contributed equally
| | - Mark Burrell
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- These authors contributed equally
| | - Jay A. Hennig
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Sara Matias
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| | - Venkatesh. N. Murthy
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| | - Samuel J. Gershman
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Naoshige Uchida
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Center for Brain Science, Harvard University, Cambridge, MA, USA
| |
Collapse
|
10
|
Drew A, Soto-Faraco S. Perceptual oddities: assessing the relationship between film editing and prediction processes. Philos Trans R Soc Lond B Biol Sci 2024; 379:20220426. [PMID: 38104604 PMCID: PMC10725757 DOI: 10.1098/rstb.2022.0426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2023] [Accepted: 10/16/2023] [Indexed: 12/19/2023] Open
Abstract
During film viewing, humans parse sequences of individual shots into larger narrative structures, often weaving transitions at edit points into an apparently seamless and continuous flow. Editing helps filmmakers manipulate visual transitions to induce feelings of fluency/disfluency, tension/relief, curiosity, expectation and several emotional responses. We propose that the perceptual dynamics induced by film editing can be captured by a predictive processing (PP) framework. We hypothesise that visual discontinuities at edit points produce discrepancies between anticipated and actual sensory input, leading to prediction error. Further, we propose that the magnitude of prediction error depends on the predictability of each shot within the narrative flow, and lay out an account based on conflict monitoring. We test this hypothesis in two empirical studies measuring electroencephalography (EEG) during passive viewing of film excerpts, as well as behavioural responses during an active edit detection task. We report the neural and behavioural modulations at editing boundaries across three levels of narrative depth, showing greater modulations for edits spanning less predictable, deeper narrative transitions. Overall, our contribution lays the groundwork for understanding film editing from a PP perspective. This article is part of the theme issue 'Art, aesthetics and predictive processing: theoretical and empirical perspectivess'.
Collapse
Affiliation(s)
- Alice Drew
- Multisensory Research Group, Centre for Brain and Cognition, Universitat Pompeu Fabra, Carrer de Ramon Trias Fargas, 25-27, 08005 Barcelona, Spain
| | - Salvador Soto-Faraco
- Multisensory Research Group, Centre for Brain and Cognition, Universitat Pompeu Fabra, Carrer de Ramon Trias Fargas, 25-27, 08005 Barcelona, Spain
- Institució Catalana de Recerca i Estudis Avançats (ICREA), 08010 Barcelona, Spain
| |
Collapse
|
11
|
Stetsenko A, Koos T. Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system. Proc Natl Acad Sci U S A 2023; 120:e2309015120. [PMID: 37903252 PMCID: PMC10636325 DOI: 10.1073/pnas.2309015120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 09/29/2023] [Indexed: 11/01/2023] Open
Abstract
The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.
Collapse
Affiliation(s)
- Anya Stetsenko
- Center for Molecular and Behavioral Neuroscience, Rutgers University, Newark, NJ07102
| | - Tibor Koos
- Center for Molecular and Behavioral Neuroscience, Rutgers University, Newark, NJ07102
| |
Collapse
|
12
|
Kim MJ, Gibson DJ, Hu D, Mahar A, Schofield CJ, Sompolpong P, Yoshida T, Tran KT, Graybiel AM. Dopamine Release Plateau and Outcome Signals in Dorsal Striatum Contrast with Classic Reinforcement Learning Formulations. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.15.553421. [PMID: 37645888 PMCID: PMC10462077 DOI: 10.1101/2023.08.15.553421] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/31/2023]
Abstract
We recorded dopamine release signals in medial and lateral sectors of the striatum as mice learned consecutive visual cue-outcome conditioning tasks including cue association, cue discrimination, reversal, and probabilistic discrimination task versions. Dopamine release responses in medial and lateral sites exhibited learning-related changes within and across phases of acquisition. These were different for the medial and lateral sites. In neither sector could these be accounted for by classic reinforcement learning as applied to dopamine-containing neuron activity. Cue responses ranged from initial sharp peaks to modulated plateau responses. In the medial sector, outcome (reward) responses during cue conditioning were minimal or, initially, negative. By contrast, in lateral sites, strong, transient dopamine release responses occurred at both cue and outcome. Prolonged, plateau release responses to cues emerged in both regions when discriminative behavioral responses became required. In most sites, we found no evidence for a transition from outcome to cue signaling, a hallmark of temporal difference reinforcement learning as applied to midbrain dopamine activity. These findings delineate reshaping of dopamine release activity during learning and suggest that current views of reward prediction error encoding need review to accommodate distinct learning-related spatial and temporal patterns of striatal dopamine release in the dorsal striatum.
Collapse
|
13
|
Takahashi YK, Zhang Z, Montesinos-Cartegena M, Kahnt T, Langdon AJ, Schoenbaum G. Expectancy-related changes in firing of dopamine neurons depend on hippocampus. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.07.19.549728. [PMID: 37781610 PMCID: PMC10541105 DOI: 10.1101/2023.07.19.549728] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
The orbitofrontal cortex (OFC) and hippocampus (HC) are both implicated in forming the cognitive or task maps that support flexible behavior. Previously, we used the dopamine neurons as a sensor or tool to measure the functional effects of OFC lesions (Takahashi et al., 2011). We recorded midbrain dopamine neurons as rats performed an odor-based choice task, in which errors in the prediction of reward were induced by manipulating the number or timing of the expected rewards across blocks of trials. We found that OFC lesions ipsilateral to the recording electrodes caused prediction errors to be degraded consistent with a loss in the resolution of the task states, particularly under conditions where hidden information was critical to sharpening the predictions. Here we have repeated this experiment, along with computational modeling of the results, in rats with ipsilateral HC lesions. The results show HC also shapes the map of our task, however unlike OFC, which provides information local to the trial, the HC appears to be necessary for estimating the upper-level hidden states based on the information that is discontinuous or separated by longer timescales. The results contrast the respective roles of the OFC and HC in cognitive mapping and add to evidence that the dopamine neurons access a rich information set from distributed regions regarding the predictive structure of the environment, potentially enabling this powerful teaching signal to support complex learning and behavior.
Collapse
Affiliation(s)
- Yuji K Takahashi
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD
| | - Zhewei Zhang
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD
| | | | - Thorsten Kahnt
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD
| | - Angela J Langdon
- Intramural Research Program, National Institute on Mental Health, Bethesda, MD
| | - Geoffrey Schoenbaum
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD
| |
Collapse
|