1
|
Balsdon T, Pisauro MA, Philiastides MG. Distinct basal ganglia contributions to learning from implicit and explicit value signals in perceptual decision-making. Nat Commun 2024; 15:5317. [PMID: 38909014 PMCID: PMC11193814 DOI: 10.1038/s41467-024-49538-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Accepted: 06/07/2024] [Indexed: 06/24/2024] Open
Abstract
Metacognitive evaluations of confidence provide an estimate of decision accuracy that could guide learning in the absence of explicit feedback. We examine how humans might learn from this implicit feedback in direct comparison with that of explicit feedback, using simultaneous EEG-fMRI. Participants performed a motion direction discrimination task where stimulus difficulty was increased to maintain performance, with intermixed explicit- and no-feedback trials. We isolate single-trial estimates of post-decision confidence using EEG decoding, and find these neural signatures re-emerge at the time of feedback together with separable signatures of explicit feedback. We identified these signatures of implicit versus explicit feedback along a dorsal-ventral gradient in the striatum, a finding uniquely enabled by an EEG-fMRI fusion. These two signals appear to integrate into an aggregate representation in the external globus pallidus, which could broadcast updates to improve cortical decision processing via the thalamus and insular cortex, irrespective of the source of feedback.
Collapse
Affiliation(s)
- Tarryn Balsdon
- Centre for Cognitive Neuroimaging, School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK.
- Laboratory of Perceptual Systems, DEC, ENS, PSL University, CNRS UMR 8248, Paris, France.
| | - M Andrea Pisauro
- Centre for Cognitive Neuroimaging, School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK
- School of Psychology, University of Plymouth, Plymouth, UK
| | - Marios G Philiastides
- Centre for Cognitive Neuroimaging, School of Psychology and Neuroscience, University of Glasgow, Glasgow, UK.
| |
Collapse
|
2
|
Janet R, Ligneul R, Losecaat-Vermeer AB, Philippe R, Bellucci G, Derrington E, Park SQ, Dreher JC. Regulation of social hierarchy learning by serotonin transporter availability. Neuropsychopharmacology 2022; 47:2205-2212. [PMID: 35945275 PMCID: PMC9630526 DOI: 10.1038/s41386-022-01378-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 06/03/2022] [Accepted: 06/30/2022] [Indexed: 11/18/2022]
Abstract
Learning one's status in a group is a fundamental process in building social hierarchies. Although animal studies suggest that serotonin (5-HT) signaling modulates learning social hierarchies, direct evidence in humans is lacking. Here we determined the relationship between serotonin transporter (SERT) availability and brain systems engaged in learning social ranks combining computational approaches with simultaneous PET-fMRI acquisition in healthy males. We also investigated the link between SERT availability and brain activity in a non-social control condition involving learning the payoffs of slot machines. Learning social ranks was modulated by the dorsal raphe nucleus (DRN) 5-HT function. BOLD ventral striatal response, tracking the rank of opponents, decreased with DRN SERT levels. Moreover, this link was specific to the social learning task. These findings demonstrate that 5-HT plays an influence on the computations required to learn social ranks.
Collapse
Affiliation(s)
- Remi Janet
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Romain Ligneul
- grid.421010.60000 0004 0453 9636Champalimaud Neuroscience Program, Champalimaud Center for the Unknown, Lisbon, Portugal
| | - Annabel B. Losecaat-Vermeer
- grid.10420.370000 0001 2286 1424Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria ,grid.7468.d0000 0001 2248 7639Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Neuroscience Research Center, 10117 Berlin, Germany
| | - Remi Philippe
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Gabriele Bellucci
- grid.419501.80000 0001 2183 0052Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Edmund Derrington
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Soyoung Q. Park
- grid.7468.d0000 0001 2248 7639Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Neuroscience Research Center, 10117 Berlin, Germany ,grid.418213.d0000 0004 0390 0098Department of Decision Neuroscience and Nutrition, German Institute of Human Nutrition (DIfE), Potsdam-Rehbrücke, Nuthetal, Germany
| | - Jean-Claude Dreher
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France.
| |
Collapse
|
3
|
Parker NF, Baidya A, Cox J, Haetzel LM, Zhukovskaya A, Murugan M, Engelhard B, Goldman MS, Witten IB. Choice-selective sequences dominate in cortical relative to thalamic inputs to NAc to support reinforcement learning. Cell Rep 2022; 39:110756. [PMID: 35584665 PMCID: PMC9218875 DOI: 10.1016/j.celrep.2022.110756] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2019] [Revised: 02/18/2022] [Accepted: 04/07/2022] [Indexed: 11/25/2022] Open
Abstract
How are actions linked with subsequent outcomes to guide choices? The nucleus accumbens, which is implicated in this process, receives glutamatergic inputs from the prelimbic cortex and midline regions of the thalamus. However, little is known about whether and how representations differ across these input pathways. By comparing these inputs during a reinforcement learning task in mice, we discovered that prelimbic cortical inputs preferentially represent actions and choices, whereas midline thalamic inputs preferentially represent cues. Choice-selective activity in the prelimbic cortical inputs is organized in sequences that persist beyond the outcome. Through computational modeling, we demonstrate that these sequences can support the neural implementation of reinforcement-learning algorithms, in both a circuit model based on synaptic plasticity and one based on neural dynamics. Finally, we test and confirm a prediction of our circuit models by direct manipulation of nucleus accumbens input neurons.
Collapse
Affiliation(s)
- Nathan F Parker
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Avinash Baidya
- Center for Neuroscience, University of California, Davis, Davis, CA 95616, USA; Department of Physics and Astronomy, University of California, Davis, Davis, CA 95616, USA
| | - Julia Cox
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Department of Neuroscience, Feinberg School of Medicine, Northwestern University, Chicago, IL 60611, USA
| | - Laura M Haetzel
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Anna Zhukovskaya
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Malavika Murugan
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Ben Engelhard
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA
| | - Mark S Goldman
- Center for Neuroscience, University of California, Davis, Davis, CA 95616, USA; Department of Neurobiology, Physiology and Behavior, University of California, Davis, Davis, CA 95616, USA; Department of Ophthalmology and Vision Science, University of California, Davis, Davis, CA 95616, USA.
| | - Ilana B Witten
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544, USA; Department of Psychology, Princeton University, Princeton, NJ 08544, USA.
| |
Collapse
|
4
|
Gmaz JM, van der Meer MAA. Context coding in the mouse nucleus accumbens modulates motivationally relevant information. PLoS Biol 2022; 20:e3001338. [PMID: 35486662 PMCID: PMC9094556 DOI: 10.1371/journal.pbio.3001338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2021] [Revised: 05/11/2022] [Accepted: 04/04/2022] [Indexed: 11/18/2022] Open
Abstract
Neural activity in the nucleus accumbens (NAc) is thought to track fundamentally value-centric quantities linked to reward and effort. However, the NAc also contributes to flexible behavior in ways that are difficult to explain based on value signals alone, raising the question of if and how nonvalue signals are encoded in NAc. We recorded NAc neural ensembles while head-fixed mice performed an odor-based biconditional discrimination task where an initial discrete cue modulated the behavioral significance of a subsequently presented reward-predictive cue. We extracted single-unit and population-level correlates related to the cues and found value-independent coding for the initial, context-setting cue. This context signal occupied a population-level coding space orthogonal to outcome-related representations and was predictive of subsequent behaviorally relevant responses to the reward-predictive cues. Together, these findings support a gating model for how the NAc contributes to behavioral flexibility and provide a novel population-level perspective from which to view NAc computations.
Collapse
Affiliation(s)
- Jimmie M. Gmaz
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, United States of America
| | - Matthijs A. A. van der Meer
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, United States of America
- * E-mail:
| |
Collapse
|
5
|
Xiao X, Deng H, Furlan A, Yang T, Zhang X, Hwang GR, Tucciarone J, Wu P, He M, Palaniswamy R, Ramakrishnan C, Ritola K, Hantman A, Deisseroth K, Osten P, Huang ZJ, Li B. A Genetically Defined Compartmentalized Striatal Direct Pathway for Negative Reinforcement. Cell 2020; 183:211-227.e20. [PMID: 32937106 DOI: 10.1016/j.cell.2020.08.032] [Citation(s) in RCA: 40] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2020] [Revised: 05/02/2020] [Accepted: 08/17/2020] [Indexed: 12/31/2022]
Abstract
The striosome compartment within the dorsal striatum has been implicated in reinforcement learning and regulation of motivation, but how striosomal neurons contribute to these functions remains elusive. Here, we show that a genetically identified striosomal population, which expresses the Teashirt family zinc finger 1 (Tshz1) and belongs to the direct pathway, drives negative reinforcement and is essential for aversive learning in mice. Contrasting a "conventional" striosomal direct pathway, the Tshz1 neurons cause aversion, movement suppression, and negative reinforcement once activated, and they receive a distinct set of synaptic inputs. These neurons are predominantly excited by punishment rather than reward and represent the anticipation of punishment or the motivation for avoidance. Furthermore, inhibiting these neurons impairs punishment-based learning without affecting reward learning or movement. These results establish a major role of striosomal neurons in behaviors reinforced by punishment and moreover uncover functions of the direct pathway unaccounted for in classic models.
Collapse
Affiliation(s)
- Xiong Xiao
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Hanfei Deng
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | | | - Tao Yang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Xian Zhang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Ga-Ram Hwang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Jason Tucciarone
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Priscilla Wu
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Miao He
- Institutes of Brain Science, State Key Laboratory of Medical Neurobiology and MOE Frontiers Center for Brain Science, Fudan University, Shanghai 200032, China
| | | | - Charu Ramakrishnan
- Howard Hughes Medical Institute (HHMI), Stanford University, Stanford, CA, USA; Department of Bioengineering and Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | | | - Adam Hantman
- HHMI Janelia Research Campus, Ashburn, VA 20147, USA
| | - Karl Deisseroth
- Howard Hughes Medical Institute (HHMI), Stanford University, Stanford, CA, USA; Department of Bioengineering and Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | - Pavel Osten
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Z Josh Huang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Bo Li
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA.
| |
Collapse
|
6
|
Elber-Dorozko L, Loewenstein Y. Striatal action-value neurons reconsidered. eLife 2018; 7:e34248. [PMID: 29848442 PMCID: PMC6008056 DOI: 10.7554/elife.34248] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2017] [Accepted: 05/13/2018] [Indexed: 11/13/2022] Open
Abstract
It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.
Collapse
Affiliation(s)
- Lotem Elber-Dorozko
- The Edmond & Lily Safra Center for Brain SciencesThe Hebrew University of JerusalemJerusalemIsrael
| | - Yonatan Loewenstein
- The Edmond & Lily Safra Center for Brain SciencesThe Hebrew University of JerusalemJerusalemIsrael
- Department of Neurobiology, The Alexander Silberman Institute of Life SciencesThe Hebrew University of JerusalemJerusalemIsrael
- The Federmann Center for the Study of RationalityThe Hebrew University of JerusalemJerusalemIsrael
| |
Collapse
|
7
|
Reward-Predictive Neural Activities in Striatal Striosome Compartments. eNeuro 2018; 5:eN-NWR-0367-17. [PMID: 29430520 PMCID: PMC5804148 DOI: 10.1523/eneuro.0367-17.2018] [Citation(s) in RCA: 37] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2017] [Revised: 01/18/2018] [Accepted: 01/25/2018] [Indexed: 02/02/2023] Open
Abstract
![]()
The striatum has been shown to play a critical role in reward prediction. It is composed of two neurochemically and anatomically distinct compartments known as the striosomes and the matrix. The striosomes comprise only about 15% of the striatum by volume and are distributed mosaically therein. Accordingly, it has been difficult to identify striosomal neurons in electrophysiological recordings and it has been unclear whether striosomal neurons, which project to midbrain dopaminergic neurons, engage in reward prediction. In this study, we utilized a mouse line (Sepw1-NP67) selectively expressing Cre in striosomal neurons, combined with endoscopic in vivo calcium imaging to selectively record activities of striosomal neurons during an odor-conditioning task. As mice learned the task, striosomal neurons in the dorsomedial striatum (DMS) showed predictive activities to odor cues that were associated with water rewards or aversive air puffs. These activities were proportional to the expected reward or air-puff intensity. Intriguingly, repeated recordings of the same striosomal neurons over a period of weeks revealed that predictive activities were learning-stage specific. That is, these activities disappeared after continuous training. Furthermore, presentations of rewards or air puffs activated some striosomal neurons. These findings suggest that the striosomes participate in reward prediction with learning stage-specific neural ensembles, and that they also send reward and aversive signals to dopaminergic neurons.
Collapse
|
8
|
Striatal Activity and Reward Relativity: Neural Signals Encoding Dynamic Outcome Valuation. eNeuro 2016; 3:eN-NWR-0022-16. [PMID: 27822506 PMCID: PMC5089537 DOI: 10.1523/eneuro.0022-16.2016] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2016] [Revised: 10/06/2016] [Accepted: 10/07/2016] [Indexed: 11/21/2022] Open
Abstract
The striatum is a key brain region involved in reward processing. Striatal activity has been linked to encoding reward magnitude and integrating diverse reward outcome information. Recent work has supported the involvement of striatum in the valuation of outcomes. The present work extends this idea by examining striatal activity during dynamic shifts in value that include different levels and directions of magnitude disparity. A novel task was used to produce diverse relative reward effects on a chain of instrumental action. Rats (Rattus norvegicus) were trained to respond to cues associated with specific outcomes varying by food pellet magnitude. Animals were exposed to single-outcome sessions followed by mixed-outcome sessions, and neural activity was compared among identical outcome trials from the different behavioral contexts. Results recording striatal activity show that neural responses to different task elements reflect incentive contrast as well as other relative effects that involve generalization between outcomes or possible influences of outcome variety. The activity that was most prevalent was linked to food consumption and post-food consumption periods. Relative encoding was sensitive to magnitude disparity. A within-session analysis showed strong contrast effects that were dependent upon the outcome received in the immediately preceding trial. Significantly higher numbers of responses were found in ventral striatum linked to relative outcome effects. Our results support the idea that relative value can incorporate diverse relationships, including comparisons from specific individual outcomes to general behavioral contexts. The striatum contains these diverse relative processes, possibly enabling both a higher information yield concerning value shifts and a greater behavioral flexibility.
Collapse
|
9
|
Effects of striatal lesions on components of choice: Reward discrimination, preference, and relative valuation. Behav Brain Res 2016; 315:130-40. [PMID: 27544873 DOI: 10.1016/j.bbr.2016.08.031] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2016] [Revised: 08/10/2016] [Accepted: 08/15/2016] [Indexed: 11/22/2022]
Abstract
The striatum is a key structure involved in reward processing and choice. Recently, we have developed a paradigm to explore how components of reward processing work together or independently during choice behavior. These components include reward discrimination, preference and relative valuation, and the goal of the present study was to determine how the striatum is involved in these dissociable components during this novel free choice paradigm. We tested choice utilizing two different outcome series with one being a more straightforward single-option discrimination anchored by a 0 reward outcome, and the other as a multi-option outcome discrimination of greater difficulty. We compared the free choice reward task to a sequential reward task and an extinction task. Striatal lesions impaired responding only in the free choice version with alterations in both appetitive and consummatory measures. Ventral striatal lesions had greater impact altering discrimination, preference and relative valuation in both the single and multi-option week studies. A major factor involved in these deficits was a significant aversion to the multi-option that contained a larger outcome option but with a longer delay to reward. Dorsal striatal lesions caused less impairment even leading to enhanced choice behavior compared to control animals during the more difficult multi-option free choice series. Overall, the results suggest that the context of action is crucial when linking striatal function to choice behavior and its diverse components. The implications include the idea that striatal involvement in decision-making is increased when responses are self-paced and diverse in a more naturalistic environment.
Collapse
|