1
|
Song MR, Lee SW. Rethinking dopamine-guided action sequence learning. Eur J Neurosci 2024; 60:3447-3465. [PMID: 38798086 DOI: 10.1111/ejn.16426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2023] [Revised: 04/21/2024] [Accepted: 05/08/2024] [Indexed: 05/29/2024]
Abstract
As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.
Collapse
Affiliation(s)
- Minryung R Song
- Department of Brain and Cognitive Sciences, KAIST, Daejeon, South Korea
| | - Sang Wan Lee
- Department of Brain and Cognitive Sciences, KAIST, Daejeon, South Korea
- Kim Jaechul Graduate School of AI, KAIST, Daejeon, South Korea
- KI for Health Science and Technology, KAIST, Daejeon, South Korea
- Center for Neuroscience-inspired AI, KAIST, Daejeon, South Korea
| |
Collapse
|
2
|
Ishizu K, Nishimoto S, Ueoka Y, Funamizu A. Localized and global representation of prior value, sensory evidence, and choice in male mouse cerebral cortex. Nat Commun 2024; 15:4071. [PMID: 38778078 PMCID: PMC11111702 DOI: 10.1038/s41467-024-48338-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Accepted: 04/26/2024] [Indexed: 05/25/2024] Open
Abstract
Adaptive behavior requires integrating prior knowledge of action outcomes and sensory evidence for making decisions while maintaining prior knowledge for future actions. As outcome- and sensory-based decisions are often tested separately, it is unclear how these processes are integrated in the brain. In a tone frequency discrimination task with two sound durations and asymmetric reward blocks, we found that neurons in the medial prefrontal cortex of male mice represented the additive combination of prior reward expectations and choices. The sensory inputs and choices were selectively decoded from the auditory cortex irrespective of reward priors and the secondary motor cortex, respectively, suggesting localized computations of task variables are required within single trials. In contrast, all the recorded regions represented prior values that needed to be maintained across trials. We propose localized and global computations of task variables in different time scales in the cerebral cortex.
Collapse
Affiliation(s)
- Kotaro Ishizu
- Institute for Quantitative Biosciences, University of Tokyo, Laboratory of Neural Computation, 1-1-1 Yayoi, Bunkyo-ku, Tokyo, 113-0032, Japan
| | - Shosuke Nishimoto
- Institute for Quantitative Biosciences, University of Tokyo, Laboratory of Neural Computation, 1-1-1 Yayoi, Bunkyo-ku, Tokyo, 113-0032, Japan
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-2, Komaba, Meguro-ku, Tokyo, 153-8902, Japan
| | - Yutaro Ueoka
- Institute for Quantitative Biosciences, University of Tokyo, Laboratory of Neural Computation, 1-1-1 Yayoi, Bunkyo-ku, Tokyo, 113-0032, Japan
| | - Akihiro Funamizu
- Institute for Quantitative Biosciences, University of Tokyo, Laboratory of Neural Computation, 1-1-1 Yayoi, Bunkyo-ku, Tokyo, 113-0032, Japan.
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, 3-8-2, Komaba, Meguro-ku, Tokyo, 153-8902, Japan.
| |
Collapse
|
3
|
Lemke SM, Celotto M, Maffulli R, Ganguly K, Panzeri S. Information flow between motor cortex and striatum reverses during skill learning. Curr Biol 2024; 34:1831-1843.e7. [PMID: 38604168 PMCID: PMC11078609 DOI: 10.1016/j.cub.2024.03.023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 02/22/2024] [Accepted: 03/14/2024] [Indexed: 04/13/2024]
Abstract
The coordination of neural activity across brain areas during a specific behavior is often interpreted as neural communication involved in controlling the behavior. However, whether information relevant to the behavior is actually transferred between areas is often untested. Here, we used information-theoretic tools to quantify how motor cortex and striatum encode and exchange behaviorally relevant information about specific reach-to-grasp movement features during skill learning in rats. We found a temporal shift in the encoding of behaviorally relevant information during skill learning, as well as a reversal in the primary direction of behaviorally relevant information flow, from cortex-to-striatum during naive movements to striatum-to-cortex during skilled movements. Standard analytical methods that quantify the evolution of overall neural activity during learning-such as changes in neural signal amplitude or the overall exchange of information between areas-failed to capture these behaviorally relevant information dynamics. Using these standard methods, we instead found a consistent coactivation of overall neural signals during movement production and a bidirectional increase in overall information propagation between areas during learning. Our results show that skill learning is achieved through a transformation in how behaviorally relevant information is routed across cortical and subcortical brain areas and that isolating the components of neural activity relevant to and informative about behavior is critical to uncover directional interactions within a coactive and coordinated network.
Collapse
Affiliation(s)
- Stefan M Lemke
- Center for Neuroscience and Cognitive Systems, Istituto Italiano di Tecnologia, Corso Bettini 31, 38068 Rovereto, Italy; Neurology Service, San Francisco Veterans Affairs Medical Center, 4150 Clement Street, San Francisco, CA 94121, USA; Department of Neurology, University of California, San Francisco, 1700 Owens Street, San Francisco, CA 94158, USA; Neuroscience Center, University of North Carolina, Chapel Hill, 116 Manning Drive, Chapel Hill, NC 27599, USA.
| | - Marco Celotto
- Center for Neuroscience and Cognitive Systems, Istituto Italiano di Tecnologia, Corso Bettini 31, 38068 Rovereto, Italy; Department of Pharmacy and Biotechnology, University of Bologna, Via Irnerio 48, 40126 Bologna, Italy; Institute of Neural Information Processing, Center for Molecular Neurobiology (ZMNH), University Medical Center Hamburg-Eppendorf (UKE), Falkenried 94, 20251 Hamburg, Germany
| | - Roberto Maffulli
- Center for Neuroscience and Cognitive Systems, Istituto Italiano di Tecnologia, Corso Bettini 31, 38068 Rovereto, Italy
| | - Karunesh Ganguly
- Neurology Service, San Francisco Veterans Affairs Medical Center, 4150 Clement Street, San Francisco, CA 94121, USA; Department of Neurology, University of California, San Francisco, 1700 Owens Street, San Francisco, CA 94158, USA
| | - Stefano Panzeri
- Institute of Neural Information Processing, Center for Molecular Neurobiology (ZMNH), University Medical Center Hamburg-Eppendorf (UKE), Falkenried 94, 20251 Hamburg, Germany.
| |
Collapse
|
4
|
Fermin ASR, Sasaoka T, Maekawa T, Ono K, Chan HL, Yamawaki S. Insula-cortico-subcortical networks predict interoceptive awareness and stress resilience. Asian J Psychiatr 2024; 95:103991. [PMID: 38484483 DOI: 10.1016/j.ajp.2024.103991] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/15/2023] [Revised: 02/25/2024] [Accepted: 02/28/2024] [Indexed: 05/13/2024]
Abstract
BACKGROUND Interoception, the neural sensing of visceral signals, and interoceptive awareness (IA), the conscious perception of interoception, are crucial for life survival functions and mental health. Resilience, the capacity to overcome adversity, has been associated with reduced interoceptive disturbances. Here, we sought evidence for our Insula Modular Active Control (IMAC) model that suggest that the insula, a brain region specialized in the processing of interoceptive information, realizes IA and contributes to resilience and mental health via cortico-subcortical connections. METHODS 64 healthy participants (32 females; ages 18-34 years) answered questionnaires that assess IA and resilience. Mental health was evaluated with the Beck Depression Inventory II that assesses depressive mood. Participants also underwent a 15 minute resting-state functional resonance imaging session. Pearson correlations and mediation analyses were used to investigate the relationship between IA and resilience and their contributions to depressive mood. We then performed insula seed-based functional connectivity analyzes to identify insula networks involved in IA, resilience and depressive mood. RESULTS We first demonstrated that resilience mediates the relationship between IA and depressive mood. Second, shared and distinct intra-insula, insula-cortical and insula-subcortical networks were associated with IA, resilience and also predicted the degree of experienced depressive mood. Third, while resilience was associated with stronger insula-precuneus, insula-cerebellum and insula-prefrontal networks, IA was linked with stronger intra-insula, insula-striatum and insula-motor networks. CONCLUSIONS Our findings help understand the roles of insula-cortico-subcortical networks in IA and resilience. These results also highlight the potential use of insula networks as biomarkers for depression prediction.
Collapse
Affiliation(s)
- Alan S R Fermin
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan.
| | - Takafumi Sasaoka
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Toru Maekawa
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Kentaro Ono
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Hui-Ling Chan
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Shigeto Yamawaki
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| |
Collapse
|
5
|
Mohebi A, Wei W, Pelattini L, Kim K, Berke JD. Dopamine transients follow a striatal gradient of reward time horizons. Nat Neurosci 2024; 27:737-746. [PMID: 38321294 PMCID: PMC11001583 DOI: 10.1038/s41593-023-01566-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Accepted: 12/21/2023] [Indexed: 02/08/2024]
Abstract
Animals make predictions to guide their behavior and update those predictions through experience. Transient increases in dopamine (DA) are thought to be critical signals for updating predictions. However, it is unclear how this mechanism handles a wide range of behavioral timescales-from seconds or less (for example, if singing a song) to potentially hours or more (for example, if hunting for food). Here we report that DA transients in distinct rat striatal subregions convey prediction errors based on distinct time horizons. DA dynamics systematically accelerated from ventral to dorsomedial to dorsolateral striatum, in the tempo of spontaneous fluctuations, the temporal integration of prior rewards and the discounting of future rewards. This spectrum of timescales for evaluative computations can help achieve efficient learning and adaptive motivation for a broad range of behaviors.
Collapse
Affiliation(s)
- Ali Mohebi
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Wei Wei
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Lilian Pelattini
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Kyoungjun Kim
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Joshua D Berke
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA.
- Department of Psychiatry and Behavioral Sciences, University of California San Francisco, San Francisco, CA, USA.
- Neuroscience Graduate Program, University of California San Francisco, San Francisco, CA, USA.
- Kavli Institute for Fundamental Neuroscience, University of California San Francisco, San Francisco, CA, USA.
- Weill Institute for Neurosciences, University of California San Francisco, San Francisco, CA, USA.
| |
Collapse
|
6
|
Ihara K, Shikano Y, Kato S, Yagishita S, Tanaka KF, Takata N. A reinforcement learning model with choice traces for a progressive ratio schedule. Front Behav Neurosci 2024; 17:1302842. [PMID: 38268795 PMCID: PMC10806202 DOI: 10.3389/fnbeh.2023.1302842] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Accepted: 12/13/2023] [Indexed: 01/26/2024] Open
Abstract
The progressive ratio (PR) lever-press task serves as a benchmark for assessing goal-oriented motivation. However, a well-recognized limitation of the PR task is that only a single data point, known as the breakpoint, is obtained from an entire session as a barometer of motivation. Because the breakpoint is defined as the final ratio of responses achieved in a PR session, variations in choice behavior during the PR task cannot be captured. We addressed this limitation by constructing four reinforcement learning models: a simple Q-learning model, an asymmetric model with two learning rates, a perseverance model with choice traces, and a perseverance model without learning. These models incorporated three behavioral choices: reinforced and non-reinforced lever presses and void magazine nosepokes, because we noticed that male mice performed frequent magazine nosepokes during PR tasks. The best model was the perseverance model, which predicted a gradual reduction in amplitudes of reward prediction errors (RPEs) upon void magazine nosepokes. We confirmed the prediction experimentally with fiber photometry of extracellular dopamine (DA) dynamics in the ventral striatum of male mice using a fluorescent protein (genetically encoded GPCR activation-based DA sensor: GRABDA2m). We verified application of the model by acute intraperitoneal injection of low-dose methamphetamine (METH) before a PR task, which increased the frequency of magazine nosepokes during the PR session without changing the breakpoint. The perseverance model captured behavioral modulation as a result of increased initial action values, which are customarily set to zero and disregarded in reinforcement learning analysis. Our findings suggest that the perseverance model reveals the effects of psychoactive drugs on choice behaviors during PR tasks.
Collapse
Affiliation(s)
- Keiko Ihara
- Division of Brain Sciences, Institute for Advanced Medical Research, Keio University School of Medicine, Tokyo, Japan
| | - Yu Shikano
- Division of Brain Sciences, Institute for Advanced Medical Research, Keio University School of Medicine, Tokyo, Japan
- Department of Biology, Stanford University, Stanford, CA, United States
| | - Sae Kato
- Division of Brain Sciences, Institute for Advanced Medical Research, Keio University School of Medicine, Tokyo, Japan
| | - Sho Yagishita
- Center for Disease Biology and Integrative Medicine, Faculty of Medicine, The University of Tokyo, Tokyo, Japan
| | - Kenji F. Tanaka
- Division of Brain Sciences, Institute for Advanced Medical Research, Keio University School of Medicine, Tokyo, Japan
| | - Norio Takata
- Division of Brain Sciences, Institute for Advanced Medical Research, Keio University School of Medicine, Tokyo, Japan
| |
Collapse
|
7
|
Lowet AS, Zheng Q, Meng M, Matias S, Drugowitsch J, Uchida N. An opponent striatal circuit for distributional reinforcement learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.02.573966. [PMID: 38260354 PMCID: PMC10802299 DOI: 10.1101/2024.01.02.573966] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/24/2024]
Abstract
Machine learning research has achieved large performance gains on a wide range of tasks by expanding the learning target from mean rewards to entire probability distributions of rewards - an approach known as distributional reinforcement learning (RL)1. The mesolimbic dopamine system is thought to underlie RL in the mammalian brain by updating a representation of mean value in the striatum2,3, but little is known about whether, where, and how neurons in this circuit encode information about higher-order moments of reward distributions4. To fill this gap, we used high-density probes (Neuropixels) to acutely record striatal activity from well-trained, water-restricted mice performing a classical conditioning task in which reward mean, reward variance, and stimulus identity were independently manipulated. In contrast to traditional RL accounts, we found robust evidence for abstract encoding of variance in the striatum. Remarkably, chronic ablation of dopamine inputs disorganized these distributional representations in the striatum without interfering with mean value coding. Two-photon calcium imaging and optogenetics revealed that the two major classes of striatal medium spiny neurons - D1 and D2 MSNs - contributed to this code by preferentially encoding the right and left tails of the reward distribution, respectively. We synthesize these findings into a new model of the striatum and mesolimbic dopamine that harnesses the opponency between D1 and D2 MSNs5-15 to reap the computational benefits of distributional RL.
Collapse
Affiliation(s)
- Adam S Lowet
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
- Program in Neuroscience, Harvard University, Boston, MA, USA
| | - Qiao Zheng
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Melissa Meng
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Sara Matias
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| | - Jan Drugowitsch
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Neurobiology, Harvard Medical School, Boston, MA, USA
| | - Naoshige Uchida
- Center for Brain Science, Harvard University, Cambridge, MA, USA
- Department of Molecular and Cellular Biology, Harvard University, Cambridge, MA, USA
| |
Collapse
|
8
|
Rios A, Nonomura S, Kato S, Yoshida J, Matsushita N, Nambu A, Takada M, Hira R, Kobayashi K, Sakai Y, Kimura M, Isomura Y. Reward expectation enhances action-related activity of nigral dopaminergic and two striatal output pathways. Commun Biol 2023; 6:914. [PMID: 37673949 PMCID: PMC10482957 DOI: 10.1038/s42003-023-05288-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2022] [Accepted: 08/25/2023] [Indexed: 09/08/2023] Open
Abstract
Neurons comprising nigrostriatal system play important roles in action selection. However, it remains unclear how this system integrates recent outcome information with current action (movement) and outcome (reward or no reward) information to achieve appropriate subsequent action. We examined how neuronal activity of substantia nigra pars compacta (SNc) and dorsal striatum reflects the level of reward expectation from recent outcomes in rats performing a reward-based choice task. Movement-related activity of direct and indirect pathway striatal projection neurons (dSPNs and iSPNs, respectively) were enhanced by reward expectation, similarly to the SNc dopaminergic neurons, in both medial and lateral nigrostriatal projections. Given the classical basal ganglia model wherein dopamine stimulates dSPNs and suppresses iSPNs through distinct dopamine receptors, dopamine might not be the primary driver of iSPN activity increasing following higher reward expectation. In contrast, outcome-related activity was affected by reward expectation in line with the classical model and reinforcement learning theory, suggesting purposive effects of reward expectation.
Collapse
Affiliation(s)
- Alain Rios
- Department of Physiology and Cell Biology, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.
| | - Satoshi Nonomura
- Department of Physiology and Cell Biology, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Aichi, 484-8506, Japan
| | - Shigeki Kato
- Department of Molecular Genetics, Institute of Biomedical Science, Fukushima Medical University, Fukushima, 960-1295, Japan
| | - Junichi Yoshida
- Dominick P. Purpura Department of Neuroscience, Albert Einstein College of Medicine, Bronx, NY, 10461, USA
| | - Natsuki Matsushita
- Division of Laboratory Animal Research, Aichi Medical University, Aichi, 480-1195, Japan
| | - Atsushi Nambu
- Division of System Neurophysiology, National Institute of Physiological Sciences and Department of Physiological Sciences, SOKENDAI, Aichi, 444-8585, Japan
| | - Masahiko Takada
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Aichi, 484-8506, Japan
| | - Riichiro Hira
- Department of Physiology and Cell Biology, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan
| | - Kazuto Kobayashi
- Department of Molecular Genetics, Institute of Biomedical Science, Fukushima Medical University, Fukushima, 960-1295, Japan
| | - Yutaka Sakai
- Brain Science Institute, Tamagawa University, Tokyo, 194-8610, Japan
| | - Minoru Kimura
- Brain Science Institute, Tamagawa University, Tokyo, 194-8610, Japan
| | - Yoshikazu Isomura
- Department of Physiology and Cell Biology, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University (TMDU), Tokyo, 113-8510, Japan.
- Brain Science Institute, Tamagawa University, Tokyo, 194-8610, Japan.
| |
Collapse
|
9
|
Blackwell KT, Doya K. Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks. PLoS Comput Biol 2023; 19:e1011385. [PMID: 37594982 PMCID: PMC10479916 DOI: 10.1371/journal.pcbi.1011385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 09/05/2023] [Accepted: 07/25/2023] [Indexed: 08/20/2023] Open
Abstract
A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.
Collapse
Affiliation(s)
- Kim T Blackwell
- Department of Bioengineering, Volgenau School of Engineering, George Mason University, Fairfax, Virginia, United States of America
| | - Kenji Doya
- Neural Computation Unit, Okinawa Institute of Science and Technology Graduate University, Okinawa, Japan
| |
Collapse
|
10
|
Nishioka T, Attachaipanich S, Hamaguchi K, Lazarus M, de Kerchove d'Exaerde A, Macpherson T, Hikida T. Error-related signaling in nucleus accumbens D2 receptor-expressing neurons guides inhibition-based choice behavior in mice. Nat Commun 2023; 14:2284. [PMID: 37085502 PMCID: PMC10121661 DOI: 10.1038/s41467-023-38025-3] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2021] [Accepted: 04/12/2023] [Indexed: 04/23/2023] Open
Abstract
Learned associations between environmental cues and the outcomes they predict (cue-outcome associations) play a major role in behavioral control, guiding not only which responses we should perform, but also which we should inhibit, in order to achieve a specific goal. The encoding of such cue-outcome associations, as well as the performance of cue-guided choice behavior, is thought to involve dopamine D1 and D2 receptor-expressing medium spiny neurons (D1-/D2-MSNs) of the nucleus accumbens (NAc). Here, using a visual discrimination task in male mice, we assessed the role of NAc D1-/D2-MSNs in cue-guided inhibition of inappropriate responding. Cell-type specific neuronal silencing and in-vivo imaging revealed NAc D2-MSNs to contribute to inhibiting behavioral responses, with activation of NAc D2-MSNs following response errors playing an important role in optimizing future choice behavior. Our findings indicate that error-signaling by NAc D2-MSNs contributes to the ability to use environmental cues to inhibit inappropriate behavior.
Collapse
Affiliation(s)
- Tadaaki Nishioka
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Suita, Japan.
- Laboratory for Developing Minds, Icahn School of Medicine at Mount Sinai, New York, NY, USA.
| | - Suthinee Attachaipanich
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Suita, Japan
| | - Kosuke Hamaguchi
- Department of Biological Sciences, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Michael Lazarus
- International Institute for Integrative Sleep Medicine (WPI-IIIS) and Faculty of Medicine, University of Tsukuba, Tsukuba, Japan
| | | | - Tom Macpherson
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Suita, Japan.
| | - Takatoshi Hikida
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Suita, Japan.
| |
Collapse
|
11
|
Choi K, Piasini E, Díaz-Hernández E, Cifuentes LV, Henderson NT, Holly EN, Subramaniyan M, Gerfen CR, Fuccillo MV. Distributed processing for value-based choice by prelimbic circuits targeting anterior-posterior dorsal striatal subregions in male mice. Nat Commun 2023; 14:1920. [PMID: 37024449 PMCID: PMC10079960 DOI: 10.1038/s41467-023-36795-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2022] [Accepted: 02/17/2023] [Indexed: 04/08/2023] Open
Abstract
Fronto-striatal circuits have been implicated in cognitive control of behavioral output for social and appetitive rewards. The functional diversity of prefrontal cortical populations is strongly dependent on their synaptic targets, with control of motor output mediated by connectivity to dorsal striatum. Despite evidence for functional diversity along the anterior-posterior striatal axis, it is unclear how distinct fronto-striatal sub-circuits support value-based choice. Here we found segregated prefrontal populations defined by anterior/posterior dorsomedial striatal target. During a feedback-based 2-alternative choice task, single-photon imaging revealed circuit-specific representations of task-relevant information with prelimbic neurons targeting anterior DMS (PL::A-DMS) robustly modulated during choices and negative outcomes, while prelimbic neurons targeting posterior DMS (PL::P-DMS) encoded internal representations of value and positive outcomes contingent on prior choice. Consistent with this distributed coding, optogenetic inhibition of PL::A-DMS circuits strongly impacted choice monitoring and responses to negative outcomes while inhibition of PL::P-DMS impaired task engagement and strategies following positive outcomes. Together our data uncover PL populations engaged in distributed processing for value-based choice.
Collapse
Affiliation(s)
- Kyuhyun Choi
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Eugenio Piasini
- Computational Neuroscience Initiative, University of Pennsylvania, Philadelphia, PA, USA
- Neural Computation Lab, International School for Advanced Studies (SISSA), Trieste, Italy
| | - Edgar Díaz-Hernández
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Luigim Vargas Cifuentes
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Neuroscience Graduate Group, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Nathan T Henderson
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Elizabeth N Holly
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Manivannan Subramaniyan
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Charles R Gerfen
- Laboratory of Systems Neuroscience, National Institute of Mental Health (NIMH), Bethesda, MD, USA
| | - Marc V Fuccillo
- Department of Neuroscience, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA.
| |
Collapse
|
12
|
Prospective and retrospective values integrated in frontal cortex drive predictive choice. Proc Natl Acad Sci U S A 2022; 119:e2206067119. [PMID: 36417435 PMCID: PMC9889848 DOI: 10.1073/pnas.2206067119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
To make a deliberate action in a volatile environment, the brain must frequently reassess the value of each action (action-value). Choice can be initially made from the experience of trial-and-errors, but once the dynamics of the environment is learned, the choice can be made from the knowledge of the environment. The action-values constructed from the experience (retrospective value) and the ones from the knowledge (prospective value) were identified in various regions of the brain. However, how and which neural circuit integrates these values and executes the chosen action remains unknown. Combining reinforcement learning and two-photon calcium imaging, we found that the preparatory activity of neurons in a part of the frontal cortex, the anterior-lateral motor (ALM) area, initially encodes retrospective value, but after extensive training, they jointly encode the retrospective and prospective value. Optogenetic inhibition of ALM preparatory activity specifically abolished the expert mice's predictive choice behavior and returned them to the novice-like state. Thus, the integrated action-value encoded in the preparatory activity of ALM plays an important role to bias the action toward the knowledge-dependent, predictive choice behavior.
Collapse
|
13
|
Transcriptomic analysis in the striatum reveals the involvement of Nurr1 in the social behavior of prenatally valproic acid-exposed male mice. Transl Psychiatry 2022; 12:324. [PMID: 35945212 PMCID: PMC9363495 DOI: 10.1038/s41398-022-02056-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/07/2022] [Revised: 06/23/2022] [Accepted: 07/01/2022] [Indexed: 11/30/2022] Open
Abstract
Autism spectrum disorder (ASD) is a neurodevelopmental disorder that exhibits neurobehavioral deficits characterized by abnormalities in social interactions, deficits in communication as well as restricted interests, and repetitive behaviors. The basal ganglia is one of the brain regions implicated as dysfunctional in ASD. In particular, the defects in corticostriatal function have been reported to be involved in the pathogenesis of ASD. Surface deformation of the striatum in the brains of patients with ASD and their correlation with behavioral symptoms was reported in magnetic resonance imaging (MRI) studies. We demonstrated that prenatal valproic acid (VPA) exposure induced synaptic and molecular changes and decreased neuronal activity in the striatum. Using RNA sequencing (RNA-Seq), we analyzed transcriptome alterations in striatal tissues from 10-week-old prenatally VPA-exposed BALB/c male mice. Among the upregulated genes, Nurr1 was significantly upregulated in striatal tissues from prenatally VPA-exposed mice. Viral knockdown of Nurr1 by shRNA significantly rescued the reduction in dendritic spine density and the number of mature dendritic spines in the striatum and markedly improved social deficits in prenatally VPA-exposed mice. In addition, treatment with amodiaquine, which is a known ligand for Nurr1, mimicked the social deficits and synaptic abnormalities in saline-exposed mice as observed in prenatally VPA-exposed mice. Furthermore, PatDp+/- mice, a commonly used ASD genetic mouse model, also showed increased levels of Nurr1 in the striatum. Taken together, these results suggest that the increase in Nurr1 expression in the striatum is a mechanism related to the changes in synaptic deficits and behavioral phenotypes of the VPA-induced ASD mouse model.
Collapse
|
14
|
Stiers P, Goulas A. Task-specific subnetworks extend from prefrontal cortex to striatum. Cortex 2022; 156:106-125. [DOI: 10.1016/j.cortex.2022.06.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Revised: 02/23/2022] [Accepted: 06/07/2022] [Indexed: 11/29/2022]
|
15
|
Fermin ASR, Friston K, Yamawaki S. An insula hierarchical network architecture for active interoceptive inference. ROYAL SOCIETY OPEN SCIENCE 2022; 9:220226. [PMID: 35774133 PMCID: PMC9240682 DOI: 10.1098/rsos.220226] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 06/09/2022] [Indexed: 05/05/2023]
Abstract
In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.
Collapse
Affiliation(s)
- Alan S. R. Fermin
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Karl Friston
- The Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London, England
| | - Shigeto Yamawaki
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| |
Collapse
|
16
|
Taniguchi T, Yamakawa H, Nagai T, Doya K, Sakagami M, Suzuki M, Nakamura T, Taniguchi A. A whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots. Neural Netw 2022; 150:293-312. [PMID: 35339010 DOI: 10.1016/j.neunet.2022.02.026] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 02/25/2022] [Accepted: 02/28/2022] [Indexed: 01/08/2023]
Abstract
Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information. In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.
Collapse
Affiliation(s)
| | - Hiroshi Yamakawa
- The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, Japan; The Whole Brain Architecture Initiative, 2-19-21 Nishikoiwa , Edogawa-ku, Tokyo, Japan; RIKEN, 6-2-3 Furuedai, Suita, Osaka, Japan
| | - Takayuki Nagai
- Osaka University, 1-3 Machikane-yama, Toyonaka, Osaka, Japan
| | - Kenji Doya
- Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Kunigami, Okinawa, Japan
| | | | - Masahiro Suzuki
- The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, Japan
| | - Tomoaki Nakamura
- The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, Japan
| | | |
Collapse
|
17
|
K Namboodiri VM, Stuber GD. The learning of prospective and retrospective cognitive maps within neural circuits. Neuron 2021; 109:3552-3575. [PMID: 34678148 PMCID: PMC8809184 DOI: 10.1016/j.neuron.2021.09.034] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/02/2021] [Revised: 08/26/2021] [Accepted: 09/16/2021] [Indexed: 11/18/2022]
Abstract
Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.
Collapse
Affiliation(s)
- Vijay Mohan K Namboodiri
- Department of Neurology, Center for Integrative Neuroscience, Kavli Institute for Fundamental Neuroscience, Neuroscience Graduate Program, University of California, San Francisco, San Francisco, CA 94158, USA.
| | - Garret D Stuber
- Center for the Neurobiology of Addiction, Pain, and Emotion, Department of Anesthesiology and Pain Medicine, Department of Pharmacology, Neuroscience Graduate Program, University of Washington, Seattle, WA 98195, USA.
| |
Collapse
|
18
|
Vandaele Y, Ottenheimer DJ, Janak PH. Dorsomedial Striatal Activity Tracks Completion of Behavioral Sequences in Rats. eNeuro 2021; 8:ENEURO.0279-21.2021. [PMID: 34725103 PMCID: PMC8607909 DOI: 10.1523/eneuro.0279-21.2021] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2021] [Revised: 09/24/2021] [Accepted: 10/13/2021] [Indexed: 11/21/2022] Open
Abstract
For proper execution of goal-directed behaviors, individuals require both a general representation of the goal and an ability to monitor their own progress toward that goal. Here, we examine how dorsomedial striatum (DMS), a region pivotal for forming associations among stimuli, actions, and outcomes, encodes the execution of goal-directed action sequences that require self-monitoring of behavior. We trained rats to complete a sequence of at least five consecutive lever presses (without visiting the reward port) to obtain a reward and recorded the activity of individual cells in DMS while rats performed the task. We found that the pattern of DMS activity gradually changed during the execution of the sequence, permitting accurate decoding of sequence progress from neural activity at a population level. Moreover, this sequence-related activity was blunted on trials where rats did not complete a sufficient number of presses. Overall, these data suggest a link between DMS activity and the execution of behavioral sequences that require monitoring of ongoing behavior.
Collapse
Affiliation(s)
- Youna Vandaele
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD 21218
| | - David J Ottenheimer
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD 21218
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, MD 21205
| | - Patricia H Janak
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, MD 21218
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, MD 21205
| |
Collapse
|
19
|
Oberto VJ, Boucly CJ, Gao H, Todorova R, Zugaro MB, Wiener SI. Distributed cell assemblies spanning prefrontal cortex and striatum. Curr Biol 2021; 32:1-13.e6. [PMID: 34699783 DOI: 10.1016/j.cub.2021.10.007] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Revised: 09/03/2021] [Accepted: 10/04/2021] [Indexed: 12/26/2022]
Abstract
Highly synchronous neuronal assembly activity is deemed essential for cognitive brain function. In theory, such synchrony could coordinate multiple brain areas performing complementary processes. However, cell assemblies have been observed only in single structures, typically cortical areas, and little is known about their synchrony with downstream subcortical structures, such as the striatum. Here, we demonstrate distributed cell assemblies activated at high synchrony (∼10 ms) spanning prefrontal cortex and striatum. In addition to including neurons at different brain hierarchical levels, surprisingly, they synchronized functionally distinct limbic and associative sub-regions. These assembly activations occurred when members shifted their firing phase relative to ongoing 4 Hz and theta rhythms, in association with high gamma oscillations. This suggests that these rhythms could mediate the emergence of cross-structural assemblies. To test for the role of assemblies in behavior, we trained the rats to perform a task requiring cognitive flexibility, alternating between two different rules in a T-maze. Overall, assembly activations were correlated with task-relevant parameters, including impending choice, reward, rule, or rule order. Moreover, these behavioral correlates were more robustly expressed by assemblies than by their individual member neurons. Finally, to verify whether assemblies can be endogenously generated, we found that they were indeed spontaneously reactivated during sleep and quiet immobility. Thus, cell assemblies are a more general coding mechanism than previously envisioned, linking distributed neocortical and subcortical areas at high synchrony.
Collapse
Affiliation(s)
- Virginie J Oberto
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | - Céline J Boucly
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | - HongYing Gao
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | - Ralitsa Todorova
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | - Michaël B Zugaro
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France
| | - Sidney I Wiener
- Center for Interdisciplinary Research in Biology (CIRB), College de France, CNRS, INSERM, Université PSL, Paris, France.
| |
Collapse
|
20
|
Zambrano D, Roelfsema PR, Bohte S. Learning continuous-time working memory tasks with on-policy neural reinforcement learning. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.11.072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
21
|
Cunningham PJ, Regier PS, Redish AD. Dorsolateral Striatal Task-initiation Bursts Represent Past Experiences More than Future Action Plans. J Neurosci 2021; 41:8051-8064. [PMID: 34376584 PMCID: PMC8460149 DOI: 10.1523/jneurosci.3080-20.2021] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2020] [Revised: 07/08/2021] [Accepted: 08/04/2021] [Indexed: 11/21/2022] Open
Abstract
The dorsolateral striatum (DLS) is involved in learning and executing procedural actions. Cell ensembles in the DLS, but not the dorsomedial striatum (DMS), exhibit a burst of firing at the start of a well-learned action sequence ("task-bracketing"). However, it is currently unclear what information is contained in these bursts. Some theories suggest that these bursts should represent the procedural action sequence itself (that they should be about future action chains), whereas others suggest that they should contain representations of the current state of the world, taking into account primarily past information. In addition, the DLS local field potential shows transient bursts of power in the 50 Hz range (γ50) around the time a learned action sequence is initiated. However, it is currently unknown how bursts of activity in DLS cell ensembles and bursts of γ50 power in the DLS local field potential are related to each other. We found that DLS bursts at lap initiation in rats represented recently experienced reward locations more than future procedural actions, indicating that task-initiation DLS bursts contain primarily retrospective, rather than prospective, information to guide procedural actions. Furthermore, representations of past reward locations increased during periods of increased γ50 power in the DLS. There was no evidence of task-initiation bursts, increased γ50 power, or retrospective reward location information in the neighboring dorsomedial striatum. These data support a role for the DLS in model-free theories of procedural decision-making over planned action-chain theories, suggesting that procedural actions derive from representations of the current and recent past.SIGNIFICANCE STATEMENT While it is well-established that the dorsolateral striatum (DLS) plays a critical role in procedural decision-making, open questions remain about the kinds of representations contained in DLS ensemble activity that guide procedural actions. We found that DLS, but not DMS, cell ensembles contained nonlocal representations of past reward locations that appear moments before task-initiation DLS bursts. These retrospective representations were temporally linked to a rise in γ50 power that also preceded the characteristic DLS burst at task-initiation. These results support models of procedural decision-making based on associations between available actions and the current state of the world over models based on planning over action-chains.
Collapse
Affiliation(s)
- Paul J Cunningham
- Department of Neuroscience, University of Minnesota, Minneapolis MN 55455
| | - Paul S Regier
- Department of Psychiatry, University of Pennsylvania Philadelphia PA 19104
| | - A David Redish
- Department of Neuroscience, University of Minnesota, Minneapolis MN 55455
| |
Collapse
|
22
|
Funamizu A. Integration of sensory evidence and reward expectation in mouse perceptual decision-making task with various sensory uncertainties. iScience 2021; 24:102826. [PMID: 34355152 PMCID: PMC8319806 DOI: 10.1016/j.isci.2021.102826] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2021] [Revised: 06/07/2021] [Accepted: 07/05/2021] [Indexed: 11/16/2022] Open
Abstract
In perceptual decision-making, prior knowledge of action outcomes is essential, especially when sensory inputs are insufficient for proper choices. Signal detection theory (SDT) shows that optimal choice bias depends not only on the prior but also the sensory uncertainty; however, it is unclear how animals integrate sensory inputs with various uncertainties and reward expectations to optimize choices. We developed a tone-frequency discrimination task for head-fixed mice in which we randomly presented either a long or short sound stimulus and biased the choice outcomes. The choice was less accurate and more biased toward the large-reward side in short- than in long-stimulus trials. Analysis with SDT found that mice did not use a separate, optimal choice threshold in different sound durations. Instead, mice updated one threshold for short and long stimuli with a simple reinforcement-learning rule. Our task in head-fixed mice helps understanding how the brain integrates sensory inputs and prior.
Collapse
Affiliation(s)
- Akihiro Funamizu
- Institute for Quantitative Biosciences, University of Tokyo, Laboratory of Neural Computation, 1-1-1 Yayoi, Bunkyo-ku, Tokyo 113-0032, Japan
- Department of Life Sciences, Graduate School of Arts and Sciences, University of Tokyo, Meguro-ku, Tokyo 153-8902, Japan
| |
Collapse
|
23
|
Bari BA, Cohen JY. Dynamic decision making and value computations in medial frontal cortex. INTERNATIONAL REVIEW OF NEUROBIOLOGY 2021; 158:83-113. [PMID: 33785157 DOI: 10.1016/bs.irn.2020.12.001] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
Abstract
Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.
Collapse
Affiliation(s)
- Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, MD, United States
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, MD, United States.
| |
Collapse
|
24
|
Siju KP, De Backer JF, Grunwald Kadow IC. Dopamine modulation of sensory processing and adaptive behavior in flies. Cell Tissue Res 2021; 383:207-225. [PMID: 33515291 PMCID: PMC7873103 DOI: 10.1007/s00441-020-03371-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/16/2020] [Accepted: 11/26/2020] [Indexed: 12/31/2022]
Abstract
Behavioral flexibility for appropriate action selection is an advantage when animals are faced with decisions that will determine their survival or death. In order to arrive at the right decision, animals evaluate information from their external environment, internal state, and past experiences. How these different signals are integrated and modulated in the brain, and how context- and state-dependent behavioral decisions are controlled are poorly understood questions. Studying the molecules that help convey and integrate such information in neural circuits is an important way to approach these questions. Many years of work in different model organisms have shown that dopamine is a critical neuromodulator for (reward based) associative learning. However, recent findings in vertebrates and invertebrates have demonstrated the complexity and heterogeneity of dopaminergic neuron populations and their functional implications in many adaptive behaviors important for survival. For example, dopaminergic neurons can integrate external sensory information, internal and behavioral states, and learned experience in the decision making circuitry. Several recent advances in methodologies and the availability of a synaptic level connectome of the whole-brain circuitry of Drosophila melanogaster make the fly an attractive system to study the roles of dopamine in decision making and state-dependent behavior. In particular, a learning and memory center-the mushroom body-is richly innervated by dopaminergic neurons that enable it to integrate multi-modal information according to state and context, and to modulate decision-making and behavior.
Collapse
Affiliation(s)
- K. P. Siju
- School of Life Sciences, Department of Molecular Life Sciences, Technical University of Munich, 85354 Freising, Germany
| | - Jean-Francois De Backer
- School of Life Sciences, Department of Molecular Life Sciences, Technical University of Munich, 85354 Freising, Germany
| | - Ilona C. Grunwald Kadow
- School of Life Sciences, Department of Molecular Life Sciences, Technical University of Munich, 85354 Freising, Germany
| |
Collapse
|
25
|
Collins AGE, Cockburn J. Beyond dichotomies in reinforcement learning. Nat Rev Neurosci 2020; 21:576-586. [PMID: 32873936 DOI: 10.1038/s41583-020-0355-6] [Citation(s) in RCA: 46] [Impact Index Per Article: 11.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/20/2020] [Indexed: 11/09/2022]
Abstract
Reinforcement learning (RL) is a framework of particular importance to psychology, neuroscience and machine learning. Interactions between these fields, as promoted through the common hub of RL, has facilitated paradigm shifts that relate multiple levels of analysis in a singular framework (for example, relating dopamine function to a computationally defined RL signal). Recently, more sophisticated RL algorithms have been proposed to better account for human learning, and in particular its oft-documented reliance on two separable systems: a model-based (MB) system and a model-free (MF) system. However, along with many benefits, this dichotomous lens can distort questions, and may contribute to an unnecessarily narrow perspective on learning and decision-making. Here, we outline some of the consequences that come from overconfidently mapping algorithms, such as MB versus MF RL, with putative cognitive processes. We argue that the field is well positioned to move beyond simplistic dichotomies, and we propose a means of refocusing research questions towards the rich and complex components that comprise learning and decision-making.
Collapse
Affiliation(s)
- Anne G E Collins
- Department of Psychology and the Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, CA, USA.
| | - Jeffrey Cockburn
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
26
|
Shiotani K, Tanisumi Y, Murata K, Hirokawa J, Sakurai Y, Manabe H. Tuning of olfactory cortex ventral tenia tecta neurons to distinct task elements of goal-directed behavior. eLife 2020; 9:57268. [PMID: 32749216 PMCID: PMC7423337 DOI: 10.7554/elife.57268] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2020] [Accepted: 08/01/2020] [Indexed: 01/22/2023] Open
Abstract
The ventral tenia tecta (vTT) is a component of the olfactory cortex and receives both bottom-up odor signals and top-down signals. However, the roles of the vTT in odor-coding and integration of inputs are poorly understood. Here, we investigated the involvement of the vTT in these processes by recording the activity from individual vTT neurons during the performance of learned odor-guided reward-directed tasks in mice. We report that individual vTT cells are highly tuned to a specific behavioral epoch of learned tasks, whereby the duration of increased firing correlated with the temporal length of the behavioral epoch. The peak time for increased firing among recorded vTT cells encompassed almost the entire temporal window of the tasks. Collectively, our results indicate that vTT cells are selectively activated during a specific behavioral context and that the function of the vTT changes dynamically in a context-dependent manner during goal-directed behaviors.
Collapse
Affiliation(s)
- Kazuki Shiotani
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan.,Research Fellow of the Japan Society for the Promotion of Science, Tokyo, Japan
| | - Yuta Tanisumi
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan.,Research Fellow of the Japan Society for the Promotion of Science, Tokyo, Japan
| | - Koshi Murata
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan.,Division of Brain Structure and Function, Faculty of Medical Sciences, University of Fukui, Fukui, Japan
| | - Junya Hirokawa
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan
| | - Yoshio Sakurai
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan
| | - Hiroyuki Manabe
- Laboratory of Neural Information, Graduate School of Brain Science, Doshisha University, Kyoto, Japan
| |
Collapse
|
27
|
Yang L, Masmanidis SC. Differential encoding of action selection by orbitofrontal and striatal population dynamics. J Neurophysiol 2020; 124:634-644. [PMID: 32727312 PMCID: PMC7500377 DOI: 10.1152/jn.00316.2020] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
Survival relies on the ability to flexibly choose between different actions according to varying environmental circumstances. Many lines of evidence indicate that action selection involves signaling in corticostriatal circuits, including the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS). While choice-specific responses have been found in individual neurons from both areas, it is unclear whether populations of OFC or DMS neurons are better at encoding an animal's choice. To address this, we trained head-fixed mice to perform an auditory guided two-alternative choice task, which required moving a joystick forward or backward. We then used silicon microprobes to simultaneously measure the spiking activity of OFC and DMS ensembles, allowing us to directly compare population dynamics between these areas within the same animals. Consistent with previous literature, both areas contained neurons that were selective for specific stimulus-action associations. However, analysis of concurrently recorded ensemble activity revealed that the animal's trial-by-trial behavior could be decoded more accurately from DMS dynamics. These results reveal substantial regional differences in encoding action selection, suggesting that DMS neural dynamics are more specialized than OFC at representing an animal's choice of action.NEW & NOTEWORTHY While previous literature shows that both orbitofrontal cortex (OFC) and dorsomedial striatum (DMS) represent information relevant to selecting specific actions, few studies have directly compared neural signals between these areas. Here we compared OFC and DMS dynamics in mice performing a two-alternative choice task. We found that the animal's choice could be decoded more accurately from DMS population activity. This work provides among the first evidence that OFC and DMS differentially represent information about an animal's selected action.
Collapse
Affiliation(s)
- Long Yang
- Department of Neurobiology, University of California Los Angeles, Los Angeles, California
| | - Sotiris C Masmanidis
- Department of Neurobiology, University of California Los Angeles, Los Angeles, California
- California Nanosystems Institute, University of California Los Angeles, Los Angeles, California
| |
Collapse
|
28
|
Guillem K, Ahmed SH. Reorganization of theta phase-locking in the orbitofrontal cortex drives cocaine choice under the influence. Sci Rep 2020; 10:8041. [PMID: 32415278 PMCID: PMC7228935 DOI: 10.1038/s41598-020-64962-w] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2020] [Accepted: 04/22/2020] [Indexed: 12/30/2022] Open
Abstract
Cortical theta oscillations of neuronal activity are a fundamental mechanism driving goal-directed behavior. We previously identified in the rat orbitofrontal cortex (OFC) a neuronal correlate of individual preferences between cocaine use and an alternative nondrug reward (i.e. saccharin). Whether theta oscillations are also associated with choice behavior between a drug and a nondrug reward remains unknown. Here we investigated the temporal structure between single unit activity and theta band oscillations (4-12 Hz) in the OFC of rats choosing between cocaine and saccharin. First, we found that the relative amplitude of theta oscillations is associated with subjective value and preference between two rewards. Second, OFC phase-locked neurons fired on opposite phase of the theta oscillation during saccharin and cocaine rewards, suggesting the existence of two separable neuronal assemblies. Finally, the pharmacological influence of cocaine at the moment of choice altered both theta band power and theta phase-locking in the OFC. That is, this drug influence shifted spike-phase relative to theta cycle and decreased the synchronization of OFC neurons relative to the theta oscillation. Overall, this study indicates that the reorganization of theta phase-locking under the influence of cocaine biases OFC neuronal assemblies in favor of cocaine choice and at the expense of a normally preferred alternative, a neuronal change that may contribute to drug preference in cocaine addiction.
Collapse
Affiliation(s)
- Karine Guillem
- Université de Bordeaux, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000, Bordeaux, France. .,CNRS, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000, Bordeaux, France.
| | - Serge H Ahmed
- Université de Bordeaux, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000, Bordeaux, France.,CNRS, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000, Bordeaux, France
| |
Collapse
|
29
|
Brand Z, Avital A. High resolution behavioral and neural activity representation using a geometrical approach. Sci Rep 2020; 10:7977. [PMID: 32409747 PMCID: PMC7224390 DOI: 10.1038/s41598-020-64726-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2019] [Accepted: 04/21/2020] [Indexed: 11/09/2022] Open
Abstract
Available tools for recording neuronal activity are limited and reductive due to massive data arising from high-frequency measurements. We have developed a method that utilizes variance within the physiological activity and includes all data points per measurement. Data is expressed geometrically in a physiologically meaningful manner, to represent a precise and detailed view of the recorded neural activity. The recorded raw data from any pair of electrodes was plotted and following a covariance calculation, an eigenvalues and chi-square distribution were used to define the ellipse which bounds 95% of the raw data. We validated our method by correlating specific behavioral observation and physiological activity with behavioral tasks that require similar body movement but potentially involve significantly different neuronal activity. Graphical representation of telemetrically recorded data generates a scatter plot with distinct elliptic geometrical properties that clearly and significantly correlated with behavior. Our reproducible approach improves on existing methods by allowing a dynamic, accurate and comprehensive correlate using an intuitive output. Long-term, it may serve as the basis for advanced machine learning applications and animal-based artificial intelligence models aimed at predicting or characterizing behavior.
Collapse
Affiliation(s)
- Zev Brand
- Behavioral Neuroscience lab, Gutwirth Building, Department of Neuroscience, Faculty of Medicine and Emek Medical Center, Technion - Israel Institute of Technology, Haifa, 32000, Israel
| | - Avi Avital
- Behavioral Neuroscience lab, Gutwirth Building, Department of Neuroscience, Faculty of Medicine and Emek Medical Center, Technion - Israel Institute of Technology, Haifa, 32000, Israel.
| |
Collapse
|
30
|
Gasser J, Pereira de Vasconcelos A, Cosquer B, Boutillier AL, Cassel JC. Shifting between response and place strategies in maze navigation: Effects of training, cue availability and functional inactivation of striatum or hippocampus in rats. Neurobiol Learn Mem 2020; 167:107131. [DOI: 10.1016/j.nlm.2019.107131] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2017] [Revised: 11/15/2019] [Accepted: 11/25/2019] [Indexed: 11/24/2022]
|
31
|
Frömer R, Dean Wolf CK, Shenhav A. Goal congruency dominates reward value in accounting for behavioral and neural correlates of value-based decision-making. Nat Commun 2019; 10:4926. [PMID: 31664035 PMCID: PMC6820735 DOI: 10.1038/s41467-019-12931-x] [Citation(s) in RCA: 34] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 10/08/2019] [Indexed: 12/22/2022] Open
Abstract
When choosing between options, whether menu items or career paths, we can evaluate how rewarding each one will be, or how congruent it is with our current choice goal (e.g., to point out the best option or the worst one.). Past decision-making research interpreted findings through the former lens, but in these experiments the most rewarding option was always most congruent with the task goal (choosing the best option). It is therefore unclear to what extent expected reward vs. goal congruency can account for choice value findings. To deconfound these two variables, we performed three behavioral studies and an fMRI study in which the task goal varied between identifying the best vs. the worst option. Contrary to prevailing accounts, we find that goal congruency dominates choice behavior and neural activity. We separately identify dissociable signals of expected reward. Our findings call for a reinterpretation of previous research on value-based choice. Decision-making research has confounded the reward value of options with their goal-congruency, as the task goal was always to pick the most rewarding option. Here, authors separately asked participants to select the least rewarding of a set of options, revealing a dominant role for goal congruency.
Collapse
Affiliation(s)
- Romy Frömer
- Cognitive, Linguistic, and Psychological Sciences, Carney Institute for Brain Science, Brown University, Providence, RI, USA.
| | - Carolyn K Dean Wolf
- Cognitive, Linguistic, and Psychological Sciences, Carney Institute for Brain Science, Brown University, Providence, RI, USA
| | - Amitai Shenhav
- Cognitive, Linguistic, and Psychological Sciences, Carney Institute for Brain Science, Brown University, Providence, RI, USA.
| |
Collapse
|
32
|
Vandaele Y, Mahajan NR, Ottenheimer DJ, Richard JM, Mysore SP, Janak PH. Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training. eLife 2019; 8:49536. [PMID: 31621583 PMCID: PMC6822989 DOI: 10.7554/elife.49536] [Citation(s) in RCA: 43] [Impact Index Per Article: 8.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2019] [Accepted: 10/16/2019] [Indexed: 12/20/2022] Open
Abstract
Hypotheses of striatal orchestration of behavior ascribe distinct functions to striatal subregions, with the dorsolateral striatum (DLS) especially implicated in habitual and skilled performance. Thus neural activity patterns recorded from the DLS, but not the dorsomedial striatum (DMS), should be correlated with habitual and automatized performance. Here, we recorded DMS and DLS neural activity in rats during training in a task promoting habitual lever pressing. Despite improving performance across sessions, clear changes in corresponding neural activity patterns were not evident in DMS or DLS during early training. Although DMS and DLS activity patterns were distinct during early training, their activity was similar following extended training. Finally, performance after extended training was not associated with DMS disengagement, as would be predicted from prior work. These results suggest that behavioral sequences may continue to engage both striatal regions long after initial acquisition, when skilled performance is consolidated.
Collapse
Affiliation(s)
- Youna Vandaele
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, United States
| | - Nagaraj R Mahajan
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, United States
| | - David J Ottenheimer
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, United States
| | - Jocelyn M Richard
- Department of Neuroscience, University of Minnesota, Minneapolis, United States
| | - Shreesh P Mysore
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, United States.,The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, United States
| | - Patricia H Janak
- Department of Psychological and Brain Sciences, Krieger School of Arts and Sciences, Johns Hopkins University, Baltimore, United States.,The Solomon H. Snyder Department of Neuroscience, Johns Hopkins School of Medicine, Johns Hopkins University, Baltimore, United States.,Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, United States
| |
Collapse
|
33
|
Rusu SI, Pennartz CMA. Learning, memory and consolidation mechanisms for behavioral control in hierarchically organized cortico-basal ganglia systems. Hippocampus 2019; 30:73-98. [PMID: 31617622 PMCID: PMC6972576 DOI: 10.1002/hipo.23167] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2018] [Revised: 09/09/2019] [Accepted: 09/11/2019] [Indexed: 01/05/2023]
Abstract
This article aims to provide a synthesis on the question how brain structures cooperate to accomplish hierarchically organized behaviors, characterized by low‐level, habitual routines nested in larger sequences of planned, goal‐directed behavior. The functioning of a connected set of brain structures—prefrontal cortex, hippocampus, striatum, and dopaminergic mesencephalon—is reviewed in relation to two important distinctions: (a) goal‐directed as opposed to habitual behavior and (b) model‐based and model‐free learning. Recent evidence indicates that the orbitomedial prefrontal cortices not only subserve goal‐directed behavior and model‐based learning, but also code the “landscape” (task space) of behaviorally relevant variables. While the hippocampus stands out for its role in coding and memorizing world state representations, it is argued to function in model‐based learning but is not required for coding of action–outcome contingencies, illustrating that goal‐directed behavior is not congruent with model‐based learning. While the dorsolateral and dorsomedial striatum largely conform to the dichotomy between habitual versus goal‐directed behavior, ventral striatal functions go beyond this distinction. Next, we contextualize findings on coding of reward‐prediction errors by ventral tegmental dopamine neurons to suggest a broader role of mesencephalic dopamine cells, viz. in behavioral reactivity and signaling unexpected sensory changes. We hypothesize that goal‐directed behavior is hierarchically organized in interconnected cortico‐basal ganglia loops, where a limbic‐affective prefrontal‐ventral striatal loop controls action selection in a dorsomedial prefrontal–striatal loop, which in turn regulates activity in sensorimotor‐dorsolateral striatal circuits. This structure for behavioral organization requires alignment with mechanisms for memory formation and consolidation. We propose that frontal corticothalamic circuits form a high‐level loop for memory processing that initiates and temporally organizes nested activities in lower‐level loops, including the hippocampus and the ripple‐associated replay it generates. The evidence on hierarchically organized behavior converges with that on consolidation mechanisms in suggesting a frontal‐to‐caudal directionality in processing control.
Collapse
Affiliation(s)
- Silviu I Rusu
- Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands.,Research Priority Program Brain and Cognition, University of Amsterdam, Amsterdam, The Netherlands
| | - Cyriel M A Pennartz
- Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands.,Research Priority Program Brain and Cognition, University of Amsterdam, Amsterdam, The Netherlands
| |
Collapse
|
34
|
Aoki S, Smith JB, Li H, Yan X, Igarashi M, Coulon P, Wickens JR, Ruigrok TJH, Jin X. An open cortico-basal ganglia loop allows limbic control over motor output via the nigrothalamic pathway. eLife 2019; 8:e49995. [PMID: 31490123 PMCID: PMC6731092 DOI: 10.7554/elife.49995] [Citation(s) in RCA: 60] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2019] [Accepted: 08/26/2019] [Indexed: 01/08/2023] Open
Abstract
Cortico-basal ganglia-thalamocortical loops are largely conceived as parallel circuits that process limbic, associative, and sensorimotor information separately. Whether and how these functionally distinct loops interact remains unclear. Combining genetic and viral approaches, we systemically mapped the limbic and motor cortico-basal ganglia-thalamocortical loops in rodents. Despite largely closed loops within each functional domain, we discovered a unidirectional influence of the limbic over the motor loop via ventral striatum-substantia nigra (SNr)-motor thalamus circuitry. Slice electrophysiology verifies that the projection from ventral striatum functionally inhibits nigro-thalamic SNr neurons. In vivo optogenetic stimulation of ventral or dorsolateral striatum to SNr pathway modulates activity in medial prefrontal cortex (mPFC) and motor cortex (M1), respectively. However, whereas the dorsolateral striatum-SNr pathway exerts little impact on mPFC, activation of the ventral striatum-SNr pathway effectively alters M1 activity. These results demonstrate an open cortico-basal ganglia loop whereby limbic information could modulate motor output through ventral striatum control of M1.
Collapse
Affiliation(s)
- Sho Aoki
- Molecular Neurobiology LaboratorySalk Institute for Biological StudiesLa JollaUnited States
- Neurobiology Research UnitOkinawa Institute of Science and TechnologyOkinawaJapan
- Department of NeuroscienceErasmus Medical Center RotterdamRotterdamNetherlands
- Japan Society for the Promotion of SciencesTokyoJapan
| | - Jared B Smith
- Molecular Neurobiology LaboratorySalk Institute for Biological StudiesLa JollaUnited States
| | - Hao Li
- Molecular Neurobiology LaboratorySalk Institute for Biological StudiesLa JollaUnited States
| | - Xunyi Yan
- Molecular Neurobiology LaboratorySalk Institute for Biological StudiesLa JollaUnited States
| | - Masakazu Igarashi
- Neurobiology Research UnitOkinawa Institute of Science and TechnologyOkinawaJapan
- Japan Society for the Promotion of SciencesTokyoJapan
| | - Patrice Coulon
- Institut des Neurosciences de la TimoneCentre National de la Recherche Scientifique (CNRS), Aix-Marseille UniversitéMarseilleFrance
| | - Jeffery R Wickens
- Neurobiology Research UnitOkinawa Institute of Science and TechnologyOkinawaJapan
| | - Tom JH Ruigrok
- Department of NeuroscienceErasmus Medical Center RotterdamRotterdamNetherlands
| | - Xin Jin
- Molecular Neurobiology LaboratorySalk Institute for Biological StudiesLa JollaUnited States
| |
Collapse
|
35
|
Willett JA, Cao J, Johnson A, Patel OH, Dorris DM, Meitzen J. The estrous cycle modulates rat caudate-putamen medium spiny neuron physiology. Eur J Neurosci 2019; 52:2737-2755. [PMID: 31278786 DOI: 10.1111/ejn.14506] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2019] [Revised: 05/16/2019] [Accepted: 06/25/2019] [Indexed: 12/27/2022]
Abstract
The neuroendocrine environment in which the brain operates is both dynamic and differs by sex. How differences in neuroendocrine state affect neuron properties has been significantly neglected in neuroscience research. Behavioral data across humans and rodents indicate that natural cyclical changes in steroid sex hormone production affect sensorimotor and cognitive behaviors in both normal and pathological contexts. These behaviors are critically mediated by the caudate-putamen. In the caudate-putamen, medium spiny neurons (MSNs) are the predominant and primary output neurons. MSNs express membrane-associated estrogen receptors and demonstrate estrogen sensitivity. However, how the cyclical hormone changes across the estrous cycle may modulate caudate-putamen MSN electrophysiological properties remains unknown. Here, we performed whole-cell patch-clamp recordings on male, diestrus female, proestrus female, and estrus female caudate-putamen MSNs. Action potential, passive membrane, and miniature excitatory post-synaptic current properties were assessed. Numerous MSN electrical properties robustly differed by cycle state, including resting membrane potential, rheobase, action potential threshold, maximum evoked action potential firing rate, and inward rectification. Strikingly, when considered independent of estrous cycle phase, all but one of these properties do not significantly differ from male MSNs. These data indicate that female caudate-putamen MSNs are sensitive to the estrous cycle, and more broadly, the importance of considering neuroendocrine state in studies of neuron physiology.
Collapse
Affiliation(s)
- Jaime A Willett
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA.,W.M. Keck Center for Behavioral Biology, North Carolina State University, Raleigh, NC, USA.,Graduate Program in Physiology, North Carolina State University, Raleigh, NC, USA.,Grass Laboratory, Marine Biological Laboratory, Woods Hole, MA, USA
| | - Jinyan Cao
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA.,W.M. Keck Center for Behavioral Biology, North Carolina State University, Raleigh, NC, USA
| | - Ashlyn Johnson
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA
| | - Opal H Patel
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA
| | - David M Dorris
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA
| | - John Meitzen
- Department of Biological Sciences, North Carolina State University, Raleigh, NC, USA.,W.M. Keck Center for Behavioral Biology, North Carolina State University, Raleigh, NC, USA.,Center for Human Health and the Environment, North Carolina State University, Raleigh, NC, USA.,Comparative Medicine Institute, North Carolina State University, Raleigh, NC, USA
| |
Collapse
|
36
|
Bari BA, Grossman CD, Lubin EE, Rajagopalan AE, Cressy JI, Cohen JY. Stable Representations of Decision Variables for Flexible Behavior. Neuron 2019; 103:922-933.e7. [PMID: 31280924 DOI: 10.1016/j.neuron.2019.06.001] [Citation(s) in RCA: 70] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2018] [Revised: 05/03/2019] [Accepted: 05/31/2019] [Indexed: 12/25/2022]
Abstract
Decisions occur in dynamic environments. In the framework of reinforcement learning, the probability of performing an action is influenced by decision variables. Discrepancies between predicted and obtained rewards (reward prediction errors) update these variables, but they are otherwise stable between decisions. Although reward prediction errors have been mapped to midbrain dopamine neurons, it is unclear how the brain represents decision variables themselves. We trained mice on a dynamic foraging task in which they chose between alternatives that delivered reward with changing probabilities. Neurons in the medial prefrontal cortex, including projections to the dorsomedial striatum, maintained persistent firing rate changes over long timescales. These changes stably represented relative action values (to bias choices) and total action values (to bias response times) with slow decay. In contrast, decision variables were weakly represented in the anterolateral motor cortex, a region necessary for generating choices. Thus, we define a stable neural mechanism to drive flexible behavior.
Collapse
Affiliation(s)
- Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| | - Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| | - Emily E Lubin
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| | - Adithya E Rajagopalan
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| | - Jianna I Cressy
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, MD 21205, USA.
| |
Collapse
|
37
|
Lintz MJ, Essig J, Zylberberg J, Felsen G. Spatial representations in the superior colliculus are modulated by competition among targets. Neuroscience 2019; 408:191-203. [PMID: 30981865 PMCID: PMC6556130 DOI: 10.1016/j.neuroscience.2019.04.002] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2018] [Revised: 03/31/2019] [Accepted: 04/01/2019] [Indexed: 12/15/2022]
Abstract
Selecting and moving to spatial targets are critical components of goal-directed behavior, yet their neural bases are not well understood. The superior colliculus (SC) is thought to contain a topographic map of contralateral space in which the activity of specific neuronal populations corresponds to particular spatial locations. However, these spatial representations are modulated by several decision-related variables, suggesting that they reflect information beyond simply the location of an upcoming movement. Here, we examine the extent to which these representations arise from competitive spatial choice. We recorded SC activity in male mice performing a behavioral task requiring orienting movements to targets for a water reward in two contexts. In "competitive" trials, either the left or right target could be rewarded, depending on which stimulus was presented at the central port. In "noncompetitive" trials, the same target (e.g., left) was rewarded throughout an entire block. While both trial types required orienting movements to the same spatial targets, only in competitive trials do targets compete for selection. We found that in competitive trials, pre-movement SC activity predicted movement to contralateral targets, as expected. However, in noncompetitive trials, some neurons lost their spatial selectivity and in others activity predicted movement to ipsilateral targets. Consistent with these findings, unilateral optogenetic inactivation of pre-movement SC activity ipsiversively biased competitive, but not noncompetitive, trials. Incorporating these results into an attractor model of SC activity points to distinct pathways for orienting movements under competitive and noncompetitive conditions, with the SC specifically required for selecting among multiple potential targets.
Collapse
Affiliation(s)
- Mario J Lintz
- Department of Physiology and Biophysics, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Neuroscience Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Medical Scientist Training Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America
| | - Jaclyn Essig
- Department of Physiology and Biophysics, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Neuroscience Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America
| | - Joel Zylberberg
- Department of Physiology and Biophysics, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Neuroscience Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America
| | - Gidon Felsen
- Department of Physiology and Biophysics, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Neuroscience Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America; Medical Scientist Training Program, University of Colorado School of Medicine, Aurora, CO 80045, United States of America.
| |
Collapse
|
38
|
Serrano W. Genetic and deep learning clusters based on neural networks for management decision structures. Neural Comput Appl 2019. [DOI: 10.1007/s00521-019-04231-8] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
39
|
Guillem K, Ahmed SH. Preference for Cocaine is Represented in the Orbitofrontal Cortex by an Increased Proportion of Cocaine Use-Coding Neurons. Cereb Cortex 2019; 28:819-832. [PMID: 28057724 DOI: 10.1093/cercor/bhw398] [Citation(s) in RCA: 33] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2016] [Accepted: 12/13/2016] [Indexed: 11/13/2022] Open
Abstract
Cocaine addiction is a harmful preference for drug use over and at the expense of other nondrug-related activities. Here we identify in the rat orbitofrontal cortex (OFC) a mechanism that explains individual preferences between cocaine use and an alternative, nondrug action. OFC neuronal activity was recorded while rats performed each of these 2 actions separately or while they chose between them. First, we found that these actions are encoded by 2 nonoverlapping neuronal populations and that the relative size of the cocaine population represented individual preferences. A larger relative size was only observed in cocaine-preferring individuals. Second, OFC neurons encoding a given individual's preferred action progressively fired more than other action-coding neurons few seconds before the preferred action was actually chosen, suggesting a prechoice neuronal competition for action selection. In cocaine-preferring rats, this manifested by a prechoice ramping-up activity in favor of the cocaine population. Finally, pharmacological manipulation of prechoice activity in favor of the cocaine population caused nondrug-preferring rats to shift their choice to cocaine. Overall, this study suggests that an individual preference for cocaine is represented in the OFC by a population size bias that systematically advantages cocaine use-coding neurons during prechoice competition for action selection.
Collapse
Affiliation(s)
- Karine Guillem
- Université de Bordeaux, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000 Bordeaux, France.,CNRS, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000 Bordeaux, France
| | - Serge H Ahmed
- Université de Bordeaux, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000 Bordeaux, France.,CNRS, Institut des Maladies Neurodégénératives, UMR 5293, 146 rue Léo-Saignat, F-33000 Bordeaux, France
| |
Collapse
|
40
|
Pauli WM, Gentile G, Collette S, Tyszka JM, O'Doherty JP. Evidence for model-based encoding of Pavlovian contingencies in the human brain. Nat Commun 2019; 10:1099. [PMID: 30846685 PMCID: PMC6405831 DOI: 10.1038/s41467-019-08922-7] [Citation(s) in RCA: 25] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2018] [Accepted: 01/16/2019] [Indexed: 12/23/2022] Open
Abstract
Prominent accounts of Pavlovian conditioning successfully approximate the frequency and intensity of conditioned responses under the assumption that learning is exclusively model-free; that animals do not develop a cognitive map of events. However, these model-free approximations fall short of comprehensively capturing learning and behavior in Pavlovian conditioning. We therefore performed multivoxel pattern analysis of high-resolution functional MRI data in human participants to test for the encoding of stimulus-stimulus associations that could support model-based computations during Pavlovian conditioning. We found that dissociable sub-regions of the striatum encode predictions of stimulus-stimulus associations and predictive value, in a manner that is directly related to learning performance. Activity patterns in the orbitofrontal cortex were also found to be related to stimulus-stimulus as well as value encoding. These results suggest that the brain encodes model-based representations during Pavlovian conditioning, and that these representations are utilized in the service of behavior.
Collapse
Affiliation(s)
- Wolfgang M Pauli
- Division of Humanities and Social Sciences, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA.
- Computation and Neural Systems Program, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA.
- Artificial Intelligence Platform, Microsoft, One Microsoft Way, Redmond, WA, 98052, USA.
| | - Giovanni Gentile
- Division of Humanities and Social Sciences, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
- Computation and Neural Systems Program, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
| | - Sven Collette
- Division of Humanities and Social Sciences, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
- Computation and Neural Systems Program, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
| | - Julian M Tyszka
- Division of Humanities and Social Sciences, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
| | - John P O'Doherty
- Division of Humanities and Social Sciences, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
- Computation and Neural Systems Program, MC 228-77, California Institute of Technology, 1200 E. California Blvd, Pasadena, CA, 91125, USA
| |
Collapse
|
41
|
Díaz-Hernández E, Contreras-López R, Sánchez-Fuentes A, Rodríguez-Sibrían L, Ramírez-Jarquín JO, Tecuapetla F. The Thalamostriatal Projections Contribute to the Initiation and Execution of a Sequence of Movements. Neuron 2018; 100:739-752.e5. [PMID: 30344045 DOI: 10.1016/j.neuron.2018.09.052] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2018] [Revised: 07/16/2018] [Accepted: 09/27/2018] [Indexed: 10/28/2022]
Abstract
One of the main inputs driving striatal activity is the thalamostriatal projection. While the hypothesis postulating that the different thalamostriatal projections contribute differentially to shape the functions of the striatum is largely accepted, existing technical limitations have hampered efforts to prove it. Here, through the use of electrophysiological recordings of antidromically photo-identified thalamostriatal neurons and the optogenetic inhibition of thalamostriatal terminals, we identify that the thalamostriatal projections from the parafascicular and the ventroposterior regions of the thalamus contribute to the smooth initiation and the appropriate execution of a sequence of movements. Our results support a model in which both thalamostriatal projections have specific contributions to the initiation and execution of sequences, highlighting the specific contribution of the ventroposterior thalamostriatal connection for the repetition of actions.
Collapse
Affiliation(s)
- Edgar Díaz-Hernández
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico
| | - Rubén Contreras-López
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico
| | - Asai Sánchez-Fuentes
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico
| | - Luis Rodríguez-Sibrían
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico
| | - Josué O Ramírez-Jarquín
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico
| | - Fatuel Tecuapetla
- Instituto de Fisiología Celular, Universidad Nacional Autónoma de México, Ciudad Universitaria, Circuito exterior s/n, 04510 Ciudad de México, CDMX, Mexico.
| |
Collapse
|
42
|
De Corte BJ, Wagner LM, Matell MS, Narayanan NS. Striatal dopamine and the temporal control of behavior. Behav Brain Res 2018; 356:375-379. [PMID: 30213664 DOI: 10.1016/j.bbr.2018.08.030] [Citation(s) in RCA: 21] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2018] [Revised: 08/08/2018] [Accepted: 08/09/2018] [Indexed: 11/17/2022]
Abstract
Striatal dopamine strongly regulates how individuals use time to guide behavior. Dopamine acts on D1- and D2- dopamine receptors in the striatum. However, the relative role of these receptors in the temporal control of behavior is unclear. To assess this, we trained rats on a task in which they decided to start and stop a series of responses based on the passage of time and evaluated how blocking D1 or D2-dopamine receptors in the dorsomedial or dorsolateral striatum impacted performance. D2 blockade delayed the decision to start and stop responding in both regions, and this effect was larger in the dorsomedial striatum. By contrast, dorsomedial D1 blockade delayed stop times, without significantly delaying start times, whereas dorsolateral D1 blockade produced no detectable effects. These findings suggest that striatal dopamine may tune decision thresholds during timing tasks. Furthermore, our data indicate that the dorsomedial striatum plays a key role in temporal control, which may be useful for localizing neural circuits that mediate the temporal control of action.
Collapse
Affiliation(s)
| | - Lucia M Wagner
- Department of Neurology, The University of Iowa, Iowa City, IA, 52242, USA; St. Olaf College, Northfield, MN, 55057, USA
| | - Matthew S Matell
- Department of Psychology, Villanova University, Villanova, PA, 19085, USA
| | | |
Collapse
|
43
|
Cano-Ramírez H, Hoffman KL. Activation of cortical and striatal regions during the expression of a naturalistic compulsive-like behavior in the rabbit. Behav Brain Res 2018; 351:168-177. [PMID: 29885848 DOI: 10.1016/j.bbr.2018.05.034] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2018] [Revised: 05/14/2018] [Accepted: 05/29/2018] [Indexed: 01/12/2023]
Abstract
Nest building behavior in the pregnant rabbit (Oryctolagus cuniculus) can serve as a model for compulsions in obsessive compulsive disorder (OCD). Previous work showed that the "straw carrying" phase of nest building (during which the rabbit repeatedly collects straw in its mouth, carries it into the nest box and deposits it there, and then returns to collect more) is associated with increased c-FOS expression (a marker of neuronal activity) in the orbitofrontal, anterior cingulate, and piriform cortices. In the present study, we quantified c-FOS expression in the caudate and putamen, as well as in the primary motor, somatosensory, and prefrontal cortices of: (1) pregnant rabbits given straw (PREG + STRAW); pregnant rabbits not given straw (PREG); (3) estrous rabbits given straw (ESTROUS + STRAW); and (4) estrous rabbits not given straw (ESTROUS). We found that straw carrying was associated with increased c-FOS expression in the dorsal putamen, ventral caudate, primary motor cortex, and somatosensory cortex. Additionally, a correlational analysis of PREG + STRAW animals revealed that these regions, along with the premotor and prelimbic cortices, were significantly intercorrelated with respect to c-FOS expression, suggesting their "coactivation" during repetitive straw carrying. By contrast, behavioral interactions of non-pregnant (ESTROUS) rabbits with straw (e.g., sniffing, nibbling it) were associated with a distinct pattern of c-FOS expression that included the medial and ventral putamen. c-FOS expression in PREG + STRAW rabbits is similar to patterns of regional brain activity in OCD patients exposed to obsession-provoking stimuli, as well as to those observed in healthy human mothers responding to infant-associated stimuli.
Collapse
Affiliation(s)
- Hugo Cano-Ramírez
- Doctorado en Ciencias Biológicas, Universidad Autónoma de Tlaxcala, Tlaxcala, Mexico; Centro de Investigación en Reproducción Animal (CIRA), Universidad Autónoma de Tlaxcala-CINVESTAV, Mexico
| | - Kurt L Hoffman
- Centro de Investigación en Reproducción Animal (CIRA), Universidad Autónoma de Tlaxcala-CINVESTAV, Mexico.
| |
Collapse
|
44
|
Elber-Dorozko L, Loewenstein Y. Striatal action-value neurons reconsidered. eLife 2018; 7:e34248. [PMID: 29848442 PMCID: PMC6008056 DOI: 10.7554/elife.34248] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2017] [Accepted: 05/13/2018] [Indexed: 11/13/2022] Open
Abstract
It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.
Collapse
Affiliation(s)
- Lotem Elber-Dorozko
- The Edmond & Lily Safra Center for Brain SciencesThe Hebrew University of JerusalemJerusalemIsrael
| | - Yonatan Loewenstein
- The Edmond & Lily Safra Center for Brain SciencesThe Hebrew University of JerusalemJerusalemIsrael
- Department of Neurobiology, The Alexander Silberman Institute of Life SciencesThe Hebrew University of JerusalemJerusalemIsrael
- The Federmann Center for the Study of RationalityThe Hebrew University of JerusalemJerusalemIsrael
| |
Collapse
|
45
|
Guillem K, Brenot V, Durand A, Ahmed SH. Neuronal representation of individual heroin choices in the orbitofrontal cortex. Addict Biol 2018; 23:880-888. [PMID: 28703355 DOI: 10.1111/adb.12536] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2017] [Revised: 05/12/2017] [Accepted: 06/21/2017] [Indexed: 02/06/2023]
Abstract
Drug addiction is a harmful preference for drug use over and at the expense of other non-drug-related activities. We previously identified in the rat orbitofrontal cortex (OFC) a mechanism that influences individual preferences between cocaine use and an alternative action rewarded by a non-drug reward (i.e. sweet water). Here, we sought to test the generality of this mechanism to a different addictive drug, heroin. OFC neuronal activity was recorded while rats responded for heroin or the alternative non-drug reward separately or while they chose between the two. First, we found that heroin-rewarded and sweet water-rewarded actions were encoded by two non-overlapping OFC neuronal populations and that the relative size of the heroin population represented individual drug choices. Second, OFC neurons encoding the preferred action-which was the non-drug action in the large majority of individuals-progressively fired more than non-preferred action-coding neurons 1 second after the onset of choice trials and around 1 second before the preferred action was actually chosen, suggesting a pre-choice neuronal competition for action selection. Together with a previous study on cocaine choice, the present study on heroin choice reveals important commonalities in how OFC neurons encode individual drug choices and preferences across different classes of drugs. It also reveals some drug-specific differences in OFC encoding activity. Notably, the proportion of neurons that non-selectively encode both the drug and the non-drug reward was higher when the drug was heroin (present study) than when it was cocaine (previous study). We will discuss the potential functional significance of these commonalities and differences in OFC neuronal activity across different drugs for understanding drug choice.
Collapse
Affiliation(s)
- Karine Guillem
- Université de Bordeaux; Institut des Maladies Neurodégénératives; France
- CNRS; Institut des Maladies Neurodégénératives; France
| | - Viridiana Brenot
- Université de Bordeaux; Institut des Maladies Neurodégénératives; France
| | - Audrey Durand
- Université de Bordeaux; Institut des Maladies Neurodégénératives; France
- CNRS; Institut des Maladies Neurodégénératives; France
| | - Serge H. Ahmed
- Université de Bordeaux; Institut des Maladies Neurodégénératives; France
- CNRS; Institut des Maladies Neurodégénératives; France
| |
Collapse
|
46
|
Stubbendorff C, Molano-Mazon M, Young AMJ, Gerdjikov TV. Synchronization in the prefrontal-striatal circuit tracks behavioural choice in a go-no-go task in rats. Eur J Neurosci 2018. [PMID: 29520856 DOI: 10.1111/ejn.13905] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Abstract
Rodent striatum is involved in sensory-motor transformations and reward-related learning. Lesion studies suggest dorsolateral striatum, dorsomedial striatum and nucleus accumbens underlie stimulus-response transformations, goal-directed behaviour and reward expectation, respectively. In addition, prefrontal inputs likely control these functions. Here, we set out to study how reward-driven behaviour is mediated by the coordinated activity of these structures in the intact brain. We implemented a discrimination task requiring rats to either respond or suppress responding on a lever after the presentation of auditory cues in order to obtain rewards. Single unit activity in the striatal subregions and pre-limbic cortex was recorded using tetrode arrays. Striatal units showed strong onset responses to auditory cues paired with an opportunity to obtain reward. Cue-onset responses in both striatum and cortex were significantly modulated by previous errors suggesting a role of these structures in maintaining appropriate motivation or action selection during ongoing behaviour. Furthermore, failure to respond to the reward-paired tones was associated with higher pre-trial coherence among striatal subregions and between cortex and striatum suggesting a task-negative corticostriatal network whose activity may be suppressed to enable processing of reward-predictive cues. Our findings highlight that coordinated activity in a distributed network including both pre-limbic cortex and multiple striatal regions underlies reward-related decisions.
Collapse
Affiliation(s)
- Christine Stubbendorff
- Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, LE1 9HN, UK.,School of Biosciences, University of Nottingham, Loughborough, UK
| | - Manuel Molano-Mazon
- Centre for Systems Neuroscience, University of Leicester, Leicester, UK.,Laboratory of Neural Computation, Istituto Italiano di Tecnologia, Rovereto, TN, Italy
| | - Andrew M J Young
- Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, LE1 9HN, UK
| | - Todor V Gerdjikov
- Department of Neuroscience, Psychology and Behaviour, University of Leicester, Leicester, LE1 9HN, UK
| |
Collapse
|
47
|
|
48
|
Achiro JM, Shen J, Bottjer SW. Neural activity in cortico-basal ganglia circuits of juvenile songbirds encodes performance during goal-directed learning. eLife 2017; 6:e26973. [PMID: 29256393 PMCID: PMC5762157 DOI: 10.7554/elife.26973] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2017] [Accepted: 12/02/2017] [Indexed: 11/13/2022] Open
Abstract
Cortico-basal ganglia circuits are thought to mediate goal-directed learning by a process of outcome evaluation to gradually select appropriate motor actions. We investigated spiking activity in core and shell subregions of the cortical nucleus LMAN during development as juvenile zebra finches are actively engaged in evaluating feedback of self-generated behavior in relation to their memorized tutor song (the goal). Spiking patterns of single neurons in both core and shell subregions during singing correlated with acoustic similarity to tutor syllables, suggesting a process of outcome evaluation. Both core and shell neurons encoded tutor similarity via either increases or decreases in firing rate, although only shell neurons showed a significant association at the population level. Tutor similarity predicted firing rates most strongly during early stages of learning, and shell but not core neurons showed decreases in response variability across development, suggesting that the activity of shell neurons reflects the progression of learning.
Collapse
Affiliation(s)
- Jennifer M Achiro
- Neuroscience Graduate ProgramUniversity of Southern CaliforniaLos AngelesUnited States
| | - John Shen
- Neuroscience Graduate ProgramUniversity of Southern CaliforniaLos AngelesUnited States
| | - Sarah W Bottjer
- Section of NeurobiologyUniversity of Southern CaliforniaLos AngelesUnited States
| |
Collapse
|
49
|
Lee Y, Kim SG, Lee B, Zhang Y, Kim Y, Kim S, Kim E, Kang H, Han K. Striatal Transcriptome and Interactome Analysis of Shank3-overexpressing Mice Reveals the Connectivity between Shank3 and mTORC1 Signaling. Front Mol Neurosci 2017; 10:201. [PMID: 28701918 PMCID: PMC5487420 DOI: 10.3389/fnmol.2017.00201] [Citation(s) in RCA: 35] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2017] [Accepted: 06/08/2017] [Indexed: 11/13/2022] Open
Abstract
Mania causes symptoms of hyperactivity, impulsivity, elevated mood, reduced anxiety and decreased need for sleep, which suggests that the dysfunction of the striatum, a critical component of the brain motor and reward system, can be causally associated with mania. However, detailed molecular pathophysiology underlying the striatal dysfunction in mania remains largely unknown. In this study, we aimed to identify the molecular pathways showing alterations in the striatum of SH3 and multiple ankyrin repeat domains 3 (Shank3)-overexpressing transgenic (TG) mice that display manic-like behaviors. The results of transcriptome analysis suggested that mammalian target of rapamycin complex 1 (mTORC1) signaling may be the primary molecular signature altered in the Shank3 TG striatum. Indeed, we found that striatal mTORC1 activity, as measured by mTOR S2448 phosphorylation, was significantly decreased in the Shank3 TG mice compared to wild-type (WT) mice. To elucidate the potential underlying mechanism, we re-analyzed previously reported protein interactomes, and detected a high connectivity between Shank3 and several upstream regulators of mTORC1, such as tuberous sclerosis 1 (TSC1), TSC2 and Ras homolog enriched in striatum (Rhes), via 94 common interactors that we denominated “Shank3-mTORC1 interactome”. We noticed that, among the 94 common interactors, 11 proteins were related to actin filaments, the level of which was increased in the dorsal striatum of Shank3 TG mice. Furthermore, we could co-immunoprecipitate Shank3, Rhes and Wiskott-Aldrich syndrome protein family verprolin-homologous protein 1 (WAVE1) proteins from the striatal lysate of Shank3 TG mice. By comparing with the gene sets of psychiatric disorders, we also observed that the 94 proteins of Shank3-mTORC1 interactome were significantly associated with bipolar disorder (BD). Altogether, our results suggest a protein interaction-mediated connectivity between Shank3 and certain upstream regulators of mTORC1 that might contribute to the abnormal striatal mTORC1 activity and to the manic-like behaviors of Shank3 TG mice.
Collapse
Affiliation(s)
- Yeunkum Lee
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea.,Department of Biomedical Sciences, College of Medicine, Korea UniversitySeoul, South Korea
| | - Sun Gyun Kim
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science (IBS)Daejeon, South Korea
| | - Bokyoung Lee
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea
| | - Yinhua Zhang
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea.,Department of Biomedical Sciences, College of Medicine, Korea UniversitySeoul, South Korea
| | - Yoonhee Kim
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea
| | - Shinhyun Kim
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea.,Department of Biomedical Sciences, College of Medicine, Korea UniversitySeoul, South Korea
| | - Eunjoon Kim
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science (IBS)Daejeon, South Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology (KAIST)Daejeon, South Korea
| | - Hyojin Kang
- HPC-enabled Convergence Technology Research Division, Korea Institute of Science and Technology InformationDaejeon, South Korea
| | - Kihoon Han
- Department of Neuroscience, College of Medicine, Korea UniversitySeoul, South Korea.,Department of Biomedical Sciences, College of Medicine, Korea UniversitySeoul, South Korea
| |
Collapse
|
50
|
Chalmers E, Luczak A, Gruber AJ. Computational Properties of the Hippocampus Increase the Efficiency of Goal-Directed Foraging through Hierarchical Reinforcement Learning. Front Comput Neurosci 2016; 10:128. [PMID: 28018203 PMCID: PMC5149552 DOI: 10.3389/fncom.2016.00128] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2016] [Accepted: 11/28/2016] [Indexed: 11/17/2022] Open
Abstract
The mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals' ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a flexible MBRL mechanism for spatial navigation that is computationally efficient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We find that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efficient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deficits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efficiently solve navigation and other difficult tasks.
Collapse
Affiliation(s)
- Eric Chalmers
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| | - Artur Luczak
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| | - Aaron J Gruber
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| |
Collapse
|