1
|
Mohebi A, Wei W, Pelattini L, Kim K, Berke JD. Dopamine transients follow a striatal gradient of reward time horizons. Nat Neurosci 2024; 27:737-746. [PMID: 38321294 PMCID: PMC11001583 DOI: 10.1038/s41593-023-01566-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2022] [Accepted: 12/21/2023] [Indexed: 02/08/2024]
Abstract
Animals make predictions to guide their behavior and update those predictions through experience. Transient increases in dopamine (DA) are thought to be critical signals for updating predictions. However, it is unclear how this mechanism handles a wide range of behavioral timescales-from seconds or less (for example, if singing a song) to potentially hours or more (for example, if hunting for food). Here we report that DA transients in distinct rat striatal subregions convey prediction errors based on distinct time horizons. DA dynamics systematically accelerated from ventral to dorsomedial to dorsolateral striatum, in the tempo of spontaneous fluctuations, the temporal integration of prior rewards and the discounting of future rewards. This spectrum of timescales for evaluative computations can help achieve efficient learning and adaptive motivation for a broad range of behaviors.
Collapse
Affiliation(s)
- Ali Mohebi
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Wei Wei
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Lilian Pelattini
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Kyoungjun Kim
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA
| | - Joshua D Berke
- Department of Neurology, University of California San Francisco, San Francisco, CA, USA.
- Department of Psychiatry and Behavioral Sciences, University of California San Francisco, San Francisco, CA, USA.
- Neuroscience Graduate Program, University of California San Francisco, San Francisco, CA, USA.
- Kavli Institute for Fundamental Neuroscience, University of California San Francisco, San Francisco, CA, USA.
- Weill Institute for Neurosciences, University of California San Francisco, San Francisco, CA, USA.
| |
Collapse
|
2
|
Chen Y, Zhu G, Yuan T, Ma R, Zhang X, Meng F, Yang A, Du T, Zhang J. Subthalamic nucleus deep brain stimulation alleviates oxidative stress via mitophagy in Parkinson's disease. NPJ Parkinsons Dis 2024; 10:52. [PMID: 38448431 PMCID: PMC10917786 DOI: 10.1038/s41531-024-00668-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 02/23/2024] [Indexed: 03/08/2024] Open
Abstract
Subthalamic nucleus deep brain stimulation (STN-DBS) has the potential to delay Parkinson's disease (PD) progression. Whether oxidative stress participates in the neuroprotective effects of DBS and related signaling pathways remains unknown. To address this, we applied STN-DBS to mice and monkey models of PD and collected brain tissue to evaluate mitophagy, oxidative stress, and related pathway. To confirm findings in animal experiments, a cohort of PD patients was recruited and oxidative stress was evaluated in cerebrospinal fluid. When PD mice received STN stimulation, the mTOR pathway was suppressed, accompanied by elevated LC3 II expression, increased mitophagosomes, and a decrease in p62 expression. The increase in mitophagy and balance of mitochondrial fission/fusion dynamics in the substantia nigra caused a marked enhancement of the antioxidant enzymes superoxide dismutase and glutathione levels. Subsequently, fewer mitochondrial apoptogenic factors were released to the cytoplasm, which resulted in a suppression of caspase activation and reservation of dopaminergic neurons. While interfaced with an mTOR activator, oxidative stress was no longer regulated by STN-DBS, with no neuroprotective effect. Similar results to those found in the rodent experiments were obtained in monkeys treated with chronic STN stimulation. Moreover, antioxidant enzymes in PD patients were increased after the operation, however, there was no relation between changes in antioxidant enzymes and motor impairment. Collectively, our study found that STN-DBS was able to increase mitophagy via an mTOR-dependent pathway, and oxidative stress was suppressed due to removal of damaged mitochondria, which was attributed to the dopaminergic neuroprotection of STN-DBS in PD.
Collapse
Affiliation(s)
- Yingchuan Chen
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
| | - Guanyu Zhu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
| | - Tianshuo Yuan
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
| | - Ruoyu Ma
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
| | - Xin Zhang
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, 100070, Beijing, China
| | - Fangang Meng
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, 100070, Beijing, China
| | - Anchao Yang
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China
| | - Tingting Du
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China.
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, 100070, Beijing, China.
| | - Jianguo Zhang
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, 100070, Beijing, China.
- Beijing Key Laboratory of Neurostimulation, 100070, Beijing, China.
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, 100070, Beijing, China.
| |
Collapse
|
3
|
Zhu T, Gallivan JP, Wolpert DM, Flanagan JR. Interaction between decision-making and motor learning when selecting reach targets in the presence of bias and noise. PLoS Comput Biol 2023; 19:e1011596. [PMID: 37917718 PMCID: PMC10703408 DOI: 10.1371/journal.pcbi.1011596] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 12/07/2023] [Accepted: 10/13/2023] [Indexed: 11/04/2023] Open
Abstract
Motor errors can have both bias and noise components. Bias can be compensated for by adaptation and, in tasks in which the magnitude of noise varies across the environment, noise can be reduced by identifying and then acting in less noisy regions of the environment. Here we examine how these two processes interact when participants reach under a combination of an externally imposed visuomotor bias and noise. In a center-out reaching task, participants experienced noise (zero-mean random visuomotor rotations) that was target-direction dependent with a standard deviation that increased linearly from a least-noisy direction. They also experienced a constant bias, a visuomotor rotation that varied (across groups) from 0 to 40 degrees. Critically, on each trial, participants could select one of three targets to reach to, thereby allowing them to potentially select targets close to the least-noisy direction. The group who experienced no bias (0 degrees) quickly learned to select targets close to the least-noisy direction. However, groups who experienced a bias often failed to identify the least-noisy direction, even though they did partially adapt to the bias. When noise was introduced after participants experienced and adapted to a 40 degrees bias (without noise) in all directions, they exhibited an improved ability to find the least-noisy direction. We developed two models-one for reach adaptation and one for target selection-that could explain participants' adaptation and target-selection behavior. Our data and simulations indicate that there is a trade-off between adaptation and selection. Specifically, because bias learning is local, participants can improve performance, through adaptation, by always selecting targets that are closest to a chosen direction. However, this comes at the expense of improving performance, through selection, by reaching toward targets in different directions to find the least-noisy direction.
Collapse
Affiliation(s)
- Tianyao Zhu
- Centre for Neuroscience Studies, Queen’s University, Kingston, Ontario, Canada
| | - Jason P. Gallivan
- Centre for Neuroscience Studies, Queen’s University, Kingston, Ontario, Canada
- Department of Psychology, Queen’s University, Kingston, Ontario, Canada
- Department of Biomedical and Molecular Sciences, Queen’s University, Kingston, Ontario, Canada
| | - Daniel M. Wolpert
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York
- Department of Neuroscience, Columbia University, New York, New York
| | - J. Randall Flanagan
- Centre for Neuroscience Studies, Queen’s University, Kingston, Ontario, Canada
- Department of Psychology, Queen’s University, Kingston, Ontario, Canada
| |
Collapse
|
4
|
Farries MA, Faust TW, Mohebi A, Berke JD. Selective encoding of reward predictions and prediction errors by globus pallidus subpopulations. Curr Biol 2023; 33:4124-4135.e5. [PMID: 37703876 PMCID: PMC10591972 DOI: 10.1016/j.cub.2023.08.042] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Revised: 07/04/2023] [Accepted: 08/15/2023] [Indexed: 09/15/2023]
Abstract
Basal ganglia (BG) circuits help guide and invigorate actions using predictions of future rewards (values). Within the BG, the globus pallidus pars externa (GPe) may play an essential role in aggregating and distributing value information. We recorded from the GPe in unrestrained rats performing both Pavlovian and instrumental tasks to obtain rewards and distinguished neuronal subtypes by their firing properties across the wake/sleep cycle and optogenetic tagging. In both tasks, the parvalbumin-positive (PV+), faster-firing "prototypical" neurons showed strong, sustained modulation by value, unlike other subtypes, including the "arkypallidal" cells that project back to striatum. Furthermore, we discovered that a distinct minority (7%) of GP cells display slower, pacemaker-like firing and encode reward prediction errors (RPEs) almost identically to midbrain dopamine neurons. These cell-specific forms of GPe value representation help define the circuit mechanisms by which the BG contribute to motivation and reinforcement learning.
Collapse
Affiliation(s)
- Michael A Farries
- Knoebel Institute for Healthy Aging, University of Denver, Denver, CO 80210, USA
| | - Thomas W Faust
- Department of Neurology, University of California, San Francisco, San Francisco, CA 94143, USA
| | - Ali Mohebi
- Department of Neurology, University of California, San Francisco, San Francisco, CA 94143, USA
| | - Joshua D Berke
- Department of Neurology, University of California, San Francisco, San Francisco, CA 94143, USA; Department of Psychiatry and Behavioral Sciences, Neuroscience Graduate Program, Kavli Institute for Fundamental Neuroscience, Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA 94143, USA.
| |
Collapse
|
5
|
Ding L. Contributions of the Basal Ganglia to Visual Perceptual Decisions. Annu Rev Vis Sci 2023; 9:385-407. [PMID: 37713277 DOI: 10.1146/annurev-vision-111022-123804] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/17/2023]
Abstract
The basal ganglia (BG) make up a prominent nexus between visual and motor-related brain regions. In contrast to the BG's well-established roles in movement control and value-based decision making, their contributions to the transformation of visual input into an action remain unclear, especially in the context of perceptual decisions based on uncertain visual evidence. This article reviews recent progress in our understanding of the BG's contributions to the formation, evaluation, and adjustment of such decisions. From theoretical and experimental perspectives, the review focuses on four key stations in the BG network, namely, the striatum, pallidum, subthalamic nucleus, and midbrain dopamine neurons, which can have different roles and together support the decision process.
Collapse
Affiliation(s)
- Long Ding
- Department of Neuroscience, University of Pennsylvania, Philadelphia, Pennsylvania, USA;
| |
Collapse
|
6
|
García-López FJ, Pastora-Bernal JM, Moreno-Morales N, Estebanez-Pérez MJ, Liñán-González A, Martín-Valero R. Virtual reality to improve low-back pain and pelvic pain during pregnancy: a pilot RCT for a multicenter randomized controlled trial. Front Med (Lausanne) 2023; 10:1206799. [PMID: 37731709 PMCID: PMC10507341 DOI: 10.3389/fmed.2023.1206799] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2023] [Accepted: 08/14/2023] [Indexed: 09/22/2023] Open
Abstract
A significant proportion of women experience low back and pelvic pain during and after pregnancy, which can negatively impact their daily lives. Various factors are attributed to these complaints, and many affected women do not receive adequate healthcare. However, there is evidence to support the use of different physiotherapeutic interventions to alleviate these conditions. Virtual reality is a promising complementary treatment to physiotherapy, particularly in improving pain perception and avoidance. The primary objective of this study is to evaluate the efficacy of a four-week program combining VR and physiotherapy compared to standard physiotherapy in pregnant women with low back and pelvic pain, in terms of improving pain avoidance, intensity, disability, and functional level. The study also aims to investigate patient satisfaction with the VR intervention. This research will be conducted through a multi-center randomized controlled clinical trial involving pregnant patients residing in the provinces of Seville and Malaga with a diagnosis of low back and pelvic pain during pregnancy. The alternative hypothesis is that the implementation of a Virtual Reality program in combination with standard physiotherapy will result in better clinical outcomes compared to the current standard intervention, which could lead to the development of new policies and interventions for these pathologies and their consequences. Clinical trial registration: clinicaltrials.gov, identifier NCT05571358.
Collapse
Affiliation(s)
| | | | - Noelia Moreno-Morales
- Department of Physiotherapy, Faculty of Health Science, University of Malaga, Malaga, Spain
| | - María-José Estebanez-Pérez
- Department of Physiotherapy, Faculty of Health Science, University of Malaga, Malaga, Spain
- Department of Physiotherapy, Faculty of Health Science, Campus of Melilla, University of Granada, Melilla, Spain
| | - Antonio Liñán-González
- Department of Nursing, Faculty of Health Science, Campus of Melilla, University of Granada, Melilla, Spain
| | - Rocío Martín-Valero
- Department of Physiotherapy, Faculty of Health Science, University of Malaga, Malaga, Spain
| |
Collapse
|
7
|
Sugiyama T, Schweighofer N, Izawa J. Reinforcement learning establishes a minimal metacognitive process to monitor and control motor learning performance. Nat Commun 2023; 14:3988. [PMID: 37422476 PMCID: PMC10329706 DOI: 10.1038/s41467-023-39536-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2022] [Accepted: 06/16/2023] [Indexed: 07/10/2023] Open
Abstract
Humans and animals develop learning-to-learn strategies throughout their lives to accelerate learning. One theory suggests that this is achieved by a metacognitive process of controlling and monitoring learning. Although such learning-to-learn is also observed in motor learning, the metacognitive aspect of learning regulation has not been considered in classical theories of motor learning. Here, we formulated a minimal mechanism of this process as reinforcement learning of motor learning properties, which regulates a policy for memory update in response to sensory prediction error while monitoring its performance. This theory was confirmed in human motor learning experiments, in which the subjective sense of learning-outcome association determined the direction of up- and down-regulation of both learning speed and memory retention. Thus, it provides a simple, unifying account for variations in learning speeds, where the reinforcement learning mechanism monitors and controls the motor learning process.
Collapse
Affiliation(s)
- Taisei Sugiyama
- Empowerment Informatics, University of Tsukuba, Tsukuba, Ibaraki, 305-8573, Japan
| | - Nicolas Schweighofer
- Biokinesiology and Physical Therapy, University of Southern California, Los Angeles, CA, 90089-9006, USA
| | - Jun Izawa
- Institute of Systems and Information Engineering, University of Tsukuba, Tsukuba, Ibaraki, 305-8573, Japan.
| |
Collapse
|
8
|
Dutriaux L, Clark NE, Papies EK, Scheepers C, Barsalou LW. The Situated Assessment Method (SAM2): Establishing individual differences in habitual behavior. PLoS One 2023; 18:e0286954. [PMID: 37347753 PMCID: PMC10287018 DOI: 10.1371/journal.pone.0286954] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2021] [Accepted: 05/28/2023] [Indexed: 06/24/2023] Open
Abstract
From the perspectives of grounded, situated, and embodied cognition, we have developed a new approach for assessing individual differences. Because this approach is grounded in two dimensions of situatedness-situational experience and the Situated Action Cycle-we refer to it as the Situated Assessment Method (SAM2). Rather than abstracting over situations during assessment of a construct (as in traditional assessment instruments), SAM2 assesses a construct in situations where it occurs, simultaneously measuring factors from the Situated Action Cycle known to influence it. To demonstrate this framework, we developed the SAM2 Habitual Behavior Instrument (SAM2 HBI). Across three studies with a total of 442 participants, the SAM2 HBI produced a robust and replicable pattern of results at both the group and individual levels. Trait-level measures of habitual behavior exhibited large reliable individual differences in the regularity of performing positive versus negative habits. Situational assessments established large effects of situations and large situation by individual interactions. Several sources of evidence demonstrated construct and content validity for SAM2 measures of habitual behavior. At both the group and individual levels, these measures were associated with factors from the Situated Action Cycle known to influence habitual behavior in the literature (consistency, automaticity, immediate reward, long-term reward). Regressions explained approximately 65% of the variance at the group level and a median of approximately 75% at the individual level. SAM2 measures further exhibited well-established interactions with personality measures for self-control and neuroticism. Cognitive-affective processes from the Situated Action Cycle explained nearly all the variance in these interactions. Finally, a composite measure of habitualness established habitual behaviors at both the group and individual levels. Additionally, a composite measure of reward was positively related to the composite measure of habitualness, increasing with self-control and decreasing with neuroticism.
Collapse
Affiliation(s)
- Léo Dutriaux
- Laboratoire sur les Interactions Cognition, Action, Émotion (LICAÉ), Université Paris Nanterre, Nanterre Cedex, France
| | - Naomi E. Clark
- School of Health and Life Sciences, Glasgow Caledonian University, Glasgow, United Kingdom
| | - Esther K. Papies
- School of Health & Wellbeing, University of Glasgow, Glasgow, United Kingdom
| | - Christoph Scheepers
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, United Kingdom
| | - Lawrence W. Barsalou
- School of Psychology and Neuroscience, University of Glasgow, Glasgow, United Kingdom
| |
Collapse
|
9
|
Whittier TT, Patrick CM, Fling BW. Somatosensory Information in Skilled Motor Performance: A Narrative Review. J Mot Behav 2023; 55:453-474. [PMID: 37245865 DOI: 10.1080/00222895.2023.2213198] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2022] [Revised: 04/26/2023] [Accepted: 05/05/2023] [Indexed: 05/30/2023]
Abstract
Historically, research aimed at improving motor performance has largely focused on the neural processes involved in motor execution due to their role in muscle activation. However, accompanying somatosensory and proprioceptive sensory information is also vitally involved in performing motor skills. Here we review research from interdisciplinary fields to provide a description for how somatosensation informs the successful performance of motor skills as well as emphasize the need for careful selection of study methods to isolate the neural processes involved in somatosensory perception. We also discuss upcoming strategies of intervention that have been used to improve performance via somatosensory targets. We believe that a greater appreciation for somatosensation's role in motor learning and control will enable researchers and practitioners to develop and apply methods for the enhancement of human performance that will benefit clinical, healthy, and elite populations alike.
Collapse
Affiliation(s)
- Tyler T Whittier
- Sensorimotor Neuroimaging Laboratory, Department of Health and Exercise Science, Colorado State University, Fort Collins, CO, USA
| | - Christopher M Patrick
- Sensorimotor Neuroimaging Laboratory, Department of Health and Exercise Science, Colorado State University, Fort Collins, CO, USA
- Molecular, Cellular and Integrative Neurosciences Program, Colorado State University, Fort Collins, CO, USA
| | - Brett W Fling
- Sensorimotor Neuroimaging Laboratory, Department of Health and Exercise Science, Colorado State University, Fort Collins, CO, USA
- Molecular, Cellular and Integrative Neurosciences Program, Colorado State University, Fort Collins, CO, USA
| |
Collapse
|
10
|
Frank SM, Maechler MR, Fogelson SV, Tse PU. Hierarchical categorization learning is associated with representational changes in the dorsal striatum and posterior frontal and parietal cortex. Hum Brain Mapp 2023; 44:3897-3912. [PMID: 37126607 DOI: 10.1002/hbm.26323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 03/27/2023] [Accepted: 04/16/2023] [Indexed: 05/03/2023] Open
Abstract
Learning and recognition can be improved by sorting novel items into categories and subcategories. Such hierarchical categorization is easy when it can be performed according to learned rules (e.g., "if car, then automatic or stick shift" or "if boat, then motor or sail"). Here, we present results showing that human participants acquire categorization rules for new visual hierarchies rapidly, and that, as they do, corresponding hierarchical representations of the categorized stimuli emerge in patterns of neural activation in the dorsal striatum and in posterior frontal and parietal cortex. Participants learned to categorize novel visual objects into a hierarchy with superordinate and subordinate levels based on the objects' shape features, without having been told the categorization rules for doing so. On each trial, participants were asked to report the category and subcategory of the object, after which they received feedback about the correctness of their categorization responses. Participants trained over the course of a one-hour-long session while their brain activation was measured using functional magnetic resonance imaging. Over the course of training, significant hierarchy learning took place as participants discovered the nested categorization rules, as evidenced by the occurrence of a learning trial, after which performance suddenly increased. This learning was associated with increased representational strength of the newly acquired hierarchical rules in a corticostriatal network including the posterior frontal and parietal cortex and the dorsal striatum. We also found evidence suggesting that reinforcement learning in the dorsal striatum contributed to hierarchical rule learning.
Collapse
Affiliation(s)
- Sebastian M Frank
- Institute for Experimental Psychology, University of Regensburg, Regensburg, Germany
| | - Marvin R Maechler
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, USA
| | - Sergey V Fogelson
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, USA
- Katz School of Science and Health, Yeshiva University, New York, New York, USA
| | - Peter U Tse
- Department of Psychological and Brain Sciences, Dartmouth College, Hanover, New Hampshire, USA
| |
Collapse
|
11
|
Leung T, Hess CW, Choate ES, Van Orden AR, Tremblay-McGaw AG, Menendez M, Boothroyd DB, Parvathinathan G, Griffin A, Caruso TJ, Stinson J, Weisman A, Liu T, Koeppen K, Koeppen K. Virtual Reality-Augmented Physiotherapy for Chronic Pain in Youth: Protocol for a Randomized Controlled Trial Enhanced With a Single-Case Experimental Design. JMIR Res Protoc 2022; 11:e40705. [PMID: 36508251 PMCID: PMC9793297 DOI: 10.2196/40705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2022] [Revised: 09/13/2022] [Accepted: 09/21/2022] [Indexed: 12/30/2022] Open
Abstract
BACKGROUND Chronic musculoskeletal (MSK) pain is a prominent health concern, resulting in pain-related disability, loss of functioning, and high health care costs. Physiotherapy rehabilitation is a gold-standard treatment for improving functioning in youth with chronic MSK pain. However, increasing physical activity can feel unattainable for many adolescents because of pain-related fear and movement avoidance. Virtual reality (VR) offers an immersive experience that can interrupt the fear-avoidance cycle and improve engagement in physiotherapy. Despite promising initial findings, data are limited and often lack the rigor required to establish VR as an evidence-based treatment for MSK pain. OBJECTIVE This trial evaluates physiorehabilitation with VR in adolescents with MSK pain. This protocol outlines the rationale, design, and implementation of a randomized controlled trial enhanced with a single-case experimental design. METHODS This study is a 2-group randomized controlled trial assessing the use of physiorehabilitation with VR in adolescents with MSK pain. The authors will collaborate with physical therapists to integrate VR into their standard clinical care. For participants enrolled in standard physiotherapy, there will be no VR integrated into their physical therapy program. Primary outcomes include physical function and engagement in VR. Secondary outcomes include pain-related fear and treatment adherence. Moreover, we will obtain clinician perspectives regarding the feasibility of integrating the intervention into the flow of clinical practice. RESULTS The pilot study implementing physiorehabilitation with VR demonstrated that high engagement and use of physiorehabilitation with VR were associated with improvements in pain, fear, avoidance, and function. Coupled with qualitative feedback from patients, families, and clinicians, the pilot study results provide support for this trial to evaluate physiorehabilitation with VR for youth with chronic MSK pain. Analysis of results from the main clinical trial will begin as recruitment progresses, and results are expected in early 2024. CONCLUSIONS Significant breakthroughs for treating MSK pain require mechanistically informed innovative approaches. Physiorehabilitation with VR provides exposure to progressive challenges, real-time feedback, and reinforcement for movement and can include activities that are difficult to achieve in the real world. It has the added benefit of sustaining patient motivation and adherence while enabling clinicians to use objective benchmarks to influence progression. These findings will inform the decision of whether to proceed with a hybrid effectiveness-dissemination trial of physiorehabilitation with VR, serving as the basis for potential large-scale implementation of physiorehabilitation with VR. TRIAL REGISTRATION ClinicalTrials.gov NCT04636177; https://clinicaltrials.gov/ct2/show/NCT04636177. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID) DERR1-10.2196/40705.
Collapse
Affiliation(s)
| | - Courtney W Hess
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Ellison S Choate
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Amanda R Van Orden
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Alexandra G Tremblay-McGaw
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Maria Menendez
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Derek B Boothroyd
- Quantitative Statistical Unit, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Gomathy Parvathinathan
- Quantitative Statistical Unit, Department of Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Anya Griffin
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Thomas J Caruso
- Department of Anesthesiology, Perioperative, and Pain Medicine, Stanford University School of Medicine, Stanford, CA, United States
| | - Jennifer Stinson
- Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, Toronto, ON, Canada.,The Research Institute, The Hospital for Sick Children, Toronto, ON, Canada
| | - Amy Weisman
- Department of Rehabilitation, Lucile Packard Children's Hospital, Stanford, CA, United States
| | - Timothy Liu
- Department of Rehabilitation, Lucile Packard Children's Hospital, Stanford, CA, United States
| | - Kurt Koeppen
- California Rehabilitation & Sports Therapy, Palo Alto, CA, United States
| | | |
Collapse
|
12
|
Jackson TB, Bernard JA. Cerebellar and basal ganglia motor network predicts trait depression and hyperactivity. Front Behav Neurosci 2022; 16:953303. [PMID: 36187378 PMCID: PMC9523104 DOI: 10.3389/fnbeh.2022.953303] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2022] [Accepted: 08/30/2022] [Indexed: 11/23/2022] Open
Abstract
In the human brain, the cerebellum (CB) and basal ganglia (BG) are implicated in cognition-, emotion-, and motor-related cortical processes and are highly interconnected, both to cortical regions via separate, trans-thalamic pathways and to each other via subcortical disynaptic pathways. We previously demonstrated a distinction between cognitive and motor CB-BG networks (CCBN, MCBN, respectively) as it relates to cortical network integration in healthy young adults, suggesting the subcortical networks separately support cortical networks. The CB and BG are also implicated in the pathophysiology of schizophrenia, Parkinson's, and compulsive behavior; thus, integration within subcortical CB-BG networks may be related to transdiagnostic symptomology. Here, we asked whether CCBN or MCBN integration predicted Achenbach Self-Report scores for anxiety, depression, intrusive thoughts, hyperactivity and inactivity, and cognitive performance in a community sample of young adults. We computed global efficiency for each CB-BG network and 7 canonical resting-state networks for all right-handed participants in the Human Connectome Project 1200 release with a complete set of preprocessed resting-state functional MRI data (N = 783). We used multivariate regression to control for substance abuse and age, and permutation testing with exchangeability blocks to control for family relationships. MCBN integration negatively predicted depression and hyperactivity, and positively predicted cortical network integration. CCBN integration predicted cortical network integration (except for the emotional network) and marginally predicted a positive relationship with hyperactivity, indicating a potential dichotomy between cognitive and motor CB-BG networks and hyperactivity. These results highlight the importance of CB-BG interactions as they relate to motivation and symptoms of depression.
Collapse
Affiliation(s)
- T. Bryan Jackson
- Department of Psychological and Brain Sciences, Texas A&M University, College Station, TX, United States
- *Correspondence: T. Bryan Jackson
| | - Jessica A. Bernard
- Department of Psychological and Brain Sciences, Texas A&M University, College Station, TX, United States
- Texas A&M Institute for Neuroscience, Texas A&M University, College Station, TX, United States
| |
Collapse
|
13
|
Ponzi A, Wickens J. Ramping activity in the striatum. Front Comput Neurosci 2022; 16:902741. [PMID: 35978564 PMCID: PMC9376361 DOI: 10.3389/fncom.2022.902741] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Accepted: 07/05/2022] [Indexed: 11/13/2022] Open
Abstract
Control of the timing of behavior is thought to require the basal ganglia (BG) and BG pathologies impair performance in timing tasks. Temporal interval discrimination depends on the ramping activity of medium spiny neurons (MSN) in the main BG input structure, the striatum, but the underlying mechanisms driving this activity are unclear. Here, we combine an MSN dynamical network model with an action selection system applied to an interval discrimination task. We find that when network parameters are appropriate for the striatum so that slowly fluctuating marginally stable dynamics are intrinsically generated, up and down ramping populations naturally emerge which enable significantly above chance task performance. We show that emergent population activity is in very good agreement with empirical studies and discuss how MSN network dysfunction in disease may alter temporal perception.
Collapse
Affiliation(s)
- Adam Ponzi
- Institute of Biophysics, Italian National Research Council, Palermo, Italy
- *Correspondence: Adam Ponzi
| | - Jeff Wickens
- Neurobiology Research Unit, Okinawa Institute of Science and Technology (OIST), Okinawa, Japan
| |
Collapse
|
14
|
Suhaimi A, Lim AWH, Chia XW, Li C, Makino H. Representation learning in the artificial and biological neural networks underlying sensorimotor integration. SCIENCE ADVANCES 2022; 8:eabn0984. [PMID: 35658033 PMCID: PMC9166289 DOI: 10.1126/sciadv.abn0984] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/03/2021] [Accepted: 04/18/2022] [Indexed: 06/15/2023]
Abstract
The integration of deep learning and theories of reinforcement learning (RL) is a promising avenue to explore novel hypotheses on reward-based learning and decision-making in humans and other animals. Here, we trained deep RL agents and mice in the same sensorimotor task with high-dimensional state and action space and studied representation learning in their respective neural networks. Evaluation of thousands of neural network models with extensive hyperparameter search revealed that learning-dependent enrichment of state-value and policy representations of the task-performance-optimized deep RL agent closely resembled neural activity of the posterior parietal cortex (PPC). These representations were critical for the task performance in both systems. PPC neurons also exhibited representations of the internally defined subgoal, a feature of deep RL algorithms postulated to improve sample efficiency. Such striking resemblance between the artificial and biological networks and their functional convergence in sensorimotor integration offers new opportunities to better understand respective intelligent systems.
Collapse
|
15
|
Fermin ASR, Friston K, Yamawaki S. An insula hierarchical network architecture for active interoceptive inference. ROYAL SOCIETY OPEN SCIENCE 2022; 9:220226. [PMID: 35774133 PMCID: PMC9240682 DOI: 10.1098/rsos.220226] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 06/09/2022] [Indexed: 05/05/2023]
Abstract
In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.
Collapse
Affiliation(s)
- Alan S. R. Fermin
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| | - Karl Friston
- The Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology, London, England
| | - Shigeto Yamawaki
- Center for Brain, Mind and Kansei Sciences Research, Hiroshima University, Hiroshima, Japan
| |
Collapse
|
16
|
MacIver MA, Finlay BL. The neuroecology of the water-to-land transition and the evolution of the vertebrate brain. Philos Trans R Soc Lond B Biol Sci 2022; 377:20200523. [PMID: 34957852 PMCID: PMC8710882 DOI: 10.1098/rstb.2020.0523] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022] Open
Abstract
The water-to-land transition in vertebrate evolution offers an unusual opportunity to consider computational affordances of a new ecology for the brain. All sensory modalities are changed, particularly a greatly enlarged visual sensorium owing to air versus water as a medium, and expanded by mobile eyes and neck. The multiplication of limbs, as evolved to exploit aspects of life on land, is a comparable computational challenge. As the total mass of living organisms on land is a hundredfold larger than the mass underwater, computational improvements promise great rewards. In water, the midbrain tectum coordinates approach/avoid decisions, contextualized by water flow and by the animal's body state and learning. On land, the relative motions of sensory surfaces and effectors must be resolved, adding on computational architectures from the dorsal pallium, such as the parietal cortex. For the large-brained and long-living denizens of land, making the right decision when the wrong one means death may be the basis of planning, which allows animals to learn from hypothetical experience before enactment. Integration of value-weighted, memorized panoramas in basal ganglia/frontal cortex circuitry, with allocentric cognitive maps of the hippocampus and its associated cortices becomes a cognitive habit-to-plan transition as substantial as the change in ecology. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.
Collapse
Affiliation(s)
- Malcolm A. MacIver
- Center for Robotics and Biosystems, Northwestern University, Evanston, IL 60208, USA
| | - Barbara L. Finlay
- Department of Psychology, Behavioral and Evolutionary Neuroscience Group, Cornell University, Ithaca, NY 14850, USA
| |
Collapse
|
17
|
Mei J, Muller E, Ramaswamy S. Informing deep neural networks by multiscale principles of neuromodulatory systems. Trends Neurosci 2022; 45:237-250. [DOI: 10.1016/j.tins.2021.12.008] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 12/04/2021] [Accepted: 12/21/2021] [Indexed: 01/19/2023]
|
18
|
de Brouwer AJ, Areshenkoff CN, Rashid MR, Flanagan JR, Poppenk J, Gallivan JP. Human Variation in Error-Based and Reinforcement Motor Learning Is Associated With Entorhinal Volume. Cereb Cortex 2021; 32:3423-3440. [PMID: 34963128 PMCID: PMC9376876 DOI: 10.1093/cercor/bhab424] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2021] [Revised: 11/01/2021] [Accepted: 11/03/2021] [Indexed: 12/31/2022] Open
Abstract
Error-based and reward-based processes are critical for motor learning and are thought to be mediated via distinct neural pathways. However, recent behavioral work in humans suggests that both learning processes can be bolstered by the use of cognitive strategies, which may mediate individual differences in motor learning ability. It has been speculated that medial temporal lobe regions, which have been shown to support motor sequence learning, also support the use of cognitive strategies in error-based and reinforcement motor learning. However, direct evidence in support of this idea remains sparse. Here we first show that better overall learning during error-based visuomotor adaptation is associated with better overall learning during the reward-based shaping of reaching movements. Given the cognitive contribution to learning in both of these tasks, these results support the notion that strategic processes, associated with better performance, drive intersubject variation in both error-based and reinforcement motor learning. Furthermore, we show that entorhinal cortex volume is larger in better learning individuals-characterized across both motor learning tasks-compared with their poorer learning counterparts. These results suggest that individual differences in learning performance during error and reinforcement learning are related to neuroanatomical differences in entorhinal cortex.
Collapse
Affiliation(s)
- Anouk J de Brouwer
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Corson N Areshenkoff
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Mohammad R Rashid
- School of Computing, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - J Randall Flanagan
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Jordan Poppenk
- Centre for Neuroscience Studies, Queen’s University, Kingston, ON K7L 3N6, Canada,Department of Psychology, Queen’s University, Kingston, ON K7L 3N6, Canada,School of Computing, Queen’s University, Kingston, ON K7L 3N6, Canada
| | - Jason P Gallivan
- Address correspondence to Jason P. Gallivan, Queen’s University, Kingston, Ontario K7L 3N6, Canada.
| |
Collapse
|
19
|
Liakoni V, Lehmann MP, Modirshanechi A, Brea J, Lutti A, Gerstner W, Preuschoff K. Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making. Neuroimage 2021; 246:118780. [PMID: 34875383 DOI: 10.1016/j.neuroimage.2021.118780] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 08/03/2021] [Accepted: 12/04/2021] [Indexed: 11/25/2022] Open
Abstract
Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland.
| | - Marco P Lehmann
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Antoine Lutti
- Laboratoire de recherche en neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Kerstin Preuschoff
- Geneva Finance Research Institute & Interfaculty Center for Affective Sciences, University of Geneva, Geneva, Switzerland
| |
Collapse
|
20
|
Subramanian A, Chitlangia S, Baths V. Reinforcement learning and its connections with neuroscience and psychology. Neural Netw 2021; 145:271-287. [PMID: 34781215 DOI: 10.1016/j.neunet.2021.10.003] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2021] [Revised: 09/26/2021] [Accepted: 10/01/2021] [Indexed: 11/19/2022]
Abstract
Reinforcement learning methods have recently been very successful at performing complex sequential tasks like playing Atari games, Go and Poker. These algorithms have outperformed humans in several tasks by learning from scratch, using only scalar rewards obtained through interaction with their environment. While there certainly has been considerable independent innovation to produce such results, many core ideas in reinforcement learning are inspired by phenomena in animal learning, psychology and neuroscience. In this paper, we comprehensively review a large number of findings in both neuroscience and psychology that evidence reinforcement learning as a promising candidate for modeling learning and decision making in the brain. In doing so, we construct a mapping between various classes of modern RL algorithms and specific findings in both neurophysiological and behavioral literature. We then discuss the implications of this observed relationship between RL, neuroscience and psychology and its role in advancing research in both AI and brain science.
Collapse
Affiliation(s)
- Ajay Subramanian
- Department of Psychology, New York University, New York, New York, 10003, USA; Cognitive Neuroscience Lab, BITS Pilani K K Birla Goa Campus, NH-17B, Zuarinagar, Goa, 403726, India.
| | - Sharad Chitlangia
- Amazon; Cognitive Neuroscience Lab, BITS Pilani K K Birla Goa Campus, NH-17B, Zuarinagar, Goa, 403726, India.
| | - Veeky Baths
- Cognitive Neuroscience Lab, BITS Pilani K K Birla Goa Campus, NH-17B, Zuarinagar, Goa, 403726, India; Department of Biological Sciences, BITS Pilani K K Birla Goa Campus, NH-17B, Zuarinagar, Goa, 403726, India.
| |
Collapse
|
21
|
Xu HA, Modirshanechi A, Lehmann MP, Gerstner W, Herzog MH. Novelty is not surprise: Human exploratory and adaptive behavior in sequential decision-making. PLoS Comput Biol 2021; 17:e1009070. [PMID: 34081705 PMCID: PMC8205159 DOI: 10.1371/journal.pcbi.1009070] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 06/15/2021] [Accepted: 05/12/2021] [Indexed: 11/19/2022] Open
Abstract
Classic reinforcement learning (RL) theories cannot explain human behavior in the absence of external reward or when the environment changes. Here, we employ a deep sequential decision-making paradigm with sparse reward and abrupt environmental changes. To explain the behavior of human participants in these environments, we show that RL theories need to include surprise and novelty, each with a distinct role. While novelty drives exploration before the first encounter of a reward, surprise increases the rate of learning of a world-model as well as of model-free action-values. Even though the world-model is available for model-based RL, we find that human decisions are dominated by model-free action choices. The world-model is only marginally used for planning, but it is important to detect surprising events. Our theory predicts human action choices with high probability and allows us to dissociate surprise, novelty, and reward in EEG signals.
Collapse
Affiliation(s)
- He A. Xu
- Laboratory of Psychophysics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Alireza Modirshanechi
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Marco P. Lehmann
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Wulfram Gerstner
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- School of Computer and Communication Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Michael H. Herzog
- Laboratory of Psychophysics, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Brain-Mind Institute, School of Life Sciences, Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| |
Collapse
|
22
|
More dynamical and more symbiotic: Cortico-striatal models of resolve, suppression, and routine habit. Behav Brain Sci 2021; 44:e41. [PMID: 33899722 DOI: 10.1017/s0140525x20000928] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
I extend Ainslie's core claims with three cortico-striatal models that respectively subserve the key constructs of resolve, suppression, and routine habit. I show that these models suggest a more dynamical and symbiotic relation among the constructs: there are more ways they interact to reinforce willpower, and the temporal dimension of the interactions can often determine the effectiveness of the reinforcement.
Collapse
|
23
|
Shin EJ, Jang Y, Kim S, Kim H, Cai X, Lee H, Sul JH, Lee SH, Chung Y, Lee D, Jung MW. Robust and distributed neural representation of action values. eLife 2021; 10:53045. [PMID: 33876728 PMCID: PMC8104958 DOI: 10.7554/elife.53045] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Accepted: 04/19/2021] [Indexed: 11/13/2022] Open
Abstract
Studies in rats, monkeys, and humans have found action-value signals in multiple regions of the brain. These findings suggest that action-value signals encoded in these brain structures bias choices toward higher expected rewards. However, previous estimates of action-value signals might have been inflated by serial correlations in neural activity and also by activity related to other decision variables. Here, we applied several statistical tests based on permutation and surrogate data to analyze neural activity recorded from the striatum, frontal cortex, and hippocampus. The results show that previously identified action-value signals in these brain areas cannot be entirely accounted for by concurrent serial correlations in neural activity and action value. We also found that neural activity related to action value is intermixed with signals related to other decision variables. Our findings provide strong evidence for broadly distributed neural signals related to action value throughout the brain.
Collapse
Affiliation(s)
- Eun Ju Shin
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Yunsil Jang
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Soyoun Kim
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
| | - Hoseok Kim
- Department of Neuroscience, Biomedicum, Karolinska Institutet, Stockholm, Sweden
| | - Xinying Cai
- New York University Shanghai, NYU-ECNU Institute of Brain and Cognitive Science at NYU Shanghai, and Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai, China
| | - Hyunjung Lee
- Department of Anatomy, Kyungpook National University School of Medicine, Daegu, Republic of Korea
| | - Jung Hoon Sul
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea
| | - Sung-Hyun Lee
- Neuroscience Graduate Program, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Yeonseung Chung
- Department of Mathematical Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, Kavli Neuroscience Discovery Institute, Department of Neuroscience, and Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, United States
| | - Min Whan Jung
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| |
Collapse
|
24
|
Min K, Lee J, Kakei S. Dynamic Modulation of a Learned Motor Skill for Its Recruitment. Front Comput Neurosci 2020; 14:457682. [PMID: 33424572 PMCID: PMC7793756 DOI: 10.3389/fncom.2020.457682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2019] [Accepted: 10/13/2020] [Indexed: 11/24/2022] Open
Abstract
Humans learn motor skills (MSs) through practice and experience and may then retain them for recruitment, which is effective as a rapid response for novel contexts. For an MS to be recruited for novel contexts, its recruitment range must be extended. In addressing this issue, we hypothesized that an MS is dynamically modulated according to the feedback context to expand its recruitment range into novel contexts, which do not involve the learning of an MS. The following two sub-issues are considered. We previously demonstrated that the learned MS could be recruited in novel contexts through its modulation, which is driven by dynamically regulating the synergistic redundancy between muscles according to the feedback context. However, this modulation is trained in the dynamics under the MS learning context. Learning an MS in a specific condition naturally causes movement deviation from the desired state when the MS is executed in a novel context. We hypothesized that this deviation can be reduced with the additional modulation of an MS, which tunes the MS-produced muscle activities by using the feedback gain signals driven by the deviation from the desired state. Based on this hypothesis, we propose a feedback gain signal-driven tuning model of a learned MS for its robust recruitment. This model is based on the neurophysiological architecture in the cortico-basal ganglia circuit, in which an MS is plausibly retained as it was learned and is then recruited by tuning its muscle control signals according to the feedback context. In this study, through computational simulation, we show that the proposed model may be used to neurophysiologically describe the recruitment of a learned MS in novel contexts.
Collapse
Affiliation(s)
- Kyuengbo Min
- Department of Psychiatry & Behavioral Sciences, Tokyo Metropolitan Institute of Medical Science, Tokyo, Japan
| | - Jongho Lee
- Department of Clinical Engineering, Faculty of Health Sciences, Komatsu University, Komatsu, Japan
| | - Shinji Kakei
- Department of Psychiatry & Behavioral Sciences, Tokyo Metropolitan Institute of Medical Science, Tokyo, Japan
| |
Collapse
|
25
|
Chen R, Goldberg JH. Actor-critic reinforcement learning in the songbird. Curr Opin Neurobiol 2020; 65:1-9. [PMID: 32898752 PMCID: PMC7769887 DOI: 10.1016/j.conb.2020.08.005] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2020] [Revised: 08/04/2020] [Accepted: 08/05/2020] [Indexed: 11/18/2022]
Abstract
It feels rewarding to ace your opponent on match point. Here, we propose common mechanisms underlie reward and performance learning. First, when a singing bird unexpectedly hits the right note, its dopamine (DA) neurons are activated as when a thirsty monkey receives an unexpected juice reward. Second, these DA signals reinforce vocal variations much as they reinforce stimulus-response associations. Third, limbic inputs to DA neurons signal the predicted quality of song syllables much like they signal the predicted reward value of a place or a stimulus during foraging. Finally, songbirds may solve difficult problems in reinforcement learning - such as credit assignment and catastrophic forgetting - with node perturbation and consolidation of reinforced vocal patterns in motor cortical circuits. Consolidation occurs downstream of a canonical 'actor-critic' circuit motif that learns to maximize performance quality in essentially the same way it learns to maximize reward: by computing and learning from prediction errors.
Collapse
Affiliation(s)
- Ruidong Chen
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, United States
| | - Jesse H Goldberg
- Department of Neurobiology and Behavior, Cornell University, Ithaca, NY 14853, United States.
| |
Collapse
|
26
|
Du T, Chen Y, Shi L, Liu D, Liu Y, Yuan T, Zhang X, Zhu G, Zhang J. Deep brain stimulation of the anterior nuclei of the thalamus relieves basal ganglia dysfunction in monkeys with temporal lobe epilepsy. CNS Neurosci Ther 2020; 27:341-351. [PMID: 33085171 PMCID: PMC7871793 DOI: 10.1111/cns.13462] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Revised: 09/19/2020] [Accepted: 09/21/2020] [Indexed: 12/20/2022] Open
Abstract
Aims Deep brain stimulation of the anterior nuclei of the thalamus (ANT‐DBS) is effective in temporal lobe epilepsy (TLE). Previous studies have shown that the basal ganglia are involved in seizure propagation in TLE, but the effects of ANT‐DBS on the basal ganglia have not been clarified. Methods ANT‐DBS was applied to monkeys with kainic acid–induced TLE using a robot‐assisted system. Behavior was monitored continuously. Immunofluorescence analysis and Western blotting were used to estimate protein expression levels in the basal ganglia and the effects of ANT stimulation. Results The seizure frequency decreased after ANT‐DBS. D1 and D2 receptor levels in the putamen and caudate were significantly higher in the ANT‐DBS group than in the epilepsy (EP) model. Neuronal loss and apoptosis were less severe in the ANT‐DBS group. Glutamate receptor 1 (GluR1) in the nucleus accumbens (NAc) shell and globus pallidus internus (GPi) increased in the EP group but decreased after ANT‐DBS. γ‐Aminobutyric acid receptor A (GABAA‐R) decreased and glutamate decarboxylase 67 (GAD67) increased in the GPi of the EP group, whereas the reverse tendencies were observed after ANT‐DBS. Conclusion ANT‐DBS exerts neuroprotective effects on the caudate and putamen, enhances D1 and D2 receptor expression, and downregulates GPi overactivation, which enhanced the antiepileptic function of the basal ganglia.
Collapse
Affiliation(s)
- Tingting Du
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing, China
| | - Yingchuan Chen
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Lin Shi
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Defeng Liu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Yuye Liu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Tianshuo Yuan
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Xin Zhang
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing, China
| | - Guanyu Zhu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China
| | - Jianguo Zhang
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing, China.,Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.,Beijing Key Laboratory of Neurostimulation, Beijing, China
| |
Collapse
|
27
|
Spatial organization of functional clusters representing reward and movement information in the striatal direct and indirect pathways. Proc Natl Acad Sci U S A 2020; 117:27004-27015. [PMID: 33055217 DOI: 10.1073/pnas.2010361117] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/05/2023] Open
Abstract
To obtain insights into striatal neural processes underlying reward-based learning and movement control, we examined spatial organizations of striatal neurons related to movement and reward-based learning. For this, we recorded the activity of direct- and indirect-pathway neurons (D1 and A2a receptor-expressing neurons, respectively) in mice engaged in probabilistic classical conditioning and open-field free exploration. We found broadly organized functional clusters of striatal neurons in the direct as well as indirect pathways for both movement- and reward-related variables. Functional clusters for different variables were partially overlapping in both pathways, but the overlap between outcome- and value-related functional clusters was greater in the indirect than direct pathway. Also, value-related spatial clusters were progressively refined during classical conditioning. Our study shows the broad and learning-dependent spatial organization of functional clusters of dorsal striatal neurons in the direct and indirect pathways. These findings further argue against the classic model of the basal ganglia and support the importance of spatiotemporal patterns of striatal neuronal ensemble activity in the control of behavior.
Collapse
|
28
|
Pamukcu A, Cui Q, Xenias HS, Berceau BL, Augustine EC, Fan I, Chalasani S, Hantman AW, Lerner TN, Boca SM, Chan CS. Parvalbumin + and Npas1 + Pallidal Neurons Have Distinct Circuit Topology and Function. J Neurosci 2020; 40:7855-7876. [PMID: 32868462 PMCID: PMC7548687 DOI: 10.1523/jneurosci.0361-20.2020] [Citation(s) in RCA: 39] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2020] [Revised: 06/23/2020] [Accepted: 07/31/2020] [Indexed: 12/19/2022] Open
Abstract
The external globus pallidus (GPe) is a critical node within the basal ganglia circuit. Phasic changes in the activity of GPe neurons during movement and their alterations in Parkinson's disease (PD) argue that the GPe is important in motor control. Parvalbumin-positive (PV+) neurons and Npas1+ neurons are the two principal neuron classes in the GPe. The distinct electrophysiological properties and axonal projection patterns argue that these two neuron classes serve different roles in regulating motor output. However, the causal relationship between GPe neuron classes and movement remains to be established. Here, by using optogenetic approaches in mice (both males and females), we showed that PV+ neurons and Npas1+ neurons promoted and suppressed locomotion, respectively. Moreover, PV+ neurons and Npas1+ neurons are under different synaptic influences from the subthalamic nucleus (STN). Additionally, we found a selective weakening of STN inputs to PV+ neurons in the chronic 6-hydroxydopamine lesion model of PD. This finding reinforces the idea that the reciprocally connected GPe-STN network plays a key role in disease symptomatology and thus provides the basis for future circuit-based therapies.SIGNIFICANCE STATEMENT The external pallidum is a key, yet an understudied component of the basal ganglia. Neural activity in the pallidum goes awry in neurologic diseases, such as Parkinson's disease. While this strongly argues that the pallidum plays a critical role in motor control, it has been difficult to establish the causal relationship between pallidal activity and motor function/dysfunction. This was in part because of the cellular complexity of the pallidum. Here, we showed that the two principal neuron types in the pallidum have opposing roles in motor control. In addition, we described the differences in their synaptic influence. Importantly, our research provides new insights into the cellular and circuit mechanisms that explain the hypokinetic features of Parkinson's disease.
Collapse
Affiliation(s)
- Arin Pamukcu
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Qiaoling Cui
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Harry S Xenias
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Brianna L Berceau
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Elizabeth C Augustine
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Isabel Fan
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Saivasudha Chalasani
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Adam W Hantman
- Janelia Research Campus, Howard Hughes Medical Institute, Ashburn, Virginia 20147
| | - Talia N Lerner
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| | - Simina M Boca
- Innovation Center for Biomedical Informatics, Georgetown University Medical Center, Washington, DC 20007
| | - C Savio Chan
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois 60611
| |
Collapse
|
29
|
Barsalou LW. Challenges and Opportunities for Grounding Cognition. J Cogn 2020; 3:31. [PMID: 33043241 PMCID: PMC7528688 DOI: 10.5334/joc.116] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2020] [Accepted: 07/20/2020] [Indexed: 01/09/2023] Open
Abstract
According to the grounded perspective, cognition emerges from the interaction of classic cognitive processes with the modalities, the body, and the environment. Rather than being an autonomous impenetrable module, cognition incorporates these other domains intrinsically into its operation. The Situated Action Cycle offers one way of understanding how the modalities, the body, and the environment become integrated to ground cognition. Seven challenges and opportunities are raised for this perspective: (1) How does cognition emerge from the Situated Action Cycle and in turn support it? (2) How can we move beyond simply equating embodiment with action, additionally establishing how embodiment arises in the autonomic, neuroendocrine, immune, cardiovascular, respiratory, digestive, and integumentary systems? (3) How can we better understand the mechanisms underlying multimodal simulation, its functions across the Situated Action Cycle, and its integration with other representational systems? (4) How can we develop and assess theoretical accounts of symbolic processing from the grounded perspective (perhaps using the construct of simulators)? (5) How can we move beyond the simplistic distinction between concrete and abstract concepts, instead addressing how concepts about the external and internal worlds pattern to support the Situated Action Cycle? (6) How do individual differences emerge from different populations of situational memories as the Situated Action Cycle manifests itself differently across individuals? (7) How can constructs from grounded cognition provide insight into the replication and generalization crises, perhaps from a quantum perspective on mechanisms (as exemplified by simulators).
Collapse
Affiliation(s)
- Lawrence W. Barsalou
- Institute of Neuroscience and Psychology, School of Psychology, University of Glasgow, Glasgow, UK
| |
Collapse
|
30
|
Xiao X, Deng H, Furlan A, Yang T, Zhang X, Hwang GR, Tucciarone J, Wu P, He M, Palaniswamy R, Ramakrishnan C, Ritola K, Hantman A, Deisseroth K, Osten P, Huang ZJ, Li B. A Genetically Defined Compartmentalized Striatal Direct Pathway for Negative Reinforcement. Cell 2020; 183:211-227.e20. [PMID: 32937106 DOI: 10.1016/j.cell.2020.08.032] [Citation(s) in RCA: 40] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2020] [Revised: 05/02/2020] [Accepted: 08/17/2020] [Indexed: 12/31/2022]
Abstract
The striosome compartment within the dorsal striatum has been implicated in reinforcement learning and regulation of motivation, but how striosomal neurons contribute to these functions remains elusive. Here, we show that a genetically identified striosomal population, which expresses the Teashirt family zinc finger 1 (Tshz1) and belongs to the direct pathway, drives negative reinforcement and is essential for aversive learning in mice. Contrasting a "conventional" striosomal direct pathway, the Tshz1 neurons cause aversion, movement suppression, and negative reinforcement once activated, and they receive a distinct set of synaptic inputs. These neurons are predominantly excited by punishment rather than reward and represent the anticipation of punishment or the motivation for avoidance. Furthermore, inhibiting these neurons impairs punishment-based learning without affecting reward learning or movement. These results establish a major role of striosomal neurons in behaviors reinforced by punishment and moreover uncover functions of the direct pathway unaccounted for in classic models.
Collapse
Affiliation(s)
- Xiong Xiao
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Hanfei Deng
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | | | - Tao Yang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Xian Zhang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Ga-Ram Hwang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Jason Tucciarone
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Priscilla Wu
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Miao He
- Institutes of Brain Science, State Key Laboratory of Medical Neurobiology and MOE Frontiers Center for Brain Science, Fudan University, Shanghai 200032, China
| | | | - Charu Ramakrishnan
- Howard Hughes Medical Institute (HHMI), Stanford University, Stanford, CA, USA; Department of Bioengineering and Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | | | - Adam Hantman
- HHMI Janelia Research Campus, Ashburn, VA 20147, USA
| | - Karl Deisseroth
- Howard Hughes Medical Institute (HHMI), Stanford University, Stanford, CA, USA; Department of Bioengineering and Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA
| | - Pavel Osten
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Z Josh Huang
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA
| | - Bo Li
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY 11724, USA.
| |
Collapse
|
31
|
Enomoto K, Matsumoto N, Inokawa H, Kimura M, Yamada H. Topographic distinction in long-term value signals between presumed dopamine neurons and presumed striatal projection neurons in behaving monkeys. Sci Rep 2020; 10:8912. [PMID: 32488042 PMCID: PMC7265398 DOI: 10.1038/s41598-020-65914-0] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2019] [Accepted: 05/12/2020] [Indexed: 12/15/2022] Open
Abstract
Nigrostriatal dopamine (DA) projections are anatomically organized along the dorsolateral-ventromedial axis, conveying long-term value signals to the striatum for shaping actions toward multiple future rewards. The present study examines whether the topographic organization of long-term value signals are observed upon activity of presumed DA neurons and presumed striatal projection neurons (phasically active neurons, PANs), as predicted based on anatomical literature. Our results indicate that DA neurons in the dorsolateral midbrain encode long-term value signals on a short timescale, while ventromedial midbrain DA neurons encode such signals on a relatively longer timescale. Activity of the PANs in the dorsal striatum is more heterogeneous for encoding long-term values, although significant differences in long-term value signals were observed between the caudate nucleus and putamen. These findings suggest that topographic DA signals for long-term values are not simply transferred to striatal neurons, possibly due to the contribution of other projections to the striatum.
Collapse
Affiliation(s)
- Kazuki Enomoto
- Department of Physiology, Kyoto Prefectural University of Medicine, Kyoto, 602-8566, Japan.,Center for Information and Neural Networks, National Institute of Information and Communications Technology, Osaka, 565-0871, Japan.,Brain Science Institute, Tamagawa University, Machida, Tokyo, 194-8610, Japan
| | - Naoyuki Matsumoto
- Department of Physiology, Kyoto Prefectural University of Medicine, Kyoto, 602-8566, Japan.,Division of Food and Health Sciences, Faculty of Environmental and Symbiotic Sciences, Prefectural University of Kumamoto, Kumamoto, 862-8502, Japan
| | - Hitoshi Inokawa
- Department of Physiology, Kyoto Prefectural University of Medicine, Kyoto, 602-8566, Japan
| | - Minoru Kimura
- Department of Physiology, Kyoto Prefectural University of Medicine, Kyoto, 602-8566, Japan.,Brain Science Institute, Tamagawa University, Machida, Tokyo, 194-8610, Japan
| | - Hiroshi Yamada
- Department of Physiology, Kyoto Prefectural University of Medicine, Kyoto, 602-8566, Japan. .,Graduate School of Comprehensive Human Sciences, University of Tsukuba, 1-1-1 Tenno-dai, Tsukuba, Ibaraki, 305-8577, Japan. .,Transborder Medical Research Center, University of Tsukuba, 1-1-1 Tenno-dai, Tsukuba, Ibaraki, 305-8577, Japan. .,Division of Biomedical Science, Faculty of Medicine, University of Tsukuba, 1-1-1 Tenno-dai, Tsukuba, Ibaraki, 305-8577, Japan.
| |
Collapse
|
32
|
Baladron J, Hamker FH. Habit learning in hierarchical cortex-basal ganglia loops. Eur J Neurosci 2020; 52:4613-4638. [PMID: 32237250 DOI: 10.1111/ejn.14730] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2020] [Revised: 03/21/2020] [Accepted: 03/22/2020] [Indexed: 12/17/2022]
Abstract
How do the multiple cortico-basal ganglia-thalamo-cortical loops interact? Are they parallel and fully independent or controlled by an arbitrator, or are they hierarchically organized? We introduce here a set of four key concepts, integrated and evaluated by means of a neuro-computational model, that bring together current ideas regarding cortex-basal ganglia interactions in the context of habit learning. According to key concept 1, each loop learns to select an intermediate objective at a different abstraction level, moving from goals in the ventral striatum to motor in the putamen. Key concept 2 proposes that the cortex integrates the basal ganglia selection with environmental information regarding the achieved objective. Key concept 3 claims shortcuts between loops, and key concept 4 predicts that loops compute their own prediction error signal for learning. Computational benefits of the key concepts are demonstrated. Contrasting with former concepts of habit learning, the loops collaborate to select goal-directed actions while training slower shortcuts develops habitual responses.
Collapse
Affiliation(s)
- Javier Baladron
- Department of Computer Science, Chemnitz University of Technology, Chemnitz, Germany
| | - Fred H Hamker
- Department of Computer Science, Chemnitz University of Technology, Chemnitz, Germany
| |
Collapse
|
33
|
Du TT, Zhu G, Chen Y, Shi L, Liu D, Liu Y, Zhang X, Zhang J. Anterior thalamic nucleus stimulation protects hippocampal neurons by activating autophagy in epileptic monkeys. Aging (Albany NY) 2020; 12:6324-6339. [PMID: 32267832 PMCID: PMC7185094 DOI: 10.18632/aging.103026] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2019] [Accepted: 02/05/2020] [Indexed: 12/29/2022]
Abstract
Deep brain stimulation of the anterior nucleus of the thalamus (ANT-DBS) is effective in treating temporal lobe epilepsy (TLE) and protects hippocampal neurons. Autophagy plays an essential role in epileptogenesis; however, the underlying effect of autophagy on ANT-DBS-mediated neuroprotection remains unclear. A monkey model of epilepsy was established by injecting kainic acid into the hippocampus and amygdala using a robot-assisted system. ANT-DBS was delivered in the chronic stage of the epileptic model and continued for 8 weeks. We found that ANT-DBS reduced the frequency of seizures and exerted neuroprotective effects via activating autophagy in hippocampal neurons. ANT-DBS increased light chain 3 (LC3) II level and co-localization of LC3 and lysosomal-associated membrane protein-1, accompanied by decreased expression of the autophagy substrate ubiquitin-binding protein p62, suggesting increased autophagosome formation. Most importantly, brain-derived neurotrophic factor (BDNF) –tropomyosin-related kinase type B (TrkB) pathway were involved in the regulation of autophagy. Both protein levels were reduced by ANT-DBS, and there was less phosphorylation of downstream regulators, extracellular signal-regulated kinase and Akt, followed by inactivation of mammalian target of rapamycin complex 1. Taken together, chronic ANT-DBS exerts neuroprotective effects on hippocampal neurons through inducing autophagy via suppressing the BDNF–TrkB pathway in a TLE monkey model.
Collapse
Affiliation(s)
- Ting-Ting Du
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing 100070, China
| | - Guanyu Zhu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Yingchuan Chen
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Lin Shi
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Defeng Liu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Yuye Liu
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China
| | - Xin Zhang
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing 100070, China
| | - Jianguo Zhang
- Department of Functional Neurosurgery, Beijing Neurosurgical Institute, Capital Medical University, Beijing 100070, China.,Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China.,Beijing Key Laboratory of Neurostimulation, Beijing 100070, China
| |
Collapse
|
34
|
Abecassis ZA, Berceau BL, Win PH, García D, Xenias HS, Cui Q, Pamukcu A, Cherian S, Hernández VM, Chon U, Lim BK, Kim Y, Justice NJ, Awatramani R, Hooks BM, Gerfen CR, Boca SM, Chan CS. Npas1 +-Nkx2.1 + Neurons Are an Integral Part of the Cortico-pallido-cortical Loop. J Neurosci 2020; 40:743-768. [PMID: 31811030 PMCID: PMC6975296 DOI: 10.1523/jneurosci.1199-19.2019] [Citation(s) in RCA: 52] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2019] [Revised: 11/21/2019] [Accepted: 11/26/2019] [Indexed: 11/21/2022] Open
Abstract
Within the basal ganglia circuit, the external globus pallidus (GPe) is critically involved in motor control. Aside from Foxp2+ neurons and ChAT+ neurons that have been established as unique neuron types, there is little consensus on the classification of GPe neurons. Properties of the remaining neuron types are poorly defined. In this study, we leverage new mouse lines, viral tools, and molecular markers to better define GPe neuron subtypes. We found that Sox6 represents a novel, defining marker for GPe neuron subtypes. Lhx6+ neurons that lack the expression of Sox6 were devoid of both parvalbumin and Npas1. This result confirms previous assertions of the existence of a unique Lhx6+ population. Neurons that arise from the Dbx1+ lineage were similarly abundant in the GPe and displayed a heterogeneous makeup. Importantly, tracing experiments revealed that Npas1+-Nkx2.1+ neurons represent the principal noncholinergic, cortically-projecting neurons. In other words, they form the pallido-cortical arm of the cortico-pallido-cortical loop. Our data further show that pyramidal-tract neurons in the cortex collateralized within the GPe, forming a closed-loop system between the two brain structures. Overall, our findings reconcile some of the discrepancies that arose from differences in techniques or the reliance on preexisting tools. Although spatial distribution and electrophysiological properties of GPe neurons reaffirm the diversification of GPe subtypes, statistical analyses strongly support the notion that these neuron subtypes can be categorized under the two principal neuron classes: PV+ neurons and Npas1+ neurons.SIGNIFICANCE STATEMENT The poor understanding of the neuronal composition in the external globus pallidus (GPe) undermines our ability to interrogate its precise behavioral and disease involvements. In this study, 12 different genetic crosses were used, hundreds of neurons were electrophysiologically characterized, and >100,000 neurons were histologically- and/or anatomically-profiled. Our current study further establishes the segregation of GPe neuron classes and illustrates the complexity of GPe neurons in adult mice. Our results support the idea that Npas1+-Nkx2.1+ neurons are a distinct GPe neuron subclass. By providing a detailed analysis of the organization of the cortico-pallidal-cortical projection, our findings establish the cellular and circuit substrates that can be important for motor function and dysfunction.
Collapse
Affiliation(s)
- Zachary A Abecassis
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Brianna L Berceau
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Phyo H Win
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Daniela García
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Harry S Xenias
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Qiaoling Cui
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Arin Pamukcu
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Suraj Cherian
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Vivian M Hernández
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Uree Chon
- Department of Neural and Behavioral Sciences, College of Medicine, Pennsylvania State University, Hershey, Pennsylvania
| | - Byung Kook Lim
- Neurobiology Section, Biological Sciences Division, University of California San Diego, La Jolla, California
| | - Yongsoo Kim
- Department of Neural and Behavioral Sciences, College of Medicine, Pennsylvania State University, Hershey, Pennsylvania
| | - Nicholas J Justice
- Center for Metabolic and degenerative disease, Institute of Molecular Medicine, University of Texas, Houston, Texas
- Department of Integrative Pharmacology, University of Texas, Houston, Texas
| | - Raj Awatramani
- Department of Neurology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois
| | - Bryan M Hooks
- Department of Neurobiology, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania
| | - Charles R Gerfen
- Laboratory of Systems Neuroscience, National Institute of Mental Health, Bethesda, Maryland, and
| | - Simina M Boca
- Innovation Center for Biomedical Informatics, Georgetown University Medical Center, Washington, District of Columbia
| | - C Savio Chan
- Department of Physiology, Feinberg School of Medicine, Northwestern University, Chicago, Illinois,
| |
Collapse
|
35
|
Abstract
AIM Chronic low back pain (CLBP) is a highly prevalent and significant cause of disability which is often resistant to pharmacological management. Virtual reality (VR) is an emerging technology with the potential to influence CLBP, and has been suggested as an alternative to opioids for pain management. VR is a goalfocused, computer-simulated reality allowing modification of the user's experience of their perceived world. MATERIALS/METHODS A narrative review of peer-reviewed literature using a systematic search strategy, and sole reviewer for data extraction. CONCLUSIONS VR has demonstrated effectiveness in reducing acute, experimental and chronic pain. This review describes the theoretical basis of the therapeutic effects of VR on CLBP via three distinct mechanisms: distraction, neuromodulation and graded exposure therapy. Furthermore, clinical application will be considered, including discussion of ethical issues associated with the technology.Implications for rehabilitationVirtual reality (VR) is suggested as an alternative for opioids in the management of acute and chronic pain.The therapeutic mechanisms of VR in chronic low back pain (CLBP) are equivocal but include distraction, neuromodulation of body perception and graded exposure therapy.VR may show greater efficacy in patients with CLBP with associated kinesiophobia.VR may show greater effect with increased immersion.
Collapse
|
36
|
Dezfouli A, Balleine BW. Learning the structure of the world: The adaptive nature of state-space and action representations in multi-stage decision-making. PLoS Comput Biol 2019; 15:e1007334. [PMID: 31490932 PMCID: PMC6750884 DOI: 10.1371/journal.pcbi.1007334] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2018] [Revised: 09/18/2019] [Accepted: 08/12/2019] [Indexed: 01/26/2023] Open
Abstract
State-space and action representations form the building blocks of decision-making processes in the brain; states map external cues to the current situation of the agent whereas actions provide the set of motor commands from which the agent can choose to achieve specific goals. Although these factors differ across environments, it is currently unknown whether or how accurately state and action representations are acquired by the agent because previous experiments have typically provided this information a priori through instruction or pre-training. Here we studied how state and action representations adapt to reflect the structure of the world when such a priori knowledge is not available. We used a sequential decision-making task in rats in which they were required to pass through multiple states before reaching the goal, and for which the number of states and how they map onto external cues were unknown a priori. We found that, early in training, animals selected actions as if the task was not sequential and outcomes were the immediate consequence of the most proximal action. During the course of training, however, rats recovered the true structure of the environment and made decisions based on the expanded state-space, reflecting the multiple stages of the task. Similarly, we found that the set of actions expanded with training, although the emergence of new action sequences was sensitive to the experimental parameters and specifics of the training procedure. We conclude that the profile of choices shows a gradual shift from simple representations to more complex structures compatible with the structure of the world.
Collapse
Affiliation(s)
- Amir Dezfouli
- School of Psychology, UNSW, Sydney, Australia
- Data61, CSIRO, Sydney, Australia
| | | |
Collapse
|
37
|
Kwak S, Jung MW. Distinct roles of striatal direct and indirect pathways in value-based decision making. eLife 2019; 8:46050. [PMID: 31310237 PMCID: PMC6658164 DOI: 10.7554/elife.46050] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2019] [Accepted: 07/09/2019] [Indexed: 12/12/2022] Open
Abstract
The striatum is critically involved in value-based decision making. However, it is unclear how striatal direct and indirect pathways work together to make optimal choices in a dynamic and uncertain environment. Here, we examined the effects of selectively inactivating D1 receptor (D1R)- or D2 receptor (D2R)-expressing dorsal striatal neurons (corresponding to direct- and indirect-pathway neurons, respectively) on mouse choice behavior in a reversal task with progressively increasing reversal frequency and a dynamic two-armed bandit task. Inactivation of either D1R- or D2R-expressing striatal neurons impaired performance in both tasks, but the pattern of altered choice behavior differed between the two animal groups. A reinforcement learning model-based analysis indicated that inactivation of D1R- and D2R-expressing striatal neurons selectively impairs value-dependent action selection and value learning, respectively. Our results suggest differential contributions of striatal direct and indirect pathways to two distinct steps in value-based decision making.
Collapse
Affiliation(s)
- Shinae Kwak
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Min Whan Jung
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| |
Collapse
|
38
|
Jensen G, Alkan Y, Ferrera VP, Terrace HS. Reward associations do not explain transitive inference performance in monkeys. SCIENCE ADVANCES 2019; 5:eaaw2089. [PMID: 32128384 PMCID: PMC7032924 DOI: 10.1126/sciadv.aaw2089] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/28/2018] [Accepted: 06/21/2019] [Indexed: 06/10/2023]
Abstract
Most accounts of behavior in nonhuman animals assume that they make choices to maximize expected reward value. However, model-free reinforcement learning based on reward associations cannot account for choice behavior in transitive inference paradigms. We manipulated the amount of reward associated with each item of an ordered list, so that maximizing expected reward value was always in conflict with decision rules based on the implicit list order. Under such a schedule, model-free reinforcement algorithms cannot achieve high levels of accuracy, even after extensive training. Monkeys nevertheless learned to make correct rule-based choices. These results show that monkeys' performance in transitive inference paradigms is not driven solely by expected reward and that appropriate inferences are made despite discordant reward incentives. We show that their choices can be explained by an abstract, model-based representation of list order, and we provide a method for inferring the contents of such representations from observed data.
Collapse
Affiliation(s)
- Greg Jensen
- Department of Psychology, Columbia University, New York, NY 10027, USA
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA
| | - Yelda Alkan
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA
- Department of Neuroscience, Columbia University, New York, NY 10027, USA
| | - Vincent P. Ferrera
- Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027, USA
- Department of Neuroscience, Columbia University, New York, NY 10027, USA
- Department of Psychology and Psychiatry, Columbia University, New York, NY 10027, USA
| | | |
Collapse
|
39
|
Constantinople CM, Piet AT, Brody CD. An Analysis of Decision under Risk in Rats. Curr Biol 2019; 29:2066-2074.e5. [PMID: 31155352 PMCID: PMC6863753 DOI: 10.1016/j.cub.2019.05.013] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2018] [Revised: 03/06/2019] [Accepted: 05/01/2019] [Indexed: 01/29/2023]
Abstract
In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled "Prospect Theory: An Analysis of Decision under Risk," which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists' normative workhorse model, Expected Utility Theory [1, 2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, prospect theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3-7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to prospect theory reproduced rats' nonlinear utility and probability weighting functions and also captured trial-by-trial learning dynamics.
Collapse
Affiliation(s)
| | - Alex T Piet
- Princeton Neuroscience Institute, Princeton University, Washington Road, Princeton, NJ 08544, USA
| | - Carlos D Brody
- Princeton Neuroscience Institute, Princeton University, Washington Road, Princeton, NJ 08544, USA; Department of Molecular Biology, Princeton University, Washington Road, Princeton, NJ 08544, USA; Howard Hughes Medical Institute, Princeton University, Washington Road, Princeton, NJ 08544, USA
| |
Collapse
|
40
|
Alemanno F, Houdayer E, Emedoli D, Locatelli M, Mortini P, Mandelli C, Raggi A, Iannaccone S. Efficacy of virtual reality to reduce chronic low back pain: Proof-of-concept of a non-pharmacological approach on pain, quality of life, neuropsychological and functional outcome. PLoS One 2019; 14:e0216858. [PMID: 31120892 PMCID: PMC6532874 DOI: 10.1371/journal.pone.0216858] [Citation(s) in RCA: 52] [Impact Index Per Article: 10.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2018] [Accepted: 04/30/2019] [Indexed: 12/29/2022] Open
Abstract
Objectives Chronic pain, such as low-back pain, can be a highly disabling condition degrading people’s quality of life (QoL). Not every patient responds to pharmacological therapies, thus alternative treatments have to be developed. The chronicity of pain can lead to a somatic dysperception, meaning a mismatch between patients’ own body perception and its actual physical state. Since clinical evaluation of pain relies on patients’ subjective reports, a body image disruption can be associated with an incorrect pain rating inducing incorrect treatment and a possible risk of drug abuse. Our aim was to reduce chronic low-back pain through a multimodal neurorehabilitative strategy using innovative technologies to help patients regain a correct body image. Methods Twenty patients with chronic low-back pain were included. Before and after treatment, patients underwent: a neurological exam; a neuro-psychological evaluation testing cognitive functions (memory, attention, executive functions) and personality traits, QoL and mood; pain ratings; sensorimotor functional abilities’ testing. Patients underwent a 6 week-neurorehabilitative treatment (total 12 sessions) using virtual reality (VRRS system, Khymeia, Italy). Treatment consisted on teaching patients to execute correct movements with the painful body parts to regain a correct body image, based on the augmented multisensory feedback (auditory, visual) provided by the VRRS. Results Our data showed significant reductions in all pain rating scale scores (p<0.05); significant improvements of QoL in the domains of physical functioning, physical role functioning, bodily pain, vitality, and social role functioning; improvements in cognitive functions (p<0.05); improvements in functional scales (p<0.05) and mood (p = 0.04). Conclusion This non-pharmacological approach was able to act on the multi-dimensional aspects of pain and improved patients’ QoL, pain intensity, mood and patient’s functional abilities.
Collapse
Affiliation(s)
- Federica Alemanno
- Department of Rehabilitation and Functional Recovery, IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy
- * E-mail:
| | - Elise Houdayer
- Department of Rehabilitation and Functional Recovery, IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy
| | - Daniele Emedoli
- Department of Rehabilitation and Functional Recovery, IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy
| | - Matteo Locatelli
- Department of Rehabilitation and Functional Recovery, IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy
| | - Pietro Mortini
- Department of Neurosurgery and Gamma Knife Radiosurgery, IRCCS San Raffaele Hospital and Vita-Salute University, Milan, Italy
| | - Carlo Mandelli
- Department of Neurosurgery and Gamma Knife Radiosurgery, IRCCS San Raffaele Hospital and Vita-Salute University, Milan, Italy
| | - Alberto Raggi
- Unit of Neurology, G.B. Morgagni – L. Pierantoni Hospital, Forlì, Italy
| | - Sandro Iannaccone
- Department of Rehabilitation and Functional Recovery, IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy
| |
Collapse
|
41
|
Jeong Y, Huh N, Lee J, Yun I, Lee JW, Lee I, Jung MW. Role of the hippocampal CA1 region in incremental value learning. Sci Rep 2018; 8:9870. [PMID: 29959363 PMCID: PMC6026161 DOI: 10.1038/s41598-018-28176-5] [Citation(s) in RCA: 15] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2018] [Accepted: 06/18/2018] [Indexed: 12/30/2022] Open
Abstract
It is generally believed that the hippocampus plays a crucial role in declarative memory—remembering facts and events—but not in gradual stimulus-response association or incremental value learning. Based on the finding that CA1 conveys strong value signals during dynamic foraging, we investigated the possibility that the hippocampus contributes to incremental value learning. Specifically, we examined effects of inactivating different subregions of the dorsal hippocampus on behavioral performance of mice performing a dynamic foraging task in a modified T-maze. A reinforcement learning model-based analysis indicated that inactivation of CA1, but not dentate gyrus, CA3, or CA2, impaired trial-by-trial updating of chosen value without affecting value-dependent action selection. As a result, it took longer for CA1-inactivated mice to bias their choices toward the higher-reward-probability target after changes in reward probability. Our results indicate, contrary to the traditional view, that the hippocampus, especially CA1, might contribute to incremental value learning under certain circumstances.
Collapse
Affiliation(s)
- Yeongseok Jeong
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, 34141, Korea
| | - Namjung Huh
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, 34141, Korea
| | - Joonyeup Lee
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, 34141, Korea.,Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, 34141, Korea
| | - Injae Yun
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, 34141, Korea
| | - Jong Won Lee
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, 34141, Korea
| | - Inah Lee
- Department of Brain and Cognitive Science, Seoul National University, Seoul, 08826, Korea
| | - Min Whan Jung
- Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, 34141, Korea. .,Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, 34141, Korea.
| |
Collapse
|
42
|
Zeng Y, Wang G, Xu B. A Basal Ganglia Network Centric Reinforcement Learning Model and Its Application in Unmanned Aerial Vehicle. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2649564] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
43
|
Min K, Iwamoto M, Kakei S, Kimpara H. Muscle Synergy-Driven Robust Motion Control. Neural Comput 2018; 30:1104-1131. [PMID: 29381443 DOI: 10.1162/neco_a_01063] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022]
Abstract
Humans are able to robustly maintain desired motion and posture under dynamically changing circumstances, including novel conditions. To accomplish this, the brain needs to optimize the synergistic control between muscles against external dynamic factors. However, previous related studies have usually simplified the control of multiple muscles using two opposing muscles, which are minimum actuators to simulate linear feedback control. As a result, they have been unable to analyze how muscle synergy contributes to motion control robustness in a biological system. To address this issue, we considered a new muscle synergy concept used to optimize the synergy between muscle units against external dynamic conditions, including novel conditions. We propose that two main muscle control policies synergistically control muscle units to maintain the desired motion against external dynamic conditions. Our assumption is based on biological evidence regarding the control of multiple muscles via the corticospinal tract. One of the policies is the group control policy (GCP), which is used to control muscle group units classified based on functional similarities in joint control. This policy is used to effectively resist external dynamic circumstances, such as disturbances. The individual control policy (ICP) assists the GCP in precisely controlling motion by controlling individual muscle units. To validate this hypothesis, we simulated the reinforcement of the synergistic actions of the two control policies during the reinforcement learning of feedback motion control. Using this learning paradigm, the two control policies were synergistically combined to result in robust feedback control under novel transient and sustained disturbances that did not involve learning. Further, by comparing our data to experimental data generated by human subjects under the same conditions as those of the simulation, we showed that the proposed synergy concept may be used to analyze muscle synergy-driven motion control robustness in humans.
Collapse
Affiliation(s)
- Kyuengbo Min
- Tokyo Metropolitan Institute of Medical Science, Tokyo 156-8506, Japan
| | | | - Shinji Kakei
- Tokyo Metropolitan Institute of Medical Science, Tokyo 156-8506, Japan
| | | |
Collapse
|
44
|
Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways. Nat Commun 2018; 9:404. [PMID: 29374173 PMCID: PMC5786099 DOI: 10.1038/s41467-017-02817-1] [Citation(s) in RCA: 45] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/27/2017] [Accepted: 12/31/2017] [Indexed: 11/30/2022] Open
Abstract
The direct and indirect pathways of the basal ganglia have long been thought to mediate behavioral promotion and inhibition, respectively. However, this classic dichotomous model has been recently challenged. To better understand neural processes underlying reward-based learning and movement control, we recorded from direct (dSPNs) and indirect (iSPNs) pathway spiny projection neurons in the dorsomedial striatum of D1-Cre and D2-Cre mice performing a probabilistic Pavlovian conditioning task. dSPNs tend to increase activity while iSPNs decrease activity as a function of reward value, suggesting the striatum represents value in the relative activity levels of dSPNs versus iSPNs. Lick offset-related activity increase is largely dSPN selective, suggesting dSPN involvement in suppressing ongoing licking behavior. Rapid responses to negative outcome and previous reward-related responses are more frequent among iSPNs than dSPNs, suggesting stronger contributions of iSPNs to outcome-dependent behavioral adjustment. These findings provide new insights into striatal neural circuit operations. Classically the direct and indirect pathways of basal ganglia are understood to have opposing roles in movement and reward learning, but recent work suggests a more complicated view. Here the authors further study indirect and direct pathway neurons, in the context of a probabilistic reward task.
Collapse
|
45
|
Pérez-Díaz F, Díaz E, Sánchez N, Vargas JP, Pearce JM, López JC. Different involvement of medial prefrontal cortex and dorso-lateral striatum in automatic and controlled processing of a future conditioned stimulus. PLoS One 2017; 12:e0189630. [PMID: 29240804 PMCID: PMC5730208 DOI: 10.1371/journal.pone.0189630] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2017] [Accepted: 11/29/2017] [Indexed: 11/30/2022] Open
Abstract
Recent studies support the idea that stimulus processing in latent inhibition can vary during the course of preexposure. Controlled attentional mechanisms are said to be important in the early stages of preexposure, while in later stages animals adopt automatic processing of the stimulus to be used for conditioning. Given this distinction, it is possible that both types of processing are governed by different neural systems, affecting differentially the retrieval of information about the stimulus. In the present study we tested if a lesion to the dorso-lateral striatum or to the medial prefrontal cortex has a selective effect on exposure to the future conditioned stimulus (CS). With this aim, animals received different amounts of exposure to the future CS. The results showed that a lesion to the medial prefrontal cortex enhanced latent inhibition in animals receiving limited preexposure to the CS, but had no effect in animals receiving extended preexposure to the CS. The lesion of the dorso-lateral striatum produced a decrease in latent inhibition, but only in animals with an extended exposure to the future conditioned stimulus. These results suggest that the dorsal striatum and medial prefrontal cortex play essential roles in controlled and automatic processes. Automatic attentional processes appear to be impaired by a lesion to the dorso-lateral striatum and facilitated by a lesion to the prefrontal cortex.
Collapse
Affiliation(s)
- Francisco Pérez-Díaz
- Animal Behav & Neurosci Lab, Dpt. Psicología Experimental, Universidad de Sevilla, c/ Camilo Jose Cela s/n, Seville, Spain
| | - Estrella Díaz
- Animal Behav & Neurosci Lab, Dpt. Psicología Experimental, Universidad de Sevilla, c/ Camilo Jose Cela s/n, Seville, Spain
| | - Natividad Sánchez
- Animal Behav & Neurosci Lab, Dpt. Psicología Experimental, Universidad de Sevilla, c/ Camilo Jose Cela s/n, Seville, Spain
| | - Juan Pedro Vargas
- Animal Behav & Neurosci Lab, Dpt. Psicología Experimental, Universidad de Sevilla, c/ Camilo Jose Cela s/n, Seville, Spain
| | - John M. Pearce
- School of Psychology, Cardiff University, Cardiff, Wales, United Kingdom
- School of Psychology, University of Sydney, Sydney, Australia
| | - Juan Carlos López
- Animal Behav & Neurosci Lab, Dpt. Psicología Experimental, Universidad de Sevilla, c/ Camilo Jose Cela s/n, Seville, Spain
| |
Collapse
|
46
|
Illuminating Neural Circuits: From Molecules to MRI. J Neurosci 2017; 37:10817-10825. [PMID: 29118210 DOI: 10.1523/jneurosci.2569-17.2017] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2017] [Revised: 09/22/2017] [Accepted: 09/26/2017] [Indexed: 11/21/2022] Open
Abstract
Neurological disease drives symptoms through pathological changes to circuit functions. Therefore, understanding circuit mechanisms that drive behavioral dysfunction is of critical importance for quantitative diagnosis and systematic treatment of neurological disease. Here, we describe key technologies that enable measurement and manipulation of neural activity and neural circuits. Applying these approaches led to the discovery of circuit mechanisms underlying pathological motor behavior, arousal regulation, and protein accumulation. Finally, we discuss how optogenetic functional magnetic resonance imaging reveals global scale circuit mechanisms, and how circuit manipulations could lead to new treatments of neurological diseases.
Collapse
|
47
|
Viejo G, Girard B, Procyk E, Khamassi M. Adaptive coordination of working-memory and reinforcement learning in non-human primates performing a trial-and-error problem solving task. Behav Brain Res 2017; 355:76-89. [PMID: 29061387 DOI: 10.1016/j.bbr.2017.09.030] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2017] [Revised: 08/08/2017] [Accepted: 09/18/2017] [Indexed: 10/18/2022]
Abstract
Accumulating evidence suggest that human behavior in trial-and-error learning tasks based on decisions between discrete actions may involve a combination of reinforcement learning (RL) and working-memory (WM). While the understanding of brain activity at stake in this type of tasks often involve the comparison with non-human primate neurophysiological results, it is not clear whether monkeys use similar combined RL and WM processes to solve these tasks. Here we analyzed the behavior of five monkeys with computational models combining RL and WM. Our model-based analysis approach enables to not only fit trial-by-trial choices but also transient slowdowns in reaction times, indicative of WM use. We found that the behavior of the five monkeys was better explained in terms of a combination of RL and WM despite inter-individual differences. The same coordination dynamics we used in a previous study in humans best explained the behavior of some monkeys while the behavior of others showed the opposite pattern, revealing a possible different dynamics of WM process. We further analyzed different variants of the tested models to open a discussion on how the long pretraining in these tasks may have favored particular coordination dynamics between RL and WM. This points towards either inter-species differences or protocol differences which could be further tested in humans.
Collapse
Affiliation(s)
- Guillaume Viejo
- Sorbonne Universités, UPMC Univ Paris 06, CNRS, Institute of Intelligent Systems and Robotics (ISIR), F-75005 Paris, France; Montreal Neurological Institute and Hospital, 3801 University Street, Montreal, Quebec, Canada.
| | - Benoît Girard
- Sorbonne Universités, UPMC Univ Paris 06, CNRS, Institute of Intelligent Systems and Robotics (ISIR), F-75005 Paris, France
| | - Emmanuel Procyk
- University of Lyon, Université Claude Bernard Lyon 1, INSERM, Stem Cell and Brain Research Institute, U1208 Lyon, France
| | - Mehdi Khamassi
- Sorbonne Universités, UPMC Univ Paris 06, CNRS, Institute of Intelligent Systems and Robotics (ISIR), F-75005 Paris, France
| |
Collapse
|
48
|
Vuong J, Devergnas A. The role of the basal ganglia in the control of seizure. J Neural Transm (Vienna) 2017; 125:531-545. [PMID: 28766041 DOI: 10.1007/s00702-017-1768-x] [Citation(s) in RCA: 35] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2017] [Accepted: 07/23/2017] [Indexed: 12/19/2022]
Abstract
Epilepsy is a network disorder and each type of seizure involves distinct cortical and subcortical network, differently implicated in the control and propagation of the ictal activity. The role of the basal ganglia has been revealed in several cases of focal and generalized seizures. Here, we review the data that show the implication of the basal ganglia in absence, temporal lobe, and neocortical seizures in animal models (rodent, cat, and non-human primate) and in human. Based on these results and the advancement of deep brain stimulation for Parkinson's disease, basal ganglia neuromodulation has been tested with some success that can be equally seen as promising or disappointing. The effect of deep brain stimulation can be considered promising with a 76% in seizure reduction in temporal lobe epilepsy patients, but also disappointing, since only few patients have become seizure free and the antiepileptic effects have been highly variable among patients. This variability could probably be explained by the heterogeneity among the patients included in these clinical studies. To illustrate the importance of specific network identification, electrophysiological activity of the putamen and caudate nucleus has been recorded during penicillin-induced pre-frontal and motor seizures in one monkey. While an increase of the firing rate was found in putamen and caudate nucleus during pre-frontal seizures, only the activity of the putamen cells was increased during motor seizures. These preliminary results demonstrate the implication of the basal ganglia in two types of neocortical seizures and the necessity of studying the network to identify the important nodes implicated in the propagation and control of each type of seizure.
Collapse
Affiliation(s)
- J Vuong
- Yerkes National Primate Research Center, Emory University, 954 Gatewood Road NE, Atlanta, GA, 30329, USA
| | - Annaelle Devergnas
- Yerkes National Primate Research Center, Emory University, 954 Gatewood Road NE, Atlanta, GA, 30329, USA. .,Department of Neurology, Emory University, Atlanta, GA, 30322, USA.
| |
Collapse
|
49
|
Chalmers E, Luczak A, Gruber AJ. Computational Properties of the Hippocampus Increase the Efficiency of Goal-Directed Foraging through Hierarchical Reinforcement Learning. Front Comput Neurosci 2016; 10:128. [PMID: 28018203 PMCID: PMC5149552 DOI: 10.3389/fncom.2016.00128] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2016] [Accepted: 11/28/2016] [Indexed: 11/17/2022] Open
Abstract
The mammalian brain is thought to use a version of Model-based Reinforcement Learning (MBRL) to guide “goal-directed” behavior, wherein animals consider goals and make plans to acquire desired outcomes. However, conventional MBRL algorithms do not fully explain animals' ability to rapidly adapt to environmental changes, or learn multiple complex tasks. They also require extensive computation, suggesting that goal-directed behavior is cognitively expensive. We propose here that key features of processing in the hippocampus support a flexible MBRL mechanism for spatial navigation that is computationally efficient and can adapt quickly to change. We investigate this idea by implementing a computational MBRL framework that incorporates features inspired by computational properties of the hippocampus: a hierarchical representation of space, “forward sweeps” through future spatial trajectories, and context-driven remapping of place cells. We find that a hierarchical abstraction of space greatly reduces the computational load (mental effort) required for adaptation to changing environmental conditions, and allows efficient scaling to large problems. It also allows abstract knowledge gained at high levels to guide adaptation to new obstacles. Moreover, a context-driven remapping mechanism allows learning and memory of multiple tasks. Simulating dorsal or ventral hippocampal lesions in our computational framework qualitatively reproduces behavioral deficits observed in rodents with analogous lesions. The framework may thus embody key features of how the brain organizes model-based RL to efficiently solve navigation and other difficult tasks.
Collapse
Affiliation(s)
- Eric Chalmers
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| | - Artur Luczak
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| | - Aaron J Gruber
- Department of Neuroscience, University of Lethbridge Lethbridge, AB, Canada
| |
Collapse
|
50
|
Babič J, Oztop E, Kawato M. Human motor adaptation in whole body motion. Sci Rep 2016; 6:32868. [PMID: 27608652 PMCID: PMC5016968 DOI: 10.1038/srep32868] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2016] [Accepted: 08/16/2016] [Indexed: 11/18/2022] Open
Abstract
The main role of the sensorimotor system of an organism is to increase the survival of the species. Therefore, to understand the adaptation and optimality mechanisms of motor control, it is necessary to study the sensorimotor system in terms of ecological fitness. We designed an experimental paradigm that exposed sensorimotor system to risk of injury. We studied human subjects performing uncon- strained squat-to-stand movements that were systematically subjected to non-trivial perturbation. We found that subjects adapted by actively compensating the perturbations, converging to movements that were different from their normal unperturbed squat-to-stand movements. Furthermore, the adapted movements had clear intrinsic inter-subject differences which could be explained by different adapta- tion strategies employed by the subjects. These results suggest that classical optimality measures of physical energy and task satisfaction should be seen as part of a hierarchical organization of optimality with safety being at the highest level. Therefore, in addition to physical energy and task fulfillment, the risk of injury and other possible costs such as neural computational overhead have to be considered when analyzing human movement.
Collapse
Affiliation(s)
- Jan Babič
- Jožef Stefan Institute, Ljubljana, Slovenia
| | | | - Mitsuo Kawato
- ATR Brain Information Communication Research Laboratory Group, Kyoto, Japan
| |
Collapse
|