1
|
Kim T, Lee SW, Lho SK, Moon SY, Kim M, Kwon JS. Neurocomputational model of compulsivity: deviating from an uncertain goal-directed system. Brain 2024; 147:2230-2244. [PMID: 38584499 PMCID: PMC11146420 DOI: 10.1093/brain/awae102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 02/18/2024] [Accepted: 03/07/2024] [Indexed: 04/09/2024] Open
Abstract
Despite a theory that an imbalance in goal-directed versus habitual systems serve as building blocks of compulsions, research has yet to delineate how this occurs during arbitration between the two systems in obsessive-compulsive disorder. Inspired by a brain model in which the inferior frontal cortex selectively gates the putamen to guide goal-directed or habitual actions, this study aimed to examine whether disruptions in the arbitration process via the fronto-striatal circuit would underlie imbalanced decision-making and compulsions in patients. Thirty patients with obsessive-compulsive disorder [mean (standard deviation) age = 26.93 (6.23) years, 12 females (40%)] and 30 healthy controls [mean (standard deviation) age = 24.97 (4.72) years, 17 females (57%)] underwent functional MRI scans while performing the two-step Markov decision task, which was designed to dissociate goal-directed behaviour from habitual behaviour. We employed a neurocomputational model to account for an uncertainty-based arbitration process, in which a prefrontal arbitrator (i.e. inferior frontal gyrus) allocates behavioural control to a more reliable strategy by selectively gating the putamen. We analysed group differences in the neural estimates of uncertainty of each strategy. We also compared the psychophysiological interaction effects of system preference (goal-directed versus habitual) on fronto-striatal coupling between groups. We examined the correlation between compulsivity score and the neural activity and connectivity involved in the arbitration process. The computational model captured the subjects' preferences between the strategies. Compared with healthy controls, patients had a stronger preference for the habitual system (t = -2.88, P = 0.006), which was attributed to a more uncertain goal-directed system (t = 2.72, P = 0.009). Before the allocation of controls, patients exhibited hypoactivity in the inferior frontal gyrus compared with healthy controls when this region tracked the inverse of uncertainty (i.e. reliability) of goal-directed behaviour (P = 0.001, family-wise error rate corrected). When reorienting behaviours to reach specific goals, patients exhibited weaker right ipsilateral ventrolateral prefronto-putamen coupling than healthy controls (P = 0.001, family-wise error rate corrected). This hypoconnectivity was correlated with more severe compulsivity (r = -0.57, P = 0.002). Our findings suggest that the attenuated top-down control of the putamen by the prefrontal arbitrator underlies compulsivity in obsessive-compulsive disorder. Enhancing fronto-striatal connectivity may be a potential neurotherapeutic approach for compulsivity and adaptive decision-making.
Collapse
Affiliation(s)
- Taekwan Kim
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Seoul 08826, Republic of Korea
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Neuroscience-inspired Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Neuroscience-inspired Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| | - Silvia Kyungjin Lho
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
| | - Sun-Young Moon
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
| | - Minah Kim
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul 03080, Republic of Korea
| | - Jun Soo Kwon
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Seoul 08826, Republic of Korea
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul 03080, Republic of Korea
| |
Collapse
|
2
|
Han D, Doya K, Li D, Tani J. Synergizing habits and goals with variational Bayes. Nat Commun 2024; 15:4461. [PMID: 38796491 DOI: 10.1038/s41467-024-48577-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 05/06/2024] [Indexed: 05/28/2024] Open
Abstract
Behaving efficiently and flexibly is crucial for biological and artificial embodied agents. Behavior is generally classified into two types: habitual (fast but inflexible), and goal-directed (flexible but slow). While these two types of behaviors are typically considered to be managed by two distinct systems in the brain, recent studies have revealed a more sophisticated interplay between them. We introduce a theoretical framework using variational Bayesian theory, incorporating a Bayesian intention variable. Habitual behavior depends on the prior distribution of intention, computed from sensory context without goal-specification. In contrast, goal-directed behavior relies on the goal-conditioned posterior distribution of intention, inferred through variational free energy minimization. Assuming that an agent behaves using a synergized intention, our simulations in vision-based sensorimotor tasks explain the key properties of their interaction as observed in experiments. Our work suggests a fresh perspective on the neural mechanisms of habits and goals, shedding light on future research in decision making.
Collapse
Affiliation(s)
- Dongqi Han
- Microsoft Research Asia, Shanghai, 200232, China.
| | - Kenji Doya
- Okinawa Institute of Science and Technology, Okinawa, 904-0495, Japan
| | - Dongsheng Li
- Microsoft Research Asia, Shanghai, 200232, China
| | - Jun Tani
- Okinawa Institute of Science and Technology, Okinawa, 904-0495, Japan
| |
Collapse
|
3
|
Robbins TW, Banca P, Belin D. From compulsivity to compulsion: the neural basis of compulsive disorders. Nat Rev Neurosci 2024; 25:313-333. [PMID: 38594324 DOI: 10.1038/s41583-024-00807-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/04/2024] [Indexed: 04/11/2024]
Abstract
Compulsive behaviour, an apparently irrational perseveration in often maladaptive acts, is a potential transdiagnostic symptom of several neuropsychiatric disorders, including obsessive-compulsive disorder and addiction, and may reflect the severe manifestation of a dimensional trait termed compulsivity. In this Review, we examine the psychological basis of compulsions and compulsivity and their underlying neural circuitry using evidence from human neuroimaging and animal models. Several main elements of this circuitry are identified, focused on fronto-striatal systems implicated in goal-directed behaviour and habits. These systems include the orbitofrontal, prefrontal, anterior cingulate and insular cortices and their connections with the basal ganglia as well as sensoriomotor and parietal cortices and cerebellum. We also consider the implications for future classification of impulsive-compulsive disorders and their treatment.
Collapse
Affiliation(s)
- Trevor W Robbins
- Behavioural and Clinical Neuroscience Institute, Department of Psychology, University of Cambridge, Cambridge, UK.
| | - Paula Banca
- Behavioural and Clinical Neuroscience Institute, Department of Psychology, University of Cambridge, Cambridge, UK
| | - David Belin
- Behavioural and Clinical Neuroscience Institute, Department of Psychology, University of Cambridge, Cambridge, UK
| |
Collapse
|
4
|
Philippe R, Janet R, Khalvati K, Rao RPN, Lee D, Dreher JC. Neurocomputational mechanisms involved in adaptation to fluctuating intentions of others. Nat Commun 2024; 15:3189. [PMID: 38609372 PMCID: PMC11014977 DOI: 10.1038/s41467-024-47491-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Accepted: 03/12/2024] [Indexed: 04/14/2024] Open
Abstract
Humans frequently interact with agents whose intentions can fluctuate between competition and cooperation over time. It is unclear how the brain adapts to fluctuating intentions of others when the nature of the interactions (to cooperate or compete) is not explicitly and truthfully signaled. Here, we use model-based fMRI and a task in which participants thought they were playing with another player. In fact, they played with an algorithm that alternated without signaling between cooperative and competitive strategies. We show that a neurocomputational mechanism with arbitration between competitive and cooperative experts outperforms other learning models in predicting choice behavior. At the brain level, the fMRI results show that the ventral striatum and ventromedial prefrontal cortex track the difference of reliability between these experts. When attributing competitive intentions, we find increased coupling between these regions and a network that distinguishes prediction errors related to competition and cooperation. These findings provide a neurocomputational account of how the brain arbitrates dynamically between cooperative and competitive intentions when making adaptive social decisions.
Collapse
Affiliation(s)
- Rémi Philippe
- CNRS-Institut des Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Lyon, France
- Université Claude Bernard Lyon 1, Lyon, France
| | - Rémi Janet
- CNRS-Institut des Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Lyon, France
- Université Claude Bernard Lyon 1, Lyon, France
| | - Koosha Khalvati
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
| | - Rajesh P N Rao
- Paul G. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA
- Center for Neurotechnology, University of Washington, Seattle, WA, USA
| | - Daeyeol Lee
- Zanvyl Krieger Mind/Brain Institute, Johns Hopkins University, Baltimore, MD, USA
- Kavli Discovery Neuroscience Institute, Johns Hopkins University, Baltimore, MD, USA
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
- Department of Neuroscience, Johns Hopkins University, Baltimore, MD, USA
| | - Jean-Claude Dreher
- CNRS-Institut des Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Lyon, France.
- Université Claude Bernard Lyon 1, Lyon, France.
| |
Collapse
|
5
|
Oyama K, Majima K, Nagai Y, Hori Y, Hirabayashi T, Eldridge MAG, Mimura K, Miyakawa N, Fujimoto A, Hori Y, Iwaoki H, Inoue KI, Saunders RC, Takada M, Yahata N, Higuchi M, Richmond BJ, Minamimoto T. Distinct roles of monkey OFC-subcortical pathways in adaptive behavior. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.17.567492. [PMID: 38076986 PMCID: PMC10705585 DOI: 10.1101/2023.11.17.567492] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/22/2023]
Abstract
To be the most successful, primates must adapt to changing environments and optimize their behavior by making the most beneficial choices. At the core of adaptive behavior is the orbitofrontal cortex (OFC) of the brain, which updates choice value through direct experience or knowledge-based inference. Here, we identify distinct neural circuitry underlying these two separate abilities. We designed two behavioral tasks in which macaque monkeys updated the values of certain items, either by directly experiencing changes in stimulus-reward associations, or by inferring the value of unexperienced items based on the task's rules. Chemogenetic silencing of bilateral OFC combined with mathematical model-fitting analysis revealed that monkey OFC is involved in updating item value based on both experience and inference. In vivo imaging of chemogenetic receptors by positron emission tomography allowed us to map projections from the OFC to the rostromedial caudate nucleus (rmCD) and the medial part of the mediodorsal thalamus (MDm). Chemogenetic silencing of the OFC-rmCD pathway impaired experience-based value updating, while silencing the OFC-MDm pathway impaired inference-based value updating. Our results thus demonstrate a dissociable contribution of distinct OFC projections to different behavioral strategies, and provide new insights into the neural basis of value-based adaptive decision-making in primates.
Collapse
Affiliation(s)
- Kei Oyama
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
- PRESTO, Japan Science and Technology Agency, Kawaguchi, Japan
| | - Kei Majima
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba, Japan
- PRESTO, Japan Science and Technology Agency, Kawaguchi, Japan
| | - Yuji Nagai
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yukiko Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Toshiyuki Hirabayashi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Koki Mimura
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
- Research Center for Medical and Health Data Science, The Institute of Statistical Mathematics, Tachikawa, Japan
| | - Naohisa Miyakawa
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Atsushi Fujimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Yuki Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Haruhiko Iwaoki
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Ken-Ichi Inoue
- Systems Neuroscience Section, Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Japan
| | - Richard C Saunders
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Masahiko Takada
- Systems Neuroscience Section, Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama, Japan
| | - Noriaki Yahata
- Institute for Quantum Life Science, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Makoto Higuchi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, USA
| | - Takafumi Minamimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba, Japan
| |
Collapse
|
6
|
Ruan Z, Seger CA, Yang Q, Kim D, Lee SW, Chen Q, Peng Z. Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder. Front Psychiatry 2023; 14:1162800. [PMID: 37304449 PMCID: PMC10250695 DOI: 10.3389/fpsyt.2023.1162800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Accepted: 05/05/2023] [Indexed: 06/13/2023] Open
Abstract
Introduction Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time. Methods A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis. Results Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal. Conclusion These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.
Collapse
Affiliation(s)
- Zhongqiang Ruan
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
| | - Carol A. Seger
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Department of Psychology, Colorado State University, Fort Collins, CO, United States
| | - Qiong Yang
- Affective Disorder Center, Affiliated Brain Hospital of Guangzhou Medical University (Guangzhou Huiai Hospital), Guangzhou, China
| | - Dongjae Kim
- Department of AI-based Convergence, College of Engineering, Dankook University, Yongin, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Qi Chen
- School of Psychology, Shenzhen University, Shenzhen, China
| | - Ziwen Peng
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou, China
- Department of Child Psychiatry, Shenzhen Kangning Hospital, Shenzhen University School of Medicine, Shenzhen, China
| |
Collapse
|
7
|
Fan C, Yao L, Zhang J, Zhen Z, Wu X. Advanced Reinforcement Learning and Its Connections with Brain Neuroscience. RESEARCH (WASHINGTON, D.C.) 2023; 6:0064. [PMID: 36939448 PMCID: PMC10017102 DOI: 10.34133/research.0064] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/27/2022] [Accepted: 01/10/2023] [Indexed: 01/22/2023]
Abstract
In recent years, brain science and neuroscience have greatly propelled the innovation of computer science. In particular, knowledge from the neurobiology and neuropsychology of the brain revolutionized the development of reinforcement learning (RL) by providing novel interpretable mechanisms of how the brain achieves intelligent and efficient decision making. Triggered by this, there has been a boom in research about advanced RL algorithms that are built upon the inspirations of brain neuroscience. In this work, to further strengthen the bidirectional link between the 2 communities and especially promote the research on modern RL technology, we provide a comprehensive survey of recent advances in the area of brain-inspired/related RL algorithms. We start with basis theories of RL, and present a concise introduction to brain neuroscience related to RL. Then, we classify these advanced RL methodologies into 3 categories according to different connections of the brain, i.e., micro-neural activity, macro-brain structure, and cognitive function. Each category is further surveyed by presenting several modern RL algorithms along with their mathematical models, correlations with the brain, and open issues. Finally, we introduce several important applications of RL algorithms, followed by the discussions of challenges and opportunities for future research.
Collapse
Affiliation(s)
- Chaoqiong Fan
- School of Artificial Intelligence,
Beijing Normal University, Beijing, China
| | - Li Yao
- School of Artificial Intelligence,
Beijing Normal University, Beijing, China
| | - Jiacai Zhang
- School of Artificial Intelligence,
Beijing Normal University, Beijing, China
| | - Zonglei Zhen
- Faculty of Psychology,
Beijing Normal University, Beijing, China
| | - Xia Wu
- School of Artificial Intelligence,
Beijing Normal University, Beijing, China
- Address correspondence to:
| |
Collapse
|
8
|
Tashjian SM, Wise T, Mobbs D. Model-based prioritization for acquiring protection. PLoS Comput Biol 2022; 18:e1010805. [PMID: 36534704 PMCID: PMC9810162 DOI: 10.1371/journal.pcbi.1010805] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2022] [Revised: 01/03/2023] [Accepted: 12/09/2022] [Indexed: 12/23/2022] Open
Abstract
Protection often involves the capacity to prospectively plan the actions needed to mitigate harm. The computational architecture of decisions involving protection remains unclear, as well as whether these decisions differ from other beneficial prospective actions such as reward acquisition. Here we compare protection acquisition to reward acquisition and punishment avoidance to examine overlapping and distinct features across the three action types. Protection acquisition is positively valenced similar to reward. For both protection and reward, the more the actor gains, the more benefit. However, reward and protection occur in different contexts, with protection existing in aversive contexts. Punishment avoidance also occurs in aversive contexts, but differs from protection because punishment is negatively valenced and motivates avoidance. Across three independent studies (Total N = 600) we applied computational modeling to examine model-based reinforcement learning for protection, reward, and punishment in humans. Decisions motivated by acquiring protection evoked a higher degree of model-based control than acquiring reward or avoiding punishment, with no significant differences in learning rate. The context-valence asymmetry characteristic of protection increased deployment of flexible decision strategies, suggesting model-based control depends on the context in which outcomes are encountered as well as the valence of the outcome.
Collapse
Affiliation(s)
- Sarah M. Tashjian
- Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
| | - Toby Wise
- Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, United Kingdom
| | - Dean Mobbs
- Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems, California Institute of Technology, Pasadena, California, United States of America
| |
Collapse
|
9
|
Dong W, Luo J, Huo H, Seger CA, Chen Q. Frontostriatal Functional Connectivity Underlies the Association between Punishment Sensitivity and Procrastination. Brain Sci 2022; 12:brainsci12091163. [PMID: 36138899 PMCID: PMC9497208 DOI: 10.3390/brainsci12091163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 08/24/2022] [Accepted: 08/27/2022] [Indexed: 11/16/2022] Open
Abstract
Procrastination is defined as putting off an intended course of action voluntarily despite the harmful consequences. Previous studies have suggested that procrastination is associated with punishment sensitivity in that high punishment sensitivity results in increased negative utility for task performance. We hypothesized the effects of punishment sensitivity on procrastination would be mediated by a network connecting the caudate nucleus and prefrontal cortex, both of which have been previously associated with self-control and emotional control during procrastination. We employed voxel-based morphometry (VBM) and resting-state functional connectivity (rsFC) to examine the neural substrates of punishment sensitivity and its relationship with procrastination (N = 268). The behavioral results indicated a strong positive correlation between measures of punishment sensitivity and procrastination. The VBM analysis revealed that the gray matter (GM) volume of the right caudate was significantly positively correlated with punishment sensitivity. The primary rsFC analysis revealed connectivity between this caudate location and the bilateral middle frontal gyrus (MFG) was significantly negatively correlated with punishment sensitivity. A mediation model indicated punishment sensitivity completely mediated the relation between functional connectivity within a caudate–bilateral MFG network and procrastination. Our results support the theory that those with higher punishment sensitivity have weaker effective emotional self-control supported by the caudate–MFG network, resulting in greater procrastination.
Collapse
Affiliation(s)
- Wenshan Dong
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Jie Luo
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Hangfeng Huo
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
| | - Carol A. Seger
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
- Department of Psychology and Program in Molecular, Cellular, and Integrative Neurosciences, Colorado State University, Fort Collins, CO 80523, USA
| | - Qi Chen
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou 510631, China
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, Guangzhou 510631, China
- Correspondence: ; Tel.: +86-186-1735-3673
| |
Collapse
|
10
|
Stress-sensitive inference of task controllability. Nat Hum Behav 2022; 6:812-822. [PMID: 35273354 DOI: 10.1038/s41562-022-01306-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Accepted: 01/17/2022] [Indexed: 11/08/2022]
Abstract
Estimating the controllability of the environment enables agents to better predict upcoming events and decide when to engage controlled action selection. How does the human brain estimate controllability? Trial-by-trial analysis of choices, decision times and neural activity in an explore-and-predict task demonstrate that humans solve this problem by comparing the predictions of an 'actor' model with those of a reduced 'spectator' model of their environment. Neural blood oxygen level-dependent responses within striatal and medial prefrontal areas tracked the instantaneous difference in the prediction errors generated by these two statistical learning models. Blood oxygen level-dependent activity in the posterior cingulate, temporoparietal and prefrontal cortices covaried with changes in estimated controllability. Exposure to inescapable stressors biased controllability estimates downward and increased reliance on the spectator model in an anxiety-dependent fashion. Taken together, these findings provide a mechanistic account of controllability inference and its distortion by stress exposure.
Collapse
|
11
|
Grossman CD, Bari BA, Cohen JY. Serotonin neurons modulate learning rate through uncertainty. Curr Biol 2022; 32:586-599.e7. [PMID: 34936883 PMCID: PMC8825708 DOI: 10.1016/j.cub.2021.12.006] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 10/11/2021] [Accepted: 12/03/2021] [Indexed: 12/20/2022]
Abstract
Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA.
| |
Collapse
|
12
|
Kim D, Jeong J, Lee SW. Prefrontal solution to the bias-variance tradeoff during reinforcement learning. Cell Rep 2021; 37:110185. [PMID: 34965420 DOI: 10.1016/j.celrep.2021.110185] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Revised: 08/09/2021] [Accepted: 12/07/2021] [Indexed: 11/17/2022] Open
Abstract
Evidence that the brain combines different value learning strategies to minimize prediction error is accumulating. However, the tradeoff between bias and variance error, which imposes different constraints on each learning strategy's performance, poses a challenge for value learning. While this tradeoff specifies the requirements for optimal learning, little has been known about how the brain deals with this issue. Here, we hypothesize that the brain adaptively resolves the bias-variance tradeoff during reinforcement learning. Our theory suggests that the solution necessitates baseline correction for prediction error, which offsets the adverse effects of irreducible error on value learning. We show behavioral evidence of adaptive control using a Markov decision task with context changes. The prediction error baseline seemingly signals context changes to improve adaptability. Critically, we identify multiplexed representations of prediction error baseline within the ventrolateral and ventromedial prefrontal cortex, key brain regions known to guide model-based and model-free reinforcement learning.
Collapse
Affiliation(s)
- Dongjae Kim
- Center for Neural Science, New York University, New York, NY, USA; Department of Psychology, New York University, New York, NY, USA
| | - Jaeseung Jeong
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KAIST Center for Neuroscience-inspired AI, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KI for Health Science and Technology, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea; KI for Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), 34141 Daejeon, Republic of Korea.
| |
Collapse
|
13
|
Macpherson T, Matsumoto M, Gomi H, Morimoto J, Uchibe E, Hikida T. Parallel and hierarchical neural mechanisms for adaptive and predictive behavioral control. Neural Netw 2021; 144:507-521. [PMID: 34601363 DOI: 10.1016/j.neunet.2021.09.009] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 07/21/2021] [Accepted: 09/06/2021] [Indexed: 12/21/2022]
Abstract
Our brain can be recognized as a network of largely hierarchically organized neural circuits that operate to control specific functions, but when acting in parallel, enable the performance of complex and simultaneous behaviors. Indeed, many of our daily actions require concurrent information processing in sensorimotor, associative, and limbic circuits that are dynamically and hierarchically modulated by sensory information and previous learning. This organization of information processing in biological organisms has served as a major inspiration for artificial intelligence and has helped to create in silico systems capable of matching or even outperforming humans in several specific tasks, including visual recognition and strategy-based games. However, the development of human-like robots that are able to move as quickly as humans and respond flexibly in various situations remains a major challenge and indicates an area where further use of parallel and hierarchical architectures may hold promise. In this article we review several important neural and behavioral mechanisms organizing hierarchical and predictive processing for the acquisition and realization of flexible behavioral control. Then, inspired by the organizational features of brain circuits, we introduce a multi-timescale parallel and hierarchical learning framework for the realization of versatile and agile movement in humanoid robots.
Collapse
Affiliation(s)
- Tom Macpherson
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Osaka, Japan
| | - Masayuki Matsumoto
- Division of Biomedical Science, Faculty of Medicine, University of Tsukuba, Tsukuba, Ibaraki, Japan
| | - Hiroaki Gomi
- NTT Communication Science Laboratories, Nippon Telegraph and Telephone Co., Kanagawa, Japan
| | - Jun Morimoto
- Department of Brain Robot Interface, ATR Computational Neuroscience Laboratories, Kyoto, Japan; Graduate School of Informatics, Kyoto University, Kyoto, Japan
| | - Eiji Uchibe
- Department of Brain Robot Interface, ATR Computational Neuroscience Laboratories, Kyoto, Japan
| | - Takatoshi Hikida
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Osaka, Japan.
| |
Collapse
|
14
|
Heo S, Sung Y, Lee SW. Effects of subclinical depression on prefrontal-striatal model-based and model-free learning. PLoS Comput Biol 2021; 17:e1009003. [PMID: 33989284 PMCID: PMC8153417 DOI: 10.1371/journal.pcbi.1009003] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2020] [Revised: 05/26/2021] [Accepted: 04/26/2021] [Indexed: 11/18/2022] Open
Abstract
Depression is characterized by deficits in the reinforcement learning (RL) process. Although many computational and neural studies have extended our knowledge of the impact of depression on RL, most focus on habitual control (model-free RL), yielding a relatively poor understanding of goal-directed control (model-based RL) and arbitration control to find a balance between the two. We investigated the effects of subclinical depression on model-based and model-free learning in the prefrontal-striatal circuitry. First, we found that subclinical depression is associated with the attenuated state and reward prediction error representation in the insula and caudate. Critically, we found that it accompanies the disrupted arbitration control between model-based and model-free learning in the predominantly inferior lateral prefrontal cortex and frontopolar cortex. We also found that depression undermines the ability to exploit viable options, called exploitation sensitivity. These findings characterize how subclinical depression influences different levels of the decision-making hierarchy, advancing previous conflicting views that depression simply influences either habitual or goal-directed control. Our study creates possibilities for various clinical applications, such as early diagnosis and behavioral therapy design.
Collapse
Affiliation(s)
- Suyeon Heo
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- Brain and Cognitive Engineering Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Yoondo Sung
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- Brain and Cognitive Engineering Program, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Health Science Technology, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Institute for Artificial Intelligence, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
- KAIST Center for Neuroscience-inspired AI, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| |
Collapse
|
15
|
|
16
|
Panayi MC, Killcross S. The Role of the Rodent Lateral Orbitofrontal Cortex in Simple Pavlovian Cue-Outcome Learning Depends on Training Experience. Cereb Cortex Commun 2021; 2:tgab010. [PMID: 34296155 PMCID: PMC8152875 DOI: 10.1093/texcom/tgab010] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Revised: 01/29/2021] [Accepted: 02/01/2021] [Indexed: 11/30/2022] Open
Abstract
The orbitofrontal cortex (OFC) is a critical structure in the flexible control of value-based behaviors. OFC dysfunction is typically only detected when task or environmental contingencies change, against a backdrop of apparently intact initial acquisition and behavior. While intact acquisition following OFC lesions in simple Pavlovian cue-outcome conditioning is often predicted by models of OFC function, this predicted null effect has not been thoroughly investigated. Here, we test the effects of lesions and temporary muscimol inactivation of the rodent lateral OFC on the acquisition of a simple single cue-outcome relationship. Surprisingly, pretraining lesions significantly enhanced acquisition after overtraining, whereas post-training lesions and inactivation significantly impaired acquisition. This impaired acquisition to the cue reflects a disruption of behavioral control and not learning since the cue could also act as an effective blocking stimulus in an associative blocking procedure. These findings suggest that even simple cue-outcome representations acquired in the absence of OFC function are impoverished. Therefore, while OFC function is often associated with flexible behavioral control in complex environments, it is also involved in very simple Pavlovian acquisition where complex cue-outcome relationships are irrelevant to task performance.
Collapse
Affiliation(s)
- Marios C Panayi
- School of Psychology, UNSW Sydney, Sydney, NSW 2052, Australia
- National Institute on Drug Abuse Intramural Research Program, Cellular Neurobiology Research Branch, Behavioral Neurophysiology Research Section, 251 Bayview Blvd., Baltimore, MD 21224, USA
| | - Simon Killcross
- School of Psychology, UNSW Sydney, Sydney, NSW 2052, Australia
| |
Collapse
|
17
|
O'Doherty JP, Lee SW, Tadayonnejad R, Cockburn J, Iigaya K, Charpentier CJ. Why and how the brain weights contributions from a mixture of experts. Neurosci Biobehav Rev 2021; 123:14-23. [PMID: 33444700 DOI: 10.1016/j.neubiorev.2020.10.022] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2020] [Revised: 09/14/2020] [Accepted: 10/26/2020] [Indexed: 12/12/2022]
Abstract
It has long been suggested that human behavior reflects the contributions of multiple systems that cooperate or compete for behavioral control. Here we propose that the brain acts as a "Mixture of Experts" in which different expert systems propose strategies for action. It will be argued that the brain determines which experts should control behavior at any one moment in time by keeping track of the reliability of the predictions within each system, and by allocating control over behavior in a manner that depends on the relative reliabilities across experts. fMRI and neurostimulation studies suggest a specific contribution of the anterior prefrontal cortex in this process. Further, such a mechanism also takes into consideration the complexity of the expert, favoring simpler over more cognitively complex experts. Results from the study of different expert systems in both experiential and social learning domains hint at the possibility that this reliability-based control mechanism is domain general, exerting control over many different expert systems simultaneously in order to produce sophisticated behavior.
Collapse
Affiliation(s)
- John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA; Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, 91125, USA.
| | - Sang Wan Lee
- Department of Bio and Brain Engineering and Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science Technology (KAIST), Daejeon, 34141, Republic of Korea
| | - Reza Tadayonnejad
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA; Division of Neuromodulation, Semel Institute for Neuroscience and Behavior, University of California, Los Angeles, CA, 90095, USA
| | - Jeff Cockburn
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| | - Kyo Iigaya
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| | - Caroline J Charpentier
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, 91125, USA
| |
Collapse
|
18
|
Masset P, Ott T, Lak A, Hirokawa J, Kepecs A. Behavior- and Modality-General Representation of Confidence in Orbitofrontal Cortex. Cell 2020; 182:112-126.e18. [PMID: 32504542 DOI: 10.1016/j.cell.2020.05.022] [Citation(s) in RCA: 42] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2019] [Revised: 02/27/2020] [Accepted: 05/11/2020] [Indexed: 02/06/2023]
Abstract
Every decision we make is accompanied by a sense of confidence about its likely outcome. This sense informs subsequent behavior, such as investing more-whether time, effort, or money-when reward is more certain. A neural representation of confidence should originate from a statistical computation and predict confidence-guided behavior. An additional requirement for confidence representations to support metacognition is abstraction: they should emerge irrespective of the source of information and inform multiple confidence-guided behaviors. It is unknown whether neural confidence signals meet these criteria. Here, we show that single orbitofrontal cortex neurons in rats encode statistical decision confidence irrespective of the sensory modality, olfactory or auditory, used to make a choice. The activity of these neurons also predicts two confidence-guided behaviors: trial-by-trial time investment and cross-trial choice strategy updating. Orbitofrontal cortex thus represents decision confidence consistent with a metacognitive process that is useful for mediating confidence-guided economic decisions.
Collapse
Affiliation(s)
- Paul Masset
- Cold Spring Harbor Laboratory, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA; Watson School of Biological Sciences, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA
| | - Torben Ott
- Cold Spring Harbor Laboratory, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA; Department of Neuroscience and Department of Psychiatry, Washington University in St. Louis, St. Louis, MO 63110, USA
| | - Armin Lak
- Department of Physiology, Anatomy and Genetics, University of Oxford, Oxford OX1 3PT, UK
| | - Junya Hirokawa
- Cold Spring Harbor Laboratory, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA
| | - Adam Kepecs
- Cold Spring Harbor Laboratory, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA; Watson School of Biological Sciences, 1 Bungtown Road, Cold Spring Harbor, NY 11724, USA; Department of Neuroscience and Department of Psychiatry, Washington University in St. Louis, St. Louis, MO 63110, USA.
| |
Collapse
|
19
|
Effects of 5-HT 2C, 5-HT 1A receptor challenges and modafinil on the initiation and persistence of gambling behaviours. Psychopharmacology (Berl) 2020; 237:1745-1756. [PMID: 32123974 PMCID: PMC7239826 DOI: 10.1007/s00213-020-05496-x] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/01/2019] [Accepted: 02/19/2020] [Indexed: 11/07/2022]
Abstract
RATIONALE Problematic patterns of gambling are characterised by loss of control and persistent gambling often to recover losses. However, little is known about the mechanisms that mediate initial choices to begin gambling and then continue to gamble in the face of losing outcomes. OBJECTIVES These experiments first assessed gambling and loss-chasing performance under different win/lose probabilities in C57Bl/6 mice, and then investigated the effects of antagonism of 5-HT2CR with SB242084, 5-HT1AR agonism with 8-OH-DPAT and modafinil, a putative cognitive enhancer. RESULTS As seen in humans and other species, mice demonstrated the expected patterns of behaviour as the odds for winning were altered increasing gambling and loss-chasing when winning was more likely. SB242084 decreased the likelihood to initially gamble, but had no effects on subsequent gambling choices in the face of repeated losses. In contrast, 8-OH-DPAT had no effects on choosing to gamble in the first place, but once started 8-OH-DPAT increased gambling choices in a dose-sensitive manner. Modafinil effects were different to the serotonergic drugs in both decreasing the propensity to initiate gambling and chase losses. CONCLUSIONS We present evidence for dissociable effects of systemic drug administration on different aspects of gambling behaviour. These data extend and reinforce the importance of serotonergic mechanisms in mediating discrete components of gambling behaviour. They further demonstrate the ability of modafinil to reduce gambling behaviour. Our work using a novel mouse paradigm may be of utility in modelling the complex psychological and neurobiological underpinnings of gambling problems, including the analysis of genetic and environmental factors.
Collapse
|