1
|
|
2
|
Bartolo R, Averbeck BB. Prefrontal Cortex Predicts State Switches during Reversal Learning. Neuron 2020; 106:1044-1054.e4. [PMID: 32315603 DOI: 10.1016/j.neuron.2020.03.024] [Citation(s) in RCA: 51] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2019] [Revised: 01/28/2020] [Accepted: 03/24/2020] [Indexed: 11/25/2022]
Abstract
Reinforcement learning allows organisms to predict future outcomes and to update their beliefs about value in the world. The dorsal-lateral prefrontal cortex (dlPFC) integrates information carried by reward circuits, which can be used to infer the current state of the world under uncertainty. Here, we explored the dlPFC computations related to updating current beliefs during stochastic reversal learning. We recorded the activity of populations up to 1,000 neurons, simultaneously, in two male macaques while they executed a two-armed bandit reversal learning task. Behavioral analyses using a Bayesian framework showed that animals inferred reversals and switched their choice preference rapidly, rather than slowly updating choice values, consistent with state inference. Furthermore, dlPFC neural populations accurately encoded choice preference switches. These results suggest that prefrontal neurons dynamically encode decisions associated with Bayesian subjective values, highlighting the role of the PFC in representing a belief about the current state of the world.
Collapse
Affiliation(s)
- Ramon Bartolo
- Laboratory of Neuropsychology, National Institute of Mental Health/National Institutes of Health, Bethesda, MD 20892-4415, USA.
| | - Bruno B Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health/National Institutes of Health, Bethesda, MD 20892-4415, USA
| |
Collapse
|
3
|
Parr AC, Coe BC, Munoz DP, Dorris MC. A novel fMRI paradigm to dissociate the behavioral and neural components of mixed-strategy decision making from non-strategic decisions in humans. Eur J Neurosci 2019; 51:1914-1927. [PMID: 31596980 DOI: 10.1111/ejn.14586] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2019] [Revised: 08/22/2019] [Accepted: 09/18/2019] [Indexed: 11/30/2022]
Abstract
During competitive interactions, such as predator-prey or team sports, the outcome of one's actions is dependent on both their own choices and those of their opponents. Success in these rivalries requires that individuals choose dynamically and unpredictably, often adopting a mixed strategy. Understanding the neural basis of strategic decision making is complicated by the fact that it recruits various cognitive processes that are often shared with non-strategic forms of decision making, such as value estimation, working memory, response inhibition, response selection, and reward processes. Although researchers have explored neural activity within key brain regions during mixed-strategy games, how brain activity differs in the context of strategic interactions versus non-strategic choices is not well understood. We developed a novel behavioral paradigm to dissociate choice behavior during mixed-strategy interactions from non-strategic choices, and we used task-based functional magnetic resonance imaging (fMRI) to contrast brain activation. In a block design, participants competed in the classic mixed-strategy game, "matching pennies," against a dynamic computer opponent designed to exploit predictability in players' response patterns. Results were contrasted with a non-strategic task that had comparable sensory input, motor output, and reward rate; thus, differences in behavior and brain activation reflect strategic processes. The mixed-strategy game was associated with activation of a distributed cortico-striatal network compared to the non-strategic task. We propose that choosing in mixed-strategy contexts requires additional cognitive demands present to a lesser degree during the control task, illustrating the strength of this design in probing function of cognitive systems beyond core sensory, motor, and reward processes.
Collapse
Affiliation(s)
- Ashley C Parr
- Centre for Neuroscience Studies, Queen's University, Kingston, ON, Canada.,Department of Psychiatry, University of Pittsburgh, Pittsburgh, Pennsylvania
| | - Brian C Coe
- Centre for Neuroscience Studies, Queen's University, Kingston, ON, Canada
| | - Douglas P Munoz
- Centre for Neuroscience Studies, Queen's University, Kingston, ON, Canada.,Department of Biomedical and Molecular Sciences, Queen's University, Kingston, ON, Canada
| | - Michael C Dorris
- Institute of Neuroscience, Shanghai Institutes for Biological Sciences, Chinese Academy of Sciences, Shanghai, China
| |
Collapse
|
4
|
Abstract
Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).
Collapse
Affiliation(s)
| | - Amitai Shenhav
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown Institute for Brain Science, Brown University
| | | |
Collapse
|
5
|
van Wingerden M, van den Bos W. Can You Trust a Rat? Using Animal Models to Investigate the Neural Basis of Trust Like Behavior. SOCIAL COGNITION 2015. [DOI: 10.1521/soco.2015.33.5.387] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022] Open
|
6
|
Takahashi H, Izuma K, Matsumoto M, Matsumoto K, Omori T. The Anterior Insula Tracks Behavioral Entropy during an Interpersonal Competitive Game. PLoS One 2015; 10:e0123329. [PMID: 26039634 PMCID: PMC4454696 DOI: 10.1371/journal.pone.0123329] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2014] [Accepted: 03/02/2015] [Indexed: 12/01/2022] Open
Abstract
In competitive situations, individuals need to adjust their behavioral strategy dynamically in response to their opponent’s behavior. In the present study, we investigated the neural basis of how individuals adjust their strategy during a simple, competitive game of matching pennies. We used entropy as a behavioral index of randomness in decision-making, because maximizing randomness is thought to be an optimal strategy in the game, according to game theory. While undergoing functional magnetic resonance imaging (fMRI), subjects played matching pennies with either a human or computer opponent in each block, although in reality they played the game with the same computer algorithm under both conditions. The winning rate of each block was also manipulated. Both the opponent (human or computer), and the winning rate, independently affected subjects’ block-wise entropy during the game. The fMRI results revealed that activity in the bilateral anterior insula was positively correlated with subjects’ (not opponent’s) behavioral entropy during the game, which indicates that during an interpersonal competitive game, the anterior insula tracked how uncertain subjects’ behavior was, rather than how uncertain subjects felt their opponent's behavior was. Our results suggest that intuitive or automatic processes based on somatic markers may be a key to optimally adjusting behavioral strategies in competitive situations.
Collapse
Affiliation(s)
- Hideyuki Takahashi
- Brain Science Institute, Tamagawa University, Machida, Tokyo, Japan
- Graduate School of Engineering, Osaka university, Suita city, Osaka, Japan
- * E-mail:
| | - Keise Izuma
- Brain Science Institute, Tamagawa University, Machida, Tokyo, Japan
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
| | - Madoka Matsumoto
- Department of Neuropsychiatry, The University of Tokyo Hospital, Hongo, Bunkyo-ku, Tokyo, Japan
| | - Kenji Matsumoto
- Brain Science Institute, Tamagawa University, Machida, Tokyo, Japan
| | - Takashi Omori
- Brain Science Institute, Tamagawa University, Machida, Tokyo, Japan
| |
Collapse
|
7
|
Rudebeck PH, Murray EA. The orbitofrontal oracle: cortical mechanisms for the prediction and evaluation of specific behavioral outcomes. Neuron 2014; 84:1143-56. [PMID: 25521376 PMCID: PMC4271193 DOI: 10.1016/j.neuron.2014.10.049] [Citation(s) in RCA: 254] [Impact Index Per Article: 25.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
Abstract
The orbitofrontal cortex (OFC) has long been associated with the flexible control of behavior and concepts such as behavioral inhibition, self-control, and emotional regulation. These ideas emphasize the suppression of behaviors and emotions, but OFC's affirmative functions have remained enigmatic. Here we review recent work that has advanced our understanding of this prefrontal area and how its functions are shaped through interaction with subcortical structures such as the amygdala. Recent findings have overturned theories emphasizing behavioral inhibition as OFC's fundamental function. Instead, new findings indicate that OFC provides predictions about specific outcomes associated with stimuli, choices, and actions, especially their moment-to-moment value based on current internal states. OFC function thereby encompasses a broad representation or model of an individual's sensory milieu and potential actions, along with their relationship to likely behavioral outcomes.
Collapse
Affiliation(s)
- Peter H Rudebeck
- Friedman Brain Institute, Icahn School of Medicine at Mount Sinai, One Gustave L. Levy Place, New York, NY 10014, USA.
| | - Elisabeth A Murray
- Section on the Neurobiology of Learning and Memory, Laboratory of Neuropsychology, National Institute of Mental Health, Building 49, Suite 1B80, 49 Convent Drive, Bethesda, MD 20892, USA.
| |
Collapse
|
8
|
Behavioral Variability through Stochastic Choice and Its Gating by Anterior Cingulate Cortex. Cell 2014; 159:21-32. [DOI: 10.1016/j.cell.2014.08.037] [Citation(s) in RCA: 123] [Impact Index Per Article: 12.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2014] [Revised: 08/22/2014] [Accepted: 08/25/2014] [Indexed: 10/24/2022]
|
9
|
Abstract
Adaptive behaviors increase the likelihood of survival and reproduction and improve the quality of life. However, it is often difficult to identify optimal behaviors in real life due to the complexity of the decision maker's environment and social dynamics. As a result, although many different brain areas and circuits are involved in decision making, evolutionary and learning solutions adopted by individual decision makers sometimes produce suboptimal outcomes. Although these problems are exacerbated in numerous neurological and psychiatric disorders, their underlying neurobiological causes remain incompletely understood. In this review, theoretical frameworks in economics and machine learning and their applications in recent behavioral and neurobiological studies are summarized. Examples of such applications in clinical domains are also discussed for substance abuse, Parkinson's disease, attention-deficit/hyperactivity disorder, schizophrenia, mood disorders, and autism. Findings from these studies have begun to lay the foundations necessary to improve diagnostics and treatment for various neurological and psychiatric disorders.
Collapse
Affiliation(s)
- Daeyeol Lee
- Department of Neurobiology, Yale University School of Medicine, New Haven, CT 06510, USA.
| |
Collapse
|
10
|
Insights from the application of computational neuroimaging to social neuroscience. Curr Opin Neurobiol 2013; 23:387-92. [PMID: 23518140 DOI: 10.1016/j.conb.2013.02.007] [Citation(s) in RCA: 37] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2012] [Revised: 01/17/2013] [Accepted: 02/15/2013] [Indexed: 11/22/2022]
Abstract
A recent approach in social neuroscience has been the application of formal computational models for a particular social-cognitive process to neuroimaging data. Here we review preliminary findings from this nascent subfield, focusing on observational learning and strategic interactions. We present evidence consistent with the existence of three distinct learning systems that may contribute to social cognition: an observational-reward-learning system involved in updating expectations of future reward based on observing rewards obtained by others, an action-observational learning system involved in learning about the action tendencies of others, and a third system engaged when it is necessary to learn about the hidden mental-states or traits of another. These three systems appear to map onto distinct neuroanatomical substrates, and depend on unique computational signals.
Collapse
|