1
|
Burk DC, Taswell C, Tang H, Averbeck BB. Computational Mechanisms Underlying Motivation to Earn Symbolic Reinforcers. J Neurosci 2024; 44:e1873232024. [PMID: 38670805 PMCID: PMC11170943 DOI: 10.1523/jneurosci.1873-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/02/2023] [Revised: 02/27/2024] [Accepted: 04/11/2024] [Indexed: 04/28/2024] Open
Abstract
Reinforcement learning is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g., money, points) that are later exchanged for primary reinforcers (e.g., food, drink). Although symbolic reinforcers are ubiquitous in our daily lives, widely used in laboratory tasks because they can be motivating, mechanisms by which they become motivating are less understood. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g., the current number of accumulated tokens, choice options, task epoch, trials since the last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to then correlate with the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n = 5 monkeys, three males and two females). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in the state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.
Collapse
Affiliation(s)
- Diana C Burk
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892-4415
| | - Craig Taswell
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892-4415
| | - Hua Tang
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892-4415
| | - Bruno B Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland 20892-4415
| |
Collapse
|
2
|
Kim T, Lee SW, Lho SK, Moon SY, Kim M, Kwon JS. Neurocomputational model of compulsivity: deviating from an uncertain goal-directed system. Brain 2024; 147:2230-2244. [PMID: 38584499 PMCID: PMC11146420 DOI: 10.1093/brain/awae102] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 02/18/2024] [Accepted: 03/07/2024] [Indexed: 04/09/2024] Open
Abstract
Despite a theory that an imbalance in goal-directed versus habitual systems serve as building blocks of compulsions, research has yet to delineate how this occurs during arbitration between the two systems in obsessive-compulsive disorder. Inspired by a brain model in which the inferior frontal cortex selectively gates the putamen to guide goal-directed or habitual actions, this study aimed to examine whether disruptions in the arbitration process via the fronto-striatal circuit would underlie imbalanced decision-making and compulsions in patients. Thirty patients with obsessive-compulsive disorder [mean (standard deviation) age = 26.93 (6.23) years, 12 females (40%)] and 30 healthy controls [mean (standard deviation) age = 24.97 (4.72) years, 17 females (57%)] underwent functional MRI scans while performing the two-step Markov decision task, which was designed to dissociate goal-directed behaviour from habitual behaviour. We employed a neurocomputational model to account for an uncertainty-based arbitration process, in which a prefrontal arbitrator (i.e. inferior frontal gyrus) allocates behavioural control to a more reliable strategy by selectively gating the putamen. We analysed group differences in the neural estimates of uncertainty of each strategy. We also compared the psychophysiological interaction effects of system preference (goal-directed versus habitual) on fronto-striatal coupling between groups. We examined the correlation between compulsivity score and the neural activity and connectivity involved in the arbitration process. The computational model captured the subjects' preferences between the strategies. Compared with healthy controls, patients had a stronger preference for the habitual system (t = -2.88, P = 0.006), which was attributed to a more uncertain goal-directed system (t = 2.72, P = 0.009). Before the allocation of controls, patients exhibited hypoactivity in the inferior frontal gyrus compared with healthy controls when this region tracked the inverse of uncertainty (i.e. reliability) of goal-directed behaviour (P = 0.001, family-wise error rate corrected). When reorienting behaviours to reach specific goals, patients exhibited weaker right ipsilateral ventrolateral prefronto-putamen coupling than healthy controls (P = 0.001, family-wise error rate corrected). This hypoconnectivity was correlated with more severe compulsivity (r = -0.57, P = 0.002). Our findings suggest that the attenuated top-down control of the putamen by the prefrontal arbitrator underlies compulsivity in obsessive-compulsive disorder. Enhancing fronto-striatal connectivity may be a potential neurotherapeutic approach for compulsivity and adaptive decision-making.
Collapse
Affiliation(s)
- Taekwan Kim
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Seoul 08826, Republic of Korea
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Neuroscience-inspired Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Center for Neuroscience-inspired Artificial Intelligence, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
- Kim Jaechul Graduate School of AI, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea
| | - Silvia Kyungjin Lho
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
| | - Sun-Young Moon
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
| | - Minah Kim
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul 03080, Republic of Korea
| | - Jun Soo Kwon
- Department of Brain and Cognitive Sciences, Seoul National University College of Natural Sciences, Seoul 08826, Republic of Korea
- Department of Neuropsychiatry, Seoul National University Hospital, Seoul 03080, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul 03080, Republic of Korea
| |
Collapse
|
3
|
Han D, Doya K, Li D, Tani J. Synergizing habits and goals with variational Bayes. Nat Commun 2024; 15:4461. [PMID: 38796491 DOI: 10.1038/s41467-024-48577-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Accepted: 05/06/2024] [Indexed: 05/28/2024] Open
Abstract
Behaving efficiently and flexibly is crucial for biological and artificial embodied agents. Behavior is generally classified into two types: habitual (fast but inflexible), and goal-directed (flexible but slow). While these two types of behaviors are typically considered to be managed by two distinct systems in the brain, recent studies have revealed a more sophisticated interplay between them. We introduce a theoretical framework using variational Bayesian theory, incorporating a Bayesian intention variable. Habitual behavior depends on the prior distribution of intention, computed from sensory context without goal-specification. In contrast, goal-directed behavior relies on the goal-conditioned posterior distribution of intention, inferred through variational free energy minimization. Assuming that an agent behaves using a synergized intention, our simulations in vision-based sensorimotor tasks explain the key properties of their interaction as observed in experiments. Our work suggests a fresh perspective on the neural mechanisms of habits and goals, shedding light on future research in decision making.
Collapse
Affiliation(s)
- Dongqi Han
- Microsoft Research Asia, Shanghai, 200232, China.
| | - Kenji Doya
- Okinawa Institute of Science and Technology, Okinawa, 904-0495, Japan
| | - Dongsheng Li
- Microsoft Research Asia, Shanghai, 200232, China
| | - Jun Tani
- Okinawa Institute of Science and Technology, Okinawa, 904-0495, Japan
| |
Collapse
|
4
|
Charpentier CJ, Wu Q, Min S, Ding W, Cockburn J, O'Doherty JP. Heterogeneity in strategy use during arbitration between experiential and observational learning. Nat Commun 2024; 15:4436. [PMID: 38789415 PMCID: PMC11126711 DOI: 10.1038/s41467-024-48548-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 05/06/2024] [Indexed: 05/26/2024] Open
Abstract
To navigate our complex social world, it is crucial to deploy multiple learning strategies, such as learning from directly experiencing action outcomes or from observing other people's behavior. Despite the prevalence of experiential and observational learning in humans and other social animals, it remains unclear how people favor one strategy over the other depending on the environment, and how individuals vary in their strategy use. Here, we describe an arbitration mechanism in which the prediction errors associated with each learning strategy influence their weight over behavior. We designed an online behavioral task to test our computational model, and found that while a substantial proportion of participants relied on the proposed arbitration mechanism, there was some meaningful heterogeneity in how people solved this task. Four other groups were identified: those who used a fixed mixture between the two strategies, those who relied on a single strategy and non-learners with irrelevant strategies. Furthermore, groups were found to differ on key behavioral signatures, and on transdiagnostic symptom dimensions, in particular autism traits and anxiety. Together, these results demonstrate how large heterogeneous datasets and computational methods can be leveraged to better characterize individual differences.
Collapse
Affiliation(s)
- Caroline J Charpentier
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
- Department of Psychology & Brain and Behavior Institute, University of Maryland, College Park, MD, USA.
| | - Qianying Wu
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Seokyoung Min
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Weilun Ding
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - Jeffrey Cockburn
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
5
|
Schaaf JV, Johansson A, Visser I, Huizenga HM. What's in a name: The role of verbalization in reinforcement learning. Psychon Bull Rev 2024:10.3758/s13423-024-02506-3. [PMID: 38769270 DOI: 10.3758/s13423-024-02506-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/25/2024] [Indexed: 05/22/2024]
Abstract
(e.g., characters or fractals) and concrete stimuli (e.g., pictures of everyday objects) are used interchangeably in the reinforcement-learning literature. Yet, it is unclear whether the same learning processes underlie learning from these different stimulus types. In two preregistered experiments (N = 50 each), we assessed whether abstract and concrete stimuli yield different reinforcement-learning performance and whether this difference can be explained by verbalization. We argued that concrete stimuli are easier to verbalize than abstract ones, and that people therefore can appeal to the phonological loop, a subcomponent of the working-memory system responsible for storing and rehearsing verbal information, while learning. To test whether this verbalization aids reinforcement-learning performance, we administered a reinforcement-learning task in which participants learned either abstract or concrete stimuli while verbalization was hindered or not. In the first experiment, results showed a more pronounced detrimental effect of hindered verbalization for concrete than abstract stimuli on response times, but not on accuracy. In the second experiment, in which we reduced the response window, results showed the differential effect of hindered verbalization between stimulus types on accuracy, not on response times. These results imply that verbalization aids learning for concrete, but not abstract, stimuli and therefore that different processes underlie learning from these types of stimuli. This emphasizes the importance of carefully considering stimulus types. We discuss these findings in light of generalizability and validity of reinforcement-learning research.
Collapse
Affiliation(s)
- Jessica V Schaaf
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
- Cognitive Neuroscience Department, Radboud University Medical Centre, Nijmegen, The Netherlands.
- Donders Institute for Brain, Cognition and Behaviour, Nijmegen, The Netherlands.
| | - Annie Johansson
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
| | - Ingmar Visser
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Yield, Research Institute for Child Development and Education, Amsterdam, The Netherlands
- ABC, Amsterdam Brain and Cognition Centre, Amsterdam, The Netherlands
| | - Hilde M Huizenga
- Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
- Yield, Research Institute for Child Development and Education, Amsterdam, The Netherlands
- ABC, Amsterdam Brain and Cognition Centre, Amsterdam, The Netherlands
| |
Collapse
|
6
|
Hamzehpour L, Bohn T, Dutsch V, Jaspers L, Grimm O. From brain to body: exploring the connection between altered reward processing and physical fitness in schizophrenia. Psychiatry Res 2024; 335:115877. [PMID: 38555826 DOI: 10.1016/j.psychres.2024.115877] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Revised: 03/11/2024] [Accepted: 03/24/2024] [Indexed: 04/02/2024]
Abstract
Understanding the underlying mechanisms that link psychopathology and physical comorbidities in schizophrenia is crucial since decreased physical fitness and overweight pose major risk factors for cardio-vascular diseases and decrease the patients' life expectancies. We hypothesize that altered reward anticipation plays an important role in this. We implemented the Monetary Incentive Delay task in a MR scanner and a fitness test battery to compare schizophrenia patients (SZ, n = 43) with sex- and age-matched healthy controls (HC, n = 36) as to reward processing and their physical fitness. We found differences in reward anticipation between SZs and HCs, whereby increased activity in HCs positively correlated with overall physical condition and negatively correlated with psychopathology. On the other handy, SZs revealed stronger activity in the posterior cingulate cortex and in cerebellar regions during reward anticipation, which could be linked to decreased overall physical fitness. These findings demonstrate that a dysregulated reward system is not only responsible for the symptomatology of schizophrenia, but might also be involved in physical comorbidities which could pave the way for future lifestyle therapy interventions.
Collapse
Affiliation(s)
- Lara Hamzehpour
- Goethe University Frankfurt, University Hospital, Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, Heinrich-Hoffmann-Straße 10 60528 Frankfurt am Main, Germany; Goethe University Frankfurt, Faculty 15 Biological Sciences, Frankfurt am Main, Germany.
| | - Tamara Bohn
- Goethe University Frankfurt, University Hospital, Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, Heinrich-Hoffmann-Straße 10 60528 Frankfurt am Main, Germany
| | - Valentin Dutsch
- Goethe University Frankfurt, University Hospital, Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, Heinrich-Hoffmann-Straße 10 60528 Frankfurt am Main, Germany
| | - Lucia Jaspers
- Goethe University Frankfurt, University Hospital, Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, Heinrich-Hoffmann-Straße 10 60528 Frankfurt am Main, Germany
| | - Oliver Grimm
- Goethe University Frankfurt, University Hospital, Department of Psychiatry, Psychosomatic Medicine and Psychotherapy, Heinrich-Hoffmann-Straße 10 60528 Frankfurt am Main, Germany
| |
Collapse
|
7
|
Wurm F, Ernst B, Steinhauser M. Surprise-minimization as a solution to the structural credit assignment problem. PLoS Comput Biol 2024; 20:e1012175. [PMID: 38805546 PMCID: PMC11175464 DOI: 10.1371/journal.pcbi.1012175] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Revised: 06/13/2024] [Accepted: 05/18/2024] [Indexed: 05/30/2024] Open
Abstract
The structural credit assignment problem arises when the causal structure between actions and subsequent outcomes is hidden from direct observation. To solve this problem and enable goal-directed behavior, an agent has to infer structure and form a representation thereof. In the scope of this study, we investigate a possible solution in the human brain. We recorded behavioral and electrophysiological data from human participants in a novel variant of the bandit task, where multiple actions lead to multiple outcomes. Crucially, the mapping between actions and outcomes was hidden and not instructed to the participants. Human choice behavior revealed clear hallmarks of credit assignment and learning. Moreover, a computational model which formalizes action selection as the competition between multiple representations of the hidden structure was fit to account for participants data. Starting in a state of uncertainty about the correct representation, the central mechanism of this model is the arbitration of action control towards the representation which minimizes surprise about outcomes. Crucially, single-trial latent-variable analysis reveals that the neural patterns clearly support central quantitative predictions of this surprise minimization model. The results suggest that neural activity is not only related to reinforcement learning under correct as well as incorrect task representations but also reflects central mechanisms of credit assignment and behavioral arbitration.
Collapse
Affiliation(s)
- Franz Wurm
- Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany
- Leiden University, Leiden, the Netherlands
- Leiden Institute for Brain and Cognition, Leiden University, Leiden, the Netherlands
| | - Benjamin Ernst
- Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany
| | | |
Collapse
|
8
|
Aquino TG, Courellis H, Mamelak AN, Rutishauser U, O Doherty JP. Encoding of Predictive Associations in Human Prefrontal and Medial Temporal Neurons During Pavlovian Appetitive Conditioning. J Neurosci 2024; 44:e1628232024. [PMID: 38423764 PMCID: PMC11044193 DOI: 10.1523/jneurosci.1628-23.2024] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Revised: 01/29/2024] [Accepted: 02/19/2024] [Indexed: 03/02/2024] Open
Abstract
Pavlovian conditioning is thought to involve the formation of learned associations between stimuli and values, and between stimuli and specific features of outcomes. Here, we leveraged human single neuron recordings in ventromedial prefrontal, dorsomedial frontal, hippocampus, and amygdala while patients of both sexes performed an appetitive Pavlovian conditioning task probing both stimulus-value and stimulus-stimulus associations. Ventromedial prefrontal cortex encoded predictive value along with the amygdala, and also encoded predictions about the identity of stimuli that would subsequently be presented, suggesting a role for neurons in this region in encoding predictive information beyond value. Unsigned error signals were found in dorsomedial frontal areas and hippocampus, potentially supporting learning of non-value related outcome features. Our findings implicate distinct human prefrontal and medial temporal neuronal populations in mediating predictive associations which could partially support model-based mechanisms during Pavlovian conditioning.
Collapse
Affiliation(s)
- Tomas G Aquino
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - Hristos Courellis
- Biological Engineering, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - Adam N Mamelak
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
| | - Ueli Rutishauser
- Department of Neurosurgery, Cedars-Sinai Medical Center, Los Angeles, California 90048
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
| | - John P O Doherty
- Computation and Neural Systems, Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, California 91125
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, California 91125
| |
Collapse
|
9
|
Terburg D, van Honk J, Schutter DJLG. Doubling down on dual systems: A cerebellum-amygdala route towards action- and outcome-based social and affective behavior. Cortex 2024; 173:175-186. [PMID: 38417390 DOI: 10.1016/j.cortex.2024.02.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 11/24/2023] [Accepted: 02/09/2024] [Indexed: 03/01/2024]
Abstract
The amygdala and cerebellum are both evolutionary preserved brain structures containing cortical as well as subcortical properties. For decades, the amygdala has been considered the fear-center of the brain, but recent advances have shown that the amygdala acts as a critical hub between cortical and subcortical systems and shapes social and affective behaviors beyond fear. Likewise, the cerebellum is a dedicated control unit that fine-tunes motor behavior to fit contextual requirements. There is however increasing evidence that the cerebellum strongly influences subcortical as well as cortical processes beyond the motor domain. These insights broadened the view on the cerebellum's functions to also include social and affective behavior. Here we explore how the amygdala and cerebellum might interact in shaping social and affective behaviors based on their roles in threat reactivity and reinforcement learning. A novel mechanistic neural framework of cerebellum-amygdala interactions will be presented which provides testable hypotheses for future social and affective neuroscientific research in humans.
Collapse
Affiliation(s)
- David Terburg
- Experimental Psychology, Helmholtz Institute, Utrecht University, the Netherlands; Department of Psychiatry and Mental Health, University of Cape Town, South Africa.
| | - Jack van Honk
- Experimental Psychology, Helmholtz Institute, Utrecht University, the Netherlands; Department of Psychiatry and Institute of Infectious Diseases and Molecular Medicine (IDM), University of Cape Town, South Africa
| | | |
Collapse
|
10
|
Kwon M, Choi H, Park H, Ahn WY, Jung YC. Neural correlates of model-based behavior in internet gaming disorder and alcohol use disorder. J Behav Addict 2024; 13:236-249. [PMID: 38460004 PMCID: PMC10988400 DOI: 10.1556/2006.2024.00006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/10/2023] [Revised: 12/26/2023] [Accepted: 02/08/2024] [Indexed: 03/11/2024] Open
Abstract
Background An imbalance between model-based and model-free decision-making systems is a common feature in addictive disorders. However, little is known about whether similar decision-making deficits appear in internet gaming disorder (IGD). This study compared neurocognitive features associated with model-based and model-free systems in IGD and alcohol use disorder (AUD). Method Participants diagnosed with IGD (n = 22) and AUD (n = 22), and healthy controls (n = 30) performed the two-stage task inside the functional magnetic resonance imaging (fMRI) scanner. We used computational modeling and hierarchical Bayesian analysis to provide a mechanistic account of their choice behavior. Then, we performed a model-based fMRI analysis and functional connectivity analysis to identify neural correlates of the decision-making processes in each group. Results The computational modeling results showed similar levels of model-based behavior in the IGD and AUD groups. However, we observed distinct neural correlates of the model-based reward prediction error (RPE) between the two groups. The IGD group exhibited insula-specific activation associated with model-based RPE, while the AUD group showed prefrontal activation, particularly in the orbitofrontal cortex and superior frontal gyrus. Furthermore, individuals with IGD demonstrated hyper-connectivity between the insula and brain regions in the salience network in the context of model-based RPE. Discussion and Conclusions The findings suggest potential differences in the neurobiological mechanisms underlying model-based behavior in IGD and AUD, albeit shared cognitive features observed in computational modeling analysis. As the first neuroimaging study to compare IGD and AUD in terms of the model-based system, this study provides novel insights into distinct decision-making processes in IGD.
Collapse
Affiliation(s)
- Mina Kwon
- Department of Psychology, Seoul National University, Seoul, South Korea
| | - Hangnyoung Choi
- Department of Psychiatry, Yonsei University College of Medicine, Seoul, South Korea
- Institute of Behavioral Science in Medicine, Yonsei University College of Medicine, Seoul, South Korea
| | - Harhim Park
- Department of Psychology, Seoul National University, Seoul, South Korea
| | - Woo-Young Ahn
- Department of Psychology, Seoul National University, Seoul, South Korea
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, South Korea
- AI Institute, Seoul National University, Seoul, South Korea
| | - Young-Chul Jung
- Department of Psychiatry, Yonsei University College of Medicine, Seoul, South Korea
- Institute of Behavioral Science in Medicine, Yonsei University College of Medicine, Seoul, South Korea
- Institute for Innovation in Digital Healthcare, Yonsei University, Seoul, South Korea
| |
Collapse
|
11
|
Rodriguez Buritica JM, Eppinger B, Heekeren HR, Crone EA, van Duijvenvoorde ACK. Observational reinforcement learning in children and young adults. NPJ SCIENCE OF LEARNING 2024; 9:18. [PMID: 38480747 PMCID: PMC10937639 DOI: 10.1038/s41539-024-00227-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/15/2022] [Accepted: 02/21/2024] [Indexed: 03/17/2024]
Abstract
Observational learning is essential for the acquisition of new behavior in educational practices and daily life and serves as an important mechanism for human cognitive and social-emotional development. However, we know little about its underlying neurocomputational mechanisms from a developmental perspective. In this study we used model-based fMRI to investigate differences in observational learning and individual learning between children and younger adults. Prediction errors (PE), the difference between experienced and predicted outcomes, related positively to striatal and ventral medial prefrontal cortex activation during individual learning and showed no age-related differences. PE-related activation during observational learning was more pronounced when outcomes were worse than predicted. Particularly, negative PE-coding in the dorsal medial prefrontal cortex was stronger in adults compared to children and was associated with improved observational learning in children and adults. The current findings pave the way to better understand observational learning challenges across development and educational settings.
Collapse
Affiliation(s)
- Julia M Rodriguez Buritica
- Department of Psychology, University of Greifswald, Greifswald, Germany.
- Berlin School of Mind and Brain & Department of Psychology, Humboldt University of Berlin, Berlin, Germany.
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany.
| | - Ben Eppinger
- Department of Psychology, University of Greifswald, Greifswald, Germany
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
- Department of Psychology, Concordia University, Montreal, Canada
- Department of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Hauke R Heekeren
- Department of Psychology, University of Greifswald, Greifswald, Germany
- Executive University Board, Universität Hamburg, Hamburg, Germany
| | - Eveline A Crone
- Department of Psychology, Education and Child Studies, Erasmus University Rotterdam, Rotterdam, Netherlands
- Institute of Psychology, Leiden University, Leiden, The Netherlands
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands
| | - Anna C K van Duijvenvoorde
- Institute of Psychology, Leiden University, Leiden, The Netherlands.
- Leiden Institute for Brain and Cognition, Leiden, The Netherlands.
| |
Collapse
|
12
|
Colas JT, O’Doherty JP, Grafton ST. Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts. PLoS Comput Biol 2024; 20:e1011950. [PMID: 38552190 PMCID: PMC10980507 DOI: 10.1371/journal.pcbi.1011950] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2023] [Accepted: 02/26/2024] [Indexed: 04/01/2024] Open
Abstract
Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - John P. O’Doherty
- Division of the Humanities and Social Sciences, California Institute of Technology, Pasadena, California, United States of America
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, California, United States of America
| | - Scott T. Grafton
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, California, United States of America
| |
Collapse
|
13
|
Heimer O, Hertz U. The spread of affective and semantic valence representations across states. Cognition 2024; 244:105714. [PMID: 38176154 DOI: 10.1016/j.cognition.2023.105714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Revised: 12/22/2023] [Accepted: 12/24/2023] [Indexed: 01/06/2024]
Abstract
In many decision problems, outcomes are not reached after a single action but rather after a series of events or states. To optimize decisions over multiple states, representations of how good or bad the outcomes are, that is, the outcomes' valence, should spread across states. One mechanism for valence spreading is a temporal, state-independent process in which a single valence representation is updated when an outcome is experienced and fades away afterwards. Each state's valence is based on its temporal proximity to the experienced outcome. An alternative, state-dependent mechanism relies on the structure of transitions between states, updating a separate valence representation for each state according to its spatial distance from the outcomes. We examined how these mechanistic accounts shape the spread of two formats of valence representation, feelings (affective valence) and knowledge (semantic valence), between states. In two pre-registered experiments (N = 585), we used a novel task in which participants move in a four-state maze, one of which contains an outcome. The participants provide self-reports of affective and semantic valence throughout the maze and after finishing it. Results show that the affective representation of negative valence is more localized in state-space than the semantic representation. We also found evidence for the relative reliance of the affective valence on a temporal, state-independent mechanism and of the semantic valence on a structured, state-dependent mechanism. Our findings provide mechanistic accounts for the differences between affective and semantic valence representations and indicate how such representations may play a role in associative learning and decision-making.
Collapse
Affiliation(s)
- Orit Heimer
- Department of Psychology, University of Haifa, Haifa, Israel.
| | - Uri Hertz
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
| |
Collapse
|
14
|
Cheng H, Chafee MV, Blackman RK, Brown JW. Monkey Prefrontal Cortex Learns to Minimize Sequence Prediction Error. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.02.28.582611. [PMID: 38464188 PMCID: PMC10925260 DOI: 10.1101/2024.02.28.582611] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
In this study, we develop a novel recurrent neural network (RNN) model of pre-frontal cortex that predicts sensory inputs, actions, and outcomes at the next time step. Synaptic weights in the model are adjusted to minimize sequence prediction error, adapting a deep learning rule similar to those of large language models. The model, called Sequence Prediction Error Learning (SPEL), is a simple RNN that predicts world state at the next time step, but that differs from standard RNNs by using its own prediction errors from the previous state predictions as inputs to the hidden units of the network. We show that the time course of sequence prediction errors generated by the model closely matched the activity time courses of populations of neurons in macaque prefrontal cortex. Hidden units in the model responded to combinations of task variables and exhibited sensitivity to changing stimulus probability in ways that closely resembled monkey prefrontal neurons. Moreover, the model generated prolonged response times to infrequent, unexpected events as did monkeys. The results suggest that prefrontal cortex may generate internal models of the temporal structure of the world even during tasks that do not explicitly depend on temporal expectation, using a sequence prediction error minimization learning rule to do so. As such, the SPEL model provides a unified, general-purpose theoretical framework for modeling the lateral prefrontal cortex.
Collapse
|
15
|
Nebe S, Kretzschmar A, Brandt MC, Tobler PN. Characterizing Human Habits in the Lab. COLLABRA. PSYCHOLOGY 2024; 10:92949. [PMID: 38463460 PMCID: PMC7615722 DOI: 10.1525/collabra.92949] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 03/12/2024]
Abstract
Habits pose a fundamental puzzle for those aiming to understand human behavior. They pervade our everyday lives and dominate some forms of psychopathology but are extremely hard to elicit in the lab. In this Registered Report, we developed novel experimental paradigms grounded in computational models, which suggest that habit strength should be proportional to the frequency of behavior and, in contrast to previous research, independent of value. Specifically, we manipulated how often participants performed responses in two tasks varying action repetition without, or separately from, variations in value. Moreover, we asked how this frequency-based habitization related to value-based operationalizations of habit and self-reported propensities for habitual behavior in real life. We find that choice frequency during training increases habit strength at test and that this form of habit shows little relation to value-based operationalizations of habit. Our findings empirically ground a novel perspective on the constituents of habits and suggest that habits may arise in the absence of external reinforcement. We further find no evidence for an overlap between different experimental approaches to measuring habits and no associations with self-reported real-life habits. Thus, our findings call for a rigorous reassessment of our understanding and measurement of human habitual behavior in the lab.
Collapse
Affiliation(s)
- Stephan Nebe
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Switzerland
| | - André Kretzschmar
- Individual Differences and Assessment, Department of Psychology, University of Zurich, Switzerland
| | - Maike C. Brandt
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Switzerland
| | - Philippe N. Tobler
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Switzerland
| |
Collapse
|
16
|
Liu Q, Zhao Y, Attanti S, Voss JL, Schoenbaum G, Kahnt T. Midbrain signaling of identity prediction errors depends on orbitofrontal cortex networks. Nat Commun 2024; 15:1704. [PMID: 38402210 PMCID: PMC10894191 DOI: 10.1038/s41467-024-45880-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2023] [Accepted: 02/06/2024] [Indexed: 02/26/2024] Open
Abstract
Outcome-guided behavior requires knowledge about the identity of future rewards. Previous work across species has shown that the dopaminergic midbrain responds to violations in expected reward identity and that the lateral orbitofrontal cortex (OFC) represents reward identity expectations. Here we used network-targeted transcranial magnetic stimulation (TMS) and functional magnetic resonance imaging (fMRI) during a trans-reinforcer reversal learning task to test the hypothesis that outcome expectations in the lateral OFC contribute to the computation of identity prediction errors (iPE) in the midbrain. Network-targeted TMS aiming at lateral OFC reduced the global connectedness of the lateral OFC and impaired reward identity learning in the first block of trials. Critically, TMS disrupted neural representations of expected reward identity in the OFC and modulated iPE responses in the midbrain. These results support the idea that iPE signals in the dopaminergic midbrain are computed based on outcome expectations represented in the lateral OFC.
Collapse
Affiliation(s)
- Qingfang Liu
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Yao Zhao
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Sumedha Attanti
- Mayo Clinic Alix School of Medicine, Scottsdale, AZ, 85259, USA
| | - Joel L Voss
- Department of Neurology, The University of Chicago, Chicago, IL, 60611, USA
| | - Geoffrey Schoenbaum
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA
| | - Thorsten Kahnt
- National Institute on Drug Abuse Intramural Research Program, Baltimore, MD, 21224, USA.
| |
Collapse
|
17
|
Hallquist MN, Hwang K, Luna B, Dombrovski AY. Reward-based option competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space. SCIENCE ADVANCES 2024; 10:eadj2219. [PMID: 38394198 PMCID: PMC10889364 DOI: 10.1126/sciadv.adj2219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 01/23/2024] [Indexed: 02/25/2024]
Abstract
Primates exploring and exploiting a continuous sensorimotor space rely on dynamic maps in the dorsal stream. Two complementary perspectives exist on how these maps encode rewards. Reinforcement learning models integrate rewards incrementally over time, efficiently resolving the exploration/exploitation dilemma. Working memory buffer models explain rapid plasticity of parietal maps but lack a plausible exploration/exploitation policy. The reinforcement learning model presented here unifies both accounts, enabling rapid, information-compressing map updates and efficient transition from exploration to exploitation. As predicted by our model, activity in human frontoparietal dorsal stream regions, but not in MT+, tracks the number of competing options, as preferred options are selectively maintained on the map, while spatiotemporally distant alternatives are compressed out. When valuable new options are uncovered, posterior β1/α oscillations desynchronize within 0.4 to 0.7 s, consistent with option encoding by competing β1-stabilized subpopulations. Together, outcomes matching locally cached reward representations rapidly update parietal maps, biasing choices toward often-sampled, rewarded options.
Collapse
Affiliation(s)
| | - Kai Hwang
- Department of Psychological and Brain Sciences, Iowa Neuroscience Institute, University of Iowa, Iowa City, IA, USA
| | - Beatriz Luna
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, USA
| | | |
Collapse
|
18
|
Park S, Kim J, Kim S. Corticostriatal activity related to performance during continuous de novo motor learning. Sci Rep 2024; 14:3731. [PMID: 38355810 PMCID: PMC10867026 DOI: 10.1038/s41598-024-54176-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/21/2023] [Accepted: 02/09/2024] [Indexed: 02/16/2024] Open
Abstract
Corticostriatal regions play a pivotal role in visuomotor learning. However, less research has been done on how fMRI activity in their subregions is related to task performance, which is provided as visual feedback during motor learning. To address this, we conducted an fMRI experiment in which participants acquired a complex de novo motor skill using continuous or binary visual feedback related to performance. We found a highly selective response related to performance in the entire striatum in both conditions and a relatively higher response in the caudate nucleus for the binary feedback condition. However, the ventromedial prefrontal cortex (vmPFC) response was significant only for the continuous feedback condition. Furthermore, we also found functional distinction of the striatal subregions in random versus goal-directed motor control. These findings underscore the substantial effects of the visual feedback indicating performance on distinct corticostriatal responses, thereby elucidating its significance in reinforcement-based motor learning.
Collapse
Affiliation(s)
- Sungbeen Park
- Department of Artificial Intelligence, Hanyang University, 222 Wangsimni-Ro Seongdong-Gu, Seoul, 04763, Republic of Korea
| | - Junghyun Kim
- Department of Data Science, Hanyang University, 222 Wangsimni-Ro Seongdong-Gu, Seoul, 04763, Republic of Korea
| | - Sungshin Kim
- Department of Artificial Intelligence, Hanyang University, 222 Wangsimni-Ro Seongdong-Gu, Seoul, 04763, Republic of Korea.
- Department of Data Science, Hanyang University, 222 Wangsimni-Ro Seongdong-Gu, Seoul, 04763, Republic of Korea.
- Center for Neuroscience Imaging Research, Institute for Basic Science, 2066 Seobu-Ro, Jangan-Gu, Suwon, 16419, Republic of Korea.
| |
Collapse
|
19
|
Kopytin G, Ivanova M, Herrojo Ruiz M, Shestakova A. Evaluating the Influence of Musical and Monetary Rewards on Decision Making through Computational Modelling. Behav Sci (Basel) 2024; 14:124. [PMID: 38392477 PMCID: PMC10886002 DOI: 10.3390/bs14020124] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 01/25/2024] [Accepted: 02/02/2024] [Indexed: 02/24/2024] Open
Abstract
A central question in behavioural neuroscience is how different rewards modulate learning. While the role of monetary rewards is well-studied in decision-making research, the influence of abstract rewards like music remains poorly understood. This study investigated the dissociable effects of these two reward types on decision making. Forty participants completed two decision-making tasks, each characterised by probabilistic associations between stimuli and rewards, with probabilities changing over time to reflect environmental volatility. In each task, choices were reinforced either by monetary outcomes (win/lose) or by the endings of musical melodies (consonant/dissonant). We applied the Hierarchical Gaussian Filter, a validated hierarchical Bayesian framework, to model learning under these two conditions. Bayesian statistics provided evidence for similar learning patterns across both reward types, suggesting individuals' similar adaptability. However, within the musical task, individual preferences for consonance over dissonance explained some aspects of learning. Specifically, correlation analyses indicated that participants more tolerant of dissonance behaved more stochastically in their belief-to-response mappings and were less likely to choose the response associated with the current prediction for a consonant ending, driven by higher volatility estimates. By contrast, participants averse to dissonance showed increased tonic volatility, leading to larger updates in reward tendency beliefs.
Collapse
Affiliation(s)
- Grigory Kopytin
- Institute for Cognitive Neuroscience, HSE University, 101000 Moscow, Russia
| | - Marina Ivanova
- Institute for Cognitive Neuroscience, HSE University, 101000 Moscow, Russia
| | - Maria Herrojo Ruiz
- Department of Psychology, Goldsmiths University of London, London SE14 6NW, UK
| | - Anna Shestakova
- Institute for Cognitive Neuroscience, HSE University, 101000 Moscow, Russia
| |
Collapse
|
20
|
Ongchoco JDK, Knobe J, Jara-Ettinger J. People's thinking plans adapt to the problem they're trying to solve. Cognition 2024; 243:105669. [PMID: 38039797 DOI: 10.1016/j.cognition.2023.105669] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/27/2023] [Revised: 11/13/2023] [Accepted: 11/16/2023] [Indexed: 12/03/2023]
Abstract
Much of our thinking focuses on deciding what to do in situations where the space of possible options is too large to evaluate exhaustively. Previous work has found that people do this by learning the general value of different behaviors, and prioritizing thinking about high-value options in new situations. Is this good-action bias always the best strategy, or can thinking about low-value options sometimes become more beneficial? Can people adapt their thinking accordingly based on the situation? And how do we know what to think about in novel events? Here, we developed a block-puzzle paradigm that enabled us to measure people's thinking plans and compare them to a computational model of rational thought. We used two distinct response methods to explore what people think about-a self-report method, in which we asked people explicitly to report what they thought about, and an implicit response time method, in which we used people's decision-making times to reveal what they thought about. Our results suggest that people can quickly estimate the apparent value of different options and use this to decide what to think about. Critically, we find that people can flexibly prioritize whether to think about high-value options (Experiments 1 and 2) or low-value options (Experiments 3, 4, and 5), depending on the problem. Through computational modeling, we show that these thinking strategies are broadly rational, enabling people to maximize the value of long-term decisions. Our results suggest that thinking plans are flexible: What we think about depends on the structure of the problems we are trying to solve.
Collapse
Affiliation(s)
| | - Joshua Knobe
- Department of Psychology, Yale University, New Haven, USA
| | | |
Collapse
|
21
|
Cisler JM, Dunsmoor JE, Fonzo GA, Nemeroff CB. Latent-state and model-based learning in PTSD. Trends Neurosci 2024; 47:150-162. [PMID: 38212163 PMCID: PMC10923154 DOI: 10.1016/j.tins.2023.12.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 12/18/2023] [Accepted: 12/18/2023] [Indexed: 01/13/2024]
Abstract
Post-traumatic stress disorder (PTSD) is characterized by altered emotional and behavioral responding following a traumatic event. In this article, we review the concepts of latent-state and model-based learning (i.e., learning and inferring abstract task representations) and discuss their relevance for clinical and neuroscience models of PTSD. Recent data demonstrate evidence for brain and behavioral biases in these learning processes in PTSD. These new data potentially recast excessive fear towards trauma cues as a problem in learning and updating abstract task representations, as opposed to traditional conceptualizations focused on stimulus-specific learning. Biases in latent-state and model-based learning may also be a common mechanism targeted in common therapies for PTSD. We highlight key knowledge gaps that need to be addressed to further elaborate how latent-state learning and its associated neurocircuitry mechanisms function in PTSD and how to optimize treatments to target these processes.
Collapse
Affiliation(s)
- Josh M Cisler
- Department of Psychiatry and Behavioral Sciences, University of Texas at Austin, Austin, TX, USA; Institute for Early Life Adversity Research, University of Texas at Austin, Austin, TX, USA.
| | - Joseph E Dunsmoor
- Department of Psychiatry and Behavioral Sciences, University of Texas at Austin, Austin, TX, USA; Institute for Early Life Adversity Research, University of Texas at Austin, Austin, TX, USA
| | - Gregory A Fonzo
- Department of Psychiatry and Behavioral Sciences, University of Texas at Austin, Austin, TX, USA; Institute for Early Life Adversity Research, University of Texas at Austin, Austin, TX, USA
| | - Charles B Nemeroff
- Department of Psychiatry and Behavioral Sciences, University of Texas at Austin, Austin, TX, USA; Institute for Early Life Adversity Research, University of Texas at Austin, Austin, TX, USA
| |
Collapse
|
22
|
Farmani S, Sharifi K, Ghazizadeh A. Cortical and subcortical substrates of minutes and days-long object value memory in humans. Cereb Cortex 2024; 34:bhae006. [PMID: 38244576 DOI: 10.1093/cercor/bhae006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2023] [Revised: 12/30/2023] [Accepted: 12/31/2023] [Indexed: 01/22/2024] Open
Abstract
Obtaining valuable objects motivates many of our daily decisions. However, the neural underpinnings of object processing based on human value memory are not yet fully understood. Here, we used whole-brain functional magnetic resonance imaging (fMRI) to examine activations due to value memory as participants passively viewed objects before, minutes after, and 1-70 days following value training. Significant value memory for objects was evident in the behavioral performance, which nevertheless faded over the days following training. Minutes after training, the occipital, ventral temporal, interparietal, and frontal areas showed strong value discrimination. Days after training, activation in the frontal, temporal, and occipital regions decreased, whereas the parietal areas showed sustained activation. In addition, days-long value responses emerged in certain subcortical regions, including the caudate, ventral striatum, and thalamus. Resting-state analysis revealed that these subcortical areas were functionally connected. Furthermore, the activation in the striatal cluster was positively correlated with participants' performance in days-long value memory. These findings shed light on the neural basis of value memory in humans with implications for object habit formation and cross-species comparisons.
Collapse
Affiliation(s)
- Sepideh Farmani
- School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM), Tehran 19395-5746, Iran
| | - Kiomars Sharifi
- School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM), Tehran 19395-5746, Iran
- Bio-Intelligence Unit, Electrical Engineering Department, Sharif University of Technology, Tehran 11365-11155, Iran
| | - Ali Ghazizadeh
- School of Cognitive Sciences, Institute for Research in Fundamental Sciences (IPM), Tehran 19395-5746, Iran
- Bio-Intelligence Unit, Electrical Engineering Department, Sharif University of Technology, Tehran 11365-11155, Iran
| |
Collapse
|
23
|
Brooks HR, Sokol-Hessner P. Multiple timescales of temporal context in risky choice: Behavioral identification and relationships to physiological arousal. PLoS One 2024; 19:e0296681. [PMID: 38241251 PMCID: PMC10798524 DOI: 10.1371/journal.pone.0296681] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Accepted: 12/15/2023] [Indexed: 01/21/2024] Open
Abstract
Context-dependence is fundamental to risky monetary decision-making. A growing body of evidence suggests that temporal context, or recent events, alters risk-taking at a minimum of three timescales: immediate (e.g. trial-by-trial), neighborhood (e.g. a group of consecutive trials), and global (e.g. task-level). To examine context effects, we created a novel monetary choice set with intentional temporal structure in which option values shifted between multiple levels of value magnitude ("contexts") several times over the course of the task. This structure allowed us to examine whether effects of each timescale were simultaneously present in risky choice behavior and the potential mechanistic role of arousal, an established correlate of risk-taking, in context-dependency. We found that risk-taking was sensitive to immediate, neighborhood, and global timescales: risk-taking decreased following large (vs. small) outcome amounts, increased following large positive (but not negative) shifts in context, and increased when cumulative earnings exceeded expectations. We quantified arousal with skin conductance responses, which were related to the global timescale, increasing with cumulative earnings, suggesting that physiological arousal captures a task-level assessment of performance. Our results both replicate and extend prior research by demonstrating that risky decision-making is consistently dynamic at multiple timescales and that the role of arousal in risk-taking extends to some, but not all timescales of context-dependence.
Collapse
Affiliation(s)
- Hayley R. Brooks
- Department of Psychology, University of Denver, Denver, Colorado, United States of America
- Department of Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, Rhode Island, United States of America
| | - Peter Sokol-Hessner
- Department of Psychology, University of Denver, Denver, Colorado, United States of America
| |
Collapse
|
24
|
Xu S, Ren W. Distinct processing of the state prediction error signals in frontal and parietal correlates in learning the environment model. Cereb Cortex 2024; 34:bhad449. [PMID: 38037370 DOI: 10.1093/cercor/bhad449] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Accepted: 10/31/2023] [Indexed: 12/02/2023] Open
Abstract
Goal-directed reinforcement learning constructs a model of how the states in the environment are connected and prospectively evaluates action values by simulating experience. State prediction error (SPE) is theorized as a crucial signal for learning the environment model. However, the underlying neural mechanisms remain unclear. Here, using electroencephalogram, we verified in a two-stage Markov task two neural correlates of SPEs: an early negative correlate transferring from frontal to central electrodes and a late positive correlate over parietal regions. Furthermore, by investigating the effects of explicit knowledge about the environment model and rewards in the environment, we found that, for the parietal correlate, rewards enhanced the representation efficiency (beta values of regression coefficient) of SPEs, whereas explicit knowledge elicited a larger SPE representation (event-related potential activity) for rare transitions. However, for the frontal and central correlates, rewards increased activities in a content-independent way and explicit knowledge enhanced activities only for common transitions. Our results suggest that the parietal correlate of SPEs is responsible for the explicit learning of state transition structure, whereas the frontal and central correlates may be involved in cognitive control. Our study provides novel evidence for distinct roles of the frontal and the parietal cortices in processing SPEs.
Collapse
Affiliation(s)
- Shuyuan Xu
- MOE Key Laboratory of Modern Teaching Technology, Shaanxi Normal University, Xi'an, Shaanxi, China
| | - Wei Ren
- MOE Key Laboratory of Modern Teaching Technology, Shaanxi Normal University, Xi'an, Shaanxi, China
- Faculty of Education, Shaanxi Normal University, Xi'an, Shaanxi, China
| |
Collapse
|
25
|
Stevenson N, Innes RJ, Boag RJ, Miletić S, Isherwood SJS, Trutti AC, Heathcote A, Forstmann BU. Joint Modelling of Latent Cognitive Mechanisms Shared Across Decision-Making Domains. COMPUTATIONAL BRAIN & BEHAVIOR 2024; 7:1-22. [PMID: 38425991 PMCID: PMC10899373 DOI: 10.1007/s42113-023-00192-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Accepted: 11/27/2023] [Indexed: 03/02/2024]
Abstract
Decision-making behavior is often understood using the framework of evidence accumulation models (EAMs). Nowadays, EAMs are applied to various domains of decision-making with the underlying assumption that the latent cognitive constructs proposed by EAMs are consistent across these domains. In this study, we investigate both the extent to which the parameters of EAMs are related between four different decision-making domains and across different time points. To that end, we make use of the novel joint modelling approach, that explicitly includes relationships between parameters, such as covariances or underlying factors, in one combined joint model. Consequently, this joint model also accounts for measurement error and uncertainty within the estimation of these relations. We found that EAM parameters were consistent between time points on three of the four decision-making tasks. For our between-task analysis, we constructed a joint model with a factor analysis on the parameters of the different tasks. Our two-factor joint model indicated that information processing ability was related between the different decision-making domains. However, other cognitive constructs such as the degree of response caution and urgency were only comparable on some domains.
Collapse
Affiliation(s)
- Niek Stevenson
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Reilly J. Innes
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Russell J. Boag
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Steven Miletić
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | | | - Anne C. Trutti
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Andrew Heathcote
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Birte U. Forstmann
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| |
Collapse
|
26
|
Badrulhisham F, Pogatzki-Zahn E, Segelcke D, Spisak T, Vollert J. Machine learning and artificial intelligence in neuroscience: A primer for researchers. Brain Behav Immun 2024; 115:470-479. [PMID: 37972877 DOI: 10.1016/j.bbi.2023.11.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/26/2023] [Revised: 10/16/2023] [Accepted: 11/08/2023] [Indexed: 11/19/2023] Open
Abstract
Artificial intelligence (AI) is often used to describe the automation of complex tasks that we would attribute intelligence to. Machine learning (ML) is commonly understood as a set of methods used to develop an AI. Both have seen a recent boom in usage, both in scientific and commercial fields. For the scientific community, ML can solve bottle necks created by complex, multi-dimensional data generated, for example, by functional brain imaging or *omics approaches. ML can here identify patterns that could not have been found using traditional statistic approaches. However, ML comes with serious limitations that need to be kept in mind: their tendency to optimise solutions for the input data means it is of crucial importance to externally validate any findings before considering them more than a hypothesis. Their black-box nature implies that their decisions usually cannot be understood, which renders their use in medical decision making problematic and can lead to ethical issues. Here, we present an introduction for the curious to the field of ML/AI. We explain the principles as commonly used methods as well as recent methodological advancements before we discuss risks and what we see as future directions of the field. Finally, we show practical examples of neuroscience to illustrate the use and limitations of ML.
Collapse
Affiliation(s)
| | - Esther Pogatzki-Zahn
- Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Muenster, Muenster, Germany
| | - Daniel Segelcke
- Department of Anaesthesiology, Intensive Care and Pain Medicine, University Hospital Muenster, Muenster, Germany
| | - Tamas Spisak
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Medicine Essen, Essen, Germany; Center for Translational Neuro- and Behavioral Sciences, Department of Neurology, University Medicine Essen, Essen, Germany
| | - Jan Vollert
- Department of Clinical and Biomedical Sciences, Faculty of Health and Life Sciences, University of Exeter, Exeter, United Kingdom; Pain Research, Department of Surgery and Cancer, Imperial College London, London, United Kingdom.
| |
Collapse
|
27
|
Pool ER, Pauli WM, Cross L, O'Doherty JP. Neural substrates of parallel devaluation-sensitive and devaluation-insensitive Pavlovian learning in humans. Nat Commun 2023; 14:8057. [PMID: 38052792 PMCID: PMC10697955 DOI: 10.1038/s41467-023-43747-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Accepted: 11/17/2023] [Indexed: 12/07/2023] Open
Abstract
We aim to differentiate the brain regions involved in the learning and encoding of Pavlovian associations sensitive to changes in outcome value from those that are not sensitive to such changes by combining a learning task with outcome devaluation, eye-tracking, and functional magnetic resonance imaging in humans. Contrary to theoretical expectation, voxels correlating with reward prediction errors in the ventral striatum and subgenual cingulate appear to be sensitive to devaluation. Moreover, regions encoding state prediction errors appear to be devaluation insensitive. We can also distinguish regions encoding predictions about outcome taste identity from predictions about expected spatial location. Regions encoding predictions about taste identity seem devaluation sensitive while those encoding predictions about an outcome's spatial location seem devaluation insensitive. These findings suggest the existence of multiple and distinct associative mechanisms in the brain and help identify putative neural correlates for the parallel expression of both devaluation sensitive and insensitive conditioned behaviors.
Collapse
Affiliation(s)
- Eva R Pool
- Swiss Center for Affective Sciences, University of Geneva, Geneva, Switzerland.
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA.
| | - Wolfgang M Pauli
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, USA
| | - Logan Cross
- Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA, USA
- Department of Computer Science, Stanford University, Palo Alto, CA, USA
| | - John P O'Doherty
- Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA, USA
- Computation and Neural Systems Program, California Institute of Technology, Pasadena, CA, USA
| |
Collapse
|
28
|
Liu L, Liu D, Guo T, Schwieter JW, Liu H. The right superior temporal gyrus plays a role in semantic-rule learning: Evidence supporting a reinforcement learning model. Neuroimage 2023; 282:120393. [PMID: 37820861 DOI: 10.1016/j.neuroimage.2023.120393] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Revised: 08/29/2023] [Accepted: 09/25/2023] [Indexed: 10/13/2023] Open
Abstract
In real-life communication, individuals use language that carries evident rewarding and punishing elements, such as praise and criticism. A common trend is to seek more praise while avoiding criticism. Furthermore, semantics is crucial for conveying information, but such semantic access to native and foreign languages is subtly distinct. To investigate how rule learning occurs in different languages and to highlight the importance of semantics in this process, we investigated both verbal and non-verbal rule learning in first (L1) and second (L2) languages using a reinforcement learning framework, including a semantic rule and a color rule. Our computational modeling on behavioral and brain imaging data revealed that individuals may be more motivated to learn and adhere to rules in an L1 compared to L2, with greater striatum activation during the outcome phase in the L1. Additionally, results on the learning rates and inverse temperature in the two rule learning tasks showed that individuals tend to be conservative and are reluctant to change their judgments regarding rule learning of semantic information. Moreover, the greater the prediction errors, the greater activation of the right superior temporal gyrus in the semantic-rule learning condition, demonstrating that such learning has differential neural correlates than symbolic rule learning. Overall, the findings provide insight into the neural mechanisms underlying rule learning in different languages, and indicate that rule learning involving verbal semantics is not a general symbolic learning that resembles a conditioned stimulus-response, but rather has its own specific characteristics.
Collapse
Affiliation(s)
- Linyan Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - Dongxue Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - Tingting Guo
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - John W Schwieter
- Language Acquisition, Multilingualism, and Cognition Laboratory / Bilingualism Matters @ Wilfrid Laurier University, Canada; Department of Linguistics and Languages, McMaster University, Canada
| | - Huanhuan Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China.
| |
Collapse
|
29
|
Kim D, Wang Z, Sakagami M, Sasaki Y, Watanabe T. Only cortical prediction error signals are involved in visual learning, despite availability of subcortical prediction error signals. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.13.566726. [PMID: 38014275 PMCID: PMC10680585 DOI: 10.1101/2023.11.13.566726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/29/2023]
Abstract
Both the midbrain systems, encompassing the ventral striatum (VS), and the cortical systems, including the dorsal anterior cingulate cortex (dACC), play roles in reinforcing and enhancing learning. However, the specific contributions of signals from these regions in learning remains unclear. To investigate this, we examined how VS and dACC are involved in visual perceptual learning (VPL) through an orientation discrimination task. In the primary experiment, subjects fasted for 5 hours before each of 14 days of training sessions and 3 days of test sessions. Subjects were rewarded with water for accurate trial responses. During the test sessions, BOLD signals were recorded from regions including VS and dACC. Although BOLD signals in both areas were associated with positive and negative RPEs, only those in dACC associated with negative RPE showed a significant correlation with performance improvement. Additionally, no significant correlation was observed between BOLD signals associated with RPEs in VS and dACC. These results suggest that although signals associated with positive and negative RPEs from both midbrain and cortical systems are readily accessible, only RPE signals in the prefrontal system, generated without linking to RPE signals in VS, are utilized for the enhancement of VPL.
Collapse
|
30
|
Eppinger B, Ruel A, Bolenz F. Diminished State Space Theory of Human Aging. PERSPECTIVES ON PSYCHOLOGICAL SCIENCE 2023:17456916231204811. [PMID: 37931229 DOI: 10.1177/17456916231204811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2023]
Abstract
Many new technologies, such as smartphones, computers, or public-access systems (like ticket-vending machines), are a challenge for older adults. One feature that these technologies have in common is that they involve underlying, partially observable, structures (state spaces) that determine the actions that are necessary to reach a certain goal (e.g., to move from one menu to another, to change a function, or to activate a new service). In this work we provide a theoretical, neurocomputational account to explain these behavioral difficulties in older adults. Based on recent findings from age-comparative computational- and cognitive-neuroscience studies, we propose that age-related impairments in complex goal-directed behavior result from an underlying deficit in the representation of state spaces of cognitive tasks. Furthermore, we suggest that these age-related deficits in adaptive decision-making are due to impoverished neural representations in the orbitofrontal cortex and hippocampus.
Collapse
Affiliation(s)
- Ben Eppinger
- Institute of Psychology, University of Greifswald
- Department of Psychology, Concordia University
- PERFORM Centre, Concordia University
- Faculty of Psychology, Technische Universität Dresden
| | - Alexa Ruel
- Department of Psychology, Concordia University
- PERFORM Centre, Concordia University
- Institute of Psychology, University of Hamburg
| | - Florian Bolenz
- Center for Adaptive Rationality, Max Planck Institute for Human Development, Berlin, Germany
- Science of Intelligence/Cluster of Excellence, Technical University of Berlin
| |
Collapse
|
31
|
Lee HJ, Lee H, Lim CY, Rhim I, Lee SH. Corrective feedback guides human perceptual decision-making by informing about the world state rather than rewarding its choice. PLoS Biol 2023; 21:e3002373. [PMID: 37939126 PMCID: PMC10659185 DOI: 10.1371/journal.pbio.3002373] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 11/20/2023] [Accepted: 10/10/2023] [Indexed: 11/10/2023] Open
Abstract
Corrective feedback received on perceptual decisions is crucial for adjusting decision-making strategies to improve future choices. However, its complex interaction with other decision components, such as previous stimuli and choices, challenges a principled account of how it shapes subsequent decisions. One popular approach, based on animal behavior and extended to human perceptual decision-making, employs "reinforcement learning," a principle proven successful in reward-based decision-making. The core idea behind this approach is that decision-makers, although engaged in a perceptual task, treat corrective feedback as rewards from which they learn choice values. Here, we explore an alternative idea, which is that humans consider corrective feedback on perceptual decisions as evidence of the actual state of the world rather than as rewards for their choices. By implementing these "feedback-as-reward" and "feedback-as-evidence" hypotheses on a shared learning platform, we show that the latter outperforms the former in explaining how corrective feedback adjusts the decision-making strategy along with past stimuli and choices. Our work suggests that humans learn about what has happened in their environment rather than the values of their own choices through corrective feedback during perceptual decision-making.
Collapse
Affiliation(s)
- Hyang-Jung Lee
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, South Korea
| | - Heeseung Lee
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, South Korea
| | - Chae Young Lim
- Department of Statistics, Seoul National University, Seoul, South Korea
| | - Issac Rhim
- Institute of Neuroscience, University of Oregon, Eugene, Oregon, United States of America
| | - Sang-Hun Lee
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, South Korea
| |
Collapse
|
32
|
Huo H, Lesage E, Dong W, Verguts T, Seger CA, Diao S, Feng T, Chen Q. The neural substrates of how model-based learning affects risk taking: Functional coupling between right cerebellum and left caudate. Brain Cogn 2023; 172:106088. [PMID: 37783018 DOI: 10.1016/j.bandc.2023.106088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 09/19/2023] [Accepted: 09/20/2023] [Indexed: 10/04/2023]
Abstract
Higher executive control capacity allows people to appropriately evaluate risk and avoid both excessive risk aversion and excessive risk-taking. The neural mechanisms underlying this relationship between executive function and risk taking are still unknown. We used voxel-based morphometry (VBM) analysis combined with resting-state functional connectivity (rs-FC) to evaluate how one component of executive function, model-based learning, relates to risk taking. We measured individuals' use of the model-based learning system with the two-step task, and risk taking with the Balloon Analogue Risk Task. Behavioral results indicated that risk taking was positively correlated with the model-based weighting parameter ω. The VBM results showed a positive association between model-based learning and gray matter volume in the right cerebellum (RCere) and left inferior parietal lobule (LIPL). Functional connectivity results suggested that the coupling between RCere and the left caudate (LCAU) was correlated with both model-based learning and risk taking. Mediation analysis indicated that RCere-LCAU functional connectivity completely mediated the effect of model-based learning on risk taking. These results indicate that learners who favor model-based strategies also engage in more appropriate risky behaviors through interactions between reward-based learning, error-based learning and executive control subserved by a caudate, cerebellar and parietal network.
Collapse
Affiliation(s)
- Hangfeng Huo
- Department of Psychology, Faculty of Education, Guangxi Normal University, Guilin, China; School of Psychology, South China Normal University, 510631 Guangzhou, China; Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China; Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| | - Elise Lesage
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| | - Wenshan Dong
- School of Psychology, South China Normal University, 510631 Guangzhou, China; Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China; Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| | - Tom Verguts
- Department of Experimental Psychology, Ghent University, Ghent, Belgium
| | - Carol A Seger
- School of Psychology, South China Normal University, 510631 Guangzhou, China; Center for Studies of Psychological Application, South China Normal University, 510631 Guangzhou, China; Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China; Department of Psychology and Program in Molecular, Cellular, and Integrative Neurosciences, Colorado State University, Fort Collins, CO, 80523, USA
| | - Sitong Diao
- School of Psychology, Shenzhen University, 518060 Shenzhen, China
| | - Tingyong Feng
- Research Center of Psychology and Social Development, Faculty of Psychology, Southwest University, Chongqing, China; Key Laboratory of Cognition and Personality, Ministry of Education, Chongqing, China.
| | - Qi Chen
- School of Psychology, Shenzhen University, 518060 Shenzhen, China.
| |
Collapse
|
33
|
Gold BP, Pearce MT, McIntosh AR, Chang C, Dagher A, Zatorre RJ. Auditory and reward structures reflect the pleasure of musical expectancies during naturalistic listening. Front Neurosci 2023; 17:1209398. [PMID: 37928727 PMCID: PMC10625409 DOI: 10.3389/fnins.2023.1209398] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 10/05/2023] [Indexed: 11/07/2023] Open
Abstract
Enjoying music consistently engages key structures of the neural auditory and reward systems such as the right superior temporal gyrus (R STG) and ventral striatum (VS). Expectations seem to play a central role in this effect, as preferences reliably vary according to listeners' uncertainty about the musical future and surprise about the musical past. Accordingly, VS activity reflects the pleasure of musical surprise, and exhibits stronger correlations with R STG activity as pleasure grows. Yet the reward value of musical surprise - and thus the reason for these surprises engaging the reward system - remains an open question. Recent models of predictive neural processing and learning suggest that forming, testing, and updating hypotheses about one's environment may be intrinsically rewarding, and that the constantly evolving structure of musical patterns could provide ample opportunity for this procedure. Consistent with these accounts, our group previously found that listeners tend to prefer melodic excerpts taken from real music when it either validates their uncertain melodic predictions (i.e., is high in uncertainty and low in surprise) or when it challenges their highly confident ones (i.e., is low in uncertainty and high in surprise). An independent research group (Cheung et al., 2019) replicated these results with musical chord sequences, and identified their fMRI correlates in the STG, amygdala, and hippocampus but not the VS, raising new questions about the neural mechanisms of musical pleasure that the present study seeks to address. Here, we assessed concurrent liking ratings and hemodynamic fMRI signals as 24 participants listened to 50 naturalistic, real-world musical excerpts that varied across wide spectra of computationally modeled uncertainty and surprise. As in previous studies, liking ratings exhibited an interaction between uncertainty and surprise, with the strongest preferences for high uncertainty/low surprise and low uncertainty/high surprise. FMRI results also replicated previous findings, with music liking effects in the R STG and VS. Furthermore, we identify interactions between uncertainty and surprise on the one hand, and liking and surprise on the other, in VS activity. Altogether, these results provide important support for the hypothesized role of the VS in deriving pleasure from learning about musical structure.
Collapse
Affiliation(s)
- Benjamin P. Gold
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, TN, United States
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
- International Laboratory for Brain, Music and Sound Research (BRAMS), Montreal, QC, Canada
- Centre for Research on Brain, Language and Music (CRBLM), Montreal, QC, Canada
- Centre for Interdisciplinary Research in Music, Media, and Technology (CIRMMT), Montreal, QC, Canada
| | - Marcus T. Pearce
- Cognitive Science Research Group, School of Electronic Engineering & Computer Science, Queen Mary University of London, London, United Kingdom
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| | - Anthony R. McIntosh
- Baycrest Centre, Rotman Research Institute, Toronto, ON, Canada
- Department of Psychology, University of Toronto, Toronto, ON, Canada
| | - Catie Chang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, TN, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| | - Alain Dagher
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
| | - Robert J. Zatorre
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
- International Laboratory for Brain, Music and Sound Research (BRAMS), Montreal, QC, Canada
- Centre for Research on Brain, Language and Music (CRBLM), Montreal, QC, Canada
- Centre for Interdisciplinary Research in Music, Media, and Technology (CIRMMT), Montreal, QC, Canada
| |
Collapse
|
34
|
Burk DC, Taswell C, Tang H, Averbeck BB. Computational mechanisms underlying motivation to earn symbolic reinforcers. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.10.11.561900. [PMID: 37873311 PMCID: PMC10592730 DOI: 10.1101/2023.10.11.561900] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/25/2023]
Abstract
Reinforcement learning (RL) is a theoretical framework that describes how agents learn to select options that maximize rewards and minimize punishments over time. We often make choices, however, to obtain symbolic reinforcers (e.g. money, points) that can later be exchanged for primary reinforcers (e.g. food, drink). Although symbolic reinforcers are motivating, little is understood about the neural or computational mechanisms underlying the motivation to earn them. In the present study, we examined how monkeys learn to make choices that maximize fluid rewards through reinforcement with tokens. The question addressed here is how the value of a state, which is a function of multiple task features (e.g. current number of accumulated tokens, choice options, task epoch, trials since last delivery of primary reinforcer, etc.), drives value and affects motivation. We constructed a Markov decision process model that computes the value of task states given task features to capture the motivational state of the animal. Fixation times, choice reaction times, and abort frequency were all significantly related to values of task states during the tokens task (n=5 monkeys). Furthermore, the model makes predictions for how neural responses could change on a moment-by-moment basis relative to changes in state value. Together, this task and model allow us to capture learning and behavior related to symbolic reinforcement.
Collapse
Affiliation(s)
- Diana C. Burk
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda MD, 20892-4415
| | - Craig Taswell
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda MD, 20892-4415
| | - Hua Tang
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda MD, 20892-4415
| | - Bruno B. Averbeck
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Bethesda MD, 20892-4415
| |
Collapse
|
35
|
Hernandez-Pavon JC, San Agustín A, Wang MC, Veniero D, Pons JL. Can we manipulate brain connectivity? A systematic review of cortico-cortical paired associative stimulation effects. Clin Neurophysiol 2023; 154:169-193. [PMID: 37634335 DOI: 10.1016/j.clinph.2023.06.016] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Revised: 05/09/2023] [Accepted: 06/16/2023] [Indexed: 08/29/2023]
Abstract
OBJECTIVE Cortico-cortical paired associative stimulation (ccPAS) is a form of dual-site transcranial magnetic stimulation (TMS) entailing a series of single-TMS pulses paired at specific interstimulus intervals (ISI) delivered to distant cortical areas. The goal of this article is to systematically review its efficacy in inducing plasticity in humans focusing on stimulation parameters and hypotheses of underlying neurophysiology. METHODS A systematic review of the literature from 2009-2023 was undertaken to identify all articles utilizing ccPAS to study brain plasticity and connectivity. Six electronic databases were searched and included. RESULTS 32 studies were identified. The studies targeted connections within the same hemisphere or between hemispheres. 28 ccPAS studies were in healthy participants, 1 study in schizophrenia, and 1 in Alzheimer's disease (AD) patients. 2 additional studies used cortico-cortical repetitive paired associative stimulation (cc-rPAS) in generalized anxiety disorder (GAD) patients. Outcome measures include electromyography (EMG), behavioral measures, electroencephalography (EEG), and functional magnetic resonance imaging (fMRI). ccPAS seems to be able to modulate brain connectivity depending on the ISI. CONCLUSIONS ccPAS can be used to modulate corticospinal excitability, brain activity, and behavior. Although the stimulation parameters used across studies reviewed in this paper are varied, ccPAS is a promising approach for basic research and potential clinical applications. SIGNIFICANCE Recent advances in neuroscience have caused a shift of interest from the study of single areas to a more complex approach focusing on networks of areas that orchestrate brain activity. Consequently, the TMS community is also witnessing a change, with a growing interest in targeting multiple brain areas rather than a single locus, as evidenced by an increasing number of papers using ccPAS. In light of this new enthusiasm for brain connectivity, this review summarizes existing literature and stimulation parameters that have proven effective in changing electrophysiological, behavioral, or neuroimaging-derived measures.
Collapse
Affiliation(s)
- Julio C Hernandez-Pavon
- Legs + Walking Lab, Shirley Ryan AbilityLab (Formerly, The Rehabilitation Institute of Chicago), Chicago, IL, USA; Center for Brain Stimulation, Shirley Ryan AbilityLab, Chicago, IL, USA; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA; Department of Psychological Sciences, Kansas State University, Manhattan, KS, USA.
| | - Arantzazu San Agustín
- Legs + Walking Lab, Shirley Ryan AbilityLab (Formerly, The Rehabilitation Institute of Chicago), Chicago, IL, USA; Center for Brain Stimulation, Shirley Ryan AbilityLab, Chicago, IL, USA; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA; Neural Rehabilitation Group, Cajal Institute, CSIC, Madrid, Spain; PhD Program in Neuroscience, Autonoma de Madrid University-Cajal Institute, Madrid 28029, Spain
| | - Max C Wang
- Department of Physical Therapy and Human Movement Science, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| | | | - Jose L Pons
- Legs + Walking Lab, Shirley Ryan AbilityLab (Formerly, The Rehabilitation Institute of Chicago), Chicago, IL, USA; Center for Brain Stimulation, Shirley Ryan AbilityLab, Chicago, IL, USA; Department of Physical Medicine and Rehabilitation, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA; Department of Biomedical Engineering, McCormick School of Engineering, Northwestern University, Chicago, IL, USA
| |
Collapse
|
36
|
Modirshanechi A, Becker S, Brea J, Gerstner W. Surprise and novelty in the brain. Curr Opin Neurobiol 2023; 82:102758. [PMID: 37619425 DOI: 10.1016/j.conb.2023.102758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2023] [Revised: 06/30/2023] [Accepted: 07/20/2023] [Indexed: 08/26/2023]
Abstract
Notions of surprise and novelty have been used in various experimental and theoretical studies across multiple brain areas and species. However, 'surprise' and 'novelty' refer to different quantities in different studies, which raises concerns about whether these studies indeed relate to the same functionalities and mechanisms in the brain. Here, we address these concerns through a systematic investigation of how different aspects of surprise and novelty relate to different brain functions and physiological signals. We review recent classifications of definitions proposed for surprise and novelty along with links to experimental observations. We show that computational modeling and quantifiable definitions enable novel interpretations of previous findings and form a foundation for future theoretical and experimental studies.
Collapse
Affiliation(s)
- Alireza Modirshanechi
- Brain-Mind Institute, School of Life Sciences, EPFL, Lausanne, Switzerland; School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland.
| | - Sophia Becker
- Brain-Mind Institute, School of Life Sciences, EPFL, Lausanne, Switzerland; School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland. https://twitter.com/sophiabecker95
| | - Johanni Brea
- Brain-Mind Institute, School of Life Sciences, EPFL, Lausanne, Switzerland; School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland
| | - Wulfram Gerstner
- Brain-Mind Institute, School of Life Sciences, EPFL, Lausanne, Switzerland; School of Computer and Communication Sciences, EPFL, Lausanne, Switzerland.
| |
Collapse
|
37
|
Peng XR, Bundil I, Schulreich S, Li SC. Neural correlates of valence-dependent belief and value updating during uncertainty reduction: An fNIRS study. Neuroimage 2023; 279:120327. [PMID: 37582418 DOI: 10.1016/j.neuroimage.2023.120327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 08/07/2023] [Accepted: 08/11/2023] [Indexed: 08/17/2023] Open
Abstract
Selective use of new information is crucial for adaptive decision-making. Combining a gamble bidding task with assessing cortical responses using functional near-infrared spectroscopy (fNIRS), we investigated potential effects of information valence on behavioral and neural processes of belief and value updating during uncertainty reduction in young adults. By modeling changes in the participants' expressed subjective values using a Bayesian model, we dissociated processes of (i) updating beliefs about statistical properties of the gamble, (ii) updating values of a gamble based on new information about its winning probabilities, as well as (iii) expectancy violation. The results showed that participants used new information to update their beliefs and values about the gambles in a quasi-optimal manner, as reflected in the selective updating only in situations with reducible uncertainty. Furthermore, their updating was valence-dependent: information indicating an increase in winning probability was underweighted, whereas information about a decrease in winning probability was updated in good agreement with predictions of the Bayesian decision theory. Results of model-based and moderation analyses showed that this valence-dependent asymmetry was associated with a distinct contribution of expectancy violation, besides belief updating, to value updating after experiencing new positive information regarding winning probabilities. In line with the behavioral results, we replicated previous findings showing involvements of frontoparietal brain regions in the different components of updating. Furthermore, this study provided novel results suggesting a valence-dependent recruitment of brain regions. Individuals with stronger oxyhemoglobin responses during value updating was more in line with predictions of the Bayesian model while integrating new information that indicates an increase in winning probability. Taken together, this study provides first results showing expectancy violation as a contributing factor to sub-optimal valence-dependent updating during uncertainty reduction and suggests limitations of normative Bayesian decision theory.
Collapse
Affiliation(s)
- Xue-Rui Peng
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop, Technische Universität Dresden, Dresden, Germany.
| | - Indra Bundil
- Cardiff University Brain Research Imaging Centre, School of Psychology, Cardiff University, Cardiff, United Kingdom
| | - Stefan Schulreich
- Department of Nutritional Sciences, Faculty of Life Sciences, University of Vienna, Vienna, Austria; Department of Cognitive Psychology, Faculty of Psychology and Human Movement Science, Universität Hamburg, Hamburg, Germany
| | - Shu-Chen Li
- Chair of Lifespan Developmental Neuroscience, Faculty of Psychology, Technische Universität Dresden, Dresden, Germany; Centre for Tactile Internet with Human-in-the-Loop, Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
38
|
Möhring L, Gläscher J. Prediction errors drive dynamic changes in neural patterns that guide behavior. Cell Rep 2023; 42:112931. [PMID: 37540597 DOI: 10.1016/j.celrep.2023.112931] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 06/13/2023] [Accepted: 07/18/2023] [Indexed: 08/06/2023] Open
Abstract
Learning describes the process by which our internal expectation models of the world are updated by surprising outcomes (prediction errors [PEs]) to improve predictions of future events. However, the mechanisms through which error signals dynamically influence existing neural representations are unknown. Here, we use functional magnetic resonance imaging (fMRI) in humans solving a two-step Markov decision task to investigate changes in neural activation patterns following PEs. Using a dynamic multivariate pattern analysis, we can show that PE-related fMRI responses in error-coding regions predict trial-by-trial changes in multivariate neural patterns in the orbitofrontal cortex, the precuneus, and the ventromedial prefrontal cortex (vmPFC). Importantly, the dynamics of these pattern changes in the vmPFC also predicted upcoming changes in choice strategies and thus highlight the importance of these pattern changes for behavior.
Collapse
Affiliation(s)
- Leon Möhring
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Martinistr. 52, 20246 Hamburg, Germany.
| | - Jan Gläscher
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Martinistr. 52, 20246 Hamburg, Germany.
| |
Collapse
|
39
|
Lee RSC, Albertella L, Christensen E, Suo C, Segrave RA, Brydevall M, Kirkham R, Liu C, Fontenelle LF, Chamberlain SR, Rotaru K, Yücel M. A Novel, Expert-Endorsed, Neurocognitive Digital Assessment Tool for Addictive Disorders: Development and Validation Study. J Med Internet Res 2023; 25:e44414. [PMID: 37624635 PMCID: PMC7615064 DOI: 10.2196/44414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Revised: 02/16/2023] [Accepted: 07/22/2023] [Indexed: 08/26/2023] Open
Abstract
BACKGROUND Many people with harmful addictive behaviors may not meet formal diagnostic thresholds for a disorder. A dimensional approach, by contrast, including clinical and community samples, is potentially key to early detection, prevention, and intervention. Importantly, while neurocognitive dysfunction underpins addictive behaviors, established assessment tools for neurocognitive assessment are lengthy and unengaging, difficult to administer at scale, and not suited to clinical or community needs. The BrainPark Assessment of Cognition (BrainPAC) Project sought to develop and validate an engaging and user-friendly digital assessment tool purpose-built to comprehensively assess the main consensus-driven constructs underpinning addictive behaviors. OBJECTIVE The purpose of this study was to psychometrically validate a gamified battery of consensus-based neurocognitive tasks against standard laboratory paradigms, ascertain test-retest reliability, and determine their sensitivity to addictive behaviors (eg, alcohol use) and other risk factors (eg, trait impulsivity). METHODS Gold standard laboratory paradigms were selected to measure key neurocognitive constructs (Balloon Analogue Risk Task [BART], Stop Signal Task [SST], Delay Discounting Task [DDT], Value-Modulated Attentional Capture [VMAC] Task, and Sequential Decision-Making Task [SDT]), as endorsed by an international panel of addiction experts; namely, response selection and inhibition, reward valuation, action selection, reward learning, expectancy and reward prediction error, habit, and compulsivity. Working with game developers, BrainPAC tasks were developed and validated in 3 successive cohorts (total N=600) and a separate test-retest cohort (N=50) via Mechanical Turk using a cross-sectional design. RESULTS BrainPAC tasks were significantly correlated with the original laboratory paradigms on most metrics (r=0.18-0.63, P<.05). With the exception of the DDT k function and VMAC total points, all other task metrics across the 5 tasks did not differ between the gamified and nongamified versions (P>.05). Out of 5 tasks, 4 demonstrated adequate to excellent test-retest reliability (intraclass correlation coefficient 0.72-0.91, P<.001; except SDT). Gamified metrics were significantly associated with addictive behaviors on behavioral inventories, though largely independent of trait-based scales known to predict addiction risk. CONCLUSIONS A purpose-built battery of digitally gamified tasks is sufficiently valid for the scalable assessment of key neurocognitive processes underpinning addictive behaviors. This validation provides evidence that a novel approach, purported to enhance task engagement, in the assessment of addiction-related neurocognition is feasible and empirically defensible. These findings have significant implications for risk detection and the successful deployment of next-generation assessment tools for substance use or misuse and other mental disorders characterized by neurocognitive anomalies related to motivation and self-regulation. Future development and validation of the BrainPAC tool should consider further enhancing convergence with established measures as well as collecting population-representative data to use clinically as normative comparisons.
Collapse
Affiliation(s)
- Rico S. C. Lee
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Australia
| | - Lucy Albertella
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Erynn Christensen
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Chao Suo
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Rebecca A. Segrave
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Maja Brydevall
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Rebecca Kirkham
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Chang Liu
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| | - Leonardo F. Fontenelle
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
- Obsessive, Compulsive, and Anxiety Spectrum Research Program, Institute of Psychiatry, Federal University of Rio de Janeiro (UFRJ)
- D’Or Institute for Research and Education, Rio de Janeiro, Brazil
| | - Samuel R. Chamberlain
- Department of Psychiatry, University of Southampton, Southampton, United Kingdom; and Southern Health NHS Foundation Trust, Southampton, UK
| | - Kristian Rotaru
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
- Monash Business School, Monash University, Melbourne, Australia
| | - Murat Yücel
- BrainPark, Turner Institute for Brain and Mental Health, Monash University, Melbourne, Australia
| |
Collapse
|
40
|
Crombie KM, Azar A, Botsford C, Heilicher M, Hiser J, Moughrabi N, Gruichich TS, Schomaker CM, Cisler JM. The influence of aerobic exercise on model-based decision making in women with posttraumatic stress disorder. JOURNAL OF MOOD AND ANXIETY DISORDERS 2023; 2:100015. [PMID: 37593142 PMCID: PMC10433398 DOI: 10.1016/j.xjmad.2023.100015] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/19/2023]
Abstract
Individuals with PTSD often exhibit deficits in executive functioning. An unexplored aspect of neurocognitive functions associated with PTSD is the type of learning system engaged in during decision-making. A model-free (MF) system is habitual in nature and involves trial-and-error learning that is often updated based on the most recent experience (e.g., repeat action if rewarded). A model-based (MB) system is goal-directed in nature and involves the development of an abstract representation of the environment to facilitate decisions (e.g., choose sequence of actions according to current contextual state and predicted outcomes). The existing neurocognitive literature on PTSD suggests the hypothesis of greater reliance on MF vs MB learning strategies when navigating their environment. While MF systems may be more cognitively efficient, they do not afford flexibility when making prospective predictions about likely outcomes of different decision-tree branches. Emerging research suggests that an acute bout of aerobic exercise improves certain aspects of neurocognition, and thereby could promote the utilization of MB over MF systems during decision making, although prior research has not yet tested this hypothesis. Accordingly, the current study administered a lab-based two-stage Markov decision-making task capable of discriminating MF vs MB decision making, in order to determine if moderate-intensity aerobic exercise (either shortly after or 30-minutes after the exercise bout has ended) promotes greater engagement in MB behavioral strategies compared to light-intensity aerobic exercise in adult women with and without PTSD (N=61). Results revealed that control women generally displayed higher levels of MB behavior that was further increased following immediate exercise, particularly moderate-intensity exercise. By contrast, the PTSD group generally displayed lower levels of MB behavior, and exhibited greater MB behavior when completing the task following moderate-intensity aerobic exercise compared to light-intensity aerobic exercise regardless of whether there was a short or long delay between exercise and the task. Additionally, women with PTSD demonstrated less impairment in MB decision-making compared to controls following moderate-intensity aerobic exercise. These results suggest that an acute bout of moderate-intensity aerobic exercise boosts MB behavior in women with PTSD, and suggests that aerobic exercise may play an important role in enhancing cognitive outcomes for PTSD.
Collapse
Affiliation(s)
- Kevin M. Crombie
- The University of Texas at Austin, Department of Psychiatry and Behavioral Sciences, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
- The University of Alabama, Department of Kinesiology, 1003 Wade Hall, Tuscaloosa, Alabama, United States of America 35487
| | - Ameera Azar
- The University of Texas at Austin, Department of Psychiatry and Behavioral Sciences, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
| | - Chloe Botsford
- University of Wisconsin – Madison, Department of Psychiatry, 6001 Research Park Boulevard, Madison, Wisconsin, United States of America 53719
| | - Mickela Heilicher
- University of Wisconsin – Madison, Department of Psychiatry, 6001 Research Park Boulevard, Madison, Wisconsin, United States of America 53719
| | - Jaryd Hiser
- University of Wisconsin – Madison, Department of Psychiatry, 6001 Research Park Boulevard, Madison, Wisconsin, United States of America 53719
- The Ohio State University, Department of Psychiatry and Behavioral Health, 1670 Upham Drive, Suite 130, Columbus, Ohio, United States of America 43210
| | - Nicole Moughrabi
- The University of Texas at Austin, Department of Psychiatry and Behavioral Sciences, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
| | - Tijana Sagorac Gruichich
- University of Wisconsin – Madison, Department of Psychiatry, 6001 Research Park Boulevard, Madison, Wisconsin, United States of America 53719
| | - Chloe M. Schomaker
- The University of Texas at Austin, Department of Psychiatry and Behavioral Sciences, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
| | - Josh M. Cisler
- The University of Texas at Austin, Department of Psychiatry and Behavioral Sciences, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
- Institute for Early Life Adversity Research, The University of Texas at Austin Dell Medical School, 1601 Trinity Street, Building B, Austin, Texas, United States of America 78712
| |
Collapse
|
41
|
Burger KS. Food reinforcement architecture: A framework for impulsive and compulsive overeating and food abuse. Obesity (Silver Spring) 2023; 31:1734-1744. [PMID: 37368515 DOI: 10.1002/oby.23792] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/30/2022] [Revised: 03/01/2023] [Accepted: 03/06/2023] [Indexed: 06/29/2023]
Abstract
Few reward-based theories address key drivers of susceptibility to food cues and consumption beyond fullness. Decision-making and habit formation are governed by reinforcement-based learning processes that, when overstimulated, can drive unregulated hedonically motivated overeating. Here, a model food reinforcement architecture is proposed that uses fundamental concepts in reinforcement and decision-making to identify maladaptive eating habits that can lead to obesity. This model is unique in that it identifies metabolic drivers of reward and incorporates neuroscience, computational decision-making, and psychology to map overeating and obesity. Food reinforcement architecture identifies two paths to overeating: a propensity for hedonic targeting of food cues contributing to impulsive overeating and lack of satiation that contributes to compulsive overeating. A combination of those paths will result in a conscious and subconscious drive to overeat independent of negative consequences, leading to food abuse and/or obesity. Use of this model to identify aberrant reinforcement learning processes and decision-making systems that can serve as markers of overeating risk may provide an opportunity for early intervention in obesity.
Collapse
Affiliation(s)
- Kyle S Burger
- Department of Nutrition, University of North Carolina, Chapel Hill, North Carolina, USA
| |
Collapse
|
42
|
Molinaro G, Collins AGE. Intrinsic rewards explain context-sensitive valuation in reinforcement learning. PLoS Biol 2023; 21:e3002201. [PMID: 37459394 PMCID: PMC10374061 DOI: 10.1371/journal.pbio.3002201] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2022] [Revised: 07/27/2023] [Accepted: 06/15/2023] [Indexed: 07/28/2023] Open
Abstract
When observing the outcome of a choice, people are sensitive to the choice's context, such that the experienced value of an option depends on the alternatives: getting $1 when the possibilities were 0 or 1 feels much better than when the possibilities were 1 or 10. Context-sensitive valuation has been documented within reinforcement learning (RL) tasks, in which values are learned from experience through trial and error. Range adaptation, wherein options are rescaled according to the range of values yielded by available options, has been proposed to account for this phenomenon. However, we propose that other mechanisms-reflecting a different theoretical viewpoint-may also explain this phenomenon. Specifically, we theorize that internally defined goals play a crucial role in shaping the subjective value attributed to any given option. Motivated by this theory, we develop a new "intrinsically enhanced" RL model, which combines extrinsically provided rewards with internally generated signals of goal achievement as a teaching signal. Across 7 different studies (including previously published data sets as well as a novel, preregistered experiment with replication and control studies), we show that the intrinsically enhanced model can explain context-sensitive valuation as well as, or better than, range adaptation. Our findings indicate a more prominent role of intrinsic, goal-dependent rewards than previously recognized within formal models of human RL. By integrating internally generated signals of reward, standard RL theories should better account for human behavior, including context-sensitive valuation and beyond.
Collapse
Affiliation(s)
- Gaia Molinaro
- Department of Psychology, University of California, Berkeley, Berkeley, California, United States of America
| | - Anne G E Collins
- Department of Psychology, University of California, Berkeley, Berkeley, California, United States of America
- Helen Wills Neuroscience Institute, University of California, Berkeley, Berkeley, California, United States of America
| |
Collapse
|
43
|
Smid CR, Ganesan K, Thompson A, Cañigueral R, Veselic S, Royer J, Kool W, Hauser TU, Bernhardt B, Steinbeis N. Neurocognitive basis of model-based decision making and its metacontrol in childhood. Dev Cogn Neurosci 2023; 62:101269. [PMID: 37352654 PMCID: PMC10329104 DOI: 10.1016/j.dcn.2023.101269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Revised: 04/16/2023] [Accepted: 06/14/2023] [Indexed: 06/25/2023] Open
Abstract
Human behavior is supported by both goal-directed (model-based) and habitual (model-free) decision-making, each differing in its flexibility, accuracy, and computational cost. The arbitration between habitual and goal-directed systems is thought to be regulated by a process known as metacontrol. However, how these systems emerge and develop remains poorly understood. Recently, we found that while children between 5 and 11 years displayed robust signatures of model-based decision-making, which increased during this developmental period, there were substantial individual differences in the display of metacontrol. Here, we inspect the neurocognitive basis of model-based decision-making and metacontrol in childhood and focus this investigation on executive functions, fluid reasoning, and brain structure. A total of 69 participants between the ages of 6-13 completed a two-step decision-making task and an extensive behavioral test battery. A subset of 44 participants also completed a structural magnetic resonance imaging scan. We find that individual differences in metacontrol are specifically associated with performance on an inhibition task and individual differences in thickness of dorsolateral prefrontal, temporal, and superior-parietal cortices. These brain regions likely reflect the involvement of cognitive processes crucial to metacontrol, such as cognitive control and contextual processing.
Collapse
Affiliation(s)
- C R Smid
- Department of Psychology and Language Sciences, University College London, United Kingdom.
| | - K Ganesan
- Department of Psychology and Language Sciences, University College London, United Kingdom
| | - A Thompson
- Department of Psychology and Language Sciences, University College London, United Kingdom
| | - R Cañigueral
- Department of Psychology and Language Sciences, University College London, United Kingdom
| | - S Veselic
- Clinical and Movement Neurosciences, Department of Motor Neuroscience, University College London, United Kingdom; Wellcome Centre for Human Neuroimaging, University College London, United Kingdom
| | - J Royer
- McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, Canada
| | - W Kool
- Department of Psychological & Brain Sciences, Washington University in St. Louis, St. Louis, MO, United States
| | - T U Hauser
- Wellcome Centre for Human Neuroimaging, University College London, United Kingdom; Max Planck University College London Centre for Computational Psychiatry and Ageing Research, United Kingdom
| | - B Bernhardt
- McConnell Brain Imaging Centre, Montreal Neurological Institute, McGill University, Montreal, Canada
| | - N Steinbeis
- Department of Psychology and Language Sciences, University College London, United Kingdom
| |
Collapse
|
44
|
Breitinger E, Dundon NM, Pokorny L, Wunram HL, Roessner V, Bender S. Contingent negative variation to tactile stimuli - differences in anticipatory and preparatory processes between participants with and without blindness. Cereb Cortex 2023; 33:7582-7594. [PMID: 36977633 DOI: 10.1093/cercor/bhad062] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 02/09/2023] [Accepted: 02/10/2023] [Indexed: 03/30/2023] Open
Abstract
People who are blind demonstrate remarkable abilities within the spared senses and compensatory enhancement of cognitive skills, underscored by substantial plastic reorganization in relevant neural areas. However, little is known about whether people with blindness form top-down models of the world on short timescales more efficiently to guide goal-oriented behavior. This electroencephalography study investigates this hypothesis at the neurophysiological level, focusing on contingent negative variation (CNV) as a marker of anticipatory and preparatory processes prior to expected events. In sum, 20 participants with blindness and 27 sighted participants completed a classic CNV task and a memory CNV task, both containing tactile stimuli to exploit the expertise of the former group. Although the reaction times in the classic CNV task did not differ between groups, participants who are blind reached higher performance rates in the memory task. This superior performance co-occurred with a distinct neurophysiological profile, relative to controls: greater late CNV amplitudes over central areas, suggesting enhanced stimulus expectancy and motor preparation prior to key events. Controls, in contrast, recruited more frontal sites, consistent with inefficient sensory-aligned control. We conclude that in more demanding cognitive contexts exploiting the spared senses, people with blindness efficiently generate task-relevant internal models to facilitate behavior.
Collapse
Affiliation(s)
- Eva Breitinger
- Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy, University of Cologne, Faculty of Medicine and University Hospital Cologne, Germany
| | - Neil M Dundon
- Department of Child and Adolescent Psychiatry, Psychotherapy, and Psychosomatics, University of Freiburg, Germany
- Department of Psychological and Brain Sciences, University of California, Santa Barbara, CA
| | - Lena Pokorny
- Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy, University of Cologne, Faculty of Medicine and University Hospital Cologne, Germany
| | - Heidrun L Wunram
- Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy, University of Cologne, Faculty of Medicine and University Hospital Cologne, Germany
- Department of Pediatrics, University of Cologne, Faculty of Medicine and University Hospital Cologne, Germany
| | - Veit Roessner
- Department of Child and Adolescent Psychiatry and Psychotherapy, Technische Universität Dresden, Faculty of Medicine, University Hospital C. G. Carus, Germany
| | - Stephan Bender
- Department of Child and Adolescent Psychiatry, Psychosomatics, and Psychotherapy, University of Cologne, Faculty of Medicine and University Hospital Cologne, Germany
| |
Collapse
|
45
|
Rosenblau G, Frolichs K, Korn CW. A neuro-computational social learning framework to facilitate transdiagnostic classification and treatment across psychiatric disorders. Neurosci Biobehav Rev 2023; 149:105181. [PMID: 37062494 PMCID: PMC10236440 DOI: 10.1016/j.neubiorev.2023.105181] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2022] [Revised: 03/14/2023] [Accepted: 04/13/2023] [Indexed: 04/18/2023]
Abstract
Social deficits are among the core and most striking psychiatric symptoms, present in most psychiatric disorders. Here, we introduce a novel social learning framework, which consists of neuro-computational models that combine reinforcement learning with various types of social knowledge structures. We outline how this social learning framework can help specify and quantify social psychopathology across disorders and provide an overview of the brain regions that may be involved in this type of social learning. We highlight how this framework can specify commonalities and differences in the social psychopathology of individuals with autism spectrum disorder (ASD), personality disorders (PD), and major depressive disorder (MDD) and improve treatments on an individual basis. We conjecture that individuals with psychiatric disorders rely on rigid social knowledge representations when learning about others, albeit the nature of their rigidity and the behavioral consequences can greatly differ. While non-clinical cohorts tend to efficiently adapt social knowledge representations to relevant environmental constraints, psychiatric cohorts may rigidly stick to their preconceived notions or overly coarse knowledge representations during learning.
Collapse
Affiliation(s)
- Gabriela Rosenblau
- Department of Psychological and Brain Sciences, George Washington University, Washington DC, USA; Autism and Neurodevelopmental Disorders Institute, George Washington University, Washington DC, USA.
| | - Koen Frolichs
- Section Social Neuroscience, Department of General Psychiatry, University of Heidelberg, Heidelberg, Germany; Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Christoph W Korn
- Section Social Neuroscience, Department of General Psychiatry, University of Heidelberg, Heidelberg, Germany; Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
| |
Collapse
|
46
|
Ruan Z, Seger CA, Yang Q, Kim D, Lee SW, Chen Q, Peng Z. Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder. Front Psychiatry 2023; 14:1162800. [PMID: 37304449 PMCID: PMC10250695 DOI: 10.3389/fpsyt.2023.1162800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Accepted: 05/05/2023] [Indexed: 06/13/2023] Open
Abstract
Introduction Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time. Methods A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis. Results Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal. Conclusion These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.
Collapse
Affiliation(s)
- Zhongqiang Ruan
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
| | - Carol A. Seger
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Department of Psychology, Colorado State University, Fort Collins, CO, United States
| | - Qiong Yang
- Affective Disorder Center, Affiliated Brain Hospital of Guangzhou Medical University (Guangzhou Huiai Hospital), Guangzhou, China
| | - Dongjae Kim
- Department of AI-based Convergence, College of Engineering, Dankook University, Yongin, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Qi Chen
- School of Psychology, Shenzhen University, Shenzhen, China
| | - Ziwen Peng
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou, China
- Department of Child Psychiatry, Shenzhen Kangning Hospital, Shenzhen University School of Medicine, Shenzhen, China
| |
Collapse
|
47
|
van Timmeren T, Piray P, Goudriaan AE, van Holst RJ. Goal-directed and habitual decision making under stress in gambling disorder: An fMRI study. Addict Behav 2023; 140:107628. [PMID: 36716563 DOI: 10.1016/j.addbeh.2023.107628] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2022] [Revised: 12/02/2022] [Accepted: 01/20/2023] [Indexed: 01/26/2023]
Abstract
The development of addictive behaviors has been suggested to be related to a transition from goal-directed to habitual decision making. Stress is a factor known to prompt habitual behavior and to increase the risk for addiction and relapse. In the current study, we therefore used functional MRI to investigate the balance between goal-directed 'model-based' and habitual 'model-free' control systems and whether acute stress would differentially shift this balance in gambling disorder (GD) patients compared to healthy controls (HCs). Using a within-subject design, 22 patients with GD and 20 HCs underwent stress induction or a control condition before performing a multistep decision-making task during fMRI. Salivary cortisol levels showed that the stress induction was successful. Contrary to our hypothesis, GD patients did not show impaired goal-directed 'model-based' decision making, which remained similar to HCs after stress induction. Bayes factors provided three times more evidence against a difference between the groups or a group-by-stress interaction on the balance between model-based and model-free decision making. Similarly, no differences were found between groups and conditions on the neural estimates of model-based or model-free decision making. These results challenge the notion that GD is related to an increased reliance on habitual (or decreased goal-directed) control, even during stress.
Collapse
Affiliation(s)
- Tim van Timmeren
- Amsterdam Institute for Addiction Research, Amsterdam UMC, Department of Psychiatry, University of Amsterdam, The Netherlands; Habit Lab, Department of Clinical Psychology, University of Amsterdam, The Netherlands; Department of Social Health and Organizational Psychology, Utrecht University, The Netherlands.
| | - Payam Piray
- Department of Psychology, University of Southern California, USA
| | - Anna E Goudriaan
- Amsterdam Institute for Addiction Research, Amsterdam UMC, Department of Psychiatry, University of Amsterdam, The Netherlands; Arkin Mental Health, The Netherlands
| | - Ruth J van Holst
- Amsterdam Institute for Addiction Research, Amsterdam UMC, Department of Psychiatry, University of Amsterdam, The Netherlands
| |
Collapse
|
48
|
Takahashi YK, Stalnaker TA, Mueller LE, Harootonian SK, Langdon AJ, Schoenbaum G. Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model. Nat Neurosci 2023; 26:830-839. [PMID: 37081296 PMCID: PMC10646487 DOI: 10.1038/s41593-023-01310-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2021] [Accepted: 03/16/2023] [Indexed: 04/22/2023]
Abstract
Dopamine neuron activity is tied to the prediction error in temporal difference reinforcement learning models. These models make significant simplifying assumptions, particularly with regard to the structure of the predictions fed into the dopamine neurons, which consist of a single chain of timepoint states. Although this predictive structure can explain error signals observed in many studies, it cannot cope with settings where subjects might infer multiple independent events and outcomes. In the present study, we recorded dopamine neurons in the ventral tegmental area in such a setting to test the validity of the single-stream assumption. Rats were trained in an odor-based choice task, in which the timing and identity of one of several rewards delivered in each trial changed across trial blocks. This design revealed an error signaling pattern that requires the dopamine neurons to access and update multiple independent predictive streams reflecting the subject's belief about timing and potentially unique identities of expected rewards.
Collapse
Affiliation(s)
- Yuji K Takahashi
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| | - Thomas A Stalnaker
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | - Lauren E Mueller
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA
| | | | - Angela J Langdon
- Intramural Research Program, National Institute of Mental Health, Bethesda, MD, USA.
| | - Geoffrey Schoenbaum
- Intramural Research Program, National Institute on Drug Abuse, Baltimore, MD, USA.
| |
Collapse
|
49
|
Characterizing habit learning in the human brain at the individual and group levels: a multi-modal MRI study. Neuroimage 2023. [DOI: 10.1016/j.neuroimage.2023.120002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/04/2023] Open
|
50
|
Na S, Rhoads SA, Yu ANC, Fiore VG, Gu X. Towards a neurocomputational account of social controllability: From models to mental health. Neurosci Biobehav Rev 2023; 148:105139. [PMID: 36940889 PMCID: PMC10106443 DOI: 10.1016/j.neubiorev.2023.105139] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Revised: 03/14/2023] [Accepted: 03/15/2023] [Indexed: 03/22/2023]
Abstract
Controllability, or the influence one has over their surroundings, is crucial for decision-making and mental health. Traditionally, controllability is operationalized in sensorimotor terms as one's ability to exercise their actions to achieve an intended outcome (also termed "agency"). However, recent social neuroscience research suggests that humans also assess if and how they can exert influence over other people (i.e., their actions, outcomes, beliefs) to achieve desired outcomes ("social controllability"). In this review, we will synthesize empirical findings and neurocomputational frameworks related to social controllability. We first introduce the concepts of contextual and perceived controllability and their respective relevance for decision-making. Then, we outline neurocomputational frameworks that can be used to model social controllability, with a focus on behavioral economic paradigms and reinforcement learning approaches. Finally, we discuss the implications of social controllability for computational psychiatry research, using delusion and obsession-compulsion as examples. Taken together, we propose that social controllability could be a key area of investigation in future social neuroscience and computational psychiatry research.
Collapse
Affiliation(s)
- Soojung Na
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States
| | - Shawn A Rhoads
- Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States
| | - Alessandra N C Yu
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States
| | - Vincenzo G Fiore
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States
| | - Xiaosi Gu
- Nash Family Department of Neuroscience, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States; Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States; Center for Computational Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY 10029, United States.
| |
Collapse
|