1
|
Tichelaar JG, Hezemans F, Bloem BR, Helmich RC, Cools R. Neural reinforcement learning signals predict recovery from impulse control disorder symptoms in Parkinson's disease. Biol Psychiatry 2024:S0006-3223(24)01434-3. [PMID: 39002875 DOI: 10.1016/j.biopsych.2024.06.027] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/08/2023] [Revised: 05/26/2024] [Accepted: 06/20/2024] [Indexed: 07/15/2024]
Abstract
BACKGROUND Impulse control disorders (ICD) in Parkinson's disease (PD) are associated with a heavy burden on patients and caretakers. While recovery can occur, ICD persists in many patients despite optimal management. The basis for this inter-individual variability in recovery is unclear and poses a major challenge to personalized health care. METHODS We adopt a computational psychiatry approach and leverage the longitudinal, prospective Personalized Parkinson Project (N=136 persons with PD, within 5 years of diagnosis) to combine dopaminergic learning theory-informed fMRI with machine learning (at baseline) to predict ICD symptom recovery after two years of follow-up. We focused on a change in QUIP-rs across the entire cohort, regardless of an ICD diagnosis. RESULTS Greater reinforcement learning signals during gain trials but not loss trials at baseline, including those in the ventral striatum, medial prefrontal cortex and the behavioral accuracy score measured while ON medication were associated with greater recovery from impulse control symptoms two years later. These signals accounted for a unique proportion of the relevant variability over and above that explained by other known factors, such as decreases in dopamine agonist use. CONCLUSIONS Our results provide a proof of principle for combining generative model-based inference of latent learning processes with machine learning-based predictive modeling of variability in clinical symptom recovery trajectories. Hence, we showed that RL modelling parameters predict recovery from ICD symptoms in PD.
Collapse
Affiliation(s)
- Jorryt G Tichelaar
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN, Nijmegen, The Netherlands; Radboud University Medical Center, Department of Neurology, Centre of Expertise for Parkinson and Movement Disorders, 6525GA, Nijmegen, The Netherlands.
| | - Frank Hezemans
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN, Nijmegen, The Netherlands; Radboud University Medical Center, Department of Psychiatry, 6525GA, Nijmegen, The Netherlands
| | - Bastiaan R Bloem
- Radboud University Medical Center, Department of Neurology, Centre of Expertise for Parkinson and Movement Disorders, 6525GA, Nijmegen, The Netherlands
| | - Rick C Helmich
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN, Nijmegen, The Netherlands; Radboud University Medical Center, Department of Neurology, Centre of Expertise for Parkinson and Movement Disorders, 6525GA, Nijmegen, The Netherlands
| | - Roshan Cools
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN, Nijmegen, The Netherlands; Radboud University Medical Center, Department of Psychiatry, 6525GA, Nijmegen, The Netherlands
| |
Collapse
|
2
|
Sacu S, Dubois M, Hezemans FH, Aggensteiner PM, Monninger M, Brandeis D, Banaschewski T, Hauser TU, Holz NE. Early-Life Adversities Are Associated With Lower Expected Value Signaling in the Adult Brain. Biol Psychiatry 2024:S0006-3223(24)01249-6. [PMID: 38636886 DOI: 10.1016/j.biopsych.2024.04.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Revised: 04/05/2024] [Accepted: 04/06/2024] [Indexed: 04/20/2024]
Abstract
BACKGROUND Early adverse experiences are assumed to affect fundamental processes of reward learning and decision making. However, computational neuroimaging studies investigating these circuits in the context of adversity are sparse and limited to studies conducted in adolescent samples, leaving the long-term effects unexplored. METHODS Using data from a longitudinal birth cohort study (n = 156; 87 female), we investigated associations between adversities and computational markers of reward learning (i.e., expected value, prediction errors). At age 33 years, all participants completed a functional magnetic resonance imaging-based passive avoidance task. Psychopathology measures were collected at the time of functional magnetic resonance imaging investigation and during the COVID-19 pandemic. We applied a principal component analysis to capture common variations across 7 adversity measures. The resulting adversity factors (factor 1: postnatal psychosocial adversities and prenatal maternal smoking; factor 2: prenatal maternal stress and obstetric adversity; factor 3: lower maternal stimulation) were linked with psychopathology and neural responses in the core reward network using multiple regression analysis. RESULTS We found that the adversity dimension primarily informed by lower maternal stimulation was linked to lower expected value representation in the right putamen, right nucleus accumbens, and anterior cingulate cortex. Expected value encoding in the right nucleus accumbens further mediated the relationship between this adversity dimension and psychopathology and predicted higher withdrawn symptoms during the COVID-19 pandemic. CONCLUSIONS Our results suggested that early adverse experiences in caregiver context might have a long-term disruptive effect on reward learning in reward-related brain regions, which can be associated with suboptimal decision making and thereby may increase the vulnerability of developing psychopathology.
Collapse
Affiliation(s)
- Seda Sacu
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; German Center for Mental Health, Mannheim, Heidelberg, and Ulm, Germany
| | - Magda Dubois
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
| | - Frank H Hezemans
- Department of Psychiatry and Psychotherapy, Medical School and University Hospital, Eberhard Karls University of Tübingen, Tübingen, Germany; Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics, Tübingen, Germany; German Center for Mental Health, Tübingen, Germany
| | - Pascal-M Aggensteiner
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; German Center for Mental Health, Mannheim, Heidelberg, and Ulm, Germany
| | - Maximilian Monninger
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany
| | - Daniel Brandeis
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; Department of Child and Adolescent Psychiatry and Psychotherapy, Psychiatric Hospital, University of Zurich, Zurich, Switzerland; Neuroscience Center Zurich, University of Zurich and ETH Zürich, Zurich, Switzerland
| | - Tobias Banaschewski
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; German Center for Mental Health, Mannheim, Heidelberg, and Ulm, Germany
| | - Tobias U Hauser
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom; Department of Psychiatry and Psychotherapy, Medical School and University Hospital, Eberhard Karls University of Tübingen, Tübingen, Germany; German Center for Mental Health, Tübingen, Germany; Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom
| | - Nathalie E Holz
- Department of Child and Adolescent Psychiatry and Psychotherapy, Central Institute of Mental Health, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; German Center for Mental Health, Mannheim, Heidelberg, and Ulm, Germany; Donders Institute for Brain, Cognition, and Behaviour, Radboud University Nijmegen, Nijmegen, the Netherlands; Department for Cognitive Neuroscience, Radboud University Medical Center, Nijmegen, the Netherlands.
| |
Collapse
|
3
|
Qiu H, Cao J, Wang R, Li X, Kuang L, Ouyang Z. Functional Abnormality of the Reward System in Depressed Adolescents and Young Adults with and without Suicidal Behavior. Brain Topogr 2024:10.1007/s10548-024-01036-4. [PMID: 38319504 DOI: 10.1007/s10548-024-01036-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2023] [Accepted: 01/16/2024] [Indexed: 02/07/2024]
Abstract
OBJECTIVE To identify local and functional connectivity abnormalities in the brain's reward network in depressed adolescents and young adults with and without suicidal behavior. METHODS Magnetic resonance imaging data were obtained from 41 major depressive disorder (MDD) patients with suicidal behavior (sMDD, males/females: 12/29), 44 MDD patients without suicidal behavior (nMDD, males/females: 13/32), and 52 healthy controls (HCs, males/females: 17/35). The Young Mania Scale, Hamilton Depression Scale, Columbia Suicide Scale, and Scale for Suicide Ideation were used to evaluate emotional state and suicidal ideation and behaviors. The amplitude of low frequency fluctuations (ALFF), regional homogeneity (ReHo) and functional connectivity of 11 regions of interest (ROIs) in the reward network were determined. RESULTS ALFF values in the vmPFC of the nMDD group were significantly lower than those in the HC group (p = 0.031). The ReHo values of the nMDD group were lower in the lVS but higher in the vmPFC than those of the HC group (P = 0.018 and 0.025, respectively). Functional connectivity of the AC with the vmPFC, lVS, rVS, and vmPFC was increased in the sMDD group compared with that in the nMDD group (P = 0.038, 0.034, 0.006, respectively). CONCLUSION Local and functional connectivity abnormalities in the reward network were found in the MDD groups. However, increased functional connectivity was found in only the sMDD group.
Collapse
Affiliation(s)
- Haitang Qiu
- Department of Mental Health, The First Affiliated Hospital of Chongqing Medical University, Chongqing, 400016, China
| | - Jun Cao
- Department of Mental Health, The First Affiliated Hospital of Chongqing Medical University, Chongqing, 400016, China
| | - Rui Wang
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China
| | - Xinke Li
- College of Medical Informatics, Chongqing Medical University, Chongqing, China
| | - Li Kuang
- Department of Mental Health, The First Affiliated Hospital of Chongqing Medical University, Chongqing, 400016, China.
| | - Zhubin Ouyang
- Department of Mental Health, The First Affiliated Hospital of Chongqing Medical University, Chongqing, 400016, China.
- Department of Radiology, The First Affiliated Hospital of Chongqing Medical University, Chongqing, China.
| |
Collapse
|
4
|
Kumar P, Dayan P, Wolfers T. From Complexity to Precision-Charting Decision-Making Through Normative Modeling. JAMA Psychiatry 2024; 81:117-118. [PMID: 38150222 DOI: 10.1001/jamapsychiatry.2023.4611] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/28/2023]
Abstract
This Viewpoint synthesizes data-driven and theory-driven approaches to normative modeling.
Collapse
Affiliation(s)
- Poornima Kumar
- Computational Psychopathology Group, Center for Depression, Anxiety, and Stress Research, McLean Hospital, Belmont, Massachusetts
- Department of Psychiatry, Harvard Medical School, Boston, Massachusetts
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
- Department of Psychiatry and Psychotherapy, University of Tübingen, Germany
| | - Thomas Wolfers
- Department of Psychiatry and Psychotherapy, University of Tübingen, Germany
- German Centre for Mental Health, Partner Site Tübingen, Germany
| |
Collapse
|
5
|
Macoveanu J, Kjærstad HL, Halvorsen KS, Fisher PM, Vinberg M, Kessing LV, Miskowiak KW. Trajectory of reward-related abnormalities in unaffected relatives of patients with bipolar disorder - A longitudinal fMRI study. J Psychiatr Res 2024; 170:217-224. [PMID: 38157669 DOI: 10.1016/j.jpsychires.2023.12.035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Revised: 11/14/2023] [Accepted: 12/20/2023] [Indexed: 01/03/2024]
Abstract
First-degree relatives of patients with bipolar disorder are at heightened risk of mood episodes, which may be attributed to the existence of endophenotypes i.e., heritable (neuro)biological changes present in patients and their unaffected relatives (UR). In this longitudinal MRI study, we aim to investigate the trajectories of aberrant reward-related functional changes identified in UR vs healthy controls (HC). Sixty-eight UR and 65 HC of similar age and gender distribution underwent MRI at baseline while performing a card guessing task. Of these, 29 UR and 36 HC were investigated with the same protocol following a 16-month period in average. We first identified brain regions showing group differences in the neural response to expected value (EV) and reward prediction error (PE) at baseline and analyzed how the reward-related response in these regions changed over time in UR vs HC. Relative to HC at baseline, UR showed lower EV signal in the right ventrolateral prefrontal cortex (vlPFC) and paracingulate gyrus and lower PE signal in the left vlPFC and dorsomedial PFC. The trajectories of these abnormalities in UR showed a normalization of the prefrontal EV signals, whereas the PE signals which correlated with depressive symptoms remained stable over time. While the UR showed both blunted EV and PE signals, none of these abnormalities increased over time, which is consistent with the observed stable mood symptoms.
Collapse
Affiliation(s)
- Julian Macoveanu
- Neurocognition and Emotion in Affective Disorders (NEAD) Centre, Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Copenhagen Affective Disorder Research Centre (CADIC), Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark.
| | - Hanne Lie Kjærstad
- Neurocognition and Emotion in Affective Disorders (NEAD) Centre, Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Copenhagen Affective Disorder Research Centre (CADIC), Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark
| | - Kaja Sofie Halvorsen
- Neurocognition and Emotion in Affective Disorders (NEAD) Centre, Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark
| | - Patrick M Fisher
- Neurobiology Research Unit, Department of Drug Design and Pharmacology, University of Copenhagen, Denmark
| | - Maj Vinberg
- Copenhagen Affective Disorder Research Centre (CADIC), Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Psychiatric Research Unit, Psychiatric Centre North Zealand, Hillerød, Denmark; Department of Clinical Medicine, University of Copenhagen, Denmark
| | - Lars Vedel Kessing
- Copenhagen Affective Disorder Research Centre (CADIC), Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Department of Clinical Medicine, University of Copenhagen, Denmark
| | - Kamilla Woznica Miskowiak
- Neurocognition and Emotion in Affective Disorders (NEAD) Centre, Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Copenhagen Affective Disorder Research Centre (CADIC), Psychiatric Centre Copenhagen, Frederiksberg Hospital, Mental Health Services, Capital Region of Denmark, Denmark; Department of Psychology, University of Copenhagen, Denmark
| |
Collapse
|
6
|
He Q, Liu JL, Eschapasse L, Zagora AK, Brown TI. The neural correlates of memory integration in value-based decision-making during human spatial navigation. Neuropsychologia 2024; 193:108758. [PMID: 38103679 DOI: 10.1016/j.neuropsychologia.2023.108758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 09/12/2023] [Accepted: 12/11/2023] [Indexed: 12/19/2023]
Abstract
In daily life, we often make decisions based on relative value of the options, and we often derive these values from segmenting or integrating the outcomes of past episodes in memory. The neural correlates involved in value-based decision-making have been extensively studied in the literature, but few studies have investigated this topic in decisions that require segmenting or integrating episodic memory from related sources, and even fewer studies examine it in the context of spatial navigation. Building on the computational models from our previous studies, the current study investigates the neural substrates involved in decisions that require people either segment or integrate wayfinding outcomes involving different goals, across virtual spatial navigation tasks with differing demands. We find that when decisions require computation of spatial distances for navigation options, but also evaluation of one's prior spatial navigation ability with the task, the estimated value of navigational choices (EV) modulates neural activity in the dorsomedial prefrontal (dmPFC) cortex and ventrolateral prefrontal (vlFPC) cortex. However, superior parietal cortex tracked EV when decision-making tasks only require spatial distance memory but not evaluation of spatial navigation ability. Our findings reveal divergent neural substrates of memory integration in value-based decision-making under different spatial processing demands.
Collapse
Affiliation(s)
- Qiliang He
- School of Psychology, Georgia Institute of Technology, USA.
| | - Jancy Ling Liu
- School of Economics, Georgia Institute of Technology, USA
| | - Lou Eschapasse
- School of Psychology, Georgia Institute of Technology, USA
| | - Anna K Zagora
- School of Biological Sciences, Georgia Institute of Technology, USA
| | | |
Collapse
|
7
|
Tranter MM, Faget L, Hnasko TS, Powell SB, Dillon DG, Barnes SA. Postnatal Phencyclidine-Induced Deficits in Decision Making Are Ameliorated by Optogenetic Inhibition of Ventromedial Orbitofrontal Cortical Glutamate Neurons. BIOLOGICAL PSYCHIATRY GLOBAL OPEN SCIENCE 2024; 4:264-274. [PMID: 38298783 PMCID: PMC10829674 DOI: 10.1016/j.bpsgos.2023.08.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2023] [Revised: 07/12/2023] [Accepted: 08/01/2023] [Indexed: 02/02/2024] Open
Abstract
Background The orbitofrontal cortex (OFC) is essential for decision making, and functional disruptions within the OFC are evident in schizophrenia. Postnatal phencyclidine (PCP) administration in rats is a neurodevelopmental manipulation that induces schizophrenia-relevant cognitive impairments. We aimed to determine whether manipulating OFC glutamate cell activity could ameliorate postnatal PCP-induced deficits in decision making. Methods Male and female Wistar rats (n = 110) were administered saline or PCP on postnatal days 7, 9, and 11. In adulthood, we expressed YFP (yellow fluorescent protein) (control), ChR2 (channelrhodopsin-2) (activation), or eNpHR 3.0 (enhanced halorhodopsin) (inhibition) in glutamate neurons within the ventromedial OFC (vmOFC). Rats were tested on the probabilistic reversal learning task once daily for 20 days while we manipulated the activity of vmOFC glutamate cells. Behavioral performance was analyzed using a Q-learning computational model of reinforcement learning. Results Compared with saline-treated rats expressing YFP, PCP-treated rats expressing YFP completed fewer reversals, made fewer win-stay responses, and had lower learning rates. We induced similar performance impairments in saline-treated rats by activating vmOFC glutamate cells (ChR2). Strikingly, PCP-induced performance deficits were ameliorated when the activity of vmOFC glutamate cells was inhibited (halorhodopsin). Conclusions Postnatal PCP-induced deficits in decision making are associated with hyperactivity of vmOFC glutamate cells. Thus, normalizing vmOFC activity may represent a potential therapeutic target for decision-making deficits in patients with schizophrenia.
Collapse
Affiliation(s)
- Michael M. Tranter
- Department of Psychiatry, University of California San Diego, La Jolla, California
- Research Service, VA San Diego Healthcare System, La Jolla, California
| | - Lauren Faget
- Department of Neurosciences, University of California San Diego, La Jolla, California
| | - Thomas S. Hnasko
- Research Service, VA San Diego Healthcare System, La Jolla, California
- Department of Neurosciences, University of California San Diego, La Jolla, California
| | - Susan B. Powell
- Department of Psychiatry, University of California San Diego, La Jolla, California
- Research Service, VA San Diego Healthcare System, La Jolla, California
| | - Daniel G. Dillon
- Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, Massachusetts
- Harvard Medical School, Boston, Massachusetts
| | - Samuel A. Barnes
- Department of Psychiatry, University of California San Diego, La Jolla, California
- Research Service, VA San Diego Healthcare System, La Jolla, California
| |
Collapse
|
8
|
Labutina N, Polyakov S, Nemtyreva L, Shuldishova A, Gizatullina O. Neural Correlates of Social Decision-Making. IRANIAN JOURNAL OF PSYCHIATRY 2024; 19:148-154. [PMID: 38420275 PMCID: PMC10896758 DOI: 10.18502/ijps.v19i1.14350] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/02/2023] [Revised: 08/13/2023] [Accepted: 09/02/2023] [Indexed: 03/02/2024]
Abstract
Objective: Recent studies have utilized innovative techniques to investigate the neural mechanisms underlying social and individual decision-making, aiming to understand how individuals respond to the world. Method : In this review, we summarized current scientific evidence concerning the neural underpinnings of social decision-making and their impact on social behavior. Results: Critical brain regions involved in social cognition and decision-making are integral to the process of social decision-making. Notably, the medial prefrontal cortex (mPFC) and temporoparietal junction (TPJ) contribute to the comprehension of others' mental states. Similarly, the posterior superior temporal sulcus (pSTS) shows heightened activity when individuals observe faces and movements. On the lateral surface of the brain, the inferior frontal gyrus (IFG) and inferior parietal sulcus (IPS) play a role in social cognition. Furthermore, the medial surface of the brain, including the amygdala, anterior cingulate cortex (ACC), and anterior insula (AI), also participates in social cognition processes. Regarding decision-making, functional magnetic resonance imaging (fMRI) studies have illuminated the involvement of a network of brain regions, encompassing the ventromedial prefrontal cortex (vmPFC), ventral striatum (VS), and nucleus accumbens (NAcc). Conclusion: Dysfunction in specific subregions of the prefrontal cortex (PFC) has been linked to various psychiatric conditions. These subregions play pivotal roles in cognitive, emotional, and social processing, and their impairment can contribute to the development and manifestation of psychiatric symptoms. A comprehensive understanding of the unique contributions of these PFC subregions to psychiatric disorders has the potential to inform the development of targeted interventions and treatments for affected individuals.
Collapse
Affiliation(s)
| | | | | | - Alina Shuldishova
- Peoples’ Friendship University of Russia (RUDN University), Moscow, Russia
| | - Olga Gizatullina
- Financial University under the Government of the Russian Federation, Moscow, Russia
| |
Collapse
|
9
|
Chase HW. A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI. Front Psychol 2023; 14:1211528. [PMID: 38187436 PMCID: PMC10768009 DOI: 10.3389/fpsyg.2023.1211528] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Accepted: 11/28/2023] [Indexed: 01/09/2024] Open
Abstract
Introduction Computational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses. Methods Simulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses. Results Initial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise. Conclusion Overall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.
Collapse
Affiliation(s)
- Henry W. Chase
- Department of Psychiatry, University of Pittsburgh School of Medicine, Pittsburgh, PA, United States
| |
Collapse
|
10
|
Park H, Doh H, Lee E, Park H, Ahn WY. The neurocognitive role of working memory load when Pavlovian motivational control affects instrumental learning. PLoS Comput Biol 2023; 19:e1011692. [PMID: 38064498 PMCID: PMC10732416 DOI: 10.1371/journal.pcbi.1011692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 12/20/2023] [Accepted: 11/15/2023] [Indexed: 12/21/2023] Open
Abstract
Research suggests that a fast, capacity-limited working memory (WM) system and a slow, incremental reinforcement learning (RL) system jointly contribute to instrumental learning. Thus, situations that strain WM resources alter instrumental learning: under WM loads, learning becomes slow and incremental, the reliance on computationally efficient learning increases, and action selection becomes more random. It is also suggested that Pavlovian learning influences people's behavior during instrumental learning by providing hard-wired instinctive responses including approach to reward predictors and avoidance of punishment predictors. However, it remains unknown how constraints on WM resources affect instrumental learning under Pavlovian influence. Thus, we conducted a functional magnetic resonance imaging (fMRI) study (N = 49) in which participants completed an instrumental learning task with Pavlovian-instrumental conflict (the orthogonalized go/no-go task) both with and without extra WM load. Behavioral and computational modeling analyses revealed that WM load reduced the learning rate and increased random choice, without affecting Pavlovian bias. Model-based fMRI analysis revealed that WM load strengthened RPE signaling in the striatum. Moreover, under WM load, the striatum showed weakened connectivity with the ventromedial and dorsolateral prefrontal cortex when computing reward expectations. These results suggest that the limitation of cognitive resources by WM load promotes slow and incremental learning through the weakened cooperation between WM and RL; such limitation also makes action selection more random, but it does not directly affect the balance between instrumental and Pavlovian systems.
Collapse
Affiliation(s)
- Heesun Park
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Hoyoung Doh
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Eunhwi Lee
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Harhim Park
- Department of Psychology, Seoul National University, Seoul, Korea
| | - Woo-Young Ahn
- Department of Psychology, Seoul National University, Seoul, Korea
- Department of Brain and Cognitive Sciences, Seoul National University, Seoul, Korea
| |
Collapse
|
11
|
Liu L, Liu D, Guo T, Schwieter JW, Liu H. The right superior temporal gyrus plays a role in semantic-rule learning: Evidence supporting a reinforcement learning model. Neuroimage 2023; 282:120393. [PMID: 37820861 DOI: 10.1016/j.neuroimage.2023.120393] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Revised: 08/29/2023] [Accepted: 09/25/2023] [Indexed: 10/13/2023] Open
Abstract
In real-life communication, individuals use language that carries evident rewarding and punishing elements, such as praise and criticism. A common trend is to seek more praise while avoiding criticism. Furthermore, semantics is crucial for conveying information, but such semantic access to native and foreign languages is subtly distinct. To investigate how rule learning occurs in different languages and to highlight the importance of semantics in this process, we investigated both verbal and non-verbal rule learning in first (L1) and second (L2) languages using a reinforcement learning framework, including a semantic rule and a color rule. Our computational modeling on behavioral and brain imaging data revealed that individuals may be more motivated to learn and adhere to rules in an L1 compared to L2, with greater striatum activation during the outcome phase in the L1. Additionally, results on the learning rates and inverse temperature in the two rule learning tasks showed that individuals tend to be conservative and are reluctant to change their judgments regarding rule learning of semantic information. Moreover, the greater the prediction errors, the greater activation of the right superior temporal gyrus in the semantic-rule learning condition, demonstrating that such learning has differential neural correlates than symbolic rule learning. Overall, the findings provide insight into the neural mechanisms underlying rule learning in different languages, and indicate that rule learning involving verbal semantics is not a general symbolic learning that resembles a conditioned stimulus-response, but rather has its own specific characteristics.
Collapse
Affiliation(s)
- Linyan Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - Dongxue Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - Tingting Guo
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China
| | - John W Schwieter
- Language Acquisition, Multilingualism, and Cognition Laboratory / Bilingualism Matters @ Wilfrid Laurier University, Canada; Department of Linguistics and Languages, McMaster University, Canada
| | - Huanhuan Liu
- Research Center of Brain and Cognitive Neuroscience, Liaoning Normal University, China; Key Laboratory of Brain and Cognitive Neuroscience, Liaoning Province, China.
| |
Collapse
|
12
|
Gold BP, Pearce MT, McIntosh AR, Chang C, Dagher A, Zatorre RJ. Auditory and reward structures reflect the pleasure of musical expectancies during naturalistic listening. Front Neurosci 2023; 17:1209398. [PMID: 37928727 PMCID: PMC10625409 DOI: 10.3389/fnins.2023.1209398] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 10/05/2023] [Indexed: 11/07/2023] Open
Abstract
Enjoying music consistently engages key structures of the neural auditory and reward systems such as the right superior temporal gyrus (R STG) and ventral striatum (VS). Expectations seem to play a central role in this effect, as preferences reliably vary according to listeners' uncertainty about the musical future and surprise about the musical past. Accordingly, VS activity reflects the pleasure of musical surprise, and exhibits stronger correlations with R STG activity as pleasure grows. Yet the reward value of musical surprise - and thus the reason for these surprises engaging the reward system - remains an open question. Recent models of predictive neural processing and learning suggest that forming, testing, and updating hypotheses about one's environment may be intrinsically rewarding, and that the constantly evolving structure of musical patterns could provide ample opportunity for this procedure. Consistent with these accounts, our group previously found that listeners tend to prefer melodic excerpts taken from real music when it either validates their uncertain melodic predictions (i.e., is high in uncertainty and low in surprise) or when it challenges their highly confident ones (i.e., is low in uncertainty and high in surprise). An independent research group (Cheung et al., 2019) replicated these results with musical chord sequences, and identified their fMRI correlates in the STG, amygdala, and hippocampus but not the VS, raising new questions about the neural mechanisms of musical pleasure that the present study seeks to address. Here, we assessed concurrent liking ratings and hemodynamic fMRI signals as 24 participants listened to 50 naturalistic, real-world musical excerpts that varied across wide spectra of computationally modeled uncertainty and surprise. As in previous studies, liking ratings exhibited an interaction between uncertainty and surprise, with the strongest preferences for high uncertainty/low surprise and low uncertainty/high surprise. FMRI results also replicated previous findings, with music liking effects in the R STG and VS. Furthermore, we identify interactions between uncertainty and surprise on the one hand, and liking and surprise on the other, in VS activity. Altogether, these results provide important support for the hypothesized role of the VS in deriving pleasure from learning about musical structure.
Collapse
Affiliation(s)
- Benjamin P. Gold
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, TN, United States
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
- International Laboratory for Brain, Music and Sound Research (BRAMS), Montreal, QC, Canada
- Centre for Research on Brain, Language and Music (CRBLM), Montreal, QC, Canada
- Centre for Interdisciplinary Research in Music, Media, and Technology (CIRMMT), Montreal, QC, Canada
| | - Marcus T. Pearce
- Cognitive Science Research Group, School of Electronic Engineering & Computer Science, Queen Mary University of London, London, United Kingdom
- Department of Clinical Medicine, Aarhus University, Aarhus, Denmark
| | - Anthony R. McIntosh
- Baycrest Centre, Rotman Research Institute, Toronto, ON, Canada
- Department of Psychology, University of Toronto, Toronto, ON, Canada
| | - Catie Chang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, United States
- Vanderbilt University Institute of Imaging Science, Vanderbilt University Medical Center, Nashville, TN, United States
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, United States
- Department of Computer Science, Vanderbilt University, Nashville, TN, United States
| | - Alain Dagher
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
| | - Robert J. Zatorre
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada
- International Laboratory for Brain, Music and Sound Research (BRAMS), Montreal, QC, Canada
- Centre for Research on Brain, Language and Music (CRBLM), Montreal, QC, Canada
- Centre for Interdisciplinary Research in Music, Media, and Technology (CIRMMT), Montreal, QC, Canada
| |
Collapse
|
13
|
Collomb-Clerc A, Gueguen MCM, Minotti L, Kahane P, Navarro V, Bartolomei F, Carron R, Regis J, Chabardès S, Palminteri S, Bastin J. Human thalamic low-frequency oscillations correlate with expected value and outcomes during reinforcement learning. Nat Commun 2023; 14:6534. [PMID: 37848435 PMCID: PMC10582006 DOI: 10.1038/s41467-023-42380-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 10/09/2023] [Indexed: 10/19/2023] Open
Abstract
Reinforcement-based adaptive decision-making is believed to recruit fronto-striatal circuits. A critical node of the fronto-striatal circuit is the thalamus. However, direct evidence of its involvement in human reinforcement learning is lacking. We address this gap by analyzing intra-thalamic electrophysiological recordings from eight participants while they performed a reinforcement learning task. We found that in both the anterior thalamus (ATN) and dorsomedial thalamus (DMTN), low frequency oscillations (LFO, 4-12 Hz) correlated positively with expected value estimated from computational modeling during reward-based learning (after outcome delivery) or punishment-based learning (during the choice process). Furthermore, LFO recorded from ATN/DMTN were also negatively correlated with outcomes so that both components of reward prediction errors were signaled in the human thalamus. The observed differences in the prediction signals between rewarding and punishing conditions shed light on the neural mechanisms underlying action inhibition in punishment avoidance learning. Our results provide insight into the role of thalamus in reinforcement-based decision-making in humans.
Collapse
Affiliation(s)
- Antoine Collomb-Clerc
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France
| | - Maëlle C M Gueguen
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, NJ, USA
| | - Lorella Minotti
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France
- Neurology Department, University Hospital of Grenoble, Grenoble, France
| | - Philippe Kahane
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France
- Neurology Department, University Hospital of Grenoble, Grenoble, France
| | - Vincent Navarro
- Sorbonne Université, Paris Brain Institute - Institut du Cerveau, ICM, INSERM, CNRS, AP-HP, Pitié-Salpêtrière Hospital, Paris, France
| | - Fabrice Bartolomei
- Timone University Hospital, Sleep Unit, Epileptology and Cerebral Rhythmology, University Hospital of Marseille, Marseille, France
- Aix Marseille University, Inserm, Institut de Neurosciences des Systèmes, Marseille, France
| | - Romain Carron
- Aix Marseille University, Inserm, Institut de Neurosciences des Systèmes, Marseille, France
- Timone University Hospital, Department of functional and stereotactic neurosurgery, University Hospital of Marseille, Marseille, France
| | - Jean Regis
- Neurosurgery Department, University Hospital of Marseille, Marseille, France
| | - Stephan Chabardès
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France
- Neurosurgery Department, University Hospital of Grenoble, Grenoble, France
| | - Stefano Palminteri
- Laboratoire de Neurosciences Cognitives Computationnelles, Département d'Etudes Cognitives, ENS, PSL, INSERM, Paris, France
| | - Julien Bastin
- Univ. Grenoble Alpes, Inserm, U1216, CHU Grenoble Alpes, Grenoble Institut Neurosciences, 38000, Grenoble, France.
| |
Collapse
|
14
|
Chakroun K, Wiehler A, Wagner B, Mathar D, Ganzer F, van Eimeren T, Sommer T, Peters J. Dopamine regulates decision thresholds in human reinforcement learning in males. Nat Commun 2023; 14:5369. [PMID: 37666865 PMCID: PMC10477234 DOI: 10.1038/s41467-023-41130-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Accepted: 08/22/2023] [Indexed: 09/06/2023] Open
Abstract
Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.
Collapse
Affiliation(s)
- Karima Chakroun
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Antonius Wiehler
- Motivation, Brain and Behavior Lab, Paris Brain Institute (ICM), Pitié-Salpêtrière Hospital, Paris, France
| | - Ben Wagner
- Chair of Cognitive Computational Neuroscience, Technical University Dresden, Dresden, Germany
| | - David Mathar
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany
| | - Florian Ganzer
- Integrated Psychiatry Winterthur, Winterthur, Switzerland
| | - Thilo van Eimeren
- Multimodal Neuroimaging Group, Department of Nuclear Medicine, University Medical Center Cologne, Cologne, Germany
| | - Tobias Sommer
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Jan Peters
- Institute for Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
- Department of Psychology, Biological Psychology, University of Cologne, Cologne, Germany.
| |
Collapse
|
15
|
Tichelaar JG, Sayalı C, Helmich RC, Cools R. Impulse control disorder in Parkinson's disease is associated with abnormal frontal value signalling. Brain 2023; 146:3676-3689. [PMID: 37192341 PMCID: PMC10473575 DOI: 10.1093/brain/awad162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Revised: 04/18/2023] [Accepted: 04/26/2023] [Indexed: 05/18/2023] Open
Abstract
Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson's disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson's disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression. One hundred and ninety-nine patients with Parkinson's disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered. These data substantiate the hypothesis that dopamine's effects on reinforcement learning in Parkinson's disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum. See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.
Collapse
Affiliation(s)
- Jorryt G Tichelaar
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN Nijmegen, The Netherlands
- Radboud University Medical Center, Department of Neurology, Centre of Expertise for Parkinson and Movement Disorders, 6525GA Nijmegen, The Netherlands
| | - Ceyda Sayalı
- The Johns Hopkins University School of Medicine, Center for Psychedelic and Consciousness Research, Baltimore, MD 21224, USA
| | - Rick C Helmich
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN Nijmegen, The Netherlands
- Radboud University Medical Center, Department of Neurology, Centre of Expertise for Parkinson and Movement Disorders, 6525GA Nijmegen, The Netherlands
| | - Roshan Cools
- Radboud University Medical Centre, Donders Institute for Brain, Cognition and Behaviour, Centre for Cognitive Neuroimaging, 6525EN Nijmegen, The Netherlands
- Radboud University Medical Center, Department of Psychiatry, 6525GA Nijmegen, The Netherlands
| |
Collapse
|
16
|
Drossel G, Brucar LR, Rawls E, Hendrickson TJ, Zilverstand A. Subtypes in addiction and their neurobehavioral profiles across three functional domains. Transl Psychiatry 2023; 13:127. [PMID: 37072391 PMCID: PMC10113211 DOI: 10.1038/s41398-023-02426-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/17/2022] [Revised: 03/28/2023] [Accepted: 03/31/2023] [Indexed: 04/20/2023] Open
Abstract
Rates of return to use in addiction treatment remain high. We argue that the development of improved treatment options will require advanced understanding of individual heterogeneity in Substance Use Disorders (SUDs). We hypothesized that considerable individual differences exist in the three functional domains underlying addiction-approach-related behavior, executive function, and negative emotionality. We included N = 593 participants from the enhanced Nathan Kline Institute-Rockland Sample community sample (ages 18-59, 67% female) that included N = 420 Controls and N = 173 with past SUDs [54% female; N = 75 Alcohol Use Disorder (AUD) only, N = 30 Cannabis Use Disorder (CUD) only, and N = 68 Multiple SUDs]. To test our a priori hypothesis that distinct neuro-behavioral subtypes exist within individuals with past SUDs, we conducted a latent profile analysis with all available phenotypic data as input (74 subscales from 18 measures), and then characterized resting-state brain function for each discovered subtype. Three subtypes with distinct neurobehavioral profiles were recovered (p < 0.05, Cohen's D: 0.4-2.8): a "Reward type" with higher approach-related behavior (N = 69); a "Cognitive type" with lower executive function (N = 70); and a "Relief type" with high negative emotionality (N = 34). For those in the Reward type, substance use mapped onto resting-state connectivity in the Value/Reward, Ventral-Frontoparietal and Salience networks; for the Cognitive type in the Auditory, Parietal Association, Frontoparietal and Salience networks; and for the Relief type in the Parietal Association, Higher Visual and Salience networks (pFDR < 0.05). Subtypes were equally distributed amongst individuals with different primary SUDs (χ2 = 4.71, p = 0.32) and gender (χ2 = 3.44, p = 0.18). Results support functionally derived subtypes, demonstrating considerable individual heterogeneity in the multi-dimensional impairments in addiction. This confirms the need for mechanism-based subtyping to inform the development of personalized addiction medicine approaches.
Collapse
Affiliation(s)
- Gunner Drossel
- Graduate Program in Neuroscience, University of Minnesota, Minneapolis, MN, USA
- Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA
| | - Leyla R Brucar
- Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA
| | - Eric Rawls
- Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA
| | - Timothy J Hendrickson
- University of Minnesota Informatics Institute, University of Minnesota, Minneapolis, MN, USA
- Masonic Institute for the Developing Brain, University of Minnesota, Minneapolis, MN, USA
| | - Anna Zilverstand
- Department of Psychiatry and Behavioral Sciences, University of Minnesota, Minneapolis, MN, USA.
- Medical Discovery Team on Addiction, University of Minnesota, Minneapolis, MN, USA.
| |
Collapse
|
17
|
Abela N, Haywood K, Di Giovanni G. Alcohol and cannabinoid binges and daily exposure to nicotine in adolescent/young adult rats induce sex-dependent long-term appetitive instrumental learning impairment. Front Behav Neurosci 2023; 17:1129866. [PMID: 36815183 PMCID: PMC9939753 DOI: 10.3389/fnbeh.2023.1129866] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2022] [Accepted: 01/16/2023] [Indexed: 02/09/2023] Open
Abstract
Adolescence is a critical developmental period, concerning anatomical, neurochemical and behavioral changes. Moreover, adolescents are more sensitive to the long-term deleterious effects of drug abuse. Binge-like consumption of alcohol and marijuana, along with tobacco smoking, is a dangerous pattern often observed in adolescents during weekends. Nevertheless, the long-term effect of their adolescent co-exposure has not been yet experimentally investigated. Long-Evans adolescent male (n = 20) and female (n = 20) rats from postnatal day 30 (P30) until P60 were daily treated with nicotine (0.3 mg/kg, i.p.), and, on two consecutive 'binging days' per week (for a total of eight times), received an intragastric ethanol solution (3 g/kg) and an intraperitoneal (i.p.) dose of cannabinoid 1/2 receptor agonist WIN55,212-2 (1.2 mg/kg). These rats were tested after treatment discontinuation at > P90 for associative food-rewarded operant learning in the two-lever conditioning chambers for six consecutive days on a fixed ratio 1 (FR1) schedule followed by another six days of daily FR2 schedule testing, after 42 days rest. We found the main effects of sex x treatment interactions in FR1 but not in FR2 experiments. Treated females show attenuated operant responses for food pellets during all FR1 and the FR2 schedule, whilst the treated males show an impairment in FR2 but not in the FR1 schedule. Moreover, the treated females' percentage of learners was significantly lower than female controls in FR1 while treated males were lower than controls in FR2. Our findings suggest that intermittent adolescent abuse of common drugs, such as alcohol and marijuana, and chronic tobacco exposure can cause significant long-term effects on motivation for natural reinforcers later in adulthood in both sexes. Females appear to be sensitive earlier to the deleterious effects of adolescent polydrug abuse, with both sexes having an increased likelihood of developing lifelong brain alterations.
Collapse
Affiliation(s)
- Norbert Abela
- Laboratory of Neurophysiology, Department of Physiology and Biochemistry, Faculty of Medicine and Surgery, University of Malta, Msida, Malta
| | - Katie Haywood
- Laboratory of Neurophysiology, Department of Physiology and Biochemistry, Faculty of Medicine and Surgery, University of Malta, Msida, Malta,Division of Neuroscience, School of Biosciences, Cardiff University, Cardiff, United Kingdom
| | - Giuseppe Di Giovanni
- Laboratory of Neurophysiology, Department of Physiology and Biochemistry, Faculty of Medicine and Surgery, University of Malta, Msida, Malta,Division of Neuroscience, School of Biosciences, Cardiff University, Cardiff, United Kingdom,*Correspondence: Giuseppe Di Giovanni, ;
| |
Collapse
|
18
|
Liebenow B, Jones R, DiMarco E, Trattner JD, Humphries J, Sands LP, Spry KP, Johnson CK, Farkas EB, Jiang A, Kishida KT. Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders. Front Psychiatry 2022; 13:886297. [PMID: 36339844 PMCID: PMC9630918 DOI: 10.3389/fpsyt.2022.886297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Accepted: 09/23/2022] [Indexed: 11/13/2022] Open
Abstract
In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.
Collapse
Affiliation(s)
- Brittany Liebenow
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Rachel Jones
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Emily DiMarco
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Jonathan D. Trattner
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Joseph Humphries
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - L. Paul Sands
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Kasey P. Spry
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Christina K. Johnson
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Evelyn B. Farkas
- Georgia State University Undergraduate Neuroscience Institute, Atlanta, GA, United States
| | - Angela Jiang
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Kenneth T. Kishida
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Neurosurgery, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Biomedical Engineering, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| |
Collapse
|
19
|
Colas JT, Dundon NM, Gerraty RT, Saragosa‐Harris NM, Szymula KP, Tanwisuth K, Tyszka JM, van Geen C, Ju H, Toga AW, Gold JI, Bassett DS, Hartley CA, Shohamy D, Grafton ST, O'Doherty JP. Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T. Hum Brain Mapp 2022; 43:4750-4790. [PMID: 35860954 PMCID: PMC9491297 DOI: 10.1002/hbm.25988] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 05/20/2022] [Accepted: 06/10/2022] [Indexed: 11/12/2022] Open
Abstract
The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| | - Neil M. Dundon
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Department of Child and Adolescent Psychiatry, Psychotherapy, and PsychosomaticsUniversity of FreiburgFreiburg im BreisgauGermany
| | - Raphael T. Gerraty
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Center for Science and SocietyColumbia UniversityNew YorkNew YorkUSA
| | - Natalie M. Saragosa‐Harris
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of CaliforniaLos AngelesCaliforniaUSA
| | - Karol P. Szymula
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Koranis Tanwisuth
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Department of PsychologyUniversity of CaliforniaBerkeleyCaliforniaUSA
| | - J. Michael Tyszka
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
| | - Camilla van Geen
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Harang Ju
- Neuroscience Graduate GroupUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Arthur W. Toga
- Laboratory of Neuro ImagingUSC Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of USC, University of Southern CaliforniaLos AngelesCaliforniaUSA
| | - Joshua I. Gold
- Department of NeuroscienceUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dani S. Bassett
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Electrical and Systems EngineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of NeurologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of PsychiatryUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Physics and AstronomyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Santa Fe InstituteSanta FeNew MexicoUSA
| | - Catherine A. Hartley
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Center for Neural ScienceNew York UniversityNew YorkNew YorkUSA
| | - Daphna Shohamy
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Kavli Institute for Brain ScienceColumbia UniversityNew YorkNew YorkUSA
| | - Scott T. Grafton
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
| | - John P. O'Doherty
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| |
Collapse
|
20
|
Abnormal Brain Networks Related to Drug and Nondrug Reward Anticipation and Outcome Processing in Stimulant Use Disorder: A Functional Connectomics Approach. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2022; 8:560-571. [PMID: 36108930 DOI: 10.1016/j.bpsc.2022.08.014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/25/2022] [Revised: 08/25/2022] [Accepted: 08/29/2022] [Indexed: 01/21/2023]
Abstract
BACKGROUND Drug addiction is associated with blunted neural responses to nondrug rewards, such as money, but heightened responses to drug cues that predict drug-reward outcomes. This dissociation underscores the role of incentive context in the attribution of motivational salience, which may reflect a narrowing toward drug-related goals. This hypothesis, however, has scarcely been investigated. METHODS To address this important scientific gap, the current study performed an empirical assessment of differences in salience attribution by comparing patients with stimulant use disorder (SUD) (n = 41) with control participants (n = 48) on network connectivity related to anticipation and outcome processing using a modified monetary incentive delay task. We hypothesized increased task-related activation and connectivity to drug rewards in patients with SUD, and reduced task-related activation and connectivity to monetary rewards during incentive processing across brain networks. RESULTS In the presence of behavioral and regional brain activation similarities, we found that patients with SUD showed significantly less connectivity involving three separate distributed networks during monetary reward anticipation, and drug and monetary reward outcome processing. No group connectivity differences for drug reward anticipation were identified. Additional graph theory analyses revealed that patients with SUD had longer path lengths across these networks, all of which positively correlated with the duration of stimulant drug use. CONCLUSIONS Specific disruptions in connectivity in networks related to the anticipation of nondrug reward together with more general dysconnectivity in the processing of rewarding outcomes suggest an insensitivity to consequences. These observations support the notion of a predominance of habitual control in patients with SUD.
Collapse
|
21
|
Macoveanu J, Stougaard ME, Kjærstad HL, Knudsen GM, Vinberg M, Kessing LV, Miskowiak KW. Trajectory of aberrant reward processing in patients with bipolar disorder - A longitudinal fMRI study. J Affect Disord 2022; 312:235-244. [PMID: 35760195 DOI: 10.1016/j.jad.2022.06.053] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/25/2022] [Revised: 05/31/2022] [Accepted: 06/20/2022] [Indexed: 11/29/2022]
Abstract
BACKGROUND Bipolar disorder (BD), and especially the mania phenotype, is characterized by heightened reward responsivity and aberrant reward processing. In this longitudinal fMRI study, we investigated neuronal response during reward anticipation as the computed expected value (EV) and outcome evaluation as reward prediction error (RPE) in recently diagnosed patients with BD. METHODS Eighty remitted patients with BD and 60 healthy controls (HC) underwent fMRI during which they performed a card guessing task. Of these, 41 patients and 36 HC were re-scanned after 16 months. We compared reward-related neural activity between groups at baseline and longitudinally and assessed the impact of mood relapse. RESULTS Patients showed lower RPE signal in areas of the ventrolateral prefrontal cortex (vlPFC) than HC. In these regions, the HC showed decrease in RPE signal over time, which was absent in patients. Patients further exhibited decreased EV signal in the occipital cortex across baseline and follow-up. Patients who remained in remission showed normalization of the EV signal at follow-up. Baseline activity in the identified regions was not associated with subsequent relapse. LIMITATIONS Follow-up scans were only available in a relatively small sample. Medication status, follow-up time and BD illness duration prior to diagnosis varied. CONCLUSIONS Lower RPE signal in the vlPFC in patients with BD at baseline and its lack of normative reduction over time may represent a trait marker of dysfunctional reward-based learning or habituation. The increase in EV signal in the occipital cortex over time in patients who remained in remission may indicate normalization of reward anticipation activity.
Collapse
Affiliation(s)
- J Macoveanu
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark
| | - M E Stougaard
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark
| | - H L Kjærstad
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark
| | - G M Knudsen
- Neurobiology Research Unit, Copenhagen University Hospital, Rigshospitalet, Denmark; Department of Clinical Medicine, University of Copenhagen, Denmark
| | - M Vinberg
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark; Mental Health Centre, Northern Zealand, Copenhagen University Hospital-Mental Health Services CPH, Copenhagen, Denmark; Department of Clinical Medicine, University of Copenhagen, Denmark
| | - L V Kessing
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark; Department of Clinical Medicine, University of Copenhagen, Denmark
| | - K W Miskowiak
- Copenhagen Affective Disorder research Centre (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital, Rigshospitalet, Denmark; Department of Psychology, University of Copenhagen, Denmark.
| |
Collapse
|
22
|
Geurts DEM, Van den Heuvel TJ, Huys QJM, Verkes RJ, Cools R. Amygdala response predicts clinical symptom reduction in patients with borderline personality disorder: A pilot fMRI study. Front Behav Neurosci 2022; 16:938403. [PMID: 36110290 PMCID: PMC9468714 DOI: 10.3389/fnbeh.2022.938403] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2022] [Accepted: 07/27/2022] [Indexed: 11/13/2022] Open
Abstract
Borderline personality disorder (BPD) is a prevalent, devastating, and heterogeneous psychiatric disorder. Treatment success is highly variable within this patient group. A cognitive neuroscientific approach to BPD might contribute to precision psychiatry by identifying neurocognitive factors that predict who will benefit from a specific treatment. Here, we build on observations that BPD is accompanied by the enhanced impact of the aversive effect on behavior and abnormal neural signaling in the amygdala. We assessed whether BPD is accompanied by abnormal aversive regulation of instrumental behavior and associated neural signaling, in a manner that is predictive of symptom reduction after therapy. We tested a clinical sample of 15 female patients with BPD, awaiting dialectical behavior therapy (DBT), and 16 matched healthy controls using fMRI and an aversive Pavlovian-to-instrumental transfer (PIT) task that assesses how instrumental behaviors are influenced by aversive Pavlovian stimuli. Patients were assessed 1 year after the start of DBT to quantify changes in BPD symptom severity. At baseline, behavioral aversive PIT and associated neural signaling did not differ between groups. However, the BOLD signal in the amygdala measured during aversive PIT was associated with symptom reduction at 1-year follow-up: higher PIT-related aversive amygdala signaling before treatment was associated with reduced clinical improvement at follow-up. Thus, within the evaluated group of BPD patients, the BOLD signal in the amygdala before treatment was related to clinical symptom reduction 1 year after the start of treatment. The results suggest that less PIT-related responsiveness of the amygdala increases the chances of treatment success. We note that the relatively small sample size is a limitation of this study and that replication is warranted.
Collapse
Affiliation(s)
- Dirk E. M. Geurts
- Centre for Cognitive Neuroimaging, Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
| | - Thom J. Van den Heuvel
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
- Department of Scelta, Expert Centre for Personality Disorders, GGNet, Nijmegen, Netherlands
| | - Quentin J. M. Huys
- Mental Health Neuroscience Department, Division of Psychiatry and Max Planck UCL Centre for Computational Psychiatry and Ageing Research, Institute of Neurology, University College London, London, United Kingdom
| | - Robbert J. Verkes
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
- Kairos Center for Forensic Psychiatry, Pro Persona Mental Health, Nijmegen, Netherlands
| | - Roshan Cools
- Centre for Cognitive Neuroimaging, Donders Institute for Brain, Cognition and Behavior, Radboud University, Nijmegen, Netherlands
- Department of Psychiatry, Radboud University Medical Center, Nijmegen, Netherlands
| |
Collapse
|
23
|
Klein S, Kruse O, Tapia León I, Van Oudenhove L, van 't Hof SR, Klucken T, Wager TD, Stark R. Cross-paradigm integration shows a common neural basis for aversive and appetitive conditioning. Neuroimage 2022; 263:119594. [PMID: 36041642 DOI: 10.1016/j.neuroimage.2022.119594] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2022] [Revised: 07/22/2022] [Accepted: 08/25/2022] [Indexed: 10/31/2022] Open
Abstract
Sharing imaging data and comparing them across different psychological tasks is becoming increasingly possible as the open science movement advances. Such cross-paradigm integration has the potential to identify commonalities in findings that neighboring areas of study thought to be paradigm-specific. However, even the integration of research from closely related paradigms, such as aversive and appetitive classical conditioning is rare - even though qualitative comparisons already hint at how similar the 'fear network' and 'reward network' may be. We aimed to validate these theories by taking a multivariate approach to assess commonalities across paradigms empirically. Specifically, we quantified the similarity of an aversive conditioning pattern derived from meta-analysis to appetitive conditioning fMRI data. We tested pattern expression in three independent appetitive conditioning studies with 29, 76 and 38 participants each. During fMRI scanning, participants in each cohorts performed an appetitive conditioning task in which a CS+ was repeatedly rewarded with money and a CS- was never rewarded. The aversive pattern was highly similar to appetitive CS+ > CS- contrast maps across samples and variations of the appetitive conditioning paradigms. Moreover, the pattern distinguished the CS+ from the CS- with above-chance accuracy in every sample. These findings provide robust empirical evidence for an underlying neural system common to appetitive and aversive learning. We believe that this approach provides a way to empirically integrate the steadily growing body of fMRI findings across paradigms.
Collapse
Affiliation(s)
- Sanja Klein
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, Giessen 35394, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, Giessen 35394, Germany; Center of Mind, Brain and Behavior, Universities of Marburg and Giessen, Marburg 35032, Germany.
| | - Onno Kruse
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, Giessen 35394, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, Giessen 35394, Germany
| | - Isabell Tapia León
- Bender Institute for Neuroimaging (BION), Justus Liebig University, Giessen 35394, Germany; Clinical Psychology and Psychotherapy, University Siegen, Siegen 57076, Germany
| | - Lukas Van Oudenhove
- Department of Chronic Diseases and Metabolism (CHROMETA), Laboratory for Brain-Gut Axis Studies (LaBGAS), Translational Research Centre for Gastrointestinal Disorders TARGID, KU Leuven, Leuven, Belgium; Leuven Brain Institute, KU Leuven, Leuven, Belgium; Department of Psychological and Brain Sciences, Cognitive and Affective Neuroscience Lab, Dartmouth College, Hanover, NH, USA
| | - Sophie R van 't Hof
- Department of Psychiatry, Amsterdam University Medical Centers, Amsterdam 1105 AZ, The Netherlands
| | - Tim Klucken
- Clinical Psychology and Psychotherapy, University Siegen, Siegen 57076, Germany
| | - Tor D Wager
- Department of Psychological and Brain Sciences, Cognitive and Affective Neuroscience Lab, Dartmouth College, Hanover, NH, USA
| | - Rudolf Stark
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, Giessen 35394, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, Giessen 35394, Germany; Center of Mind, Brain and Behavior, Universities of Marburg and Giessen, Marburg 35032, Germany
| |
Collapse
|
24
|
Analysis of individual differences in neurofeedback training illuminates successful self-regulation of the dopaminergic midbrain. Commun Biol 2022; 5:845. [PMID: 35986202 PMCID: PMC9391365 DOI: 10.1038/s42003-022-03756-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Accepted: 07/21/2022] [Indexed: 11/27/2022] Open
Abstract
The dopaminergic midbrain is associated with reinforcement learning, motivation and decision-making – functions often disturbed in neuropsychiatric disorders. Previous research has shown that dopaminergic midbrain activity can be endogenously modulated via neurofeedback. However, the robustness of endogenous modulation, a requirement for clinical translation, is unclear. Here, we examine whether the activation of particular brain regions associates with successful regulation transfer when feedback is no longer available. Moreover, to elucidate mechanisms underlying effective self-regulation, we study the relation of successful transfer with learning (temporal difference coding) outside the midbrain during neurofeedback training and with individual reward sensitivity in a monetary incentive delay (MID) task. Fifty-nine participants underwent neurofeedback training either in standard (Study 1 N = 15, Study 2 N = 28) or control feedback group (Study 1, N = 16). We find that successful self-regulation is associated with prefrontal reward sensitivity in the MID task (N = 25), with a decreasing relation between prefrontal activity and midbrain learning signals during neurofeedback training and with increased activity within cognitive control areas during transfer. The association between midbrain self-regulation and prefrontal temporal difference and reward sensitivity suggests that reinforcement learning contributes to successful self-regulation. Our findings provide insights in the control of midbrain activity and may facilitate individually tailoring neurofeedback training. Analysis of real-time fMRI data from 59 participants undergoing neurofeedback training suggests that reinforcement learning contributes to successful self-regulation in the dopaminergic midbrain.
Collapse
|
25
|
Corlett PR, Mollick JA, Kober H. Meta-analysis of human prediction error for incentives, perception, cognition, and action. Neuropsychopharmacology 2022; 47:1339-1349. [PMID: 35017672 PMCID: PMC9117315 DOI: 10.1038/s41386-021-01264-3] [Citation(s) in RCA: 25] [Impact Index Per Article: 12.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 09/07/2021] [Revised: 12/23/2021] [Accepted: 12/28/2021] [Indexed: 12/30/2022]
Abstract
Prediction errors (PEs) are a keystone for computational neuroscience. Their association with midbrain neural firing has been confirmed across species and has inspired the construction of artificial intelligence that can outperform humans. However, there is still much to learn. Here, we leverage the wealth of human PE data acquired in the functional neuroimaging setting in service of a deeper understanding, using an MKDA (multi-level kernel-based density) meta-analysis. Studies were identified with Google Scholar, and we included studies with healthy adult participants that reported activation coordinates corresponding to PEs published between 1999-2018. Across 264 PE studies that have focused on reward, punishment, action, cognition, and perception, consistent with domain-general theoretical models of prediction error we found midbrain PE signals during cognitive and reward learning tasks, and an insula PE signal for perceptual, social, cognitive, and reward prediction errors. There was evidence for domain-specific error signals--in the visual hierarchy during visual perception, and the dorsomedial prefrontal cortex during social inference. We assessed bias following prior neuroimaging meta-analyses and used family-wise error correction for multiple comparisons. This organization of computation by region will be invaluable in building and testing mechanistic models of cognitive function and dysfunction in machines, humans, and other animals. Limitations include small sample sizes and ROI masking in some included studies, which we addressed by weighting each study by sample size, and directly comparing whole brain vs. ROI-based results.
Collapse
Affiliation(s)
| | | | - Hedy Kober
- Department of Psychiatry, Yale University, New Haven, CT, USA.
| |
Collapse
|
26
|
Martinez-Saito M, Gorina E. Learning under social versus nonsocial uncertainty: A meta-analytic approach. Hum Brain Mapp 2022; 43:4185-4206. [PMID: 35620870 PMCID: PMC9374892 DOI: 10.1002/hbm.25948] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Revised: 04/08/2022] [Accepted: 05/04/2022] [Indexed: 01/10/2023] Open
Abstract
Much of the uncertainty that clouds our understanding of the world springs from the covert values and intentions held by other people. Thus, it is plausible that specialized mechanisms that compute learning signals under uncertainty of exclusively social origin operate in the brain. To test this hypothesis, we scoured academic databases for neuroimaging studies involving learning under uncertainty, and performed a meta‐analysis of brain activation maps that compared learning in the face of social versus nonsocial uncertainty. Although most of the brain activations associated with learning error signals were shared between social and nonsocial conditions, we found some evidence for functional segregation of error signals of exclusively social origin during learning in limited regions of ventrolateral prefrontal cortex and insula. This suggests that most behavioral adaptations to navigate social environments are reused from frontal and subcortical areas processing generic value representation and learning, but that a specialized circuitry might have evolved in prefrontal regions to deal with social context representation and strategic action.
Collapse
Affiliation(s)
| | - Elena Gorina
- Department of Cognitive and Brain Sciences, Hebrew University of Jerusalem, Jerusalem, Israel
| |
Collapse
|
27
|
Oren S, Tittgemeyer M, Rigoux L, Schlamann M, Schonberg T, Kuzmanovic B. Neural Encoding of Food and Monetary Reward Delivery. Neuroimage 2022; 257:119335. [PMID: 35643268 DOI: 10.1016/j.neuroimage.2022.119335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 05/23/2022] [Accepted: 05/24/2022] [Indexed: 11/18/2022] Open
Abstract
Different types of rewards such as food and money can similarly drive our behavior owing to shared brain processes encoding their subjective value. However, while the value of money is abstract and needs to be learned, the value of food is rooted in the innate processing of sensory properties and nutritional utilization. Yet, the actual consumption of food and the receipt of money have never been directly contrasted in the same experiment, questioning what unique neural processes differentiate those reward types. To fill this gap, we examined the distinct and common neural responses to the delivery of food and monetary rewards during fMRI. In a novel experimental approach, we parametrically manipulated the subjective value of food and monetary rewards by modulating the quantities of administered palatable milkshake and monetary gains. The receipt of increasing amounts of milkshake and money recruited the ventral striatum and the ventromedial prefrontal cortex, previously associated with value encoding. Notably, the consumption and the subsequent evaluation of increasing quantities of milkshake relative to money revealed an extended recruitment of brain regions related to taste, somatosensory processing, and salience. Moreover, we detected a decline of reward encoding in the ventral tegmental area, nucleus accumbens, and vmPFC, indicating that these regions may be susceptible to time-dependent effects upon accumulation of food and money rewards. Relative to monetary gains, the consumption and evaluation of palatable milkshakes engaged complex neural processing over and above value tracking, emphasizing the critical contribution of taste and other sensory properties to the processing of food rewards. Furthermore, our results highlight the need to closely monitor metabolic states and neural responses to the accumulation of rewards to pinpoint the mechanisms underlying time-dependent dynamics of reward-related processing.
Collapse
Affiliation(s)
- Shiran Oren
- Sagol School of Neuroscience, Tel Aviv University, P.O. Box 39040, Tel Aviv 6997801, Israel; Translational Neurocircuitry Group, Max Planck Institute for Metabolism Research, Gleuelerstr. 50, Cologne 50931, Germany
| | - Marc Tittgemeyer
- Translational Neurocircuitry Group, Max Planck Institute for Metabolism Research, Gleuelerstr. 50, Cologne 50931, Germany; Excellence Cluster on Cellular Stress Responses in Aging Associated Diseases (CECAD), University of Cologne, Cologne 50931, Germany
| | - Lionel Rigoux
- Translational Neurocircuitry Group, Max Planck Institute for Metabolism Research, Gleuelerstr. 50, Cologne 50931, Germany
| | - Marc Schlamann
- Institute for Diagnostic and Interventional Radiology, University Hospital Cologne, Kerpenerstr. 62, Cologne 50937, Germany
| | - Tom Schonberg
- Sagol School of Neuroscience, Tel Aviv University, P.O. Box 39040, Tel Aviv 6997801, Israel; Department of Neurobiology, The George S. Wise Faculty of Life Sciences, P.O. Box 39040, Tel Aviv 6997801, Israel
| | - Bojana Kuzmanovic
- Translational Neurocircuitry Group, Max Planck Institute for Metabolism Research, Gleuelerstr. 50, Cologne 50931, Germany.
| |
Collapse
|
28
|
Dennison JB, Sazhin D, Smith DV. Decision neuroscience and neuroeconomics: Recent progress and ongoing challenges. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2022; 13:e1589. [PMID: 35137549 PMCID: PMC9124684 DOI: 10.1002/wcs.1589] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Revised: 11/28/2021] [Accepted: 12/21/2021] [Indexed: 01/10/2023]
Abstract
In the past decade, decision neuroscience and neuroeconomics have developed many new insights in the study of decision making. This review provides an overarching update on how the field has advanced in this time period. Although our initial review a decade ago outlined several theoretical, conceptual, methodological, empirical, and practical challenges, there has only been limited progress in resolving these challenges. We summarize significant trends in decision neuroscience through the lens of the challenges outlined for the field and review examples where the field has had significant, direct, and applicable impacts across economics and psychology. First, we review progress on topics including reward learning, explore-exploit decisions, risk and ambiguity, intertemporal choice, and valuation. Next, we assess the impacts of emotion, social rewards, and social context on decision making. Then, we follow up with how individual differences impact choices and new exciting developments in the prediction and neuroforecasting of future decisions. Finally, we consider how trends in decision-neuroscience research reflect progress toward resolving past challenges, discuss new and exciting applications of recent research, and identify new challenges for the field. This article is categorized under: Psychology > Reasoning and Decision Making Psychology > Emotion and Motivation.
Collapse
Affiliation(s)
- Jeffrey B Dennison
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| | - Daniel Sazhin
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| | - David V Smith
- Department of Psychology, Temple University, Philadelphia, Pennsylvania, USA
| |
Collapse
|
29
|
Yun JY, Lee YI, Park S, Choi JM, Choi SH, Jang JH. Functional activation of insula and dorsal anterior cingulate for conflict control against larger monetary loss in young adults with subthreshold depression: a preliminary study. Sci Rep 2022; 12:6956. [PMID: 35484391 PMCID: PMC9050651 DOI: 10.1038/s41598-022-10989-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Accepted: 04/15/2022] [Indexed: 11/08/2022] Open
Abstract
Subthreshold depression (StD) is associated with higher risk of later developing major depressive disorder (MDD). Deficits of goal-directed behaviors regarding the motional, motivational, and conflict control are found in MDD. The current study examined neural underpinning of conflict control against monetary punishment in StD compared to MDD and healthy controls (HC). Seventy-one participants (HC, n = 27; StD, n = 21; MDD, n = 23) in their mid-20's completed self-reports. Preprocessing of functional magnetic resonance imaging acquired for the Simon task against larger or smaller monetary punishment was conducted using ENIGMA HALFpipe version 1.2.1. Neural correlates of conflict control against monetary punishment that could vary with either diagnosis or PHQ-9 total score were examined using a general linear model of FSL. Simon effect was effective for reaction time and accuracy in every subgroup of diagnosis and regardless of the size of monetary punishment. Conflict control against larger monetary loss was associated with higher functional activation of left insula in StD than HC and MDD. StD showed lower functional activation of left dorsal anterior cingulate (dACC) than MDD for conflict control against larger monetary loss. For conflict control against smaller monetary loss, StD demonstrated higher functional activation of left paracentral lobule and right putamen compared to HC. Directed acyclic graphs showed directional associations from suicidal ideation, sadness, and concentration difficulty to functional activation of paracentral lobule, ventromedial prefrontal cortex (vmPFC), and thalamus for conflict control against monetary loss. Differential functional activation of insula and dACC for conflict control against larger monetary loss could be a brain phenotype of StD. Item-level depressive symptoms of suicidal ideation, sadness, and concentration difficulty could be reflected in the conflict control-related functional activation of paracentral lobule (against smaller monetary loss), vmPFC and thalamus (against larger monetary loss), respectively.
Collapse
Affiliation(s)
- Je-Yeon Yun
- Seoul National University Hospital, Seoul, Republic of Korea
- Yeongeon Student Support Center, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Yoonji Irene Lee
- Department of Psychiatry, Seoul National University Hospital, Seoul, Republic of Korea
| | - Susan Park
- Department of Psychiatry, Seoul National University Hospital, Seoul, Republic of Korea
| | - Jong Moon Choi
- Department of Psychology, Louisiana State University, Baton Rouge, USA
| | - Soo-Hee Choi
- Department of Psychiatry, Seoul National University Hospital, Seoul, Republic of Korea
- Department of Psychiatry, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Joon Hwan Jang
- Department of Psychiatry, Seoul National University Health Service Center, Gwanak-ro 1, Gwanak-gu, Seoul, 08826, Republic of Korea.
- Department of Human Systems Medicine, Seoul National University College of Medicine, Seoul, Republic of Korea.
| |
Collapse
|
30
|
Motivational signals disrupt metacognitive signals in the human ventromedial prefrontal cortex. Commun Biol 2022; 5:244. [PMID: 35304877 PMCID: PMC8933484 DOI: 10.1038/s42003-022-03197-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2021] [Accepted: 02/24/2022] [Indexed: 12/15/2022] Open
Abstract
A growing body of evidence suggests that, during decision-making, BOLD signal in the ventromedial prefrontal cortex (VMPFC) correlates both with motivational variables – such as incentives and expected values – and metacognitive variables – such as confidence judgments – which reflect the subjective probability of being correct. At the behavioral level, we recently demonstrated that the value of monetary stakes bias confidence judgments, with gain (respectively loss) prospects increasing (respectively decreasing) confidence judgments, even for similar levels of difficulty and performance. If and how this value-confidence interaction is reflected in the VMPFC remains unknown. Here, we used an incentivized perceptual decision-making fMRI task that dissociates key decision-making variables, thereby allowing to test several hypotheses about the role of the VMPFC in the value-confidence interaction. While our initial analyses seemingly indicate that the VMPFC combines incentives and confidence to form an expected value signal, we falsified this conclusion with a meticulous dissection of qualitative activation patterns. Rather, our results show that strong VMPFC confidence signals observed in trials with gain prospects are disrupted in trials with no – or negative (loss) – monetary prospects. Deciphering how decision variables are represented and interact at finer scales seems necessary to better understand biased (meta)cognition. The human ventromedial prefrontal cortex helps to determine value and confidence in certain decisions, but only in situations when there is a potential for a (monetary) reward.
Collapse
|
31
|
Liakoni V, Lehmann MP, Modirshanechi A, Brea J, Lutti A, Gerstner W, Preuschoff K. Brain signals of a Surprise-Actor-Critic model: Evidence for multiple learning modules in human decision making. Neuroimage 2021; 246:118780. [PMID: 34875383 DOI: 10.1016/j.neuroimage.2021.118780] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Revised: 08/03/2021] [Accepted: 12/04/2021] [Indexed: 11/25/2022] Open
Abstract
Learning how to reach a reward over long series of actions is a remarkable capability of humans, and potentially guided by multiple parallel learning modules. Current brain imaging of learning modules is limited by (i) simple experimental paradigms, (ii) entanglement of brain signals of different learning modules, and (iii) a limited number of computational models considered as candidates for explaining behavior. Here, we address these three limitations and (i) introduce a complex sequential decision making task with surprising events that allows us to (ii) dissociate correlates of reward prediction errors from those of surprise in functional magnetic resonance imaging (fMRI); and (iii) we test behavior against a large repertoire of model-free, model-based, and hybrid reinforcement learning algorithms, including a novel surprise-modulated actor-critic algorithm. Surprise, derived from an approximate Bayesian approach for learning the world-model, is extracted in our algorithm from a state prediction error. Surprise is then used to modulate the learning rate of a model-free actor, which itself learns via the reward prediction error from model-free value estimation by the critic. We find that action choices are well explained by pure model-free policy gradient, but reaction times and neural data are not. We identify signatures of both model-free and surprise-based learning signals in blood oxygen level dependent (BOLD) responses, supporting the existence of multiple parallel learning modules in the brain. Our results extend previous fMRI findings to a multi-step setting and emphasize the role of policy gradient and surprise signalling in human learning.
Collapse
Affiliation(s)
- Vasiliki Liakoni
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland.
| | - Marco P Lehmann
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Alireza Modirshanechi
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Johanni Brea
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Antoine Lutti
- Laboratoire de recherche en neuroimagerie (LREN), Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Wulfram Gerstner
- École Polytechnique Fédérale de Lausanne (EPFL), School of Computer and Communication Sciences and School of Life Sciences, Lausanne, Switzerland
| | - Kerstin Preuschoff
- Geneva Finance Research Institute & Interfaculty Center for Affective Sciences, University of Geneva, Geneva, Switzerland
| |
Collapse
|
32
|
Brown VM, Zhu L, Solway A, Wang JM, McCurry KL, King-Casas B, Chiu PH. Reinforcement Learning Disruptions in Individuals With Depression and Sensitivity to Symptom Change Following Cognitive Behavioral Therapy. JAMA Psychiatry 2021; 78:1113-1122. [PMID: 34319349 PMCID: PMC8319827 DOI: 10.1001/jamapsychiatry.2021.1844] [Citation(s) in RCA: 41] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 01/07/2023]
Abstract
IMPORTANCE Major depressive disorder is prevalent and impairing. Parsing neurocomputational substrates of reinforcement learning in individuals with depression may facilitate a mechanistic understanding of the disorder and suggest new cognitive therapeutic targets. OBJECTIVE To determine associations among computational model-derived reinforcement learning parameters, depression symptoms, and symptom changes after treatment. DESIGN, SETTING, AND PARTICIPANTS In this mixed cross-sectional-cohort study, individuals performed reward and loss variants of a probabilistic learning task during functional magnetic resonance imaging at baseline and follow-up. A volunteer sample with and without a depression diagnosis was recruited from the community. Participants were assessed from July 2011 to February 2017, and data were analyzed from May 2017 to May 2021. MAIN OUTCOMES AND MEASURES Computational model-based analyses of participants' choices assessed a priori hypotheses about associations between components of reward-based and loss-based learning with depression symptoms. Changes in both learning parameters and symptoms were then assessed in a subset of participants who received cognitive behavioral therapy (CBT). RESULTS Of 101 included adults, 69 (68.3%) were female, and the mean (SD) age was 34.4 (11.2) years. A total of 69 participants with a depression diagnosis and 32 participants without a depression diagnosis were included at baseline; 48 participants (28 with depression who received CBT and 20 without depression) were included at follow-up (mean [SD] of 115.1 [15.6] days). Computational model-based analyses of behavioral choices and neural data identified associations of learning with symptoms during reward learning and loss learning, respectively. During reward learning only, anhedonia (and not negative affect or arousal) was associated with model-derived learning parameters (learning rate: posterior mean regression β = -0.14; 95% credible interval [CrI], -0.12 to -0.03; outcome sensitivity: posterior mean regression β = 0.18; 95% CrI, 0.02 to 0.37) and neural learning signals (moderation of association between striatal prediction error and expected value signals: t97 = -2.10; P = .04). During loss learning only, negative affect (and not anhedonia or arousal) was associated with learning parameters (outcome shift: posterior mean regression β = -0.11; 95% CrI, -0.20 to -0.01) and disrupted neural encoding of learning signals (association with subgenual anterior cingulate prediction error signals: r = -0.28; P = .005). Symptom improvement following CBT was associated with normalization of learning parameters that were disrupted at baseline (reward learning rate: posterior mean regression β = 0.15; 90% CrI, 0.001 to 0.41; loss outcome shift: posterior mean regression β = 0.42; 90% CrI, 0.09 to 0.77). CONCLUSIONS AND RELEVANCE In this study, the mapping of reinforcement learning components to symptoms of major depression revealed mechanistic features associated with these symptoms and points to possible learning-based therapeutic processes and targets.
Collapse
Affiliation(s)
- Vanessa M. Brown
- Department of Psychology, Virginia Tech, Blacksburg,Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke,Department of Psychiatry, University of Pittsburgh School of Medicine, Pittsburgh, Pennsylvania
| | - Lusha Zhu
- Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke,School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, PKU-IDG/McGovern Institute for Brain Research, Peking-Tsinghua Center for Life Sciences, Peking University, Beijing, China
| | - Alec Solway
- Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke
| | - John M. Wang
- Department of Psychology, Virginia Tech, Blacksburg,Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke
| | - Katherine L. McCurry
- Department of Psychology, Virginia Tech, Blacksburg,Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke
| | - Brooks King-Casas
- Department of Psychology, Virginia Tech, Blacksburg,Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke,Virginia Tech-Wake Forest University School of Biomedical Engineering and Sciences, Blacksburg
| | - Pearl H. Chiu
- Department of Psychology, Virginia Tech, Blacksburg,Fralin Biomedical Research Institute at VTC, Virginia Tech, Roanoke
| |
Collapse
|
33
|
Ellner D, Hallam B, Frie JA, Thorpe HHA, Shoaib M, Kayir H, Jenkins BW, Khokhar JY. Discordant Effects of Cannabinoid 2 Receptor Antagonism/Inverse Agonism During Adolescence on Pavlovian and Instrumental Reward Learning in Adult Male Rats. Front Synaptic Neurosci 2021; 13:732402. [PMID: 34526887 PMCID: PMC8437373 DOI: 10.3389/fnsyn.2021.732402] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Accepted: 08/12/2021] [Indexed: 12/03/2022] Open
Abstract
The endocannabinoid system is responsible for regulating a spectrum of physiological activities and plays a critical role in the developing brain. During adolescence, the endocannabinoid system is particularly sensitive to external insults that may change the brain’s developmental trajectory. Cannabinoid receptor type 2 (CB2R) was initially thought to predominantly function in the peripheral nervous system, but more recent studies have implicated its role in the mesolimbic pathway, a network largely attributed to reward circuitry and reward motivated behavior, which undergoes extensive changes during adolescence. It is therefore important to understand how CB2R modulation during adolescence can impact reward-related behaviors in adulthood. In this study, adolescent male rats (postnatal days 28–41) were exposed to a low or high dose of the CB2R antagonist/inverse agonist SR144528 and Pavlovian autoshaping and instrumental conditional behavioral outcomes were measured in adulthood. SR144528-treated rats had significantly slower acquisition of the autoshaping task, seen by less lever pressing behavior over time [F(2, 19) = 5.964, p = 0.010]. Conversely, there was no effect of adolescent SR144528 exposure on instrumental conditioning. These results suggest that modulation of the CB2R in adolescence differentially impacts reward-learning behaviors in adulthood.
Collapse
Affiliation(s)
- Danna Ellner
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Bryana Hallam
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Jude A Frie
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Hayley H A Thorpe
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Muhammad Shoaib
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Hakan Kayir
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Bryan W Jenkins
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| | - Jibran Y Khokhar
- Department of Biomedical Sciences, University of Guelph, Guelph, ON, Canada
| |
Collapse
|
34
|
Sequeira SL, Silk JS, Hutchinson E, Jones NP, Ladouceur CD. Neural Responses to Social Reward Predict Depressive Symptoms in Adolescent Girls During the COVID-19 Pandemic. J Pediatr Psychol 2021; 46:915-926. [PMID: 34270756 PMCID: PMC8344736 DOI: 10.1093/jpepsy/jsab037] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2020] [Revised: 03/17/2021] [Accepted: 03/18/2021] [Indexed: 11/12/2022] Open
Abstract
OBJECTIVE Adolescent depression is increasing during the COVID-19 pandemic, possibly related to dramatic social changes. Individual-level factors that contribute to social functioning, such as temperament and neural reactivity to social feedback, may confer risk for or resilience against depressive symptoms during the pandemic. METHODS Ninety-three girls (12-17 years) oversampled for high shy/fearful temperament were recruited from a longitudinal study for a follow-up COVID-19 study. During the parent study (2016-2018), participants completed a functional magnetic resonance imaging task eliciting neural activity to performance-related social feedback. Depressive symptoms were assessed during the parent study and COVID-19 follow-up (April-May 2020). In 65 participants with complete data, we examined how interactions between temperament and neural activation to social reward or punishment in a socio-affective brain network predict depressive symptoms during COVID-19. RESULTS Depressive symptoms increased during COVID-19. Significant interactions between temperament and caudate, putamen, and insula activation to social reward were found. Girls high in shy/fearful temperament showed negative associations between neural activation to social reward and COVID-19 depressive symptoms, whereas girls lower in shy/fearful temperament showed positive associations. CONCLUSIONS Girls high in shy/fearful temperament with reduced neural activation to social reward may be less likely to engage socially, which could be detrimental during the pandemic when social interactions are limited. In contrast, girls lower in shy/fearful temperament with heightened neural reactivity to social reward may be highly motivated to engage socially, which could also be detrimental with limited social opportunities. In both cases, improving social connection during the pandemic may attenuate or prevent depressive symptoms.
Collapse
Affiliation(s)
| | | | | | - Neil P Jones
- Department of Psychiatry, University of Pittsburgh, USA
| | | |
Collapse
|
35
|
Lu HY, Lorenc ES, Zhu H, Kilmarx J, Sulzer J, Xie C, Tobler PN, Watrous AJ, Orsborn AL, Lewis-Peacock J, Santacruz SR. Multi-scale neural decoding and analysis. J Neural Eng 2021; 18. [PMID: 34284369 PMCID: PMC8840800 DOI: 10.1088/1741-2552/ac160f] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2020] [Accepted: 07/20/2021] [Indexed: 12/15/2022]
Abstract
Objective. Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment. Approach. We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes. Main results. We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches. Significance. This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.
Collapse
Affiliation(s)
- Hung-Yun Lu
- The University of Texas at Austin, Biomedical Engineering, Austin, TX, United States of America
| | - Elizabeth S Lorenc
- The University of Texas at Austin, Psychology, Austin, TX, United States of America.,The University of Texas at Austin, Institute for Neuroscience, Austin, TX, United States of America
| | - Hanlin Zhu
- Rice University, Electrical and Computer Engineering, Houston, TX, United States of America
| | - Justin Kilmarx
- The University of Texas at Austin, Mechanical Engineering, Austin, TX, United States of America
| | - James Sulzer
- The University of Texas at Austin, Mechanical Engineering, Austin, TX, United States of America.,The University of Texas at Austin, Institute for Neuroscience, Austin, TX, United States of America
| | - Chong Xie
- Rice University, Electrical and Computer Engineering, Houston, TX, United States of America
| | - Philippe N Tobler
- University of Zurich, Neuroeconomics and Social Neuroscience, Zurich, Switzerland
| | - Andrew J Watrous
- The University of Texas at Austin, Neurology, Austin, TX, United States of America
| | - Amy L Orsborn
- University of Washington, Electrical and Computer Engineering, Seattle, WA, United States of America.,University of Washington, Bioengineering, Seattle, WA, United States of America.,Washington National Primate Research Center, Seattle, WA, United States of America
| | - Jarrod Lewis-Peacock
- The University of Texas at Austin, Psychology, Austin, TX, United States of America.,The University of Texas at Austin, Institute for Neuroscience, Austin, TX, United States of America
| | - Samantha R Santacruz
- The University of Texas at Austin, Biomedical Engineering, Austin, TX, United States of America.,The University of Texas at Austin, Institute for Neuroscience, Austin, TX, United States of America
| |
Collapse
|
36
|
Ficco L, Mancuso L, Manuello J, Teneggi A, Liloia D, Duca S, Costa T, Kovacs GZ, Cauda F. Disentangling predictive processing in the brain: a meta-analytic study in favour of a predictive network. Sci Rep 2021; 11:16258. [PMID: 34376727 PMCID: PMC8355157 DOI: 10.1038/s41598-021-95603-5] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Accepted: 07/28/2021] [Indexed: 02/07/2023] Open
Abstract
According to the predictive coding (PC) theory, the brain is constantly engaged in predicting its upcoming states and refining these predictions through error signals. Despite extensive research investigating the neural bases of this theory, to date no previous study has systematically attempted to define the neural mechanisms of predictive coding across studies and sensory channels, focussing on functional connectivity. In this study, we employ a coordinate-based meta-analytical approach to address this issue. We first use the Activation Likelihood Estimation (ALE) algorithm to detect spatial convergence across studies, related to prediction error and encoding. Overall, our ALE results suggest the ultimate role of the left inferior frontal gyrus and left insula in both processes. Moreover, we employ a meta-analytic connectivity method (Seed-Voxel Correlations Consensus). This technique reveals a large, bilateral predictive network, which resembles large-scale networks involved in task-driven attention and execution. In sum, we find that: (i) predictive processing seems to occur more in certain brain regions than others, when considering different sensory modalities at a time; (ii) there is no evidence, at the network level, for a distinction between error and prediction processing.
Collapse
Affiliation(s)
- Linda Ficco
- Focuslab, Department of Psychology, University of Turin, Turin, Italy.
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy.
- Department for General Psychology and Cognitive Neuroscience, Friedrich Schiller University Jena, Am Steiger 3/Haus 1, 07743, Jena, Germany.
| | - Lorenzo Mancuso
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Jordi Manuello
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Alessia Teneggi
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Donato Liloia
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Sergio Duca
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Tommaso Costa
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| | - Gyula Zoltán Kovacs
- Department of Biological Psychology and Cognitive Neuroscience, Institute for Psychology, Friedrich-Schiller University of Jena, Jena, Germany
| | - Franco Cauda
- Focuslab, Department of Psychology, University of Turin, Turin, Italy
- GCS-fMRI, Koelliker Hospital and Department of Psychology, University of Turin, Turin, Italy
| |
Collapse
|
37
|
Meta-analytic clustering dissociates brain activity and behavior profiles across reward processing paradigms. COGNITIVE AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2021; 20:215-235. [PMID: 31872334 DOI: 10.3758/s13415-019-00763-7] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Reward learning is a ubiquitous cognitive mechanism guiding adaptive choices and behaviors, and when impaired, can lead to considerable mental health consequences. Reward-related functional neuroimaging studies have begun to implicate networks of brain regions essential for processing various peripheral influences (e.g., risk, subjective preference, delay, social context) involved in the multifaceted reward processing construct. To provide a more complete neurocognitive perspective on reward processing that synthesizes findings across the literature while also appreciating these peripheral influences, we used emerging meta-analytic techniques to elucidate brain regions, and in turn networks, consistently engaged in distinct aspects of reward processing. Using a data-driven, meta-analytic, k-means clustering approach, we dissociated seven meta-analytic groupings (MAGs) of neuroimaging results (i.e., brain activity maps) from 749 experimental contrasts across 176 reward processing studies involving 13,358 healthy participants. We then performed an exploratory functional decoding approach to gain insight into the putative functions associated with each MAG. We identified a seven-MAG clustering solution that represented dissociable patterns of convergent brain activity across reward processing tasks. Additionally, our functional decoding analyses revealed that each of these MAGs mapped onto discrete behavior profiles that suggested specialized roles in predicting value (MAG-1 & MAG-2) and processing a variety of emotional (MAG-3), external (MAG-4 & MAG-5), and internal (MAG-6 & MAG-7) influences across reward processing paradigms. These findings support and extend aspects of well-accepted reward learning theories and highlight large-scale brain network activity associated with distinct aspects of reward processing.
Collapse
|
38
|
Dombrovski AY, Hallquist MN. Search for solutions, learning, simulation, and choice processes in suicidal behavior. WILEY INTERDISCIPLINARY REVIEWS. COGNITIVE SCIENCE 2021; 13:e1561. [PMID: 34008338 PMCID: PMC9285563 DOI: 10.1002/wcs.1561] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/12/2020] [Revised: 03/06/2021] [Accepted: 04/07/2021] [Indexed: 12/25/2022]
Abstract
Suicide may be viewed as an unfortunate outcome of failures in decision processes. Such failures occur when the demands of a crisis exceed a person's capacity to (i) search for options, (ii) learn and simulate possible futures, and (iii) make advantageous value‐based choices. Can individual‐level decision deficits and biases drive the progression of the suicidal crisis? Our overview of the evidence on this question is informed by clinical theory and grounded in reinforcement learning and behavioral economics. Cohort and case–control studies provide strong evidence that limited cognitive capacity and particularly impaired cognitive control are associated with suicidal behavior, imposing cognitive constraints on decision‐making. We conceptualize suicidal ideation as an element of impoverished consideration sets resulting from a search for solutions under cognitive constraints and mood‐congruent Pavlovian influences, a view supported by mostly indirect evidence. More compelling is the evidence of impaired learning in people with a history of suicidal behavior. We speculate that an inability to simulate alternative futures using one's model of the world may undermine alternative solutions in a suicidal crisis. The hypothesis supported by the strongest evidence is that the selection of suicide over alternatives is facilitated by a choice process undermined by randomness. Case–control studies using gambling tasks, armed bandits, and delay discounting support this claim. Future experimental studies will need to uncover real‐time dynamics of choice processes in suicidal people. In summary, the decision process framework sheds light on neurocognitive mechanisms that facilitate the progression of the suicidal crisis. This article is categorized under:Economics > Individual Decision‐Making Psychology > Emotion and Motivation Psychology > Learning Neuroscience > Behavior
Collapse
Affiliation(s)
| | - Michael N Hallquist
- Department of Psychology and Neuroscience, University of North Carolina, Chapel Hill, North Carolina, USA
| |
Collapse
|
39
|
Shin EJ, Jang Y, Kim S, Kim H, Cai X, Lee H, Sul JH, Lee SH, Chung Y, Lee D, Jung MW. Robust and distributed neural representation of action values. eLife 2021; 10:53045. [PMID: 33876728 PMCID: PMC8104958 DOI: 10.7554/elife.53045] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2019] [Accepted: 04/19/2021] [Indexed: 11/13/2022] Open
Abstract
Studies in rats, monkeys, and humans have found action-value signals in multiple regions of the brain. These findings suggest that action-value signals encoded in these brain structures bias choices toward higher expected rewards. However, previous estimates of action-value signals might have been inflated by serial correlations in neural activity and also by activity related to other decision variables. Here, we applied several statistical tests based on permutation and surrogate data to analyze neural activity recorded from the striatum, frontal cortex, and hippocampus. The results show that previously identified action-value signals in these brain areas cannot be entirely accounted for by concurrent serial correlations in neural activity and action value. We also found that neural activity related to action value is intermixed with signals related to other decision variables. Our findings provide strong evidence for broadly distributed neural signals related to action value throughout the brain.
Collapse
Affiliation(s)
- Eun Ju Shin
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Yunsil Jang
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Soyoun Kim
- Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, Republic of Korea
| | - Hoseok Kim
- Department of Neuroscience, Biomedicum, Karolinska Institutet, Stockholm, Sweden
| | - Xinying Cai
- New York University Shanghai, NYU-ECNU Institute of Brain and Cognitive Science at NYU Shanghai, and Shanghai Key Laboratory of Brain Functional Genomics (Ministry of Education), School of Psychology and Cognitive Science, East China Normal University, Shanghai, China
| | - Hyunjung Lee
- Department of Anatomy, Kyungpook National University School of Medicine, Daegu, Republic of Korea
| | - Jung Hoon Sul
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea
| | - Sung-Hyun Lee
- Neuroscience Graduate Program, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Yeonseung Chung
- Department of Mathematical Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| | - Daeyeol Lee
- The Zanvyl Krieger Mind/Brain Institute, Kavli Neuroscience Discovery Institute, Department of Neuroscience, and Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, United States
| | - Min Whan Jung
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science, Daejeon, Republic of Korea.,Department of Biological Sciences, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea
| |
Collapse
|
40
|
Reward and fictive prediction error signals in ventral striatum: asymmetry between factual and counterfactual processing. Brain Struct Funct 2021; 226:1553-1569. [PMID: 33839955 DOI: 10.1007/s00429-021-02270-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2020] [Accepted: 03/27/2021] [Indexed: 10/21/2022]
Abstract
Reward prediction error, the difference between the expected and obtained reward, is known to act as a reinforcement learning neural signal. In the current study, we propose a model fitting approach that combines behavioral and neural data to fit computational models of reinforcement learning. Briefly, we penalized subject-specific fitted parameters that moved away too far from the group median, except when that deviation led to an improvement in the model's fit to neural responses. By means of a probabilistic monetary learning task and fMRI, we compared our approach with standard model fitting methods. Q-learning outperformed actor-critic at both behavioral and neural level, although the inclusion of neuroimaging data into model fitting improved the fit of actor-critic models. We observed both action-value and state-value prediction error signals in the striatum, while standard model fitting approaches failed to capture state-value signals. Finally, left ventral striatum correlated with reward prediction error while right ventral striatum with fictive prediction error, suggesting a functional hemispheric asymmetry regarding prediction-error driven learning.
Collapse
|
41
|
Rupprechter S, Stankevicius A, Huys QJM, Series P, Steele JD. Abnormal reward valuation and event-related connectivity in unmedicated major depressive disorder. Psychol Med 2021; 51:795-803. [PMID: 31907081 DOI: 10.1017/s0033291719003799] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
Abstract
BACKGROUND Experience of emotion is closely linked to valuation. Mood can be viewed as a bias to experience positive or negative emotions and abnormally biased subjective reward valuation and cognitions are core characteristics of major depression. METHODS Thirty-four unmedicated subjects with major depressive disorder and controls estimated the probability that fractal stimuli were associated with reward, based on passive observations, so they could subsequently choose the higher of either their estimated fractal value or an explicitly presented reward probability. Using model-based functional magnetic resonance imaging, we estimated each subject's internal value estimation, with psychophysiological interaction analysis used to examine event-related connectivity, testing hypotheses of abnormal reward valuation and cingulate connectivity in depression. RESULTS Reward value encoding in the hippocampus and rostral anterior cingulate was abnormal in depression. In addition, abnormal decision-making in depression was associated with increased anterior mid-cingulate activity and a signal in this region encoded the difference between the values of the two options. This localised decision-making and its impairment to the anterior mid-cingulate cortex (aMCC) consistent with theories of cognitive control. Notably, subjects with depression had significantly decreased event-related connectivity between the aMCC and rostral cingulate regions during decision-making, implying impaired communication between the neural substrates of expected value estimation and decision-making in depression. CONCLUSIONS Our findings support the theory that abnormal neural reward valuation plays a central role in major depressive disorder (MDD). To the extent that emotion reflects valuation, abnormal valuation could explain abnormal emotional experience in MDD, reflect a core pathophysiological process and be a target of treatment.
Collapse
Affiliation(s)
- S Rupprechter
- Institute for Adaptive and Neural Computation, University of Edinburgh, Edinburgh, UK
| | - A Stankevicius
- Institute for Adaptive and Neural Computation, University of Edinburgh, Edinburgh, UK
| | - Q J M Huys
- Max Planck Centre for Computational Psychiatry and Ageing Research, UCL, London, UK
- Camden and Islington NHS Foundation Trust, London, UK
| | - P Series
- Institute for Adaptive and Neural Computation, University of Edinburgh, Edinburgh, UK
| | - J D Steele
- Division of Imaging Science and Technology, Medical School, University of Dundee, Dundee, UK
- Department of Neurology, Ninewells Hospital, NHS Tayside, Dundee, UK
| |
Collapse
|
42
|
Unraveling the Temporal Dynamics of Reward Signals in Music-Induced Pleasure with TMS. J Neurosci 2021; 41:3889-3899. [PMID: 33782048 DOI: 10.1523/jneurosci.0727-20.2020] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2020] [Revised: 12/18/2020] [Accepted: 12/21/2020] [Indexed: 11/21/2022] Open
Abstract
Music's ability to induce feelings of pleasure has been the subject of intense neuroscientific research lately. Prior neuroimaging studies have shown that music-induced pleasure engages cortico-striatal circuits related to the anticipation and receipt of biologically relevant rewards/incentives, but these reports are necessarily correlational. Here, we studied both the causal role of this circuitry and its temporal dynamics by applying transcranial magnetic stimulation (TMS) over the left dorsolateral PFC combined with fMRI in 17 male and female participants. Behaviorally, we found that, in accord with previous findings, excitation of fronto-striatal pathways enhanced subjective reports of music-induced pleasure and motivation, whereas inhibition of the same circuitry led to the reduction of both. fMRI activity patterns indicated that these behavioral changes were driven by bidirectional TMS-induced alteration of fronto-striatal function. Specifically, changes in activity in the NAcc predicted modulation of both hedonic and motivational responses, with a dissociation between pre-experiential versus experiential components of musical reward. In addition, TMS-induced changes in the fMRI functional connectivity between the NAcc and frontal and auditory cortices predicted the degree of modulation of hedonic responses. These results indicate that the engagement of cortico-striatal pathways and the NAcc, in particular, is indispensable to experience rewarding feelings from music.SIGNIFICANCE STATEMENT Neuroimaging studies have shown that music-induced pleasure engages cortico-striatal circuits involved in the processing of biologically relevant rewards. Yet, these reports are necessarily correlational. Here, we studied both the causal role of this circuitry and its temporal dynamics by combining brain stimulation over the frontal cortex with functional imaging. Behaviorally, we found that excitation and inhibition of fronto-striatal pathways enhanced and disrupted, respectively, subjective reports of music-induced pleasure and motivation. These changes were associated with changes in NAcc activity and NAcc coupling with frontal and auditory cortices, dissociating between pre-experimental versus experiential components of musical reward. These results indicate that the engagement of cortico-striatal pathways, and the NAcc in particular, is indispensable to experience rewarding feeling from music.
Collapse
|
43
|
Thompson K, Nahmias E, Fani N, Kvaran T, Turner J, Tone E. The Prisoner's Dilemma paradigm provides a neurobiological framework for the social decision cascade. PLoS One 2021; 16:e0248006. [PMID: 33735226 PMCID: PMC7971531 DOI: 10.1371/journal.pone.0248006] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2020] [Accepted: 02/17/2021] [Indexed: 11/18/2022] Open
Abstract
To function during social interactions, we must be able to consider and coordinate our actions with other people's perspectives. This process unfolds from decision-making, to anticipation of that decision's consequences, to feedback about those consequences, in what can be described as a "cascade" of three phases. The iterated Prisoner's Dilemma (iPD) task, an economic-exchange game used to illustrate how people achieve stable cooperation over repeated interactions, provides a framework for examining this "social decision cascade". In the present study, we examined neural activity associated with the three phases of the cascade, which can be isolated during iPD game rounds. While undergoing functional magnetic resonance imaging (fMRI), 31 adult participants made a) decisions about whether to cooperate with a co-player for a monetary reward, b) anticipated the co-player's decision, and then c) learned the co-player's decision. Across all three phases, participants recruited the temporoparietal junction (TPJ) and the dorsomedial prefrontal cortex (dmPFC), regions implicated in numerous facets of social reasoning such as perspective-taking and the judgement of intentions. Additionally, a common distributed neural network underlies both decision-making and feedback appraisal; however, differences were identified in the magnitude of recruitment between both phases. Furthermore, there was limited evidence that anticipation following the decision to defect evoked a neural signature that is distinct from the signature of anticipation following the decision to cooperate. This study is the first to delineate the neural substrates of the entire social decision cascade in the context of the iPD game.
Collapse
Affiliation(s)
- Khalil Thompson
- Department of Psychology, Georgia State University, Atlanta, Georgia, United States of America
| | - Eddy Nahmias
- Department of Psychology, Georgia State University, Atlanta, Georgia, United States of America
| | - Negar Fani
- Department of Psychiatry and Behavioral Sciences, Emory University, Atlanta, Georgia, United States of America
| | - Trevor Kvaran
- Department of Psychology, Georgia State University, Atlanta, Georgia, United States of America
| | - Jessica Turner
- Department of Psychology, Georgia State University, Atlanta, Georgia, United States of America
| | - Erin Tone
- Department of Psychology, Georgia State University, Atlanta, Georgia, United States of America
| |
Collapse
|
44
|
Abstract
Abstract
Purpose of Review
Current theories of alcohol use disorders (AUD) highlight the importance of Pavlovian and instrumental learning processes mainly based on preclinical animal studies. Here, we summarize available evidence for alterations of those processes in human participants with AUD with a focus on habitual versus goal-directed instrumental learning, Pavlovian conditioning, and Pavlovian-to-instrumental transfer (PIT) paradigms.
Recent Findings
The balance between habitual and goal-directed control in AUD participants has been studied using outcome devaluation or sequential decision-making procedures, which have found some evidence of reduced goal-directed/model-based control, but little evidence for stronger habitual responding. The employed Pavlovian learning and PIT paradigms have shown considerable differences regarding experimental procedures, e.g., alcohol-related or conventional reinforcers or stimuli.
Summary
While studies of basic learning processes in human participants with AUD support a role of Pavlovian and instrumental learning mechanisms in the development and maintenance of drug addiction, current studies are characterized by large variability regarding methodology, sample characteristics, and results, and translation from animal paradigms to human research remains challenging. Longitudinal approaches with reliable and ecologically valid paradigms of Pavlovian and instrumental processes, including alcohol-related cues and outcomes, are warranted and should be combined with state-of-the-art imaging techniques, computational approaches, and ecological momentary assessment methods.
Collapse
|
45
|
Dhingra I, Zhang S, Zhornitsky S, Wang W, Le TM, Li CSR. Sex differences in neural responses to reward and the influences of individual reward and punishment sensitivity. BMC Neurosci 2021; 22:12. [PMID: 33639845 PMCID: PMC7913329 DOI: 10.1186/s12868-021-00618-3] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2020] [Accepted: 02/16/2021] [Indexed: 12/21/2022] Open
Abstract
BACKGROUND Men and women show differences in sensitivity to reward and punishment, which may impact behavior in health and disease. However, the neural bases of these sex differences remain under-investigated. Here, by combining functional magnetic resonance imaging (fMRI) and a variant of the Monetary Incentive Delay Task (MIDT), we examined sex differences in the neural responses to wins and losses and how individual reward and punishment sensitivity modulates these regional activities. METHODS Thirty-sex men and 27 women participated in the fMRI study. We assessed sensitivity to punishment (SP) and sensitivity to reward (SR) with the Sensitivity to Punishment and Sensitivity to Reward Questionnaire (SPSRQ). In the MIDT, participants pressed a button to collect reward ($1, 1¢, or nil), with the reaction time window titrated across trials so participants achieved a success rate of approximately 67%. We processed the Imaging data with published routines and evaluated the results with a corrected threshold. RESULTS Women showed higher SP score than men and men showed higher SR score than women. Men relative to women showed higher response to the receipt of dollar or cent reward in bilateral orbitofrontal and visual cortex. Men as compared to women also showed higher response to dollar loss in bilateral orbitofrontal cortex. Further, in whole-brain regressions, women relative to men demonstrated more significant modulation by SP in the neural responses to wins and larger wins, and the sex differences were confirmed by slope tests. CONCLUSIONS Together, men showed higher SR and neural sensitivity to both wins, large or small, and losses than women. Individual differences in SP were associated with diminished neural responses to wins and larger wins in women only. These findings highlight how men and women may differ in reward-related brain activations in the MIDT and add to the imaging literature of sex differences in cognitive and affective functions.
Collapse
Affiliation(s)
- Isha Dhingra
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA
| | - Sheng Zhang
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA
| | - Simon Zhornitsky
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA
| | - Wuyi Wang
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA
| | - Thang M Le
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA
| | - Chiang-Shan R Li
- Department of Psychiatry, Yale University School of Medicine, New Haven, CT, 06520, USA.
- Department of Neuroscience, Yale University School of Medicine, New Haven, CT, 06520, USA.
- Interdepartmental Neuroscience Program, Yale University, New Haven, CT, 06520, USA.
- Connecticut Mental Health Center S112, 34 Park Street, New Haven, CT, 06519-1109, USA.
| |
Collapse
|
46
|
Peciña M, Dombrovski AY, Price R, Karim HT. Understanding the Neurocomputational Mechanisms of Antidepressant Placebo Effects. JOURNAL OF PSYCHIATRY AND BRAIN SCIENCE 2021; 6:e210001. [PMID: 33732892 PMCID: PMC7963355 DOI: 10.20900/jpbs.20210001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
Abstract
Over the last two decades, neuroscientists have used antidepressant placebo probes to examine the biological mechanisms implicated in antidepressant placebo effects. However, findings from these studies have not yet elucidated a model-based theory that would explain the mechanism through which antidepressant expectancies evolve to induce persistent mood changes. Emerging evidence suggests that antidepressant placebo effects may be informed by models of reinforcement learning (RL). Such that an individual's expectation of improvement is updated with the arrival of new sensory evidence, by incorporating a reward prediction error (RPE), which signals the mismatch between the expected (expected value) and perceived improvement. Consistent with this framework, neuroimaging studies of antidepressant placebo effects have demonstrated placebo-induced μ-opioid activation and increased blood-oxygen-level dependent (BOLD) responses in regions tracking expected values (e.g., ventromedial prefrontal cortex (vmPFC)) and RPEs (e.g., ventral striatum (VS)). In this study, we will demonstrate the causal contribution of reward learning signals (expected values and RPEs) to antidepressant placebo effects by experimentally manipulating expected values using transcranial magnetic stimulation (TMS) targeting the vmPFC and μ-opioid striatal RPE signal using pharmacological approaches. We hypothesized that antidepressant placebo expectancies are represented in the vmPFC (expected value) and updated by means of μ-opioid-modulated striatal learning signal. In a 3 × 3 factorial double-blind design, we will randomize 120 antidepressant-free individuals with depressive symptoms to one of three between-subject opioid conditions: the μ-opioid agonist buprenorphine, the μ-opioid antagonist naltrexone, or an inert pill. Within each arm, individuals will be assigned to receive three within-subject counterbalanced forms of TMS targeting the vmPFC-intermittent Theta Burst Stimulation (TBS) expected to potentiate the vmPFC, continuous TBS expected to de-potentiate the vmPFC, or sham TBS. These experimental manipulations will be used to modulate trial-by-trial reward learning signals and related brain activity during the Antidepressant Placebo functional MRI (fMRI) Task to address the following aims: (1) investigate the relationship between reward learning signals within the vmPFC-VS circuit and antidepressant placebo effects; (2) examine the causal contribution of vmPFC expected value computations to antidepressant placebo effects; and (3) investigate the causal contribution of μ-opioid-modulated striatal RPEs to antidepressant placebo effects. The proposed study will be the first to investigate the causal contribution of μ-opioid-modulated vmPFC-VS learning signals to antidepressant placebo responses, paving the way for developing novel treatments modulating learning processes and objective means of quantifying and potentially reducing placebo effects during drug development. TRIAL REGISTRATION ClinicalTrials.gov Identifier: NCT04276259.
Collapse
Affiliation(s)
- Marta Peciña
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, 15213, USA
| | | | - Rebecca Price
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, 15213, USA
| | - Helmet T. Karim
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, 15213, USA
| |
Collapse
|
47
|
A ventral striatal prediction error signal in human fear extinction learning. Neuroimage 2021; 229:117709. [PMID: 33460800 DOI: 10.1016/j.neuroimage.2020.117709] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2020] [Revised: 11/30/2020] [Accepted: 12/19/2020] [Indexed: 12/21/2022] Open
Abstract
Animal studies have shown that the prediction error (PE) signal that drives fear extinction learning is encoded by phasic activity of midbrain dopamine (DA) neurons. Thus, the extinction PE resembles the appetitive PE that drives reward learning. In humans, fear extinction learning is less well understood. Using computational neuroimaging, a previous study from our group reported hemodynamic activity in the left ventral putamen, a subregion of the ventral striatum (VS), to correlate with a PE function derived from a formal associative learning model. The activity was modulated by genetic variation in a DA-related gene. To conceptually replicate and extend this finding, we here asked whether an extinction PE (EPE) signal in the left ventral putamen can also be observed when genotype information is not taken into account. Using an optimized experimental design for model estimation, we again observed EPE-related activity in the same striatal region, indicating that activation of this region is a feature of human extinction learning. We further observed significant EPE signals across wider parts of the VS as well as in frontal cortical areas. These results may suggest that the prediction errors during extinction learning are available to larger parts of the brain, as has also been observed in human neuroimaging studies of reward PE signaling. Conclusive evidence that the human EPE signal is of DAergic nature is still outstanding.
Collapse
|
48
|
Mas-Herrero E, Maini L, Sescousse G, Zatorre RJ. Common and distinct neural correlates of music and food-induced pleasure: A coordinate-based meta-analysis of neuroimaging studies. Neurosci Biobehav Rev 2021; 123:61-71. [PMID: 33440196 DOI: 10.1016/j.neubiorev.2020.12.008] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2020] [Revised: 11/11/2020] [Accepted: 12/12/2020] [Indexed: 12/31/2022]
Abstract
Neuroimaging studies have shown that, despite the abstractness of music, it may mimic biologically rewarding stimuli (e.g., food) in its ability to engage the brain's reward circuitry. However, due to the lack of research comparing music and other types of reward, it is unclear to what extent the recruitment of reward-related structures overlaps among domains. To achieve this goal, we performed a coordinate-based meta-analysis of 38 neuroimaging studies (703 subjects) comparing the brain responses specifically to music and food-induced pleasure. Both engaged a common set of brain regions, including the ventromedial prefrontal cortex, ventral striatum, and insula. Yet, comparative analyses indicated a partial dissociation in the engagement of the reward circuitry as a function of the type of reward, as well as additional reward type-specific activations in brain regions related to perception, sensory processing, and learning. These results support the idea that hedonic reactions rely on the engagement of a common reward network, yet through specific routes of access depending on the modality and nature of the reward.
Collapse
Affiliation(s)
- Ernest Mas-Herrero
- Cognition and Brain Plasticity Unit, Bellvitge Biomedical Research Institute, L'Hospitalet de Llobregat, 08907, Barcelona, Spain; Department of Cognition, Development and Education Psychology, University of Barcelona, 08035, Barcelona, Spain.
| | - Larissa Maini
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada; Michael G. DeGroote School of Medicine, McMaster University, Hamilton, Ontario, Canada
| | - Guillaume Sescousse
- Lyon Neuroscience Research Center - INSERM U1028 - CNRS UMR5292, PSYR2 Team, University of Lyon, Lyon, France
| | - Robert J Zatorre
- Montreal Neurological Institute, McGill University, Montreal, QC, Canada; International Laboratory for Brain, Music, and Sound Research (BRAMS), Montreal, QC, Canada.
| |
Collapse
|
49
|
Macoveanu J, Kjaerstad HL, Chase HW, Frangou S, Knudsen GM, Vinberg M, Kessing LV, Miskowiak KW. Abnormal prefrontal cortex processing of reward prediction errors in recently diagnosed patients with bipolar disorder and their unaffected relatives. Bipolar Disord 2020; 22:849-859. [PMID: 32301215 DOI: 10.1111/bdi.12915] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/17/2022]
Abstract
OBJECTIVE Bipolar disorder (BD) has been associated with abnormal reward functioning including pleasure-seeking and impulsivity. Here we sought to clarify whether these changes can be attributed to abnormalities in the neural processing of reward valuation or error prediction. Moreover, we tested whether abnormalities in these processes are associated with familial vulnerability to BD. METHODS We obtained functional magnetic resonance imaging data from patients with recently diagnosed BD (n = 85), their unaffected first-degree relatives (n = 44), and healthy control participants (n = 66) while they were performing a monetary card game. We used a region-of-interest approach to test for group differences in the activation of the midbrain, the ventral striatum, and the prefrontal cortex during reward valuation and error prediction. RESULTS Patients with BD showed decreased prediction error signal in ventrolateral prefrontal cortex and the unaffected relatives showed decreased prediction error signal in the supplementary motor area in comparison to healthy controls. There were no significant group differences in the activation of the ventral striatum during the task. In healthy controls, prediction error signal in dorsal anterior cingulate cortex correlated with an out-of-scanner measure of motor inhibition but this association was absent in patients and relatives. CONCLUSIONS The findings indicate that abnormal reward processing in BD is primarily related to deficits in the engagement of prefrontal regions involved in inhibitory control during error prediction. In contrast, deficient activation in supplementary motor cortex involved in planning of movement emerged as a familial vulnerability to BD.
Collapse
Affiliation(s)
- Julian Macoveanu
- Copenhagen Affective Disorder Research Center (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark
| | - Hanne L Kjaerstad
- Copenhagen Affective Disorder Research Center (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark
| | - Henry W Chase
- Department of Psychiatry, University of Pittsburgh School of Medicine, Pittsburgh, PA, USA
| | - Sophia Frangou
- Department of Psychiatry, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Gitte M Knudsen
- Neurobiology Research Unit, Copenhagen University Hospital, Rigshospitalet, Denmark.,Faculty of Health and Medical Science, University of Copenhagen, Copenhagen, Denmark
| | - Maj Vinberg
- Copenhagen Affective Disorder Research Center (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.,Faculty of Health and Medical Science, University of Copenhagen, Copenhagen, Denmark
| | - Lars V Kessing
- Copenhagen Affective Disorder Research Center (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.,Faculty of Health and Medical Science, University of Copenhagen, Copenhagen, Denmark
| | - Kamilla W Miskowiak
- Copenhagen Affective Disorder Research Center (CADIC), Psychiatric Centre Copenhagen, Copenhagen University Hospital Rigshospitalet, Copenhagen, Denmark.,Department of Psychology, University of Copenhagen, Copenhagen, Denmark
| |
Collapse
|
50
|
Dombrovski AY, Luna B, Hallquist MN. Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma. Nat Commun 2020; 11:5407. [PMID: 33106508 PMCID: PMC7589536 DOI: 10.1038/s41467-020-18864-0] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2020] [Accepted: 08/20/2020] [Indexed: 12/15/2022] Open
Abstract
When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.
Collapse
Affiliation(s)
| | - Beatriz Luna
- Department of Psychiatry, University of Pittsburgh, Pittsburgh, PA, 15213, USA
| | - Michael N Hallquist
- Department of Psychology, Penn State University, University Park, PA, 16801, USA.
- Department of Psychology and Neuroscience, University of North Carolina, Chapel Hill, NC, 27599-3270, USA.
| |
Collapse
|