1
|
Hahn A, Reed MB, Murgaš M, Vraka C, Klug S, Schmidt C, Godbersen GM, Eggerstorfer B, Gomola D, Silberbauer LR, Nics L, Philippe C, Hacker M, Lanzenberger R. Dynamics of human serotonin synthesis differentially link to reward anticipation and feedback. Mol Psychiatry 2024:10.1038/s41380-024-02696-1. [PMID: 39179904 DOI: 10.1038/s41380-024-02696-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/09/2024] [Revised: 07/26/2024] [Accepted: 08/12/2024] [Indexed: 08/26/2024]
Abstract
Serotonin (5-HT) plays an essential role in reward processing, however, the possibilities to investigate 5-HT action in humans during emotional stimulation are particularly limited. Here we demonstrate the feasibility of assessing reward-specific dynamics in 5-HT synthesis using functional PET (fPET), combining its molecular specificity with the high temporal resolution of blood oxygen level dependent (BOLD) fMRI. Sixteen healthy volunteers underwent simultaneous fPET/fMRI with the radioligand [11C]AMT, a substrate for tryptophan hydroxylase. During the scan, participants completed the monetary incentive delay task and arterial blood samples were acquired for quantifying 5-HT synthesis rates. BOLD fMRI was recorded as a proxy of neuronal activation, allowing differentiation of reward anticipation and feedback. Monetary gain and loss resulted in substantial increases in 5-HT synthesis in the ventral striatum (VStr, +21% from baseline) and the anterior insula (+41%). In the VStr, task-specific 5-HT synthesis was further correlated with BOLD signal changes during reward feedback (ρ = -0.65), but not anticipation. Conversely, 5-HT synthesis in the anterior insula correlated with BOLD reward anticipation (ρ = -0.61), but not feedback. In sum, we provide a robust tool to identify task-induced changes in 5-HT action in humans, linking the dynamics of 5-HT synthesis to distinct phases of reward processing in a regionally specific manner. Given the relevance of altered reward processing in psychiatric disorders such as addiction, depression and schizophrenia, our approach offers a tailored assessment of impaired 5-HT signaling during cognitive and emotional processing.
Collapse
Affiliation(s)
- Andreas Hahn
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria.
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria.
| | - Murray B Reed
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Matej Murgaš
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Chrysoula Vraka
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Sebastian Klug
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Clemens Schmidt
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Godber M Godbersen
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Benjamin Eggerstorfer
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - David Gomola
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Leo R Silberbauer
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria
| | - Lukas Nics
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Cécile Philippe
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Marcus Hacker
- Department of Biomedical Imaging and Image-guided Therapy, Division of Nuclear Medicine, Medical University of Vienna, Vienna, Austria
| | - Rupert Lanzenberger
- Department of Psychiatry and Psychotherapy, Medical University of Vienna, Vienna, Austria.
- Comprehensive Center for Clinical Neurosciences and Mental Health (C3NMH), Medical University of Vienna, Vienna, Austria.
| |
Collapse
|
2
|
Phalip A, Netser S, Wagner S. Understanding the neurobiology of social behavior through exploring brain-wide dynamics of neural activity. Neurosci Biobehav Rev 2024; 165:105856. [PMID: 39159735 DOI: 10.1016/j.neubiorev.2024.105856] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2024] [Revised: 08/11/2024] [Accepted: 08/14/2024] [Indexed: 08/21/2024]
Abstract
Social behavior is highly complex and adaptable. It can be divided into multiple temporal stages: detection, approach, and consummatory behavior. Each stage can be further divided into several cognitive and behavioral processes, such as perceiving social cues, evaluating the social and non-social contexts, and recognizing the internal/emotional state of others. Recent studies have identified numerous brain-wide circuits implicated in social behavior and suggested the existence of partially overlapping functional brain networks underlying various types of social and non-social behavior. However, understanding the brain-wide dynamics underlying social behavior remains challenging, and several brain-scale dynamics (macro-, meso-, and micro-scale levels) need to be integrated. Here, we suggest leveraging new tools and concepts to explore social brain networks and integrate those different levels. These include studying the expression of immediate-early genes throughout the entire brain to impartially define the structure of the neuronal networks involved in a given social behavior. Then, network dynamics could be investigated using electrode arrays or multi-channel fiber photometry. Finally, tools like high-density silicon probes and miniscopes can probe neural activity in specific areas and across neuronal populations at the single-cell level.
Collapse
Affiliation(s)
- Adèle Phalip
- Sagol Department of Neurobiology, Faculty of Natural Sciences, University of Haifa, Haifa, Israel.
| | - Shai Netser
- Sagol Department of Neurobiology, Faculty of Natural Sciences, University of Haifa, Haifa, Israel
| | - Shlomo Wagner
- Sagol Department of Neurobiology, Faculty of Natural Sciences, University of Haifa, Haifa, Israel
| |
Collapse
|
3
|
Finkelman T, Furman-Haran E, Aberg KC, Paz R, Tal A. Inhibitory mechanisms in the prefrontal-cortex differentially mediate Putamen activity during valence-based learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.07.29.605168. [PMID: 39131397 PMCID: PMC11312490 DOI: 10.1101/2024.07.29.605168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Indexed: 08/13/2024]
Abstract
Learning from appetitive and aversive stimuli is important for survival. It involves interactions between the prefrontal cortex and subcortical structures, with inhibition playing a crucial role. However, direct evidence for this in humans is limited. Here, we overcome the difficulty of measuring inhibition in the human brain and find that GABA, the main inhibitory neurotransmitter, affects how the dACC interacts with subcortical structures during appetitive and aversive learning differently. We used 7T magnetic resonance spectroscopy (MRS) to track GABA levels in the dACC alongside whole-brain fMRI scans while participants engaged in appetitive and aversive learning tasks. During appetitive learning, dACC GABA levels were negatively correlated with learning performance and BOLD activity measured from the dACC and the Putamen. While under aversive learning, dACC GABA concentration negatively correlated with the functional connectivity between the dACC and the Putamen. Our results show that inhibition in the dACC mediates appetitive and aversive learning in humans through distinct mechanisms.
Collapse
Affiliation(s)
- Tal Finkelman
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel
- Department of Chemical and Biological Physics, Weizmann Institute of Science, Rehovot, Israel
| | - Edna Furman-Haran
- Life Sciences Core Facilities, Weizmann Institute of Science, Rehovot, Israel
| | - Kristoffer C Aberg
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel
| | - Rony Paz
- Department of Brain Sciences, Weizmann Institute of Science, Rehovot, Israel
| | - Assaf Tal
- Department of Biomedical Engineering, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
4
|
Koch C, Zika O, Bruckner R, Schuck NW. Influence of surprise on reinforcement learning in younger and older adults. PLoS Comput Biol 2024; 20:e1012331. [PMID: 39141681 PMCID: PMC11346965 DOI: 10.1371/journal.pcbi.1012331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 08/26/2024] [Accepted: 07/16/2024] [Indexed: 08/16/2024] Open
Abstract
Surprise is a key component of many learning experiences, and yet its precise computational role, and how it changes with age, remain debated. One major challenge is that surprise often occurs jointly with other variables, such as uncertainty and outcome probability. To assess how humans learn from surprising events, and whether aging affects this process, we studied choices while participants learned from bandits with either Gaussian or bi-modal outcome distributions, which decoupled outcome probability, uncertainty, and surprise. A total of 102 participants (51 older, aged 50-73; 51 younger, 19-30 years) chose between three bandits, one of which had a bimodal outcome distribution. Behavioral analyses showed that both age-groups learned the average of the bimodal bandit less well. A trial-by-trial analysis indicated that participants performed choice reversals immediately following large absolute prediction errors, consistent with heightened sensitivity to surprise. This effect was stronger in older adults. Computational models indicated that learning rates in younger as well as older adults were influenced by surprise, rather than uncertainty, but also suggested large interindividual variability in the process underlying learning in our task. Our work bridges between behavioral economics research that has focused on how outcomes with low probability affect choice in older adults, and reinforcement learning work that has investigated age differences in the effects of uncertainty and suggests that older adults overly adapt to surprising events, even when accounting for probability and uncertainty effects.
Collapse
Affiliation(s)
- Christoph Koch
- Max Planck Institute for Human Development, Berlin, Germany
- Institute of Psychology, Universität Hamburg, Hamburg, Germany
| | - Ondrej Zika
- Max Planck Institute for Human Development, Berlin, Germany
- Max Planck UCL Centre for Computational Psychiatry and Aging Research, Berlin, Germany, and London, United Kingdom
| | - Rasmus Bruckner
- Max Planck Institute for Human Development, Berlin, Germany
- Department of Education and Psychology, Freie Universität Berlin, Berlin, Germany
| | - Nicolas W. Schuck
- Max Planck Institute for Human Development, Berlin, Germany
- Institute of Psychology, Universität Hamburg, Hamburg, Germany
- Max Planck UCL Centre for Computational Psychiatry and Aging Research, Berlin, Germany, and London, United Kingdom
| |
Collapse
|
5
|
Lee RS, Sagiv Y, Engelhard B, Witten IB, Daw ND. A feature-specific prediction error model explains dopaminergic heterogeneity. Nat Neurosci 2024; 27:1574-1586. [PMID: 38961229 DOI: 10.1038/s41593-024-01689-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2022] [Accepted: 05/22/2024] [Indexed: 07/05/2024]
Abstract
The hypothesis that midbrain dopamine (DA) neurons broadcast a reward prediction error (RPE) is among the great successes of computational neuroscience. However, recent results contradict a core aspect of this theory: specifically that the neurons convey a scalar, homogeneous signal. While the predominant family of extensions to the RPE model replicates the classic model in multiple parallel circuits, we argue that these models are ill suited to explain reports of heterogeneity in task variable encoding across DA neurons. Instead, we introduce a complementary 'feature-specific RPE' model, positing that individual ventral tegmental area DA neurons report RPEs for different aspects of an animal's moment-to-moment situation. Further, we show how our framework can be extended to explain patterns of heterogeneity in action responses reported among substantia nigra pars compacta DA neurons. This theory reconciles new observations of DA heterogeneity with classic ideas about RPE coding while also providing a new perspective of how the brain performs reinforcement learning in high-dimensional environments.
Collapse
Affiliation(s)
- Rachel S Lee
- Princeton Neuroscience Institute, Princeton, NJ, USA
| | - Yotam Sagiv
- Princeton Neuroscience Institute, Princeton, NJ, USA
| | - Ben Engelhard
- Princeton Neuroscience Institute, Princeton, NJ, USA
| | | | - Nathaniel D Daw
- Princeton Neuroscience Institute, Princeton, NJ, USA.
- Department of Psychology, Princeton University, Princeton, NJ, USA.
| |
Collapse
|
6
|
Feng YY, Bromberg-Martin ES, Monosov IE. Dorsal raphe neurons integrate the values of reward amount, delay, and uncertainty in multi-attribute decision-making. Cell Rep 2024; 43:114341. [PMID: 38878290 DOI: 10.1016/j.celrep.2024.114341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Revised: 03/27/2024] [Accepted: 05/23/2024] [Indexed: 06/25/2024] Open
Abstract
The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when confronting reward uncertainty. However, it has been unclear whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider these attributes to make a choice. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes, and this population tended to integrate the attributes in a manner that reflected monkeys' preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how the DRN participates in value computations, guiding theories about the role of the DRN in decision-making and psychiatric disease.
Collapse
Affiliation(s)
- Yang-Yang Feng
- Department of Neuroscience, Washington University School of Medicine, St. Louis, MO, USA; Department of Biomedical Engineering, Washington University, St. Louis, MO, USA
| | | | - Ilya E Monosov
- Department of Neuroscience, Washington University School of Medicine, St. Louis, MO, USA; Department of Biomedical Engineering, Washington University, St. Louis, MO, USA; Washington University Pain Center, Washington University, St. Louis, MO, USA; Department of Neurosurgery, Washington University, St. Louis, MO, USA; Department of Electrical Engineering, Washington University, St. Louis, MO, USA.
| |
Collapse
|
7
|
Macías-Cortés E. Understanding Why Homeopathic Medicines are Used for Menopause: Searching for Insights into Neuroendocrine Features. HOMEOPATHY 2024; 113:54-66. [PMID: 37399836 DOI: 10.1055/s-0043-1769734] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/05/2023]
Abstract
BACKGROUND Menopause is a physiological event that marks the end of a woman's reproductive stage in life. Vasomotor symptoms and changes in mood are among its most important effects. Homeopathy has been used for many years in treating menopausal complaints, though clinical and pre-clinical research in this field is limited. Homeopathy often bases its prescription on neuropsychiatric symptoms, but it is unknown if homeopathic medicines (HMs) exert a neuroendocrine effect that causes an improvement in vasomotor symptoms and mood during menopause. OBJECTIVES The study's objectives were to address the pathophysiological changes of menopause that could help in the understanding of the possible effect of HMs at a neuroendocrine level, to review the current evidence for two of the most frequently prescribed HMs for menopause (Lachesis mutus and Sepia officinalis), and to discuss the future directions of research in this field. METHODS An extensive literature search for the pathophysiologic events of menopause and depression, as well as for the current evidence for HMs in menopause and depression, was performed. RESULTS Neuroendocrine changes are involved in the pathophysiology of vasomotor symptoms and changes in mood during menopause. Gonadal hormones modulate neurotransmitter systems. Both play a role in mood disorders and temperature regulation. It has been demonstrated that Gelsemium sempervirens, Ignatia amara and Chamomilla matricaria exert anxiolytic effects in rodent models. Lachesis mutus and Sepia officinalis are frequently prescribed for important neuropsychiatric and vasomotor symptoms. Dopamine, a neurotransmitter involved in mood, is among the constituents of the ink of the common cuttlefish, Sepia officinalis. CONCLUSION Based on all the pathophysiologic events of menopause and the improvement in menopausal complaints that certain HMs show in daily practice, these medicines might have a direct or indirect neuroendocrine effect in the body, possibly triggered via an as-yet unidentified biological mechanism. Many unanswered questions in this field require further pre-clinical and clinical research.
Collapse
Affiliation(s)
- Emma Macías-Cortés
- Outpatient Homeopathy Service, Hospital Juárez de México, Secretaría de Salud, Mexico City, Mexico
| |
Collapse
|
8
|
Illera JC, Jimenez-Blanco F, Centenera L, Gil-Cabrera F, Crespo B, Lopez PR, Silvan G, Caceres S. Addressing Combative Behaviour in Spanish Bulls by Measuring Hormonal Indicators. Vet Sci 2024; 11:182. [PMID: 38668449 PMCID: PMC11053816 DOI: 10.3390/vetsci11040182] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 02/02/2024] [Accepted: 04/19/2024] [Indexed: 04/29/2024] Open
Abstract
The fighting bull is characterised by its natural aggressiveness, but the physiological mechanisms that underlie its aggressive behaviour are poorly studied. This study determines the hormonal component of aggressiveness in fighting bulls by analysing their behaviour during a fight and correlating it to their serotonin, dopamine and testosterone levels. We also determine whether aggressive behaviour can be estimated in calves. Using 195 animals, samples were obtained when the animals were calves and after 5 years. Aggressiveness scores were obtained by an observational method during bullfights, and serotonin, dopamine and testosterone levels were determined in all animals using validated enzyme immunoassay kits. The results revealed a strong correlation of serotonin and dopamine levels with aggressiveness scores in bulls during fights, but no correlation was found with respect to testosterone. These correlations led to established cut-off point and linear regression curves to obtain expected aggressiveness scores for calves at shoeing. There were no significant differences between the expected scores obtained in calves and the observed scores in bulls. Therefore, this study demonstrates that hormone determination in calves may be a great indicator of combativeness in bulls and can reliably be used in the selection of fighting bulls.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Gema Silvan
- Department Animal Physiology, Veterinary Medicine School, Complutense University of Madrid (UCM), 28040 Madrid, Spain; (J.C.I.); (F.J.-B.); (L.C.); (F.G.-C.); (B.C.); (P.R.L.); (S.C.)
| | | |
Collapse
|
9
|
Gilmour W, Mackenzie G, Feile M, Tayler-Grint L, Suveges S, Macfarlane JA, Macleod AD, Marshall V, Grunwald IQ, Steele JD, Gilbertson T. Impaired value-based decision-making in Parkinson's disease apathy. Brain 2024; 147:1362-1376. [PMID: 38305691 PMCID: PMC10994558 DOI: 10.1093/brain/awae025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2023] [Revised: 12/07/2023] [Accepted: 01/13/2024] [Indexed: 02/03/2024] Open
Abstract
Apathy is a common and disabling complication of Parkinson's disease characterized by reduced goal-directed behaviour. Several studies have reported dysfunction within prefrontal cortical regions and projections from brainstem nuclei whose neuromodulators include dopamine, serotonin and noradrenaline. Work in animal and human neuroscience have confirmed contributions of these neuromodulators on aspects of motivated decision-making. Specifically, these neuromodulators have overlapping contributions to encoding the value of decisions, and influence whether to explore alternative courses of action or persist in an existing strategy to achieve a rewarding goal. Building upon this work, we hypothesized that apathy in Parkinson's disease should be associated with an impairment in value-based learning. Using a four-armed restless bandit reinforcement learning task, we studied decision-making in 75 volunteers; 53 patients with Parkinson's disease, with and without clinical apathy, and 22 age-matched healthy control subjects. Patients with apathy exhibited impaired ability to choose the highest value bandit. Task performance predicted an individual patient's apathy severity measured using the Lille Apathy Rating Scale (R = -0.46, P < 0.001). Computational modelling of the patient's choices confirmed the apathy group made decisions that were indifferent to the learnt value of the options, consistent with previous reports of reward insensitivity. Further analysis demonstrated a shift away from exploiting the highest value option and a reduction in perseveration, which also correlated with apathy scores (R = -0.5, P < 0.001). We went on to acquire functional MRI in 59 volunteers; a group of 19 patients with and 20 without apathy and 20 age-matched controls performing the Restless Bandit Task. Analysis of the functional MRI signal at the point of reward feedback confirmed diminished signal within ventromedial prefrontal cortex in Parkinson's disease, which was more marked in apathy, but not predictive of their individual apathy severity. Using a model-based categorization of choice type, decisions to explore lower value bandits in the apathy group activated prefrontal cortex to a similar degree to the age-matched controls. In contrast, Parkinson's patients without apathy demonstrated significantly increased activation across a distributed thalamo-cortical network. Enhanced activity in the thalamus predicted individual apathy severity across both patient groups and exhibited functional connectivity with dorsal anterior cingulate cortex and anterior insula. Given that task performance in patients without apathy was no different to the age-matched control subjects, we interpret the recruitment of this network as a possible compensatory mechanism, which compensates against symptomatic manifestation of apathy in Parkinson's disease.
Collapse
Affiliation(s)
- William Gilmour
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| | - Graeme Mackenzie
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| | - Mathias Feile
- Rehabilitation Psychiatry, Murray Royal Hospital, Perth PH2 7BH, UK
| | | | - Szabolcs Suveges
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - Jennifer A Macfarlane
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Medical Physics, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
- SINAPSE, University of Glasgow, Imaging Centre of Excellence, Level 2, Queen Elizabeth University Hospital, Glasgow G51 4TF, Scotland, UK
| | - Angus D Macleod
- Institute of Applied Health Sciences, School of Medicine, University of Aberdeen, Foresterhill, Aberdeen AB24 2ZD, UK
- Department of Neurology, Aberdeen Royal Infirmary, Foresterhill, Aberdeen AB24 2ZD, UK
| | - Vicky Marshall
- Institute of Neurological Sciences, Queen Elizabeth University Hospital, Glasgow G51 4TF, UK
| | - Iris Q Grunwald
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - J Douglas Steele
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
| | - Tom Gilbertson
- Division of Imaging Science and Technology, Ninewells Hospital and Medical School, University of Dundee, Dundee DD1 9SY, UK
- Department of Neurology, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK
| |
Collapse
|
10
|
Carvalheiro J, Philiastides MG. Distinct spatiotemporal brainstem pathways of outcome valence during reward- and punishment-based learning. Cell Rep 2023; 42:113589. [PMID: 38100353 DOI: 10.1016/j.celrep.2023.113589] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 10/05/2023] [Accepted: 11/30/2023] [Indexed: 12/17/2023] Open
Abstract
Learning to seek rewards and avoid punishments, based on positive and negative choice outcomes, is essential for human survival. Yet, the neural underpinnings of outcome valence in the human brainstem and the extent to which they differ in reward and punishment learning contexts remain largely elusive. Here, using simultaneously acquired electroencephalography and functional magnetic resonance imaging data, we show that during reward learning the substantia nigra (SN)/ventral tegmental area (VTA) and locus coeruleus are initially activated following negative outcomes, while the VTA subsequently re-engages exhibiting greater responses for positive than negative outcomes, consistent with an early arousal/avoidance response and a later value-updating process, respectively. During punishment learning, we show that distinct raphe nucleus and SN subregions are activated only by negative outcomes with a sustained post-outcome activity across time, supporting the involvement of these brainstem subregions in avoidance behavior. Finally, we demonstrate that the coupling of these brainstem structures with other subcortical and cortical areas helps to shape participants' serial choice behavior in each context.
Collapse
Affiliation(s)
- Joana Carvalheiro
- School of Psychology and Neuroscience, University of Glasgow, Glasgow G12 8QB, UK; Centre for Cognitive Neuroimaging, University of Glasgow, Glasgow G12 8QB, UK.
| | - Marios G Philiastides
- School of Psychology and Neuroscience, University of Glasgow, Glasgow G12 8QB, UK; Centre for Cognitive Neuroimaging, University of Glasgow, Glasgow G12 8QB, UK.
| |
Collapse
|
11
|
Friedenberger Z, Harkin E, Tóth K, Naud R. Silences, spikes and bursts: Three-part knot of the neural code. J Physiol 2023; 601:5165-5193. [PMID: 37889516 DOI: 10.1113/jp281510] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 09/28/2023] [Indexed: 10/28/2023] Open
Abstract
When a neuron breaks silence, it can emit action potentials in a number of patterns. Some responses are so sudden and intense that electrophysiologists felt the need to single them out, labelling action potentials emitted at a particularly high frequency with a metonym - bursts. Is there more to bursts than a figure of speech? After all, sudden bouts of high-frequency firing are expected to occur whenever inputs surge. The burst coding hypothesis advances that the neural code has three syllables: silences, spikes and bursts. We review evidence supporting this ternary code in terms of devoted mechanisms for burst generation, synaptic transmission and synaptic plasticity. We also review the learning and attention theories for which such a triad is beneficial.
Collapse
Affiliation(s)
- Zachary Friedenberger
- Brain and Mind Institute, Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Centre for Neural Dynamics and Artifical Intelligence, Department of Physics, University of Ottawa, Ottawa, Ontario, Ottawa
| | - Emerson Harkin
- Brain and Mind Institute, Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Katalin Tóth
- Brain and Mind Institute, Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Ontario, Canada
| | - Richard Naud
- Brain and Mind Institute, Department of Cellular and Molecular Medicine, University of Ottawa, Ottawa, Ontario, Canada
- Centre for Neural Dynamics and Artifical Intelligence, Department of Physics, University of Ottawa, Ottawa, Ontario, Ottawa
| |
Collapse
|
12
|
Sands LP, Jiang A, Liebenow B, DiMarco E, Laxton AW, Tatter SB, Montague PR, Kishida KT. Subsecond fluctuations in extracellular dopamine encode reward and punishment prediction errors in humans. SCIENCE ADVANCES 2023; 9:eadi4927. [PMID: 38039368 PMCID: PMC10691773 DOI: 10.1126/sciadv.adi4927] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Accepted: 11/03/2023] [Indexed: 12/03/2023]
Abstract
In the mammalian brain, midbrain dopamine neuron activity is hypothesized to encode reward prediction errors that promote learning and guide behavior by causing rapid changes in dopamine levels in target brain regions. This hypothesis (and alternatives regarding dopamine's role in punishment-learning) has limited direct evidence in humans. We report intracranial, subsecond measurements of dopamine release in human striatum measured, while volunteers (i.e., patients undergoing deep brain stimulation surgery) performed a probabilistic reward and punishment learning choice task designed to test whether dopamine release encodes only reward prediction errors or whether dopamine release may also encode adaptive punishment learning signals. Results demonstrate that extracellular dopamine levels can encode both reward and punishment prediction errors within distinct time intervals via independent valence-specific pathways in the human brain.
Collapse
Affiliation(s)
- L. Paul Sands
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
- Department of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - Angela Jiang
- Department of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - Brittany Liebenow
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
- Department of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - Emily DiMarco
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
- Department of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - Adrian W. Laxton
- Department of Neurosurgery, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - Stephen B. Tatter
- Department of Neurosurgery, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| | - P. Read Montague
- Wellcome Centre for Human Neuroimaging, University College London, WC1N 3BG London, UK
- Fralin Biomedical Research Institute, Virginia Tech, Roanoke, VA 24016, USA
- Department of Physics, Virginia Tech, Blacksburg, VA 24061, USA
| | - Kenneth T. Kishida
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
- Department of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
- Department of Neurosurgery, Wake Forest School of Medicine, Winston-Salem, NC 27101, USA
| |
Collapse
|
13
|
Yin C, Li B, Gao T. Differential effects of reward and punishment on reinforcement-based motor learning and generalization. J Neurophysiol 2023; 130:1150-1161. [PMID: 37791387 DOI: 10.1152/jn.00242.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2023] [Revised: 09/13/2023] [Accepted: 09/28/2023] [Indexed: 10/05/2023] Open
Abstract
Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.
Collapse
Affiliation(s)
- Cong Yin
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| | - Biao Li
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| | - Tian Gao
- School of Kinesiology and Health, Capital University of Physical Education and Sports, Beijing, People's Republic of China
| |
Collapse
|
14
|
Traktirov DS, Nazarov IR, Artemova VS, Gainetdinov RR, Pestereva NS, Karpenko MN. Alterations in Serotonin Neurotransmission in Hyperdopaminergic Rats Lacking the Dopamine Transporter. Biomedicines 2023; 11:2881. [PMID: 38001881 PMCID: PMC10669523 DOI: 10.3390/biomedicines11112881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2023] [Revised: 10/17/2023] [Accepted: 10/19/2023] [Indexed: 11/26/2023] Open
Abstract
Biogenic amines dopamine (DA) and serotonin (5-HT) are among the most significant monoaminergic neurotransmitters in the central nervous system (CNS). Separately, the physiological roles of DA and 5-HT have been studied in detail, and progress has been made in understanding their roles in normal and various pathological conditions (Parkinson's disease, schizophrenia, addiction, depression, etc.). In this article we showed that knockout of the gene encoding DAT leads not only to a profound dysregulation of dopamine neurotransmission in the striatum but also in the midbrain, prefrontal cortex, hippocampus, medulla oblongata and spinal cord. Furthermore, significant changes were observed in the production of mRNA of enzymes of monoamine metabolism, as well as to a notable alteration in the tissue level of serotonin, most clearly manifested in the cerebellum and the spinal cord. The observed region-specific changes in the tissue levels of serotonin and in the expression of dopamine and serotonergic metabolism enzymes in rats with an excess of dopamine can indicate important consequences for the pharmacotherapy of drugs that modulate the dopaminergic system. The drugs that affect the dopaminergic system could potently affect the serotonergic system, and this fact is important to consider when predicting their possible therapeutic or side effects.
Collapse
Affiliation(s)
- Dmitrii S. Traktirov
- Department of Physiology (Pavlov’s), Institute of Experimental Medicine, 197022 St. Petersburg, Russia (M.N.K.)
| | - Ilya R. Nazarov
- Faculty of Biology, St. Petersburg State University, 199034 St. Petersburg, Russia
| | - Valeria S. Artemova
- Department of Physiology (Pavlov’s), Institute of Experimental Medicine, 197022 St. Petersburg, Russia (M.N.K.)
- Institute of Biomedical Systems and Biotechnologies, Peter the Great St. Petersburg Polytechnic University, 195251 St. Petersburg, Russia
| | - Raul R. Gainetdinov
- Institute of Translational Biomedicine, St. Petersburg University Hospital, St. Petersburg State University, 199034 St. Petersburg, Russia;
| | - Nina S. Pestereva
- Department of Physiology (Pavlov’s), Institute of Experimental Medicine, 197022 St. Petersburg, Russia (M.N.K.)
| | - Marina N. Karpenko
- Department of Physiology (Pavlov’s), Institute of Experimental Medicine, 197022 St. Petersburg, Russia (M.N.K.)
- Institute of Biomedical Systems and Biotechnologies, Peter the Great St. Petersburg Polytechnic University, 195251 St. Petersburg, Russia
| |
Collapse
|
15
|
Lloyd K, Dayan P. Reframing dopamine: A controlled controller at the limbic-motor interface. PLoS Comput Biol 2023; 19:e1011569. [PMID: 37847681 PMCID: PMC10610519 DOI: 10.1371/journal.pcbi.1011569] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 10/27/2023] [Accepted: 10/03/2023] [Indexed: 10/19/2023] Open
Abstract
Pavlovian influences notoriously interfere with operant behaviour. Evidence suggests this interference sometimes coincides with the release of the neuromodulator dopamine in the nucleus accumbens. Suppressing such interference is one of the targets of cognitive control. Here, using the examples of active avoidance and omission behaviour, we examine the possibility that direct manipulation of the dopamine signal is an instrument of control itself. In particular, when instrumental and Pavlovian influences come into conflict, dopamine levels might be affected by the controlled deployment of a reframing mechanism that recasts the prospect of possible punishment as an opportunity to approach safety, and the prospect of future reward in terms of a possible loss of that reward. We operationalize this reframing mechanism and fit the resulting model to rodent behaviour from two paradigmatic experiments in which accumbens dopamine release was also measured. We show that in addition to matching animals' behaviour, the model predicts dopamine transients that capture some key features of observed dopamine release at the time of discriminative cues, supporting the idea that modulation of this neuromodulator is amongst the repertoire of cognitive control strategies.
Collapse
Affiliation(s)
- Kevin Lloyd
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany
- University of Tübingen, Tübingen, Germany
| |
Collapse
|
16
|
Feng YY, Bromberg-Martin ES, Monosov IE. Dorsal raphe neurons signal integrated value during multi-attribute decision-making. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.08.17.553745. [PMID: 37662243 PMCID: PMC10473596 DOI: 10.1101/2023.08.17.553745] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/05/2023]
Abstract
The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when grappling with reward uncertainty. However, whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider all these attributes to make a choice, is unclear. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes. Remarkably, these neurons commonly integrated offer attributes in a manner that reflected monkeys' overall preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how DRN participates in integrated value computations, guiding theories of DRN in decision-making and psychiatric disease.
Collapse
Affiliation(s)
- Yang-Yang Feng
- Department of Neuroscience, Washington University School of Medicine, St. Louis, Missouri, USA
- Department of Biomedical Engineering, Washington University, St. Louis, Missouri, USA
| | | | - Ilya E. Monosov
- Department of Neuroscience, Washington University School of Medicine, St. Louis, Missouri, USA
- Department of Biomedical Engineering, Washington University, St. Louis, Missouri, USA
- Washington University Pain Center, Washington University, St. Louis, Missouri, USA
- Department of Neurosurgery, Washington University, St. Louis, Missouri, USA
- Department of Electrical Engineering, Washington University, St. Louis, Missouri, USA
| |
Collapse
|
17
|
Michely J, Martin IM, Dolan RJ, Hauser TU. Boosting Serotonin Increases Information Gathering by Reducing Subjective Cognitive Costs. J Neurosci 2023; 43:5848-5855. [PMID: 37524494 PMCID: PMC10423044 DOI: 10.1523/jneurosci.1416-22.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2022] [Revised: 06/01/2023] [Accepted: 06/06/2023] [Indexed: 08/02/2023] Open
Abstract
Serotonin is implicated in the valuation of aversive costs, such as delay or physical effort. However, its role in governing sensitivity to cognitive effort, for example, deliberation costs during information gathering, is unclear. We show that treatment with a serotonergic antidepressant in healthy human individuals of either sex enhances a willingness to gather information when trying to maximize reward. Using computational modeling, we show this arises from a diminished sensitivity to subjective deliberation costs during the sampling process. This result is consistent with the notion that serotonin alleviates sensitivity to aversive costs in a domain-general fashion, with implications for its potential contribution to a positive impact on motivational deficits in psychiatric disorders.SIGNIFICANCE STATEMENT Gathering information about the world is essential for successfully navigating it. However, sampling information is costly, and we need to balance between gathering too little and too much information. The neurocomputational mechanisms underlying this arbitration between a putative gain, such as reward, and the associated costs, such as allocation of cognitive resources, remain unclear. In this study, we show that week-long daily treatment with a serotonergic antidepressant enhances a willingness to gather information when trying to maximize reward. Computational modeling indicates this arises from a reduced perception of aversive costs, rendering information gathering less cognitively effortful. This finding points to a candidate mechanism by which serotonergic treatment might help alleviate motivational deficits in a range of mental illnesses.
Collapse
Affiliation(s)
- Jochen Michely
- Department of Psychiatry and Neurosciences, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Berlin, 10117 Germany
- Berlin Institute of Health at Charité-Universitätsmedizin Berlin, BIH Charité Clinician Scientist Program, Berlin, 10117 Germany
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, WC1B 5EH, United Kingdom
- Wellcome Centre for Human Neuroimaging, University College London, London, WC1N 3BG, United Kingdom
| | - Ingrid M Martin
- Wellcome Centre for Human Neuroimaging, University College London, London, WC1N 3BG, United Kingdom
- Institute of Cognitive Neuroscience, University College London, London, WC1N 3AZ, United Kingdom
| | - Raymond J Dolan
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, WC1B 5EH, United Kingdom
- Wellcome Centre for Human Neuroimaging, University College London, London, WC1N 3BG, United Kingdom
| | - Tobias U Hauser
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, WC1B 5EH, United Kingdom
- Wellcome Centre for Human Neuroimaging, University College London, London, WC1N 3BG, United Kingdom
- Department of Psychiatry and Psychotherapy, Medical School and University Hospital, Eberhard Karls University of Tübingen, 72076 Tübingen, Germany
- German Center for Mental Health (DZPG)
| |
Collapse
|
18
|
Haines N, Kvam PD, Turner BM. Explaining the description-experience gap in risky decision-making: learning and memory retention during experience as causal mechanisms. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2023:10.3758/s13415-023-01099-z. [PMID: 37291409 DOI: 10.3758/s13415-023-01099-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/28/2023] [Indexed: 06/10/2023]
Abstract
When making decisions based on probabilistic outcomes, people guide their behavior using knowledge gathered through both indirect descriptions and direct experience. Paradoxically, how people obtain information significantly impacts apparent preferences. A ubiquitous example is the description-experience gap: individuals seemingly overweight low probability events when probabilities are described yet underweight them when probabilities must be experienced firsthand. A leading explanation for this fundamental gap in decision-making is that probabilities are weighted differently when learned through description relative to experience, yet a formal theoretical account of the mechanism responsible for such weighting differences remains elusive. We demonstrate how various learning and memory retention models incorporating neuroscientifically motivated learning mechanisms can explain why probability weighting and valuation parameters often are found to vary across description and experience. In a simulation study, we show how learning through experience can lead to systematically biased estimates of probability weighting when using a traditional cumulative prospect theory model. We then use hierarchical Bayesian modeling and Bayesian model comparison to show how various learning and memory retention models capture participants' behavior over and above changes in outcome valuation and probability weighting, accounting for description and experience-based decisions in a within-subject experiment. We conclude with a discussion of how substantive models of psychological processes can lead to insights that heuristic statistical models fail to capture.
Collapse
Affiliation(s)
- Nathaniel Haines
- The Ohio State University, Columbus, OH, USA.
- Bayesian Beginnings LLC, Columbus, USA.
| | | | | |
Collapse
|
19
|
Steding J, Ritschel F, Boehm I, Geisler D, King JA, Roessner V, Smolka MN, Zepf FD, Ehrlich S. The effects of acute tryptophan depletion on instrumental reward learning in anorexia nervosa - an fMRI study. Psychol Med 2023; 53:3426-3436. [PMID: 35343412 PMCID: PMC10277771 DOI: 10.1017/s0033291721005493] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Revised: 10/15/2021] [Accepted: 12/20/2021] [Indexed: 11/07/2022]
Abstract
BACKGROUND The serotonin (5-HT) hypothesis of anorexia nervosa (AN) posits that individuals predisposed toward or recovered from AN (recAN) have a central nervous hyperserotonergic state and therefore restrict food intake as a means to reduce 5-HT availability (via diminished tryptophan-derived precursor supply) and alleviate associated negative mood states. Importantly, the 5-HT system has also been generally implicated in reward processing, which has also been shown to be altered in AN. METHODS In this double-blind crossover study, 22 individuals recAN and 25 healthy control participants (HC) underwent functional magnetic resonance imaging (fMRI) while performing an established instrumental reward learning paradigm during acute tryptophan depletion (ATD; a dietary intervention that lowers central nervous 5-HT availability) as well as a sham depletion. RESULTS On a behavioral level, the main effects of reward and ATD were evident, but no group differences were found. fMRI analyses revealed a group × ATD × reward level interaction in the ventral anterior insula during reward anticipation as well as in the medial orbitofrontal cortex during reward consumption. DISCUSSION The precise pattern of results is suggestive of a 'normalization' of reward-related neural responses during ATD in recAN compared to HC. Our results lend further evidence to the 5-HT hypothesis of AN. Decreasing central nervous 5-HT synthesis and availability during ATD and possibly also by dieting may be a means to normalize 5-HT availability and associated brain processes.
Collapse
Affiliation(s)
- Julius Steding
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
| | - Franziska Ritschel
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
| | - Ilka Boehm
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
| | - Daniel Geisler
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
| | - Joseph A. King
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
| | - Veit Roessner
- Department of Child and Adolescent Psychiatry, Faculty of Medicine, University Hospital C. G. Carus, Technische Universität Dresden, Dresden, Germany
| | - Michael N. Smolka
- Department of Psychiatry and Neuroimaging Center, Technische Universität Dresden, Dresden, Germany
| | - Florian Daniel Zepf
- Department of Child and Adolescent Psychiatry, Psychosomatic Medicine and Psychotherapy, Jena University Hospital, Friedrich Schiller University, Jena, Germany
| | - Stefan Ehrlich
- Translational Developmental Neuroscience Section, Division of Psychological and Social Medicine and Developmental Neurosciences, Faculty of Medicine, TU Dresden, Dresden, Germany
- Eating Disorder Treatment and Research Center, Department of Child and Adolescent Psychiatry, Faculty of Medicine, Technische Universität Dresden, Dresden, Germany
| |
Collapse
|
20
|
Saaiman D, Brand L, de Brouwe G, Janse van Rensburg H, Terre'Blanche G, Legoabe L, Krahe T, Wolmarans D. Striatal adenosine A 2A receptor involvement in normal and large nest building deer mice: perspectives on compulsivity and anxiety. Behav Brain Res 2023; 449:114492. [PMID: 37172739 DOI: 10.1016/j.bbr.2023.114492] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2023] [Revised: 04/25/2023] [Accepted: 05/09/2023] [Indexed: 05/15/2023]
Abstract
Obsessive-compulsive disorder (OCD) is characterized by recurring obsessive thoughts and repetitive behaviors that are often associated with anxiety and perturbations in cortico-striatal signaling. Given the suboptimal response of OCD to current serotonergic interventions, there is a need to better understand the psychobiological mechanisms that may underlie the disorder. In this regard, investigations into adenosinergic processes might be fruitful. Indeed, adenosine modulates both anxiety- and motor behavioral output. Thus, we aimed to explore the potential associations between compulsive-like large nest building (LNB) behavior in deer mice, anxiety and adenosinergic processes. From an initial pool of 120 adult deer mice, 34 normal nest building (NNB)- and 32 LNB-expressing mice of both sexes were selected and exposed to either a normal water (wCTRL) or vehicle control (vCTRL), lorazepam (LOR) or istradefylline (ISTRA) for 7- (LOR) or 28 days after which nesting assessment was repeated and animals screened for anxiety-like behavior in an anxiogenic open field. Mice were then euthanized, the striatal tissue removed on ice and the adenosine A2A receptor expression quantified. Our findings indicate that NNB and LNB behavior are not distinctly associated with measures of generalized anxiety and that ISTRA-induced changes in nesting expression are dissociated from changes in anxiety scores. Further, data from this investigation show that nesting in deer mice is directly related to striatal adenosine signaling, and that LNB is founded upon a lower degree of adenosinergic A2A stimulation.
Collapse
Affiliation(s)
- D Saaiman
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmacology, North-West University, Potchefstroom, South Africa
| | - L Brand
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmacology, North-West University, Potchefstroom, South Africa
| | - G de Brouwe
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmacology, North-West University, Potchefstroom, South Africa
| | - H Janse van Rensburg
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmaceutical Chemistry, North-West University, Potchefstroom, South Africa
| | - G Terre'Blanche
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmaceutical Chemistry, North-West University, Potchefstroom, South Africa
| | - L Legoabe
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmaceutical Chemistry, North-West University, Potchefstroom, South Africa
| | - T Krahe
- Department of Psychology, Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Rio de Janeiro, Brazil
| | - D Wolmarans
- Center of Excellence for Pharmaceutical Sciences, Department of Pharmacology, North-West University, Potchefstroom, South Africa.
| |
Collapse
|
21
|
Seok JW, Soltis-Vaughan B, Lew BJ, Ahmad A, Blair RJR, Hwang S. Psychopharmacological treatment of disruptive behavior in youths: systematic review and network meta-analysis. Sci Rep 2023; 13:6921. [PMID: 37117632 PMCID: PMC10147946 DOI: 10.1038/s41598-023-33979-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Accepted: 04/21/2023] [Indexed: 04/30/2023] Open
Abstract
To conduct a systematic review of the comparative efficacy of various psychotropic medications for the treatment of disruptive behavior (DBs) in youths. To this aim, we systematically reviewed randomized clinical trials (RCTs) of various psychotropic medications targeting symptoms of DBs and applied network meta-analysis to investigate their relative efficacy. Fifty-five RCTs meeting the inclusion criteria were selected. To predict and interpret relative treatment efficacy, we compared the efficacy of various psychotropic medications prescribed for DB symptoms based on their mechanism of action. Network meta-analysis revealed that for reducing DBs, second-generation antipsychotics, stimulants, and non-stimulant ADHD medications were more efficacious than placebo, and second-generation antipsychotics were the most efficacious. The dopaminergic modulation of top-down inhibitory process by these medications is discussed in this review. This study offers information on the relative efficacy of various psychotropic medications for the treatment of DB, and insight into a potential neurobiological underpinning for those symptoms. It also illustrates the potential utility of these neurobiological mechanisms as a target for future treatment studies.
Collapse
Affiliation(s)
- Ji-Woo Seok
- Department of Psychiatry, University of Nebraska Medical Center, 985578 Nebraska Medical Center, Omaha, NE, 68198-5578, USA
- Digital Health Research Division, Korea Institute of Oriental Medicine, Daejeon, South Korea
| | - Brigette Soltis-Vaughan
- Department of Psychiatry, University of Nebraska Medical Center, 985578 Nebraska Medical Center, Omaha, NE, 68198-5578, USA
| | - Brandon J Lew
- Department of Psychiatry, University of Nebraska Medical Center, 985578 Nebraska Medical Center, Omaha, NE, 68198-5578, USA
| | - Aatiya Ahmad
- Department of Psychiatry, University of Nebraska Medical Center, 985578 Nebraska Medical Center, Omaha, NE, 68198-5578, USA
| | - R J R Blair
- Child and Adolescent Mental Health Centre, Mental Health Services, Capital Region of Denmark, Copenhagen, Denmark
| | - Soonjo Hwang
- Department of Psychiatry, University of Nebraska Medical Center, 985578 Nebraska Medical Center, Omaha, NE, 68198-5578, USA.
| |
Collapse
|
22
|
Chao YS, Parrilla-Carrero J, Eid M, Culver OP, Jackson TB, Lipat R, Taniguchi M, Jhou TC. Innate cocaine-seeking vulnerability arising from loss of serotonin-mediated aversive effects of cocaine in rats. Cell Rep 2023; 42:112404. [PMID: 37083325 DOI: 10.1016/j.celrep.2023.112404] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2022] [Revised: 01/11/2023] [Accepted: 04/02/2023] [Indexed: 04/22/2023] Open
Abstract
Cocaine blocks dopamine reuptake, thereby producing rewarding effects that are widely studied. However, cocaine also blocks serotonin uptake, which we show drives, in rats, individually variable aversive effects that depend on serotonin 2C receptors (5-HT2CRs) in the rostromedial tegmental nucleus (RMTg), a major GABAergic afferent to midbrain dopamine neurons. 5-HT2CRs produce depolarizing effects in RMTg neurons that are particularly strong in some rats, leading to aversive effects that reduce acquisition of and relapse to cocaine seeking. In contrast, 5-HT2CR signaling is largely lost after cocaine exposure in other rats, leading to reduced aversive effects and increased cocaine seeking. These results suggest a serotonergic biological marker of cocaine-seeking vulnerability that can be targeted to modulate drug seeking.
Collapse
Affiliation(s)
- Ying S Chao
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC 29425, USA
| | | | - Maya Eid
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY 10029, USA
| | - Oliver P Culver
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Tyler B Jackson
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Rachel Lipat
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Makoto Taniguchi
- Department of Neuroscience, Medical University of South Carolina, Charleston, SC 29425, USA
| | - Thomas C Jhou
- Department of Anatomy and Neurobiology, University of Maryland School of Medicine, Baltimore, MD 21201, USA.
| |
Collapse
|
23
|
Abstract
It has been over 50 years since the original serotonin hypothesis was proposed by the British Psychiatrist Alec Coppen. Recently, some authors have questioned the validity of the hypothesis. In this narrative review, we summarise the evidence for the serotonin hypothesis of depression, focusing on psychopharmacology and molecular imaging, as well as systems-level neuroscience.
Collapse
Affiliation(s)
- Sameer Jauhar
- Department of Psychological Medicine, IoPPN, King's College, London, UK
| | - Philip J Cowen
- Department of Psychiatry, University of Oxford, Warneford Hospital Oxford, Oxon, UK
| | - Michael Browning
- Department of Psychiatry, University of Oxford, Warneford Hospital Oxford, Oxon, UK
| |
Collapse
|
24
|
Sands LP, Jiang A, Jones RE, Trattner JD, Kishida KT. Valence-partitioned learning signals drive choice behavior and phenomenal subjective experience in humans. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.03.17.533213. [PMID: 36993384 PMCID: PMC10055186 DOI: 10.1101/2023.03.17.533213] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
How the human brain generates conscious phenomenal experience is a fundamental problem. In particular, it is unknown how variable and dynamic changes in subjective affect are driven by interactions with objective phenomena. We hypothesize a neurocomputational mechanism that generates valence-specific learning signals associated with 'what it is like' to be rewarded or punished. Our hypothesized model maintains a partition between appetitive and aversive information while generating independent and parallel reward and punishment learning signals. This valence-partitioned reinforcement learning (VPRL) model and its associated learning signals are shown to predict dynamic changes in 1) human choice behavior, 2) phenomenal subjective experience, and 3) BOLD-imaging responses that implicate a network of regions that process appetitive and aversive information that converge on the ventral striatum and ventromedial prefrontal cortex during moments of introspection. Our results demonstrate the utility of valence-partitioned reinforcement learning as a neurocomputational basis for investigating mechanisms that may drive conscious experience.
Collapse
Affiliation(s)
- L. Paul Sands
- Dept. of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
| | - Angela Jiang
- Dept. of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
| | - Rachel E. Jones
- Dept. of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
| | - Jonathan D. Trattner
- Dept. of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
| | - Kenneth T. Kishida
- Dept. of Physiology and Pharmacology, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
- Neuroscience Graduate Program, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
- Dept. of Neurosurgery, Wake Forest School of Medicine, Winston-Salem NC, 27101, US
| |
Collapse
|
25
|
Bono J, Zannone S, Pedrosa V, Clopath C. Learning predictive cognitive maps with spiking neurons during behavior and replays. eLife 2023; 12:e80671. [PMID: 36927625 PMCID: PMC10019888 DOI: 10.7554/elife.80671] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Accepted: 01/12/2023] [Indexed: 03/18/2023] Open
Abstract
The hippocampus has been proposed to encode environments using a representation that contains predictive information about likely future states, called the successor representation. However, it is not clear how such a representation could be learned in the hippocampal circuit. Here, we propose a plasticity rule that can learn this predictive map of the environment using a spiking neural network. We connect this biologically plausible plasticity rule to reinforcement learning, mathematically and numerically showing that it implements the TD-lambda algorithm. By spanning these different levels, we show how our framework naturally encompasses behavioral activity and replays, smoothly moving from rate to temporal coding, and allows learning over behavioral timescales with a plasticity rule acting on a timescale of milliseconds. We discuss how biological parameters such as dwelling times at states, neuronal firing rates and neuromodulation relate to the delay discounting parameter of the TD algorithm, and how they influence the learned representation. We also find that, in agreement with psychological studies and contrary to reinforcement learning theory, the discount factor decreases hyperbolically with time. Finally, our framework suggests a role for replays, in both aiding learning in novel environments and finding shortcut trajectories that were not experienced during behavior, in agreement with experimental data.
Collapse
Affiliation(s)
- Jacopo Bono
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Sara Zannone
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Victor Pedrosa
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| | - Claudia Clopath
- Department of Bioengineering, Imperial College LondonLondonUnited Kingdom
| |
Collapse
|
26
|
Valente EEL, Klotz JL, Egert-McLean AM, Costa GW, May JB, Harmon DL. Influence of intra-abomasal administration of L-DOPA on circulating catecholamines and feed intake in cattle. FRONTIERS IN ANIMAL SCIENCE 2023. [DOI: 10.3389/fanim.2023.1127575] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/17/2023] Open
Abstract
Dopamine has multiple physiological functions including feed intake control in which it can act as an anorectic or orexigenic agent. This study had the objective to evaluate intra-abomasal administration of L-DOPA (levodopa; L-3,4-dihydroxyphenylalanine) from -Mucuna pruriens on circulating catecholamines, indicators of energy metabolism and feed intake in cattle. Eight Holstein steers (340 ± 20 kg) fitted with ruminal cannula were used in a replicated 4 x 4 Latin Square design experiment. Intra-abomasal infusion of L-DOPA at 0, 0.5, 1 and 2 mg/kg BW was carried out for seven days and blood samples were collected at 0, 30, 60, 120, 240 and 480 min from L-DOPA infusion on day 7. The area under the curve (AUC) of plasma L-DOPA and free dopamine increased quadratically with the administration of L-DOPA. However, the AUC of plasma total dopamine had a positive linear response with the increase of L-DOPA. Conversely, the serum 5-hydroxytriptophan (5-HTP), plasma serotonin, serum serotonin, serum tyrosine, plasma glucose and plasma free fatty acids were not affected by the intra-abomasal infusion of L-DOPA. The circulating concentration of the epinephrine, norepinephrine, serotonin, glucose and free fatty acids did not change with L-DOPA infusion. It can be concluded that intra-abomasal L-DOPA administration produced a strong increase in circulating dopamine with no change in energy metabolites and feed intake in cattle.
Collapse
|
27
|
Harkin EF, Lynn MB, Payeur A, Boucher JF, Caya-Bissonnette L, Cyr D, Stewart C, Longtin A, Naud R, Béïque JC. Temporal derivative computation in the dorsal raphe network revealed by an experimentally driven augmented integrate-and-fire modeling framework. eLife 2023; 12:72951. [PMID: 36655738 PMCID: PMC9977298 DOI: 10.7554/elife.72951] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2021] [Accepted: 12/19/2022] [Indexed: 01/20/2023] Open
Abstract
By means of an expansive innervation, the serotonin (5-HT) neurons of the dorsal raphe nucleus (DRN) are positioned to enact coordinated modulation of circuits distributed across the entire brain in order to adaptively regulate behavior. Yet the network computations that emerge from the excitability and connectivity features of the DRN are still poorly understood. To gain insight into these computations, we began by carrying out a detailed electrophysiological characterization of genetically identified mouse 5-HT and somatostatin (SOM) neurons. We next developed a single-neuron modeling framework that combines the realism of Hodgkin-Huxley models with the simplicity and predictive power of generalized integrate-and-fire models. We found that feedforward inhibition of 5-HT neurons by heterogeneous SOM neurons implemented divisive inhibition, while endocannabinoid-mediated modulation of excitatory drive to the DRN increased the gain of 5-HT output. Our most striking finding was that the output of the DRN encodes a mixture of the intensity and temporal derivative of its input, and that the temporal derivative component dominates this mixture precisely when the input is increasing rapidly. This network computation primarily emerged from prominent adaptation mechanisms found in 5-HT neurons, including a previously undescribed dynamic threshold. By applying a bottom-up neural network modeling approach, our results suggest that the DRN is particularly apt to encode input changes over short timescales, reflecting one of the salient emerging computations that dominate its output to regulate behavior.
Collapse
Affiliation(s)
- Emerson F Harkin
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - Michael B Lynn
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - Alexandre Payeur
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
- Department of Physics, University of OttawaOttawaCanada
| | - Jean-François Boucher
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - Léa Caya-Bissonnette
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - Dominic Cyr
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - Chloe Stewart
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| | - André Longtin
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
- Department of Physics, University of OttawaOttawaCanada
| | - Richard Naud
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
- Department of Physics, University of OttawaOttawaCanada
| | - Jean-Claude Béïque
- Brain and Mind Research Institute, Centre for Neural Dynamics, Department of Cellular and Molecular Medicine, University of OttawaOttawaCanada
| |
Collapse
|
28
|
Bao S, Lei Y. Memory decay and generalization following distinct motor learning mechanisms. J Neurophysiol 2022; 128:1534-1545. [PMID: 36321731 DOI: 10.1152/jn.00105.2022] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022] Open
Abstract
Motor skill learning is considered to arise out of contributions from multiple learning mechanisms, including error-based learning (EBL), use-dependent learning (UDL), and reinforcement learning (RL). These learning mechanisms exhibit dissociable roles and engage different neural circuits during skill acquisition. However, it remains largely unknown how a newly formed motor memory acquired through each learning mechanism decays over time and whether distinct learning mechanisms produce different generalization patterns. Here, we used variants of reaching paradigms that dissociated these learning mechanisms to examine the time course of memory decay following each learning and the generalization patterns of each learning. We found that motor memories acquired through these learning mechanisms decayed as a function of time. Notably, 15 min, 6 h, and 24 h after acquisition, the memory of EBL decayed much greater than that of RL. The memory acquired through UDL faded away within a few minutes. Motor memories formed through EBL and RL for given movement directions generalized to untrained movement directions, with the generalization of EBL being greater than that of RL. In contrast, motor memory of UDL could not generalize to untrained movement directions. These results suggest that distinct learning mechanisms exhibit different patterns of memory decay and generalization.NEW & NOTEWORTHY Motor skill learning is likely to involve error-based learning, use-dependent plasticity, and operant reinforcement. Here, we showed that these dissociable learning mechanisms exhibited distinct patterns of memory decay and generalization. With a better understanding of the characteristics of these learning mechanisms, it becomes possible to regulate each learning process separately to improve neurological rehabilitation.
Collapse
Affiliation(s)
- Shancheng Bao
- Department of Kinesiology & Sport Management, Texas A&M University, College Station, Texas
| | - Yuming Lei
- Department of Kinesiology & Sport Management, Texas A&M University, College Station, Texas
| |
Collapse
|
29
|
Janet R, Ligneul R, Losecaat-Vermeer AB, Philippe R, Bellucci G, Derrington E, Park SQ, Dreher JC. Regulation of social hierarchy learning by serotonin transporter availability. Neuropsychopharmacology 2022; 47:2205-2212. [PMID: 35945275 PMCID: PMC9630526 DOI: 10.1038/s41386-022-01378-2] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Revised: 06/03/2022] [Accepted: 06/30/2022] [Indexed: 11/18/2022]
Abstract
Learning one's status in a group is a fundamental process in building social hierarchies. Although animal studies suggest that serotonin (5-HT) signaling modulates learning social hierarchies, direct evidence in humans is lacking. Here we determined the relationship between serotonin transporter (SERT) availability and brain systems engaged in learning social ranks combining computational approaches with simultaneous PET-fMRI acquisition in healthy males. We also investigated the link between SERT availability and brain activity in a non-social control condition involving learning the payoffs of slot machines. Learning social ranks was modulated by the dorsal raphe nucleus (DRN) 5-HT function. BOLD ventral striatal response, tracking the rank of opponents, decreased with DRN SERT levels. Moreover, this link was specific to the social learning task. These findings demonstrate that 5-HT plays an influence on the computations required to learn social ranks.
Collapse
Affiliation(s)
- Remi Janet
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Romain Ligneul
- grid.421010.60000 0004 0453 9636Champalimaud Neuroscience Program, Champalimaud Center for the Unknown, Lisbon, Portugal
| | - Annabel B. Losecaat-Vermeer
- grid.10420.370000 0001 2286 1424Neuropsychopharmacology and Biopsychology Unit, Department of Cognition, Emotion, and Methods in Psychology, Faculty of Psychology, University of Vienna, Vienna, Austria ,grid.7468.d0000 0001 2248 7639Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Neuroscience Research Center, 10117 Berlin, Germany
| | - Remi Philippe
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Gabriele Bellucci
- grid.419501.80000 0001 2183 0052Department of Computational Neuroscience, Max Planck Institute for Biological Cybernetics, Tübingen, Germany
| | - Edmund Derrington
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France
| | - Soyoung Q. Park
- grid.7468.d0000 0001 2248 7639Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, Neuroscience Research Center, 10117 Berlin, Germany ,grid.418213.d0000 0004 0390 0098Department of Decision Neuroscience and Nutrition, German Institute of Human Nutrition (DIfE), Potsdam-Rehbrücke, Nuthetal, Germany
| | - Jean-Claude Dreher
- CNRS-Institut de Sciences Cognitives Marc Jeannerod, UMR5229, Neuroeconomics, reward, and decision making laboratory, Bron, France.
| |
Collapse
|
30
|
Liebenow B, Jones R, DiMarco E, Trattner JD, Humphries J, Sands LP, Spry KP, Johnson CK, Farkas EB, Jiang A, Kishida KT. Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders. Front Psychiatry 2022; 13:886297. [PMID: 36339844 PMCID: PMC9630918 DOI: 10.3389/fpsyt.2022.886297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Accepted: 09/23/2022] [Indexed: 11/13/2022] Open
Abstract
In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.
Collapse
Affiliation(s)
- Brittany Liebenow
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Rachel Jones
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Emily DiMarco
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Jonathan D. Trattner
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Joseph Humphries
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - L. Paul Sands
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Kasey P. Spry
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Christina K. Johnson
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Evelyn B. Farkas
- Georgia State University Undergraduate Neuroscience Institute, Atlanta, GA, United States
| | - Angela Jiang
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| | - Kenneth T. Kishida
- Neuroscience Graduate Program, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Physiology and Pharmacology, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Neurosurgery, Wake Forest University School of Medicine, Winston-Salem, NC, United States
- Department of Biomedical Engineering, Wake Forest University School of Medicine, Winston-Salem, NC, United States
| |
Collapse
|
31
|
Colas JT, Dundon NM, Gerraty RT, Saragosa‐Harris NM, Szymula KP, Tanwisuth K, Tyszka JM, van Geen C, Ju H, Toga AW, Gold JI, Bassett DS, Hartley CA, Shohamy D, Grafton ST, O'Doherty JP. Reinforcement learning with associative or discriminative generalization across states and actions: fMRI at 3 T and 7 T. Hum Brain Mapp 2022; 43:4750-4790. [PMID: 35860954 PMCID: PMC9491297 DOI: 10.1002/hbm.25988] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2022] [Revised: 05/20/2022] [Accepted: 06/10/2022] [Indexed: 11/12/2022] Open
Abstract
The model-free algorithms of "reinforcement learning" (RL) have gained clout across disciplines, but so too have model-based alternatives. The present study emphasizes other dimensions of this model space in consideration of associative or discriminative generalization across states and actions. This "generalized reinforcement learning" (GRL) model, a frugal extension of RL, parsimoniously retains the single reward-prediction error (RPE), but the scope of learning goes beyond the experienced state and action. Instead, the generalized RPE is efficiently relayed for bidirectional counterfactual updating of value estimates for other representations. Aided by structural information but as an implicit rather than explicit cognitive map, GRL provided the most precise account of human behavior and individual differences in a reversal-learning task with hierarchical structure that encouraged inverse generalization across both states and actions. Reflecting inference that could be true, false (i.e., overgeneralization), or absent (i.e., undergeneralization), state generalization distinguished those who learned well more so than action generalization. With high-resolution high-field fMRI targeting the dopaminergic midbrain, the GRL model's RPE signals (alongside value and decision signals) were localized within not only the striatum but also the substantia nigra and the ventral tegmental area, including specific effects of generalization that also extend to the hippocampus. Factoring in generalization as a multidimensional process in value-based learning, these findings shed light on complexities that, while challenging classic RL, can still be resolved within the bounds of its core computations.
Collapse
Affiliation(s)
- Jaron T. Colas
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| | - Neil M. Dundon
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
- Department of Child and Adolescent Psychiatry, Psychotherapy, and PsychosomaticsUniversity of FreiburgFreiburg im BreisgauGermany
| | - Raphael T. Gerraty
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Center for Science and SocietyColumbia UniversityNew YorkNew YorkUSA
| | - Natalie M. Saragosa‐Harris
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of CaliforniaLos AngelesCaliforniaUSA
| | - Karol P. Szymula
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Koranis Tanwisuth
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Department of PsychologyUniversity of CaliforniaBerkeleyCaliforniaUSA
| | - J. Michael Tyszka
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
| | - Camilla van Geen
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Department of PsychologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Harang Ju
- Neuroscience Graduate GroupUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Arthur W. Toga
- Laboratory of Neuro ImagingUSC Stevens Neuroimaging and Informatics Institute, Keck School of Medicine of USC, University of Southern CaliforniaLos AngelesCaliforniaUSA
| | - Joshua I. Gold
- Department of NeuroscienceUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
| | - Dani S. Bassett
- Department of BioengineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Electrical and Systems EngineeringUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of NeurologyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of PsychiatryUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Department of Physics and AstronomyUniversity of PennsylvaniaPhiladelphiaPennsylvaniaUSA
- Santa Fe InstituteSanta FeNew MexicoUSA
| | - Catherine A. Hartley
- Department of PsychologyNew York UniversityNew YorkNew YorkUSA
- Center for Neural ScienceNew York UniversityNew YorkNew YorkUSA
| | - Daphna Shohamy
- Department of PsychologyColumbia UniversityNew YorkNew YorkUSA
- Zuckerman Mind Brain Behavior Institute, Columbia UniversityNew YorkNew YorkUSA
- Kavli Institute for Brain ScienceColumbia UniversityNew YorkNew YorkUSA
| | - Scott T. Grafton
- Department of Psychological and Brain SciencesUniversity of CaliforniaSanta BarbaraCaliforniaUSA
| | - John P. O'Doherty
- Division of the Humanities and Social SciencesCalifornia Institute of TechnologyPasadenaCaliforniaUSA
- Computation and Neural Systems Program, California Institute of TechnologyPasadenaCaliforniaUSA
| |
Collapse
|
32
|
Bauer EE, Agbemafle I, Reddy MB, Clark PJ. Edible insects prevent changes to brain monoamine profiles from malnourishment in weaned rats. Nutr Neurosci 2022:1-13. [PMID: 36125026 DOI: 10.1080/1028415x.2022.2104692] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/14/2022]
Abstract
Background: Childhood malnutrition can have devastating consequences on health, behavior, and cognition. Edible insects are sustainable low cost high protein and iron nutritious foods that can prevent malnutrition. However, it is unclear whether insect-based diets may help prevent changes to brain neurochemistry associated with malnutrition.Materials and Methods: Weanling male Sprague-Dawley rats were malnourished by feeding a low protein-iron diet (LPI, 5% protein and ∼2 ppm Fe) for 3 weeks or nourished by feeding a sufficient protein-iron diet (SPI, 15% protein 20 ppm FeSO4) for the duration of the study. Following 3 weeks of LPI diet, three subsets of the malnourished rats were placed on repletion diets supplemented with cricket, palm weevil larvae, or the SPI diet for 2 weeks, while the remaining rats continued the LPI diet for an additional 2 weeks. Monoamine-related neurochemicals (e.g. serotonin (5-HT), dopamine (DA), norepinephrine) and select monoamine metabolites were measured in the hypothalamus, hippocampus, striatum, and prefrontal cortex using Ultra High-Performance Liquid Chromatography.Results: Five weeks of LPI diets disrupted brain monoamines, most notable in the hypothalamus. Two weeks supplementation with cricket and palm weevil larvae diets prevented changes to measures of 5-HT and DA turnover in the hippocampus and hypothalamus. Moreover, these insect diets prevented the malnutrition-induced imbalance of 5-HT and DA metabolites in the hippocampus, striatum, and hypothalamus.Conclusion: Edible insects such as cricket and palm weevil larvae could be sustainable nutrition intervention to prevent behavioral and cognitive impairment associated abnormal brain monoamine activities that results from early life malnutrition.
Collapse
Affiliation(s)
- Ella E. Bauer
- Department of Food Science and Human Nutrition, Iowa State University, Ames, IA, USA
| | - Isaac Agbemafle
- Department of Food Science and Human Nutrition, Iowa State University, Ames, IA, USA
| | - Manju B. Reddy
- Department of Food Science and Human Nutrition, Iowa State University, Ames, IA, USA
| | - Peter J. Clark
- Department of Food Science and Human Nutrition, Iowa State University, Ames, IA, USA
| |
Collapse
|
33
|
Nussenbaum K, Velez JA, Washington BT, Hamling HE, Hartley CA. Flexibility in valenced reinforcement learning computations across development. Child Dev 2022; 93:1601-1615. [PMID: 35596654 PMCID: PMC9831067 DOI: 10.1111/cdev.13791] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
Abstract
Optimal integration of positive and negative outcomes during learning varies depending on an environment's reward statistics. The present study investigated the extent to which children, adolescents, and adults (N = 142 8-25 year-olds, 55% female, 42% White, 31% Asian, 17% mixed race, and 8% Black; data collected in 2021) adapt their weighting of better-than-expected and worse-than-expected outcomes when learning from reinforcement. Participants made choices across two contexts: one in which weighting positive outcomes more heavily than negative outcomes led to better performance, and one in which the reverse was true. Reinforcement learning modeling revealed that across age, participants shifted their valence biases in accordance with environmental structure. Exploratory analyses revealed strengthening of context-dependent flexibility with increasing age.
Collapse
Affiliation(s)
| | | | | | | | - Catherine A. Hartley
- Corresponding Author: Catherine A. Hartley, Department of Psychology, New York University, 6 Washington Place, Room 871A, New York, NY, 10003.
| |
Collapse
|
34
|
Shine JM, O’Callaghan C, Walpola IC, Wainstein G, Taylor N, Aru J, Huebner B, John YJ. Understanding the effects of serotonin in the brain through its role in the gastrointestinal tract. Brain 2022; 145:2967-2981. [DOI: 10.1093/brain/awac256] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 06/12/2022] [Accepted: 06/14/2022] [Indexed: 11/12/2022] Open
Abstract
Abstract
The neuromodulatory arousal system imbues the nervous system with the flexibility and robustness required to facilitate adaptive behaviour. While there are well-understood mechanisms linking dopamine, noradrenaline and acetylcholine to distinct behavioural states, similar conclusions have not been as readily available for serotonin. Fascinatingly, despite clear links between serotonergic function and cognitive capacities as diverse as reward processing, exploration, and the psychedelic experience, over 95% of the serotonin in the body is released in the gastrointestinal tract, where it controls digestive muscle contractions (peristalsis). Here, we argue that framing neural serotonin as a rostral extension of the gastrointestinal serotonergic system dissolves much of the mystery associated with the central serotonergic system. Specifically, we outline that central serotonin activity mimics the effects of a digestion/satiety circuit mediated by hypothalamic control over descending serotonergic nuclei in the brainstem. We review commonalities and differences between these two circuits, with a focus on the heterogeneous expression of different classes of serotonin receptors in the brain. Much in the way that serotonin-induced peristalsis facilitates the work of digestion, serotonergic influences over cognition can be reframed as performing the work of cognition. Extending this analogy, we argue that the central serotonergic system allows the brain to arbitrate between different cognitive modes as a function of serotonergic tone: low activity facilitates cognitive automaticity, whereas higher activity helps to identify flexible solutions to problems, particularly if and when the initial responses fail. This perspective sheds light on otherwise disparate capacities mediated by serotonin, and also helps to understand why there are such pervasive links between serotonergic pathology and the symptoms of psychiatric disorders.
Collapse
Affiliation(s)
| | | | - Ishan C Walpola
- Prince of Wales Hospital , Randwick, New South Wales , Australia
| | | | | | - Jaan Aru
- University of Tartu , Tartu , Estonia
| | | | | |
Collapse
|
35
|
Jepma M, Roy M, Ramlakhan K, van Velzen M, Dahan A. Different brain systems support learning from received and avoided pain during human pain-avoidance learning. eLife 2022; 11:74149. [PMID: 35731646 PMCID: PMC9217130 DOI: 10.7554/elife.74149] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2021] [Accepted: 06/07/2022] [Indexed: 12/14/2022] Open
Abstract
Both unexpected pain and unexpected pain absence can drive avoidance learning, but whether they do so via shared or separate neural and neurochemical systems is largely unknown. To address this issue, we combined an instrumental pain-avoidance learning task with computational modeling, functional magnetic resonance imaging (fMRI), and pharmacological manipulations of the dopaminergic (100 mg levodopa) and opioidergic (50 mg naltrexone) systems (N = 83). Computational modeling provided evidence that untreated participants learned more from received than avoided pain. Our dopamine and opioid manipulations negated this learning asymmetry by selectively increasing learning rates for avoided pain. Furthermore, our fMRI analyses revealed that pain prediction errors were encoded in subcortical and limbic brain regions, whereas no-pain prediction errors were encoded in frontal and parietal cortical regions. However, we found no effects of our pharmacological manipulations on the neural encoding of prediction errors. Together, our results suggest that human pain-avoidance learning is supported by separate threat- and safety-learning systems, and that dopamine and endogenous opioids specifically regulate learning from successfully avoided pain.
Collapse
Affiliation(s)
- Marieke Jepma
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands.,Department of Psychology, Leiden University, Leiden, Netherlands.,Leiden Institute for Brain and Cognition, Leiden, Netherlands
| | - Mathieu Roy
- Department of Psychology, McGill University, Montreal, Canada.,Alan Edwards Centre for Research on Pain, McGill University, Montreal, Canada
| | - Kiran Ramlakhan
- Department of Psychology, Leiden University, Leiden, Netherlands.,Department of Research and Statistics, Municipality of Amsterdam, Amsterdam, Netherlands
| | - Monique van Velzen
- Department of Anesthesiology, Leiden University Medical Center, Leiden, Netherlands
| | - Albert Dahan
- Department of Anesthesiology, Leiden University Medical Center, Leiden, Netherlands
| |
Collapse
|
36
|
Brain-inspired meta-reinforcement learning cognitive control in conflictual inhibition decision-making task for artificial agents. Neural Netw 2022; 154:283-302. [DOI: 10.1016/j.neunet.2022.06.020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2021] [Revised: 06/09/2022] [Accepted: 06/16/2022] [Indexed: 11/21/2022]
|
37
|
Krichmar JL, Hwu TJ. Design Principles for Neurorobotics. Front Neurorobot 2022; 16:882518. [PMID: 35692490 PMCID: PMC9174684 DOI: 10.3389/fnbot.2022.882518] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Accepted: 04/19/2022] [Indexed: 11/13/2022] Open
Abstract
In their book “How the Body Shapes the Way We Think: A New View of Intelligence,” Pfeifer and Bongard put forth an embodied approach to cognition. Because of this position, many of their robot examples demonstrated “intelligent” behavior despite limited neural processing. It is our belief that neurorobots should attempt to follow many of these principles. In this article, we discuss a number of principles to consider when designing neurorobots and experiments using robots to test brain theories. These principles are strongly inspired by Pfeifer and Bongard, but build on their design principles by grounding them in neuroscience and by adding principles based on neuroscience research. Our design principles fall into three categories. First, organisms must react quickly and appropriately to events. Second, organisms must have the ability to learn and remember over their lifetimes. Third, organisms must weigh options that are crucial for survival. We believe that by following these design principles a robot's behavior will be more naturalistic and more successful.
Collapse
Affiliation(s)
- Jeffrey L. Krichmar
- Department of Cognitive Sciences, University of California, Irvine, Irvine, CA, United States
- Department of Computer Science, University of California, Irvine, Irvine, CA, United States
- *Correspondence: Jeffrey L. Krichmar
| | | |
Collapse
|
38
|
Wert-Carvajal C, Reneaux M, Tchumatchenko T, Clopath C. Dopamine and serotonin interplay for valence-based spatial learning. Cell Rep 2022; 39:110645. [PMID: 35417691 DOI: 10.1016/j.celrep.2022.110645] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2021] [Revised: 10/31/2021] [Accepted: 03/17/2022] [Indexed: 11/17/2022] Open
Abstract
Dopamine (DA) and serotonin (5-HT) are important neuromodulators of synaptic plasticity that have been linked to learning from positive or negative outcomes or valence-based learning. In the hippocampus, both affect long-term plasticity but play different roles in encoding uncertainty or predicted reward. DA has been related to positive valence, from reward consumption or avoidance behavior, and 5-HT to aversive encoding. We propose DA produces overall LTP while 5-HT elicits LTD. Here, we compare two reward-modulated spike timing-dependent plasticity (R-STDP) rules to describe the action of these neuromodulators. We examined their role in cognitive performance and flexibility for computational models of the Morris water maze task and reversal learning. Our results show that the interplay of DA and 5-HT improves learning performance and can explain experimental evidence. This study reinforces the importance of neuromodulation in determining the direction of plasticity.
Collapse
Affiliation(s)
- Carlos Wert-Carvajal
- Bioengineering Department, Imperial College London, London SW7 2AZ, UK; Theory of Neural Dynamics Group, Max Planck Institute for Brain Research, 60438 Frankfurt, Germany; Institute of Experimental Epileptology and Cognition Research, Life and Brain Center, University of Bonn Medical Center, 53127 Bonn, Germany
| | - Melissa Reneaux
- Bioengineering Department, Imperial College London, London SW7 2AZ, UK
| | - Tatjana Tchumatchenko
- Theory of Neural Dynamics Group, Max Planck Institute for Brain Research, 60438 Frankfurt, Germany; Institute of Experimental Epileptology and Cognition Research, Life and Brain Center, University of Bonn Medical Center, 53127 Bonn, Germany; Institute of Physiological Chemistry, University of Mainz Medical Center, 55131 Mainz, Germany.
| | - Claudia Clopath
- Bioengineering Department, Imperial College London, London SW7 2AZ, UK.
| |
Collapse
|
39
|
Dagher M, Perrotta KA, Erwin SA, Hachisuka A, Iyer R, Masmanidis SC, Yang H, Andrews AM. Optogenetic Stimulation of Midbrain Dopamine Neurons Produces Striatal Serotonin Release. ACS Chem Neurosci 2022; 13:946-958. [PMID: 35312275 PMCID: PMC9040469 DOI: 10.1021/acschemneuro.1c00715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022] Open
Abstract
Targeting neurons with light-driven opsins is widely used to investigate cell-specific responses. We transfected midbrain dopamine neurons with the excitatory opsin Chrimson. Extracellular basal and stimulated neurotransmitter levels in the dorsal striatum were measured by microdialysis in awake mice. Optical activation of dopamine cell bodies evoked terminal dopamine release in the striatum. Multiplexed analysis of dialysate samples revealed that the evoked dopamine was accompanied by temporally coupled increases in striatal 3-methoxytyramine, an extracellular dopamine metabolite, and in serotonin. We investigated a mechanism for dopamine-serotonin interactions involving striatal dopamine receptors. However, the evoked serotonin associated with optical stimulation of dopamine neurons was not abolished by striatal D1- or D2-like receptor inhibition. Although the mechanisms underlying the coupling of striatal dopamine and serotonin remain unclear, these findings illustrate advantages of multiplexed measurements for uncovering functional interactions between neurotransmitter systems. Furthermore, they suggest that the output of optogenetic manipulations may extend beyond opsin-expressing neuronal populations.
Collapse
Affiliation(s)
- Merel Dagher
- Molecular Toxicology Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Katie A. Perrotta
- Department of Chemistry and Biochemistry, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Sara A. Erwin
- Molecular Toxicology Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Ayaka Hachisuka
- Department of Neurobiology, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Rahul Iyer
- Department of Electrical Engineering, University of California, Los Angeles, Los Angeles, CA, 94720
| | - Sotiris C. Masmanidis
- Department of Neurobiology, University of California, Los Angeles, Los Angeles, CA 90095, United States
- Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA 90095, United States
- California Nanosystems Institute, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Hongyan Yang
- Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neuroscience & Human Behavior, and Hatos Center for Neuropharmacology, University of California, Los Angeles, Los Angeles, CA 90095, United States
| | - Anne M. Andrews
- Molecular Toxicology Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA 90095, United States
- Department of Chemistry and Biochemistry, University of California, Los Angeles, Los Angeles, CA 90095, United States
- Neuroscience Interdepartmental Program, University of California, Los Angeles, Los Angeles, CA 90095, United States
- Department of Psychiatry and Biobehavioral Sciences, Semel Institute for Neuroscience & Human Behavior, and Hatos Center for Neuropharmacology, University of California, Los Angeles, Los Angeles, CA 90095, United States
- California Nanosystems Institute, University of California, Los Angeles, Los Angeles, CA 90095, United States
| |
Collapse
|
40
|
Di Rosa E, Mapelli D, Ronconi L, Macchia E, Gentili C, Bisiacchi P, Edelstyn N. Anxiety predicts impulsive-compulsive behaviours in Parkinson's disease: Clinical relevance and theoretical implications. J Psychiatr Res 2022; 148:220-229. [PMID: 35134729 DOI: 10.1016/j.jpsychires.2022.01.052] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/25/2021] [Revised: 01/13/2022] [Accepted: 01/26/2022] [Indexed: 10/19/2022]
Abstract
Patients with Parkinson's disease (PD) often present symptoms of anxiety, depression and apathy. These negative affect manifestations have been recently associated with the presence of impulsive compulsive behaviours (ICBs). However, their relation with the use of dopamine replacement therapy (DRT), a renewed risk factor for ICBs, is still not fully understood. Elucidating the role of these different ICBs predictors in PD could inform both prevention/intervention recommendations as well as theoretical models. In the present study, we have analysed data collected in 417 PD patients, 50 patients with Parkinsonian symptoms but with scan without evidence of dopaminergic deficit (SWEDD), and 185 healthy controls (HC). We examined each patient's clinical profile over a two-year time window, investigating the role of both negative affect and DRT on ICBs. Results confirmed the presence of higher levels of anxiety in both the clinical groups, and of higher level of ICBs in SWEDD patients, respect to both PD and HC. Mixed model analyses revealed a statistically significant association between anxiety and ICBs in the SWEDD patients who did not take any DRT. Findings suggest the independence between anxiety and DRT in ICBs development, and provide new evidence for the motivational opponency theoretical framework.
Collapse
Affiliation(s)
- Elisa Di Rosa
- Department of General Psychology, University of Padua, Padua, Italy; School of Psychology, Keele University, Staffordshire, UK.
| | - Daniela Mapelli
- Department of General Psychology, University of Padua, Padua, Italy
| | - Lucia Ronconi
- Department of General Psychology, University of Padua, Padua, Italy
| | - Eleonora Macchia
- Department of General Psychology, University of Padua, Padua, Italy
| | - Claudio Gentili
- Department of General Psychology, University of Padua, Padua, Italy; Padua Neuroscience Centre, University of Padua, Padua, Italy; Centro di Ateneo dei Servizi Clinici Universitari Psicologici, University of Padua, Padua, Italy
| | - Patrizia Bisiacchi
- Department of General Psychology, University of Padua, Padua, Italy; Padua Neuroscience Centre, University of Padua, Padua, Italy
| | - Nicky Edelstyn
- School of Psychology, Keele University, Staffordshire, UK
| |
Collapse
|
41
|
Grossman CD, Cohen JY. Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making. Annu Rev Neurosci 2022; 45:317-337. [PMID: 35363533 DOI: 10.1146/annurev-neuro-092021-125059] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Nervous systems evolved to effectively navigate the dynamics of the environment to achieve their goals. One framework used to study this fundamental problem arose in the study of learning and decision-making. In this framework, the demands of effective behavior require slow dynamics-on the scale of seconds to minutes-of networks of neurons. Here, we review the phenomena and mechanisms involved. Using vignettes from a few species and areas of the nervous system, we view neuromodulators as key substrates for temporal scaling of neuronal dynamics. Expected final online publication date for the Annual Review of Neuroscience, Volume 45 is July 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, and Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA;
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, and Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, Baltimore, Maryland, USA;
| |
Collapse
|
42
|
The role of state uncertainty in the dynamics of dopamine. Curr Biol 2022; 32:1077-1087.e9. [PMID: 35114098 PMCID: PMC8930519 DOI: 10.1016/j.cub.2022.01.025] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2021] [Revised: 11/22/2021] [Accepted: 01/10/2022] [Indexed: 11/22/2022]
Abstract
Reinforcement learning models of the basal ganglia map the phasic dopamine signal to reward prediction errors (RPEs). Conventional models assert that, when a stimulus predicts a reward with fixed delay, dopamine activity during the delay should converge to baseline through learning. However, recent studies have found that dopamine ramps up before reward in certain conditions even after learning, thus challenging the conventional models. In this work, we show that sensory feedback causes an unbiased learner to produce RPE ramps. Our model predicts that when feedback gradually decreases during a trial, dopamine activity should resemble a "bump," whose ramp-up phase should, furthermore, be greater than that of conditions where the feedback stays high. We trained mice on a virtual navigation task with varying brightness, and both predictions were empirically observed. In sum, our theoretical and experimental results reconcile the seemingly conflicting data on dopamine behaviors under the RPE hypothesis.
Collapse
|
43
|
Gilbert JR, Wusinich C, Zarate CA. A Predictive Coding Framework for Understanding Major Depression. Front Hum Neurosci 2022; 16:787495. [PMID: 35308621 PMCID: PMC8927302 DOI: 10.3389/fnhum.2022.787495] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2021] [Accepted: 02/14/2022] [Indexed: 12/17/2022] Open
Abstract
Predictive coding models of brain processing propose that top-down cortical signals promote efficient neural signaling by carrying predictions about incoming sensory information. These "priors" serve to constrain bottom-up signal propagation where prediction errors are carried via feedforward mechanisms. Depression, traditionally viewed as a disorder characterized by negative cognitive biases, is associated with disrupted reward prediction error encoding and signaling. Accumulating evidence also suggests that depression is characterized by impaired local and long-range prediction signaling across multiple sensory domains. This review highlights the electrophysiological and neuroimaging evidence for disrupted predictive processing in depression. The discussion is framed around the manner in which disrupted generative predictions about the sensorium could lead to depressive symptomatology, including anhedonia and negative bias. In particular, the review focuses on studies of sensory deviance detection and reward processing, highlighting research evidence for both disrupted generative predictions and prediction error signaling in depression. The role of the monoaminergic and glutamatergic systems in predictive coding processes is also discussed. This review provides a novel framework for understanding depression using predictive coding principles and establishes a foundational roadmap for potential future research.
Collapse
Affiliation(s)
- Jessica R. Gilbert
- Experimental Therapeutics and Pathophysiology Branch, National Institute of Mental Health, National Institutes of Health, Bethesda, MD, United States
| | | | | |
Collapse
|
44
|
Robson DN, Li JM. A dynamical systems view of neuroethology: Uncovering stateful computation in natural behaviors. Curr Opin Neurobiol 2022; 73:102517. [PMID: 35217311 DOI: 10.1016/j.conb.2022.01.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2021] [Revised: 01/06/2022] [Accepted: 01/11/2022] [Indexed: 11/03/2022]
Abstract
State-dependent computation is key to cognition in both biological and artificial systems. Alan Turing recognized the power of stateful computation when he created the Turing machine with theoretically infinite computational capacity in 1936. Independently, by 1950, ethologists such as Tinbergen and Lorenz also began to implicitly embed rudimentary forms of state-dependent computation to create qualitative models of internal drives and naturally occurring animal behaviors. Here, we reformulate core ethological concepts in explicitly dynamical systems terms for stateful computation. We examine, based on a wealth of recent neural data collected during complex innate behaviors across species, the neural dynamics that determine the temporal structure of internal states. We will also discuss the degree to which the brain can be hierarchically partitioned into nested dynamical systems and the need for a multi-dimensional state-space model of the neuromodulatory system that underlies motivational and affective states.
Collapse
Affiliation(s)
- Drew N Robson
- Max Planck Institute for Biological Cybernetics, Tuebingen, Germany.
| | - Jennifer M Li
- Max Planck Institute for Biological Cybernetics, Tuebingen, Germany.
| |
Collapse
|
45
|
Grossman CD, Bari BA, Cohen JY. Serotonin neurons modulate learning rate through uncertainty. Curr Biol 2022; 32:586-599.e7. [PMID: 34936883 PMCID: PMC8825708 DOI: 10.1016/j.cub.2021.12.006] [Citation(s) in RCA: 40] [Impact Index Per Article: 20.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2020] [Revised: 10/11/2021] [Accepted: 12/03/2021] [Indexed: 12/20/2022]
Abstract
Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.
Collapse
Affiliation(s)
- Cooper D Grossman
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Bilal A Bari
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA
| | - Jeremiah Y Cohen
- The Solomon H. Snyder Department of Neuroscience, Brain Science Institute, Kavli Neuroscience Discovery Institute, The Johns Hopkins University School of Medicine, 725 N. Wolfe Street, Baltimore, MD 21205, USA.
| |
Collapse
|
46
|
Gupta A, Bhatt RR, Rivera-Cancel A, Makkar R, Kragel PA, Rodriguez T, Graner JL, Alaverdyan A, Hamadani K, Vora P, Naliboff B, Labus JS, LaBar KS, Mayer EA, Zucker N. Complex functional brain network properties in anorexia nervosa. J Eat Disord 2022; 10:13. [PMID: 35123579 PMCID: PMC8817538 DOI: 10.1186/s40337-022-00534-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/08/2021] [Accepted: 01/19/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Anorexia nervosa (AN) is a disorder characterized by an incapacitating fear of weight gain and by a disturbance in the way the body is experienced, facets that motivate dangerous weight loss behaviors. Multimodal neuroimaging studies highlight atypical neural activity in brain networks involved in interoceptive awareness and reward processing. METHODS The current study used resting-state neuroimaging to model the architecture of large-scale functional brain networks and characterize network properties of individual brain regions to clinical measures. Resting-state neuroimaging was conducted in 62 adolescents, 22 (21 female) with a history of AN and 40 (39 female) healthy controls (HCs). Sensorimotor and basal ganglia regions, as part of a 165-region whole-brain network, were investigated. Subject-specific functional brain networks were computed to index centrality. A contrast analysis within the general linear model covarying for age was performed. Correlations between network properties and behavioral measures were conducted (significance q < .05). RESULTS Compared to HCs, AN had lower connectivity from sensorimotor regions, and greater connectivity from the left caudate nucleus to the right postcentral gyrus. AN demonstrated lower sensorimotor centrality, but higher basal ganglia centrality. Sensorimotor connectivity dyads and centrality exhibited negative correlations with body dissatisfaction and drive for thinness, two essential features of AN. CONCLUSIONS These findings suggest that AN is associated with greater communication from the basal ganglia, and lower information propagation in sensorimotor cortices. This is consistent with the clinical presentation of AN, where individuals exhibit patterns of rigid habitual behavior that is not responsive to bodily needs, and seem "disconnected" from their bodies.
Collapse
Affiliation(s)
- Arpana Gupta
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA. .,David Geffen School of Medicine, UCLA, Los Angeles, CA, 90095, USA. .,Vatche and Tamar Manoukian Division of Digestive Diseases, UCLA, Los Angeles, CA, 90095, USA.
| | - Ravi R Bhatt
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA.,Imaging Genetics Center, Mark and Mary Stevens Institute for Neuroimaging and Informatics, Keck School of Medicine at USC, University of Southern California, Los Angeles, USA
| | | | - Rishi Makkar
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA
| | | | - Thomas Rodriguez
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA
| | - John L Graner
- Department of Psychology and Neuroscience, Duke University, Durham, USA
| | - Anita Alaverdyan
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA
| | - Kareem Hamadani
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA
| | - Priten Vora
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA
| | - Bruce Naliboff
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA.,David Geffen School of Medicine, UCLA, Los Angeles, CA, 90095, USA.,Vatche and Tamar Manoukian Division of Digestive Diseases, UCLA, Los Angeles, CA, 90095, USA
| | - Jennifer S Labus
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA.,David Geffen School of Medicine, UCLA, Los Angeles, CA, 90095, USA.,Vatche and Tamar Manoukian Division of Digestive Diseases, UCLA, Los Angeles, CA, 90095, USA
| | - Kevin S LaBar
- Department of Psychology and Neuroscience, Duke University, Durham, USA
| | - Emeran A Mayer
- G. Oppenheimer Center for Neurobiology of Stress and Resilience, UCLA, Los Angeles, CA, 90095, USA.,David Geffen School of Medicine, UCLA, Los Angeles, CA, 90095, USA.,Vatche and Tamar Manoukian Division of Digestive Diseases, UCLA, Los Angeles, CA, 90095, USA.,Ahmanson-Lovelace Brain Mapping Center, UCLA, Los Angeles, CA, 90095, USA
| | - Nancy Zucker
- Department of Psychiatry and Behavioral Sciences, Duke University, Durham, USA.,Department of Psychology and Neuroscience, Duke University, Durham, USA
| |
Collapse
|
47
|
Yee DM, Leng X, Shenhav A, Braver TS. Aversive motivation and cognitive control. Neurosci Biobehav Rev 2022; 133:104493. [PMID: 34910931 PMCID: PMC8792354 DOI: 10.1016/j.neubiorev.2021.12.016] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2021] [Revised: 11/12/2021] [Accepted: 12/09/2021] [Indexed: 02/03/2023]
Abstract
Aversive motivation plays a prominent role in driving individuals to exert cognitive control. However, the complexity of behavioral responses attributed to aversive incentives creates significant challenges for developing a clear understanding of the neural mechanisms of this motivation-control interaction. We review the animal learning, systems neuroscience, and computational literatures to highlight the importance of experimental paradigms that incorporate both motivational context manipulations and mixed motivational components (e.g., bundling of appetitive and aversive incentives). Specifically, we postulate that to understand aversive incentive effects on cognitive control allocation, a critical contextual factor is whether such incentives are associated with negative reinforcement or punishment. We further illustrate how the inclusion of mixed motivational components in experimental paradigms enables increased precision in the measurement of aversive influences on cognitive control. A sharpened experimental and theoretical focus regarding the manipulation and assessment of distinct motivational dimensions promises to advance understanding of the neural, monoaminergic, and computational mechanisms that underlie the interaction of motivation and cognitive control.
Collapse
Affiliation(s)
- Debbie M Yee
- Cognitive, Linguistic, and Psychological Sciences, Brown University, USA; Carney Institute for Brain Science, Brown University, USA; Department of Psychological and Brain Sciences, Washington University in Saint Louis, USA.
| | - Xiamin Leng
- Cognitive, Linguistic, and Psychological Sciences, Brown University, USA; Carney Institute for Brain Science, Brown University, USA
| | - Amitai Shenhav
- Cognitive, Linguistic, and Psychological Sciences, Brown University, USA; Carney Institute for Brain Science, Brown University, USA
| | - Todd S Braver
- Department of Psychological and Brain Sciences, Washington University in Saint Louis, USA
| |
Collapse
|
48
|
Rosenbaum GM, Grassie HL, Hartley CA. Valence biases in reinforcement learning shift across adolescence and modulate subsequent memory. eLife 2022; 11:e64620. [PMID: 35072624 PMCID: PMC8786311 DOI: 10.7554/elife.64620] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2020] [Accepted: 12/24/2021] [Indexed: 12/12/2022] Open
Abstract
As individuals learn through trial and error, some are more influenced by good outcomes, while others weight bad outcomes more heavily. Such valence biases may also influence memory for past experiences. Here, we examined whether valence asymmetries in reinforcement learning change across adolescence, and whether individual learning asymmetries bias the content of subsequent memory. Participants ages 8-27 learned the values of 'point machines,' after which their memory for trial-unique images presented with choice outcomes was assessed. Relative to children and adults, adolescents overweighted worse-than-expected outcomes during learning. Individuals' valence biases modulated incidental memory, such that those who prioritized worse- (or better-) than-expected outcomes during learning were also more likely to remember images paired with these outcomes, an effect reproduced in an independent dataset. Collectively, these results highlight age-related changes in the computation of subjective value and demonstrate that a valence-asymmetric valuation process influences how information is prioritized in episodic memory.
Collapse
Affiliation(s)
- Gail M Rosenbaum
- Department of Psychology, New York UniversityNew YorkUnited States
| | - Hannah L Grassie
- Department of Psychology, New York UniversityNew YorkUnited States
| | - Catherine A Hartley
- Department of Psychology, New York UniversityNew YorkUnited States
- Center for Neural Science, New York UniversityNew YorkUnited States
| |
Collapse
|
49
|
Dorsal Raphe 5-HT Neurons Utilize, But Do Not Generate, Negative Aversive Prediction Errors. eNeuro 2022; 9:ENEURO.0132-21.2022. [PMID: 35078807 PMCID: PMC8868025 DOI: 10.1523/eneuro.0132-21.2022] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 12/29/2021] [Accepted: 01/11/2022] [Indexed: 11/21/2022] Open
Abstract
The dorsal raphe nucleus (DRN) contains the largest population of serotonin (5-HT) neurons in the central nervous system. 5-HT, synthesized via tryptophan hydroxylase 2 (Tph2), is a widely functioning neuromodulator implicated in fear learning. Here, we sought to investigate whether DRN 5-HT is necessary to reduce fear via negative prediction error (–PE). Using male and female TPH2-cre rats, DRNtph2+ cells were selectively deleted via cre-caspase (rAAV5-Flex-taCasp3-TEVp) in experiment 1. Rats then underwent fear discrimination during which three cues were associated with unique foot shock probabilities: safety p = 0.00, uncertainty p = 0.375, and danger p = 1.00. Rats then received selective extinction to the uncertainty cue, a behavioral manipulation designed to probe –PE. Deleting DRNtph2+ cells had no impact on initial discrimination but slowed selective extinction. In experiment 2, we used a within-subjects optogenetic inhibition design to causally implicate DRNtph2+ cells in prediction error signaling. Male and female TPH2-cre rats received intra-DRN infusions of cre-dependent halorhodopsin (rAAV5-Ef1a-DIO-eNpHR3.0-eYFP) or cre-YFP. DRNtph2+ cells were inhibited specifically during the time of prediction error or a control period. Illumination during either positive prediction error (+PE) or control periods had no impact on fear to the uncertainty cue. Inhibition of DRNtph2+ cells at the time of –PE did not impact immediate fear, but facilitated selective extinction in postillumination sessions. Together, these results demonstrate a role for DRNtph2+ cells in using, but not generating, –PE to weaken cue-shock associations.
Collapse
|
50
|
Lim TV, Cardinal RN, Bullmore ET, Robbins TW, Ersche KD. Impaired Learning From Negative Feedback in Stimulant Use Disorder: Dopaminergic Modulation. Int J Neuropsychopharmacol 2021; 24:867-878. [PMID: 34197589 PMCID: PMC8598302 DOI: 10.1093/ijnp/pyab041] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Revised: 06/17/2021] [Accepted: 06/30/2021] [Indexed: 12/30/2022] Open
Abstract
BACKGROUND Drug-induced alterations to the dopamine system in stimulant use disorder (SUD) are hypothesized to impair reinforcement learning (RL). Computational modeling enables the investigation of the latent processes of RL in SUD patients, which could elucidate the nature of their impairments. METHODS We investigated RL in 44 SUD patients and 41 healthy control participants using a probabilistic RL task that assesses learning from reward and punishment separately. In an independent sample, we determined the modulatory role of dopamine in RL following a single dose of the dopamine D2/3 receptor antagonist amisulpride (400 mg) and the agonist pramipexole (0.5 mg) in a randomised, double-blind, placebo-controlled, crossover design. We analyzed task performance using computational modelling and hypothesized that RL impairments in SUD patients would be differentially modulated by a dopamine D2/3 receptor antagonist and agonist. RESULTS Computational analyses in both samples revealed significantly reduced learning rates from punishment in SUD patients compared with healthy controls, whilst their reward learning rates were not measurably impaired. In addition, the dopaminergic receptor agents modulated RL parameters differentially in both groups. Both amisulpride and pramipexole impaired RL parameters in healthy participants, but ameliorated learning from punishment in SUD patients. CONCLUSION Our findings suggest that RL impairments seen in SUD patients are associated with altered dopamine function.
Collapse
Affiliation(s)
- Tsen Vei Lim
- Department of Psychiatry, University of Cambridge, Cambridge, UK
| | - Rudolf N Cardinal
- Department of Psychiatry, University of Cambridge, Cambridge, UK
- Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK
| | - Edward T Bullmore
- Department of Psychiatry, University of Cambridge, Cambridge, UK
- Cambridgeshire and Peterborough NHS Foundation Trust, Cambridge, UK
| | - Trevor W Robbins
- Department of Psychology, University of Cambridge, Cambridge, UK
| | - Karen D Ersche
- Department of Psychiatry, University of Cambridge, Cambridge, UK
- Institute of Systems Neuroscience, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|