1
|
Bosulu J, Pezzulo G, Hétu S. Needing: An Active Inference Process for Physiological Motivation. J Cogn Neurosci 2024; 36:2011-2028. [PMID: 38940737 DOI: 10.1162/jocn_a_02209] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/29/2024]
Abstract
Need states are internal states that arise from deprivation of crucial biological stimuli. They direct motivation, independently of external learning. Despite their separate origin, they interact with reward processing systems that respond to external stimuli. This article aims to illuminate the functioning of the needing system through the lens of active inference, a framework for understanding brain and cognition. We propose that need states exert a pervasive influence on the organism, which in active inference terms translates to a "pervasive surprise"-a measure of the distance from the organism's preferred state. Crucially, we define needing as an active inference process that seeks to reduce this pervasive surprise. Through a series of simulations, we demonstrate that our proposal successfully captures key aspects of the phenomenology and neurobiology of needing. We show that as need states increase, the tendency to occupy preferred states strengthens, independently of external reward prediction. Furthermore, need states increase the precision of states (stimuli and actions) leading to preferred states, suggesting their ability to amplify the value of reward cues and rewards themselves. Collectively, our model and simulations provide valuable insights into the directional and underlying influence of need states, revealing how this influence amplifies the wanting or liking associated with relevant stimuli.
Collapse
Affiliation(s)
- Juvenal Bosulu
- Université de Montréal
- Centre interdisciplinaire de recherche sur le cerveau et l'apprentissage (CIRCA), Montréal, Québec, Canada
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies (ISTC-CNR), Rome, Italy
| | - Sébastien Hétu
- Université de Montréal
- Centre interdisciplinaire de recherche sur le cerveau et l'apprentissage (CIRCA), Montréal, Québec, Canada
| |
Collapse
|
2
|
Gueguen MCM, Anlló H, Bonagura D, Kong J, Hafezi S, Palminteri S, Konova AB. Recent Opioid Use Impedes Range Adaptation in Reinforcement Learning in Human Addiction. Biol Psychiatry 2024; 95:974-984. [PMID: 38101503 PMCID: PMC11065633 DOI: 10.1016/j.biopsych.2023.12.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 11/22/2023] [Accepted: 12/08/2023] [Indexed: 12/17/2023]
Abstract
BACKGROUND Drugs like opioids are potent reinforcers thought to co-opt value-based decisions by overshadowing other rewarding outcomes, but how this happens at a neurocomputational level remains elusive. Range adaptation is a canonical process of fine-tuning representations of value based on reward context. Here, we tested whether recent opioid exposure impacts range adaptation in opioid use disorder, potentially explaining why shifting decision making away from drug taking during this vulnerable period is so difficult. METHODS Participants who had recently (<90 days) used opioids (n = 34) or who had abstained from opioid use for ≥ 90 days (n = 20) and comparison control participants (n = 44) completed a reinforcement learning task designed to induce robust contextual modulation of value. Two models were used to assess the latent process that participants engaged while making their decisions: 1) a Range model that dynamically tracks context and 2) a standard Absolute model that assumes stationary, objective encoding of value. RESULTS Control participants and ≥90-days-abstinent participants with opioid use disorder exhibited choice patterns consistent with range-adapted valuation. In contrast, participants with recent opioid use were more prone to learn and encode value on an absolute scale. Computational modeling confirmed the behavior of most control participants and ≥90-days-abstinent participants with opioid use disorder (75%), but a minority in the recent use group (38%), was better fit by the Range model than the Absolute model. Furthermore, the degree to which participants relied on range adaptation correlated with duration of continuous abstinence and subjective craving/withdrawal. CONCLUSIONS Reduced context adaptation to available rewards could explain difficulty deciding about smaller (typically nondrug) rewards in the aftermath of drug exposure.
Collapse
Affiliation(s)
- Maëlle C M Gueguen
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, New Jersey; Intercultural Cognitive Network, Tokyo, Japan
| | - Hernán Anlló
- Intercultural Cognitive Network, Tokyo, Japan; Watanabe Laboratory, School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan; Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale U960, École Normale Supérieure-Université de Recherche Paris Science et Lettres, Paris, France
| | - Darla Bonagura
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, New Jersey; Intercultural Cognitive Network, Tokyo, Japan
| | - Julia Kong
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, New Jersey
| | - Sahar Hafezi
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, New Jersey
| | - Stefano Palminteri
- Intercultural Cognitive Network, Tokyo, Japan; Laboratoire de Neurosciences Cognitives et Computationnelles, Institut National de la Santé et de la Recherche Médicale U960, École Normale Supérieure-Université de Recherche Paris Science et Lettres, Paris, France
| | - Anna B Konova
- Department of Psychiatry, Brain Health Institute and University Behavioral Health Care, Rutgers University-New Brunswick, Piscataway, New Jersey; Intercultural Cognitive Network, Tokyo, Japan.
| |
Collapse
|
3
|
Eshel N, Touponse GC, Wang AR, Osterman AK, Shank AN, Groome AM, Taniguchi L, Cardozo Pinto DF, Tucciarone J, Bentzley BS, Malenka RC. Striatal dopamine integrates cost, benefit, and motivation. Neuron 2024; 112:500-514.e5. [PMID: 38016471 PMCID: PMC10922131 DOI: 10.1016/j.neuron.2023.10.038] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Revised: 07/06/2023] [Accepted: 10/26/2023] [Indexed: 11/30/2023]
Abstract
Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.
Collapse
Affiliation(s)
- Neir Eshel
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
| | - Gavin C Touponse
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Allan R Wang
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Amber K Osterman
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Amei N Shank
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Alexandra M Groome
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Lara Taniguchi
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Daniel F Cardozo Pinto
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Jason Tucciarone
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Brandon S Bentzley
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA
| | - Robert C Malenka
- Nancy Pritzker Laboratory, Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, CA, USA.
| |
Collapse
|
4
|
Payzan-LeNestour E, Doran J. Craving money? Evidence from the laboratory and the field. SCIENCE ADVANCES 2024; 10:eadi5034. [PMID: 38215199 PMCID: PMC10786414 DOI: 10.1126/sciadv.adi5034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/29/2023] [Accepted: 12/14/2023] [Indexed: 01/14/2024]
Abstract
Continuing to gamble despite harmful consequences has plagued human life in many ways, from loss-chasing in problem gamblers to reckless investing during stock market bubbles. Here, we propose that these anomalies in human behavior can sometimes reflect Pavlovian perturbations on instrumental behavior. To show this, we combined key elements of Pavlovian psychology literature and standard economic theory into a single model. In it, when a gambling cue such as a gaming machine or a financial asset repeatedly delivers a good outcome, the agent may start engaging with the cue even when the expected value is negative. Next, we transported the theoretical framework into an experimental task and found that participants behaved like the agent in our model. Last, we applied the model to the domain of real-world financial trading and discovered an asset-pricing anomaly suggesting that market participants are susceptible to the purported Pavlovian bias.
Collapse
Affiliation(s)
| | - James Doran
- University of New South Wales Business School, UNSW Sydney, Kensington NSW 2052, Australia
| |
Collapse
|
5
|
Fraser KM, Collins VL, Wolff AR, Ottenheimer DJ, Bornhoft KN, Pat F, Chen BJ, Janak PH, Saunders BT. Contexts facilitate dynamic value encoding in the mesolimbic dopamine system. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.11.05.565687. [PMID: 37961363 PMCID: PMC10635154 DOI: 10.1101/2023.11.05.565687] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.
Collapse
Affiliation(s)
- Kurt M Fraser
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | | | - Amy R Wolff
- Department of Neuroscience, University of Minnesota
| | | | | | - Fiona Pat
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | - Bridget J Chen
- Department of Psychological and Brain Sciences, Johns Hopkins University
| | - Patricia H Janak
- Department of Psychological and Brain Sciences, Johns Hopkins University
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University
| | - Benjamin T Saunders
- Department of Neuroscience, University of Minnesota
- Medical Discovery Team on Addiction, University of Minnesota
| |
Collapse
|
6
|
Iglesias AG, Chiu AS, Wong J, Campus P, Li F, Liu ZN, Bhatti JK, Patel SA, Deisseroth K, Akil H, Burgess CR, Flagel SB. Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping. J Neurosci 2023; 43:7376-7392. [PMID: 37709540 PMCID: PMC10621773 DOI: 10.1523/jneurosci.0848-23.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2023] [Revised: 08/08/2023] [Accepted: 09/08/2023] [Indexed: 09/16/2023] Open
Abstract
The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.
Collapse
Affiliation(s)
- Amanda G Iglesias
- Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan 48104
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Alvin S Chiu
- Neuroscience Graduate Program, University of Michigan, Ann Arbor, Michigan 48104
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Jason Wong
- College of Literature, Science, and the Arts, University of Michigan, Ann Arbor, Michigan 48104
| | - Paolo Campus
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Fei Li
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Zitong Nemo Liu
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Jasmine K Bhatti
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Shiv A Patel
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Karl Deisseroth
- Department of Bioengineering, Stanford University, Stanford, California 94305
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, California 94305
- Howard Hughes Medical Institute, Stanford University, Stanford, California 94305
| | - Huda Akil
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
- Department of Psychiatry, University of Michigan, Ann Arbor, Michigan 48104
| | - Christian R Burgess
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
| | - Shelly B Flagel
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor, Michigan 48104
- Department of Psychiatry, University of Michigan, Ann Arbor, Michigan 48104
| |
Collapse
|
7
|
Kalhan S, Garrido MI, Hester R, Redish AD. Reward prediction-errors weighted by cue salience produces addictive behaviours in simulations, with asymmetrical learning and steeper delay discounting. Neural Netw 2023; 168:631-650. [PMID: 37844522 DOI: 10.1016/j.neunet.2023.09.032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 07/23/2023] [Accepted: 09/19/2023] [Indexed: 10/18/2023]
Abstract
Dysfunction in learning and motivational systems are thought to contribute to addictive behaviours. Previous models have suggested that dopaminergic roles in learning and motivation could produce addictive behaviours through pharmacological manipulations that provide excess dopaminergic signalling towards these learning and motivational systems. Redish (2004) suggested a role based on dopaminergic signals of value prediction error, while (Zhang et al., 2009) suggested a role based on dopaminergic signals of motivation. However, both models present significant limitations. They do not explain the reduced sensitivity to drug-related costs/negative consequences, the increased impulsivity generally found in people with a substance use disorder, craving behaviours, and non-pharmacological dependence, all of which are key hallmarks of addictive behaviours. Here, we propose a novel mathematical definition of salience, that combines aspects of dopamine's role in both learning and motivation within the reinforcement learning framework. Using a single parameter regime, we simulated addictive behaviours that the (Zhang et al., 2009; Redish, 2004) models also produce but we went further in simulating the downweighting of drug-related negative prediction-errors, steeper delay discounting of drug rewards, craving behaviours and aspects of behavioural/non-pharmacological addictions. The current salience model builds on our recently proposed conceptual theory that salience modulates internal representation updating and may contribute to addictive behaviours by producing misaligned internal representations (Kalhan et al., 2021). Critically, our current mathematical model of salience argues that the seemingly disparate learning and motivational aspects of dopaminergic functioning may interact through a salience mechanism that modulates internal representation updating.
Collapse
Affiliation(s)
- Shivam Kalhan
- University of Melbourne, School of Psychological Sciences, Melbourne, Victoria, Australia.
| | - Marta I Garrido
- University of Melbourne, School of Psychological Sciences, Melbourne, Victoria, Australia; Graeme Clark Institute for Biomedical Engineering, Melbourne, Victoria, Australia
| | - Robert Hester
- University of Melbourne, School of Psychological Sciences, Melbourne, Victoria, Australia
| | - A David Redish
- Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA
| |
Collapse
|
8
|
Berridge KC. Separating desire from prediction of outcome value. Trends Cogn Sci 2023; 27:932-946. [PMID: 37543439 PMCID: PMC10527990 DOI: 10.1016/j.tics.2023.07.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Revised: 07/10/2023] [Accepted: 07/14/2023] [Indexed: 08/07/2023]
Abstract
Individuals typically want what they expect to like, often based on memories of previous positive experiences. However, in some situations desire can decouple completely from memories and from learned predictions of outcome value. The potential for desire to separate from prediction arises from independent operating rules that control motivational incentive salience. Incentive salience, or 'wanting', is a type of mesolimbic desire that evolved for adaptive goals, but can also generate maladaptive addictions. Two proof-of-principle examples are presented here to show how motivational 'wanting' can soar above memory-based predictions of outcome value: (i) 'wanting what is remembered to be disgusting', and (ii) 'wanting what is predicted to hurt'. Consequently, even outcomes remembered and predicted to be negatively aversive can become positively 'wanted'. Similarly, in human addictions, people may experience powerful cue-triggered cravings for outcomes that are not predicted to be enjoyable.
Collapse
Affiliation(s)
- Kent C Berridge
- Department of Psychology, University of Michigan, Ann Arbor, MI 48109, USA.
| |
Collapse
|
9
|
Iglesias AG, Chiu AS, Wong J, Campus P, Li F, Liu Z(N, Patel SA, Deisseroth K, Akil H, Burgess CR, Flagel SB. Inhibition of dopamine neurons prevents incentive value encoding of a reward cue: With revelations from deep phenotyping. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.05.03.539324. [PMID: 37205506 PMCID: PMC10187226 DOI: 10.1101/2023.05.03.539324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
The survival of an organism is dependent on their ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the nucleus accumbens is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCut revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient towards and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues. Significance Statement Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCut revealed that cue-directed behaviors do not emerge without VTA dopamine. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of VTA dopamine during cue presentation to encode the incentive value of reward cues.
Collapse
Affiliation(s)
- Amanda G. Iglesias
- Neuroscience Graduate Program, University of Michigan, Ann Arbor 48104, Michigan
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Alvin S. Chiu
- Neuroscience Graduate Program, University of Michigan, Ann Arbor 48104, Michigan
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Jason Wong
- College of Literature, Science, and the Arts, University of Michigan, Ann Arbor 48104, Michigan
| | - Paolo Campus
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Fei Li
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Zitong (Nemo) Liu
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Shiv A. Patel
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Karl Deisseroth
- Department of Bioengineering, Stanford University, Stanford 94305, California
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford 94305, California
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford 94305, California
- Howard Hughes Medical Institute, Stanford University, Stanford 94305, California
| | - Huda Akil
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
- Department of Psychiatry, University of Michigan, Ann Arbor 48104, Michigan
| | - Christian R. Burgess
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
| | - Shelly B. Flagel
- Michigan Neuroscience Institute, University of Michigan, Ann Arbor 48104, Michigan
- Department of Psychiatry, University of Michigan, Ann Arbor 48104, Michigan
| |
Collapse
|
10
|
Wylie J, Gantman A. Doesn't everybody jaywalk? On codified rules that are seldom followed and selectively punished. Cognition 2023; 231:105323. [PMID: 36410059 DOI: 10.1016/j.cognition.2022.105323] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2022] [Revised: 10/28/2022] [Accepted: 11/03/2022] [Indexed: 11/19/2022]
Abstract
Rules are meant to apply equally to all within their jurisdiction. However, some rules are frequently broken without consequence for most. These rules are only occasionally enforced, often at the discretion of a third-party observer. We propose that these rules-whose violations are frequent, and enforcement is rare-constitute a unique subclass of explicitly codified rules, which we call 'phantom rules' (e.g., proscribing jaywalking). Their apparent punishability is ambiguous and particularly susceptible to third-party motives. Across six experiments, (N = 1440) we validated the existence of phantom rules and found evidence for their motivated enforcement. First, people played a modified Dictator Game with a novel frequently broken and rarely enforced rule (i.e., a phantom rule). People enforced this rule more often when the "dictator" was selfish (vs. fair) even though the rule only proscribed fractional offers (not selfishness). Then we turned to third person judgments of the U.S. legal system. We found these violations are recognizable to participants as both illegal and commonplace (Experiment 2), differentiable from violations of prototypical laws (Experiments 3) and enforced in a motivated way (Experiments 4a and 4b). Phantom rule violations (but not prototypical legal violations) are seen as more justifiably punished when the rule violator has also violated a social norm (vs. rule violation alone)-unless the motivation to punish has been satiated (Experiment 5). Phantom rules are frequently broken, codified rules. Consequently, their apparent punishability is ambiguous, and their enforcement is particularly susceptible to third party motives.
Collapse
Affiliation(s)
| | - Ana Gantman
- Brooklyn College, USA; The Graduate Center, City University of New York, USA
| |
Collapse
|
11
|
Read SJ, Miller LC. Behavioral regulation relies on interacting forces and predictive models. J Pers 2023. [PMID: 36696137 DOI: 10.1111/jopy.12815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2021] [Revised: 01/15/2023] [Accepted: 01/18/2023] [Indexed: 01/26/2023]
Abstract
OBJECTIVE We discuss how our recent neural network model of personality and motivation can explain many aspects of the regulation of behavior. METHOD Contrary to approaches that focus on a goal-corrected, set-point, and discrepancy-reducing mechanism, we argue that many aspects of regulation can be understood in terms of two other mechanisms. First, many aspects of the stability and coherence of personality, as well as the dynamics of personality, can be understood in terms of the interaction of forces within organized motivational systems, and their interaction with features of the environment and interoceptive states, that identify an individual's current needs. This has been described as a settling point or equilibrium of forces model, rather than a set-point architecture. Second, regulation has been shown to depend also on the use of predictive models of the world, either learned or innate. Such models can be thought of as feedforward models, in contrast to the feedback models characteristic of set-point, goal-corrected systems. RESULTS AND CONCLUSIONS We describe a neural network model of these processes that simulates the behavior over time and situations of an individual and shows how important regulatory processes can operate through a process of interactive forces and predictive models of the world.
Collapse
Affiliation(s)
- Stephen J Read
- Department of Psychology, University of Southern California, Los Angeles, California, USA
| | - Lynn C Miller
- School of Communication, USC Annenberg School for Communication and Journalism, University of Southern California, Los Angeles, California, USA
| |
Collapse
|
12
|
Marshall AT, Halbout B, Munson CN, Hutson C, Ostlund SB. Flexible control of Pavlovian-instrumental transfer based on expected reward value. JOURNAL OF EXPERIMENTAL PSYCHOLOGY. ANIMAL LEARNING AND COGNITION 2023; 49:14-30. [PMID: 36795420 PMCID: PMC10561628 DOI: 10.1037/xan0000348] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/17/2023]
Abstract
The Pavlovian-instrumental transfer (PIT) paradigm is widely used to assay the motivational influence of reward-predictive cues, reflected by their ability to invigorate instrumental behavior. Leading theories assume that a cue's motivational properties are tied to predicted reward value. We outline an alternative view that recognizes that reward-predictive cues may suppress rather than motivate instrumental behavior under certain conditions, an effect termed positive conditioned suppression. We posit that cues signaling imminent reward delivery tend to inhibit instrumental behavior, which is exploratory by nature, in order to facilitate efficient retrieval of the expected reward. According to this view, the motivation to engage in instrumental behavior during a cue should be inversely related to the value of the predicted reward, since there is more to lose by failing to secure a high-value reward than a low-value reward. We tested this hypothesis in rats using a PIT protocol known to induce positive conditioned suppression. In Experiment 1, cues signaling different reward magnitudes elicited distinct response patterns. Whereas the one-pellet cue increased instrumental behavior, cues signaling three or nine pellets suppressed instrumental behavior and elicited high levels of food-port activity. Experiment 2 found that reward-predictive cues suppressed instrumental behavior and increased food-port activity in a flexible manner that was disrupted by post-training reward devaluation. Further analyses suggest that these findings were not driven by overt competition between the instrumental and food-port responses. We discuss how the PIT task may provide a useful tool for studying cognitive control over cue-motivated behavior in rodents. (PsycInfo Database Record (c) 2023 APA, all rights reserved).
Collapse
Affiliation(s)
- Andrew T Marshall
- Children's Hospital Los Angeles, Department of Pediatrics, University of Southern California
| | - Briac Halbout
- Department of Anesthesiology and Perioperative Care, Department of Neurobiology and Behavior, Irvine Center for Addiction Neuroscience (ICAN), Center for the Neurobiology of Learning and Memory (CNLM), Center for Neural Circuit Mapping (CNCM), University of California Irvine School of Medicine
| | - Christy N Munson
- Department of Anesthesiology and Perioperative Care, Department of Neurobiology and Behavior, Irvine Center for Addiction Neuroscience (ICAN), Center for the Neurobiology of Learning and Memory (CNLM), Center for Neural Circuit Mapping (CNCM), University of California Irvine School of Medicine
| | - Collin Hutson
- Department of Anesthesiology and Perioperative Care, Department of Neurobiology and Behavior, Irvine Center for Addiction Neuroscience (ICAN), Center for the Neurobiology of Learning and Memory (CNLM), Center for Neural Circuit Mapping (CNCM), University of California Irvine School of Medicine
| | - Sean B Ostlund
- Department of Anesthesiology and Perioperative Care, Department of Neurobiology and Behavior, Irvine Center for Addiction Neuroscience (ICAN), Center for the Neurobiology of Learning and Memory (CNLM), Center for Neural Circuit Mapping (CNCM), University of California Irvine School of Medicine
| |
Collapse
|
13
|
Stussi Y, Pool ER. Multicomponential affective processes modulating food-seeking behaviors. Curr Opin Behav Sci 2022. [DOI: 10.1016/j.cobeha.2022.101226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
14
|
Bosulu J, Allaire MA, Tremblay-Grénier L, Luo Y, Eickhoff S, Hétu S. "Wanting" versus "needing" related value: An fMRI meta-analysis. Brain Behav 2022; 12:e32713. [PMID: 36000558 PMCID: PMC9480935 DOI: 10.1002/brb3.2713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Accepted: 07/04/2022] [Indexed: 11/30/2022] Open
Abstract
Consumption and its excesses are sometimes explained by imbalance of need or lack of control over "wanting." "Wanting" assigns value to cues that predict rewards, whereas "needing" assigns value to biologically significant stimuli that one is deprived of. Here we aimed at studying how the brain activation patterns related to value of "wanted" stimuli differs from that of "needed" stimuli using activation likelihood estimation neuroimaging meta-analysis approaches. We used the perception of a cue predicting a reward for "wanting" related value and the perception of food stimuli in a hungry state as a model for "needing" related value. We carried out separate, contrasts, and conjunction meta-analyses to identify differences and similarities between "wanting" and "needing" values. Our overall results for "wanting" related value show consistent activation of the ventral tegmental area, striatum, and pallidum, regions that both activate behavior and direct choice, while for "needing" related value, we found an overall consistent activation of the middle insula and to some extent the caudal-ventral putamen, regions that only direct choice. Our study suggests that wanting has more control on consumption and behavioral activation.
Collapse
Affiliation(s)
- Juvenal Bosulu
- Faculté Des Arts et des Sciences, Université de Montréal, Montréal, Canada
| | | | | | - Yi Luo
- School of Psychology and Cognitive Science, East China Normal University, Shanghai, China
| | - Simon Eickhoff
- Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine University Düsseldorf, Düsseldorf, Germany.,Institute of Neuroscience and Medicine, Brain & Behaviour (INM-7), Research Centre Jülich, Jülich, Germany
| | - Sébastien Hétu
- Faculté Des Arts et des Sciences, Université de Montréal, Montréal, Canada
| |
Collapse
|
15
|
Oyama K, Hori Y, Mimura K, Nagai Y, Eldridge MAG, Saunders RC, Miyakawa N, Hirabayashi T, Hori Y, Inoue KI, Suhara T, Takada M, Higuchi M, Richmond BJ, Minamimoto T. Chemogenetic Disconnection between the Orbitofrontal Cortex and the Rostromedial Caudate Nucleus Disrupts Motivational Control of Goal-Directed Action. J Neurosci 2022; 42:6267-6275. [PMID: 35794012 PMCID: PMC9374131 DOI: 10.1523/jneurosci.0229-22.2022] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 04/20/2022] [Accepted: 06/05/2022] [Indexed: 11/21/2022] Open
Abstract
The orbitofrontal cortex (OFC) and its major downstream target within the basal ganglia-the rostromedial caudate nucleus (rmCD)-are involved in reward-value processing and goal-directed behavior. However, a causal contribution of the pathway linking these two structures to goal-directed behavior has not been established. Using the chemogenetic technology of designer receptors exclusively activated by designer drugs with a crossed inactivation design, we functionally and reversibly disrupted interactions between the OFC and rmCD in two male macaque monkeys. We injected an adeno-associated virus vector expressing an inhibitory designer receptor, hM4Di, into the OFC and contralateral rmCD, the expression of which was visualized in vivo by positron emission tomography and confirmed by postmortem immunohistochemistry. Functional disconnection of the OFC and rmCD resulted in a significant and reproducible loss of sensitivity to the cued reward value for goal-directed action. This decreased sensitivity was most prominent when monkeys had accumulated a certain amount of reward. These results provide causal evidence that the interaction between the OFC and the rmCD is needed for motivational control of action on the basis of the relative reward value and internal drive. This finding extends the current understanding of the physiological basis of psychiatric disorders in which goal-directed behavior is affected, such as obsessive-compulsive disorder.SIGNIFICANCE STATEMENT In daily life, we routinely adjust the speed and accuracy of our actions on the basis of the value of expected reward. Abnormalities in these kinds of motivational adjustments might be related to behaviors seen in psychiatric disorders such as obsessive-compulsive disorder. In the current study, we show that the connection from the orbitofrontal cortex to the rostromedial caudate nucleus is essential for motivational control of action in monkeys. This finding expands our knowledge about how the primate brain controls motivation and behavior and provides a particular insight into disorders like obsessive-compulsive disorder in which altered connectivity between the orbitofrontal cortex and the striatum has been implicated.
Collapse
Affiliation(s)
- Kei Oyama
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Yukiko Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Koki Mimura
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Yuji Nagai
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Mark A G Eldridge
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, Maryland 20892
| | - Richard C Saunders
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, Maryland 20892
| | - Naohisa Miyakawa
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Toshiyuki Hirabayashi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Yuki Hori
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Ken-Ichi Inoue
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama 484-8506, Japan
| | - Tetsuya Suhara
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Masahiko Takada
- Center for the Evolutionary Origins of Human Behavior, Kyoto University, Inuyama 484-8506, Japan
| | - Makoto Higuchi
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, Bethesda, Maryland 20892
| | - Takafumi Minamimoto
- Department of Functional Brain Imaging, National Institutes for Quantum Science and Technology, Chiba 263-8555, Japan
| |
Collapse
|
16
|
Uchida Y, Hikida T, Yamashita Y. Computational Mechanisms of Osmoregulation: A Reinforcement Learning Model for Sodium Appetite. Front Neurosci 2022; 16:857009. [PMID: 35663557 PMCID: PMC9160331 DOI: 10.3389/fnins.2022.857009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2022] [Accepted: 04/15/2022] [Indexed: 11/30/2022] Open
Abstract
Homeostatic control with oral nutrient intake is a vital complex system involving the orderly interactions between the external and internal senses, behavioral control, reward learning, and decision-making. Sodium appetite is a representative system and has been intensively investigated in animal models of homeostatic systems and oral nutrient intake. However, the system-level mechanisms for regulating sodium intake behavior and homeostatic control remain unclear. In the current study, we attempted to provide a mechanistic understanding of sodium appetite behavior by using a computational model, the homeostatic reinforcement learning model, in which homeostatic behaviors are interpreted as reinforcement learning processes. Through simulation experiments, we confirmed that our homeostatic reinforcement learning model successfully reproduced homeostatic behaviors by regulating sodium appetite. These behaviors include the approach and avoidance behaviors to sodium according to the internal states of individuals. In addition, based on the assumption that the sense of taste is a predictor of changes in the internal state, the homeostatic reinforcement learning model successfully reproduced the previous paradoxical observations of the intragastric infusion test, which cannot be explained by the classical drive reduction theory. Moreover, we extended the homeostatic reinforcement learning model to multimodal data, and successfully reproduced the behavioral tests in which water and sodium appetite were mediated by each other. Finally, through an experimental simulation of chemical manipulation in a specific neural population in the brain stem, we proposed a testable hypothesis for the function of neural circuits involving sodium appetite behavior. The study results support the idea that osmoregulation via sodium appetitive behavior can be understood as a reinforcement learning process, and provide a mechanistic explanation for the underlying neural mechanisms of decision-making related to sodium appetite and homeostatic behavior.
Collapse
Affiliation(s)
- Yuuki Uchida
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
- Medical and Dental Sciences, Graduate School of Medical and Dental Sciences, Tokyo Medical and Dental University, Tokyo, Japan
| | - Takatoshi Hikida
- Laboratory for Advanced Brain Functions, Institute for Protein Research, Osaka University, Osaka, Japan
- *Correspondence: Takatoshi Hikida,
| | - Yuichi Yamashita
- Department of Information Medicine, National Institute of Neuroscience, National Center of Neurology and Psychiatry, Tokyo, Japan
- Yuichi Yamashita,
| |
Collapse
|
17
|
Huang J, Zhang Z, Feng W, Zhao Y, Aldanondo A, de Brito Sanchez MG, Paoli M, Rolland A, Li Z, Nie H, Lin Y, Zhang S, Giurfa M, Su S. Food wanting is mediated by transient activation of dopaminergic signaling in the honey bee brain. Science 2022; 376:508-512. [PMID: 35482873 DOI: 10.1126/science.abn9920] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
The biological bases of wanting have been characterized in mammals, but whether an equivalent wanting system exists in insects remains unknown. In this study, we focused on honey bees, which perform intensive foraging activities to satisfy colony needs, and sought to determine whether foragers leave the hive driven by specific expectations about reward and whether they recollect these expectations during their waggle dances. We monitored foraging and dance behavior and simultaneously quantified and interfered with biogenic amine signaling in the bee brain. We show that a dopamine-dependent wanting system is activated transiently in the bee brain by increased appetite and individual recollection of profitable food sources, both en route to the goal and during waggle dances. Our results show that insects share with mammals common neural mechanisms for encoding wanting of stimuli with positive hedonic value.
Collapse
Affiliation(s)
- Jingnan Huang
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Zhaonan Zhang
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Wangjiang Feng
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Yuanhong Zhao
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Anna Aldanondo
- Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), University of Toulouse, CNRS, UPS, 31062 Toulouse cedex 9, France
| | - Maria Gabriela de Brito Sanchez
- Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), University of Toulouse, CNRS, UPS, 31062 Toulouse cedex 9, France
| | - Marco Paoli
- Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), University of Toulouse, CNRS, UPS, 31062 Toulouse cedex 9, France
| | - Angele Rolland
- Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), University of Toulouse, CNRS, UPS, 31062 Toulouse cedex 9, France
| | - Zhiguo Li
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Hongyi Nie
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Yan Lin
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| | - Shaowu Zhang
- Australian Research Council Centre of Excellence in Vision Science, Research School of Biology, College of Medicine, Biology and Environment, The Australian National University, Canberra, ACT 2601, Australia
| | - Martin Giurfa
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China.,Centre de Recherches sur la Cognition Animale, Centre de Biologie Intégrative (CBI), University of Toulouse, CNRS, UPS, 31062 Toulouse cedex 9, France.,Institut Universitaire de France (IUF), Paris, France
| | - Songkun Su
- College of Animal Sciences (College of Bee Science), Fujian Agriculture and Forestry University, Fuzhou 350002, China
| |
Collapse
|
18
|
van den Ende MW, Epskamp S, Lees MH, van der Maas HL, Wiers RW, Sloot PM. A review of mathematical modeling of addiction regarding both (neuro-) psychological processes and the social contagion perspectives. Addict Behav 2022; 127:107201. [PMID: 34959078 DOI: 10.1016/j.addbeh.2021.107201] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Revised: 07/04/2021] [Accepted: 11/22/2021] [Indexed: 12/16/2022]
Abstract
Addiction is a complex biopsychosocial phenomenon, impacted by biological predispositions, psychological processes, and the social environment. Using mathematical and computational models that allow for surrogative reasoning may be a promising avenue for gaining a deeper understanding of this complex behavior. This paper reviews and classifies a selection of formal models of addiction focusing on the intra- and inter-individual dynamics, i.e., (neuro) psychological models and social models. We find that these modeling approaches to addiction are too disjoint and argue that in order to unravel the complexities of biopsychosocial processes of addiction, models should integrate intra- and inter-individual factors.
Collapse
|
19
|
Chou T, D’Orsogna MR. A mathematical model of reward-mediated learning in drug addiction. CHAOS (WOODBURY, N.Y.) 2022; 32:021102. [PMID: 35232044 PMCID: PMC8816520 DOI: 10.1063/5.0082997] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2023]
Abstract
Substances of abuse are known to activate and disrupt neuronal circuits in the brain reward system. We propose a simple and easily interpretable dynamical systems model to describe the neurobiology of drug addiction that incorporates the psychiatric concepts of reward prediction error, drug-induced incentive salience, and opponent process theory. Drug-induced dopamine releases activate a biphasic reward response with pleasurable, positive "a-processes" (euphoria, rush) followed by unpleasant, negative "b-processes" (cravings, withdrawal). Neuroadaptive processes triggered by successive intakes enhance the negative component of the reward response, which the user compensates for by increasing drug dose and/or intake frequency. This positive feedback between physiological changes and drug self-administration leads to habituation, tolerance, and, eventually, to full addiction. Our model gives rise to qualitatively different pathways to addiction that can represent a diverse set of user profiles (genetics, age) and drug potencies. We find that users who have, or neuroadaptively develop, a strong b-process response to drug consumption are most at risk for addiction. Finally, we include possible mechanisms to mitigate withdrawal symptoms, such as through the use of methadone or other auxiliary drugs used in detoxification.
Collapse
Affiliation(s)
- Tom Chou
- Department of Computational Medicine, UCLA, Los Angeles, California 90095-1766, USA
| | - Maria R. D’Orsogna
- Department of Mathematics, California State University at Northridge, Los Angeles, California 91130-8313, USA
- Also at: Department of Computational Medicine, UCLA, Los Angeles, CA 90095-1766, USA. Author to whom correspondence should be addressed:
| |
Collapse
|
20
|
A Computational View on the Nature of Reward and Value in Anhedonia. Curr Top Behav Neurosci 2021; 58:421-441. [PMID: 34935117 DOI: 10.1007/7854_2021_290] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
Anhedonia - a common feature of depression and other neuropsychiatric disorders - encompasses a reduction in the subjective experience and anticipation of rewarding events, and a reduction in the motivation to seek out such events. The presence of anhedonia often predicts or accompanies treatment resistance, and as such better interventions and treatments are important. Yet the mechanisms giving rise to anhedonia are not well understood. In this chapter, we briefly review existing computational conceptualisations of anhedonia. We argue that they are mostly descriptive and fail to provide an explanatory account of why anhedonia may occur. Working within the framework of reinforcement learning, we examine two potential computational mechanisms that could give rise to anhedonic phenomena. First, we show how anhedonia can arise in multi-dimensional drive-reduction settings through a trade-off between different rewards or needs. We then generalise this in terms of model-based value inference and identify a key role for associational belief structure. We close with a brief discussion of treatment implications of both of these conceptualisations. In summary, computational accounts of anhedonia have provided a useful descriptive framework. Recent advances in reinforcement learning suggest promising avenues by which the mechanisms underlying anhedonia may be teased apart, potentially motivating novel approaches to treatment.
Collapse
|
21
|
Quantifying the instrumental and noninstrumental underpinnings of Pavlovian responding with the Price equation. Psychon Bull Rev 2021; 29:1295-1306. [PMID: 34918283 DOI: 10.3758/s13423-021-02047-z] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/28/2021] [Indexed: 11/08/2022]
Abstract
The Price equation is a mathematical expression of selectionist and non-selectionist pressures on biological, cultural, and behavioral change. We use it here to specify instrumental and noninstrumental behaviors as they arise within the context of the Pavlovian autoshaping procedure, for rats trained under reward certainty and reward uncertainty. The point of departure for this endeavor is that some portion of autoshaped behavior referred to as goal-tracking appears instrumental-a function of resource attainment (the individual approaches the location where the unconditioned stimulus is to be delivered). By contrast, some other portion of autoshaped behavior referred to as sign-tracking is noninstrumental-irrelevant to making contact with the to-be-delivered unconditioned stimulus. A Price equation model is proposed that unifies our understanding of Pavlovian autoshaping behavior by isolating operant and respondent influences on goal-tracking (instrumental) and sign-tracking (noninstrumental) behavior.
Collapse
|
22
|
Modeling incentive salience in Pavlovian learning more parsimoniously using a multiple attribute model. COGNITIVE AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2021; 22:244-257. [PMID: 34676496 DOI: 10.3758/s13415-021-00953-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 09/02/2021] [Indexed: 11/08/2022]
Abstract
We present a multi-attribute incentive salience (MAIS) model as a computational account of incentive salience in model-based Pavlovian learning. A model of incentive salience as a joint function of reward value and physiological state has been previously proposed by Zhang et al. (2009). In that model, the function takes additive or multiplicative forms depending on whether a preference shifts from positive to negative or vice versa. We demonstrate that arbitrarily varying this function is unnecessary to explain observed data. A multiplicative function is sufficient if one takes into account empirical data suggesting the incentive salience function for an incentive is comprised of multiple physiological signals. We compare our model to the previously proposed model on two datasets. We find the MAIS model predicts the outcomes equally well, fits empirical data describing multiple sensory representations of a single stimulus, better approximates the dual-structure appetitive-aversive nature of the reward system, is compatible with existing knowledge about incentive salience in Pavlovian learning, and better describes revaluation in Pavlovian learning. This model addresses a call (Dayan & Berridge, 2014) for algorithmic and computational models of model-based Pavlovian learning that consistently and fully explain empirical observations. Because a multi-attribute model is relevant even for simple Pavlovian associations, it should be useful in a wide variety of decision-making contexts, including agent modeling and addiction research.
Collapse
|
23
|
Wang LL, Lam CYT, Huang J, Cheung EFC, Lui SSY, Chan RCK. Range-Adaptive Value Representation in Different Stages of Schizophrenia: A Proof of Concept Study. Schizophr Bull 2021; 47:1524-1533. [PMID: 34420057 PMCID: PMC8530390 DOI: 10.1093/schbul/sbab099] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
Amotivation is related to value representation. A comprehensive account of amotivation requires a mechanistic understanding of how the brain exploits external information to represent value. To achieve maximal value discriminability, brain valuation system will dynamically adapt its coding sensitivity to the range of values available in any given condition, so-called range adaptive coding. We administered an experimental task to 30 patients with chronic schizophrenia (C-SCZ), 30 first-episode schizophrenia (FE-SCZ), 34 individuals with high social anhedonia (HSoA), and their paired controls to assess range adaptation ability. C-SCZ patients exhibited over-adaptation and their performances were negatively correlated with avolition symptoms and positive symptoms and positively correlated with blunted-affect symptoms and self-reported consummatory interpersonal pleasure scores, though the results were non-significant. FE-SCZ patients exhibited reduced adaptation, which was significantly and negatively correlated with avolition symptoms and positively correlated with the overall proportion of choosing to exert more effort. Although HSoA participants exhibited comparable range adaptation to controls, their performances were significantly and negatively correlated with the proportion of choosing to exert more effort under the lowest value condition. Our results suggest that different stages of schizophrenia spectrum showed distinct range adaptation patterns. Range adaptation impairments may index a possible underlying mechanism for amotivation symptoms in FE-SCZ and more complicated and pervasive effects on clinical symptoms in C-SCZ.
Collapse
Affiliation(s)
- Ling-Ling Wang
- Neuropsychology and Applied Cognitive Neuroscience Laboratory; CAS Key Laboratory of Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Christina Y T Lam
- Castle Peak Hospital, Hong Kong Special Administrative Region, China
| | - Jia Huang
- Neuropsychology and Applied Cognitive Neuroscience Laboratory; CAS Key Laboratory of Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| | - Eric F C Cheung
- Castle Peak Hospital, Hong Kong Special Administrative Region, China
| | - Simon S Y Lui
- Castle Peak Hospital, Hong Kong Special Administrative Region, China
- Department of Psychiatry, The University of Hong Kong, Hong Kong Special Administrative Region, China
| | - Raymond C K Chan
- Neuropsychology and Applied Cognitive Neuroscience Laboratory; CAS Key Laboratory of Mental Health, Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- Department of Psychology, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
24
|
Effort-motivated behavior resolves paradoxes in appetitive conditioning. Behav Processes 2021; 193:104525. [PMID: 34601051 DOI: 10.1016/j.beproc.2021.104525] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2021] [Revised: 09/22/2021] [Accepted: 09/28/2021] [Indexed: 11/23/2022]
Abstract
Motivated behavior has long been studied by psychologists, ethologists, and neuroscientists. To date, many scientists agree with the view that cue and reward attraction is the product of a dopamine-dependent unconscious process called incentive salience or "wanting". This process allows the influence of multiple factors such as hunger and odors on motivational attraction. In some cases, however, the resulting motivated behavior differs from what the incentive salience hypothesis would predict. I argue that seeking behavior under reward uncertainty illustrates this situation: Organisms do not just "want" (appetite-based attraction) cues that are inconsistent or associated with reward occasionally, they "hope" that those cues will consistently predict reward procurement in the ongoing trial. Said otherwise, they become motivated to invest time and energy to find consistent cue-reward associations despite no guarantee of success (effort-based attraction). A multi-test comparison of performance between individuals trained under uncertainty and certainty reveals behavioral paradoxes suggesting that the concept of incentive salience cannot fully account for responding to inconsistent cues. A mathematical model explains how appetite-based and effort-based attractions might combine their effects.
Collapse
|
25
|
Computational Mechanisms of Addiction: Recent Evidence and Its Relevance to Addiction Medicine. CURRENT ADDICTION REPORTS 2021. [DOI: 10.1007/s40429-021-00399-z] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
|
26
|
Ostlund SB, Marshall AT. Probing the role of reward expectancy in Pavlovian-instrumental transfer. Curr Opin Behav Sci 2021. [DOI: 10.1016/j.cobeha.2021.04.021] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
27
|
Hori Y, Mimura K, Nagai Y, Fujimoto A, Oyama K, Kikuchi E, Inoue KI, Takada M, Suhara T, Richmond BJ, Minamimoto T. Single caudate neurons encode temporally discounted value for formulating motivation for action. eLife 2021; 10:e61248. [PMID: 34328413 PMCID: PMC8352586 DOI: 10.7554/elife.61248] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2020] [Accepted: 07/29/2021] [Indexed: 11/27/2022] Open
Abstract
The term 'temporal discounting' describes both choice preferences and motivation for delayed rewards. Here we show that neuronal activity in the dorsal part of the primate caudate head (dCDh) signals the temporally discounted value needed to compute the motivation for delayed rewards. Macaque monkeys performed an instrumental task, in which visual cues indicated the forthcoming size and delay duration before reward. Single dCDh neurons represented the temporally discounted value without reflecting changes in the animal's physiological state. Bilateral pharmacological or chemogenetic inactivation of dCDh markedly distorted the normal task performance based on the integration of reward size and delay, but did not affect the task performance for different reward sizes without delay. These results suggest that dCDh is involved in encoding the integrated multi-dimensional information critical for motivation.
Collapse
Affiliation(s)
- Yukiko Hori
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Koki Mimura
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Yuji Nagai
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Atsushi Fujimoto
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Kei Oyama
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Erika Kikuchi
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Ken-ichi Inoue
- Systems Neuroscience Section, Primate Research Institute, Kyoto UniversityInuyamaJapan
| | - Masahiko Takada
- Systems Neuroscience Section, Primate Research Institute, Kyoto UniversityInuyamaJapan
| | - Tetsuya Suhara
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| | - Barry J Richmond
- Laboratory of Neuropsychology, National Institute of Mental Health, National Institutes of Health, Department of Health and Human ServicesBethesdaUnited States
| | - Takafumi Minamimoto
- Department of Functional Brain Imaging, National Institutes for Quantum and Radiological Science and TechnologyChibaJapan
| |
Collapse
|
28
|
Abstract
The rise of computational modeling in the past decade has led to a substantial increase in the number of papers that report parameter estimates of computational cognitive models. A common application of computational cognitive models is to quantify individual differences in behavior by estimating how these are expressed in differences in parameters. For these inferences to hold, models need to be identified, meaning that one set of parameters is most likely, given the behavior under consideration. For many models, model identification can be achieved up to a scaling constraint, which means that under the assumption that one parameter has a specific value, all remaining parameters are identified. In the current note, we argue that this scaling constraint implies a strong assumption about the cognitive process that the model is intended to explain, and warn against an overinterpretation of the associative relations found in this way. We will illustrate these points using signal detection theory, reinforcement learning models, and the linear ballistic accumulator model, and provide suggestions for a clearer interpretation of modeling results.
Collapse
|
29
|
Warlow SM, Berridge KC. Incentive motivation: 'wanting' roles of central amygdala circuitry. Behav Brain Res 2021; 411:113376. [PMID: 34023307 DOI: 10.1016/j.bbr.2021.113376] [Citation(s) in RCA: 29] [Impact Index Per Article: 9.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/03/2021] [Revised: 05/16/2021] [Accepted: 05/18/2021] [Indexed: 12/28/2022]
Abstract
The central nucleus of amygdala (CeA) mediates positively-valenced reward motivation as well as negatively-valenced fear. Optogenetic or neurochemical stimulation of CeA circuitry can generate intense incentive motivation to pursue and consume a paired natural food, sex, or addictive drug reward, and even create maladaptive 'wanting what hurts' such as attraction to a shock rod. Evidence indicates CeA stimulations selectively amplify incentive motivation ('wanting') but not hedonic impact ('liking') of the same reward. Further, valence flips can occur for CeA contributions to motivational salience. That is, CeA stimulation can promote either incentive motivation or fearful motivation, even in the same individual, depending on situation. These findings may carry implications for understanding CeA roles in neuropsychiatric disorders involving aberrant motivational salience, ranging from addiction to paranoia and anxiety disorders.
Collapse
Affiliation(s)
- Shelley M Warlow
- Department of Neurosciences, University of California San Diego, La Jolla, CA, USA.
| | - Kent C Berridge
- Department of Psychology, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
30
|
Ravichandran S, Bhatt RR, Pandit B, Osadchiy V, Alaverdyan A, Vora P, Stains J, Naliboff B, Mayer EA, Gupta A. Alterations in reward network functional connectivity are associated with increased food addiction in obese individuals. Sci Rep 2021; 11:3386. [PMID: 33564081 PMCID: PMC7873272 DOI: 10.1038/s41598-021-83116-0] [Citation(s) in RCA: 24] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2020] [Accepted: 01/07/2021] [Indexed: 12/19/2022] Open
Abstract
Functional neuroimaging studies in obesity have identified alterations in the connectivity within the reward network leading to decreased homeostatic control of ingestive behavior. However, the neural mechanisms underlying sex differences in the prevalence of food addiction in obesity is unknown. The aim of the study was to identify functional connectivity alterations associated with: (1) Food addiction, (2) Sex- differences in food addiction, (3) Ingestive behaviors. 150 participants (females: N = 103, males: N = 47; food addiction: N = 40, no food addiction: N = 110) with high BMI ≥ 25 kg/m2 underwent functional resting state MRIs. Participants were administered the Yale Food Addiction Scale (YFAS), to determine diagnostic criteria for food addiction (YFAS Symptom Count ≥ 3 with clinically significant impairment or distress), and completed ingestive behavior questionnaires. Connectivity differences were analyzed using a general linear model in the CONN Toolbox and images were segmented using the Schaefer 400, Harvard-Oxford Subcortical, and Ascending Arousal Network atlases. Significant connectivities and clinical variables were correlated. Statistical significance was corrected for multiple comparisons at q < .05. (1) Individuals with food addiction had greater connectivity between brainstem regions and the orbital frontal gyrus compared to individuals with no food addiction. (2) Females with food addiction had greater connectivity in the salience and emotional regulation networks and lowered connectivity between the default mode network and central executive network compared to males with food addiction. (3) Increased connectivity between regions of the reward network was positively associated with scores on the General Food Cravings Questionnaire-Trait, indicative of greater food cravings in individuals with food addiction. Individuals with food addiction showed greater connectivity between regions of the reward network suggesting dysregulation of the dopaminergic pathway. Additionally, greater connectivity in the locus coeruleus could indicate that the maladaptive food behaviors displayed by individuals with food addiction serve as a coping mechanism in response to pathological anxiety and stress. Sex differences in functional connectivity suggest that females with food addiction engage more in emotional overeating and less cognitive control and homeostatic processing compared to males. These mechanistic pathways may have clinical implications for understanding the sex-dependent variability in response to diet interventions.
Collapse
Affiliation(s)
- Soumya Ravichandran
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
| | - Ravi R Bhatt
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
- Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, University of Southern California, Los Angeles, USA
| | - Bilal Pandit
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
| | - Vadim Osadchiy
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
- David Geffen School of Medicine At UCLA, Los Angeles, USA
| | - Anita Alaverdyan
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
| | - Priten Vora
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
| | - Jean Stains
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
- David Geffen School of Medicine At UCLA, Los Angeles, USA
- Vatche and Tamar Manoukian Division of Digestive Diseases, Los Angeles, USA
| | - Bruce Naliboff
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
- David Geffen School of Medicine At UCLA, Los Angeles, USA
- Vatche and Tamar Manoukian Division of Digestive Diseases, Los Angeles, USA
- UCLA Microbiome Center, Los Angeles, USA
| | - Emeran A Mayer
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA
- David Geffen School of Medicine At UCLA, Los Angeles, USA
- Vatche and Tamar Manoukian Division of Digestive Diseases, Los Angeles, USA
- UCLA Microbiome Center, Los Angeles, USA
- Ahmanson-Lovelace Brain Mapping Center, University of California Los Angeles (UCLA), Los Angeles, USA
| | - Arpana Gupta
- G. Oppenheimer Family Center for Neurobiology of Stress and Resilience, Ingestive Behavior and Obesity Program, CHS 42-210 MC737818, 10833 Le Conte Avenue, Los Angeles, USA.
- David Geffen School of Medicine At UCLA, Los Angeles, USA.
- Vatche and Tamar Manoukian Division of Digestive Diseases, Los Angeles, USA.
- UCLA Microbiome Center, Los Angeles, USA.
| |
Collapse
|
31
|
Shuvaev SA, Tran NB, Stephenson-Jones M, Li B, Koulakov AA. Neural Networks With Motivation. Front Syst Neurosci 2021; 14:609316. [PMID: 33536879 PMCID: PMC7848953 DOI: 10.3389/fnsys.2020.609316] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2020] [Accepted: 12/08/2020] [Indexed: 11/13/2022] Open
Abstract
Animals rely on internal motivational states to make decisions. The role of motivational salience in decision making is in early stages of mathematical understanding. Here, we propose a reinforcement learning framework that relies on neural networks to learn optimal ongoing behavior for dynamically changing motivation values. First, we show that neural networks implementing Q-learning with motivational salience can navigate in environment with dynamic rewards without adjustments in synaptic strengths when the needs of an agent shift. In this setting, our networks may display elements of addictive behaviors. Second, we use a similar framework in hierarchical manager-agent system to implement a reinforcement learning algorithm with motivation that both infers motivational states and behaves. Finally, we show that, when trained in the Pavlovian conditioning setting, the responses of the neurons in our model resemble previously published neuronal recordings in the ventral pallidum, a basal ganglia structure involved in motivated behaviors. We conclude that motivation allows Q-learning networks to quickly adapt their behavior to conditions when expected reward is modulated by agent's dynamic needs. Our approach addresses the algorithmic rationale of motivation and makes a step toward better interpretability of behavioral data via inference of motivational dynamics in the brain.
Collapse
Affiliation(s)
- Sergey A. Shuvaev
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, United States
| | - Ngoc B. Tran
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, United States
| | - Marcus Stephenson-Jones
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, United States
- Sainsbury Wellcome Centre, University College London, London, United Kingdom
| | - Bo Li
- Cold Spring Harbor Laboratory, Cold Spring Harbor, NY, United States
| | | |
Collapse
|
32
|
Blain B, Rutledge RB. Momentary subjective well-being depends on learning and not reward. eLife 2020; 9:57977. [PMID: 33200989 PMCID: PMC7755387 DOI: 10.7554/elife.57977] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/17/2020] [Accepted: 11/16/2020] [Indexed: 01/20/2023] Open
Abstract
Subjective well-being or happiness is often associated with wealth. Recent studies suggest that momentary happiness is associated with reward prediction error, the difference between experienced and predicted reward, a key component of adaptive behaviour. We tested subjects in a reinforcement learning task in which reward size and probability were uncorrelated, allowing us to dissociate between the contributions of reward and learning to happiness. Using computational modelling, we found convergent evidence across stable and volatile learning tasks that happiness, like behaviour, is sensitive to learning-relevant variables (i.e. probability prediction error). Unlike behaviour, happiness is not sensitive to learning-irrelevant variables (i.e. reward prediction error). Increasing volatility reduces how many past trials influence behaviour but not happiness. Finally, depressive symptoms reduce happiness more in volatile than stable environments. Our results suggest that how we learn about our world may be more important for how we feel than the rewards we actually receive. Many people believe they would be happier if only they had more money. And events such as winning the lottery or receiving a large pay rise do make people happy, at least temporarily. But recent studies suggest that the main factor driving happiness on such occasions is not the size of the reward received. Instead, it is how well that reward matches up with expectations. Receiving a 10% pay rise when you were expecting 1% will make you feel happier than receiving 10% when you had been expecting 20%. This difference between an expected and an actual reward is referred to as a reward prediction error. Reward prediction errors have a key role in learning. They motivate people to repeat behaviours that led to unexpectedly large rewards. But they also enable people to update their beliefs about the world, which is rewarding in itself. Could it be that reward prediction errors are associated with happiness mainly because they help us understand the world a little better than before? To test this idea, Blain and Rutledge designed a task in which the likelihood of receiving a reward was unrelated to the size of the reward. This study design makes it possible to separate out the contributions of learning versus reward to moment-by-moment happiness. In the task, volunteers had to decide which of two cars would win a race. In the ‘stable’ condition, one of the cars always had an 80% chance of winning. In the ‘volatile’ condition, one car had an 80% chance of winning for the first 20 trials. The other car then had an 80% chance of winning for the next 20 trials. The volunteers were not told these probabilities in advance, but had to work them out by playing the game. However, on every trial, the volunteers were shown the reward they would receive if they chose either of the cars and that car went on to win. The size of the rewards varied at random and was unrelated to the likelihood of a car winning. Every few trials, the volunteers were asked to indicate their current level of happiness on a scale. The results showed that volunteers were happier after winning than after losing. On average they were also happier in the stable condition than in the volatile condition. This was especially true for volunteers with pre-existing symptoms of depression. Moreover, happiness after wins did not depend on how large the reward they got was, but instead simply on how surprised they were to win. These results suggest that how we learn about the world around us can be more important for how we feel than rewards we receive directly. Measuring happiness in various types of environment could help us understand factors affecting mental health. The current results suggest, for example, that uncertain environments may be especially unpleasant for people with depression. Further research is needed to understand why this might be the case. In the real world, rewards are often uncertain and infrequent, but learning may nevertheless have the potential to boost happiness.
Collapse
Affiliation(s)
- Bastien Blain
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom
| | - Robb B Rutledge
- Max Planck UCL Centre for Computational Psychiatry and Ageing Research, University College London, London, United Kingdom.,Wellcome Centre for Human Neuroimaging, University College London, London, United Kingdom.,Department of Psychology, Yale University, New Haven, United States
| |
Collapse
|
33
|
Ottenheimer DJ, Wang K, Tong X, Fraser KM, Richard JM, Janak PH. Reward activity in ventral pallidum tracks satiety-sensitive preference and drives choice behavior. SCIENCE ADVANCES 2020; 6:6/45/eabc9321. [PMID: 33148649 PMCID: PMC7673692 DOI: 10.1126/sciadv.abc9321] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Accepted: 09/18/2020] [Indexed: 06/11/2023]
Abstract
A key function of the nervous system is producing adaptive behavior across changing conditions, like physiological state. Although states like thirst and hunger are known to impact decision-making, the neurobiology of this phenomenon has been studied minimally. Here, we tracked evolving preference for sucrose and water as rats proceeded from a thirsty to sated state. As rats shifted from water choices to sucrose choices across the session, the activity of a majority of neurons in the ventral pallidum, a region crucial for reward-related behaviors, closely matched the evolving behavioral preference. The timing of this signal followed the pattern of a reward prediction error, occurring at the cue or the reward depending on when reward identity was revealed. Additionally, optogenetic stimulation of ventral pallidum neurons at the time of reward was able to reverse behavioral preference. Our results suggest that ventral pallidum neurons guide reward-related decisions across changing physiological states.
Collapse
Affiliation(s)
- David J Ottenheimer
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University, Baltimore, MD, USA
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Karen Wang
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Xiao Tong
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Kurt M Fraser
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
| | - Jocelyn M Richard
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA
| | - Patricia H Janak
- The Solomon H. Snyder Department of Neuroscience, Johns Hopkins University, Baltimore, MD, USA.
- Department of Psychological and Brain Sciences, Johns Hopkins University, Baltimore, MD, USA
- Kavli Neuroscience Discovery Institute, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
34
|
FitzGibbon L, Lau JKL, Murayama K. The seductive lure of curiosity: information as a motivationally salient reward. Curr Opin Behav Sci 2020. [DOI: 10.1016/j.cobeha.2020.05.014] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/15/2022]
|
35
|
Pimontel MA, Kanellopoulos D, Gunning FM. Neuroanatomical Abnormalities in Older Depressed Adults With Apathy: A Systematic Review. J Geriatr Psychiatry Neurol 2020; 33:289-303. [PMID: 31635522 DOI: 10.1177/0891988719882100] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
OBJECTIVE Apathy is a common phenomenon in late-life depression and is associated with poor outcomes. Apathy is often unrecognized in older depressed adults, and efficacious treatment options are lacking. This review provides a systematic review of the neuroanatomical abnormalities associated with apathy in late-life depression. In addition, the review summarizes the neuroimaging findings from studies of neurodegenerative and focal brain injury conditions that frequently present with apathy. The goal is to elucidate cerebral network abnormalities that give rise to apathy in older adults with mood disturbances and to inform future treatment targets. METHOD Systematic literature review. RESULTS The few studies that have directly examined the neuroanatomical abnormalities of apathy in late-life depression suggest disturbances in the anterior cingulate cortex, insula, orbital and dorsal prefrontal cortex, striatum, and limbic structures (ie, amygdala, thalamus, and hippocampus). Studies examining the neuroanatomical correlates of apathy in other aging populations are consistent with the pattern observed in late-life depression. CONCLUSIONS Apathy in late-life depression appears to be accompanied by neuroanatomical abnormalities in the salience and reward networks. These network findings are consistent with that observed in individuals presenting with apathy in other aging-related conditions. These findings may inform future treatments that target apathy.
Collapse
Affiliation(s)
- Monique A Pimontel
- Graduate Center, City University of New York, New York, NY, USA.,Department of Psychiatry, Weill Cornell Medicine, New York, NY, USA
| | | | - Faith M Gunning
- Department of Psychiatry, Weill Cornell Medicine, New York, NY, USA
| |
Collapse
|
36
|
Duarte IC, Coelho G, Brito-Costa S, Cayolla R, Afonso S, Castelo-Branco M. Ventral Caudate and Anterior Insula Recruitment During Value Estimation of Passionate Rewarding Cues. Front Neurosci 2020; 14:678. [PMID: 32848534 PMCID: PMC7403482 DOI: 10.3389/fnins.2020.00678] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2020] [Accepted: 06/02/2020] [Indexed: 12/25/2022] Open
Abstract
“Wanting”, a component of reward processing, is a motivational property that guides decision making in goal-oriented behavior. This includes behavior aiming at supporting relational bonds, even at the group level. Accordingly, group belongingness works as this motivational property, which is fundamentally different from romantic or maternal love. While primary rewards (or learned associations, such as money) have been largely used to study the conceptual framework associated with “wanting,” other cues triggering behavior, such as passionate motives, are less well-studied. We investigated the neural correlates of value estimation of a passion-driven incentive in neuropsychologically defined football fans. We asked the participants (n = 57) to compute the value of football tickets (the cues that trigger passionate behavior in this “tribal love” context). The trials were all different, comprising tickets for different matches. The participants had no restrictions on the amount to be introduced. This enabled a parametric functional magnetic resonance imaging design based on the explicit estimated value given by the participants in a trial-by-trial approach. Using a whole-brain approach (to prevent biased focus on value-related regions), only the activity in the ventral caudate and left anterior insula showed a critical relationship with the reported value. Higher normalized values led to more activity in the striatum and left insula. The parametric map shows that these regions encode the magnitude of incentive by indexing self-relevant value. Other regions were involved in value computation, such as the ventromedial prefrontal cortex, lateral orbitofrontal cortex, and dorsolateral prefrontal cortex, but did not exhibit parametric patterns. The involvement of the nucleus accumbens in value estimation was only found in region of interest -based analysis, which emphasizes the role of the ventral caudate for the presently studied social “reinforcer” cue.
Collapse
Affiliation(s)
- Isabel Catarina Duarte
- Coimbra Institute for Biomedical Imaging and Translational Research, Institute for Nuclear Sciences Applied to Health, Faculty of Medicine, University of Coimbra, Coimbra, Portugal
| | - Gonçalo Coelho
- Coimbra Institute for Biomedical Imaging and Translational Research, Institute for Nuclear Sciences Applied to Health, Faculty of Medicine, University of Coimbra, Coimbra, Portugal
| | - Sónia Brito-Costa
- Human Potential Development Center, Institute of Applied Research, Polytechnic Institute of Coimbra, Coimbra, Portugal
| | - Ricardo Cayolla
- Department of Economics, Management, Industrial Engineering and Tourism, University of Aveiro, Aveiro, Portugal.,Porto Business School, University of Porto, Porto, Portugal
| | - Sónia Afonso
- Coimbra Institute for Biomedical Imaging and Translational Research, Institute for Nuclear Sciences Applied to Health, Faculty of Medicine, University of Coimbra, Coimbra, Portugal
| | - Miguel Castelo-Branco
- Coimbra Institute for Biomedical Imaging and Translational Research, Institute for Nuclear Sciences Applied to Health, Faculty of Medicine, University of Coimbra, Coimbra, Portugal
| |
Collapse
|
37
|
Mollick JA, Kober H. Computational models of drug use and addiction: A review. JOURNAL OF ABNORMAL PSYCHOLOGY 2020; 129:544-555. [PMID: 32757599 PMCID: PMC7416739 DOI: 10.1037/abn0000503] [Citation(s) in RCA: 32] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).
Collapse
Affiliation(s)
- Jessica A Mollick
- Clinical and Affective Neuroscience Lab, Department of Psychiatry, Yale University
| | - Hedy Kober
- Clinical and Affective Neuroscience Lab, Department of Psychiatry, Yale University
| |
Collapse
|
38
|
Klein S, Kruse O, Markert C, Tapia León I, Strahler J, Stark R. Subjective reward value of visual sexual stimuli is coded in human striatum and orbitofrontal cortex. Behav Brain Res 2020; 393:112792. [PMID: 32598998 DOI: 10.1016/j.bbr.2020.112792] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2020] [Revised: 06/23/2020] [Accepted: 06/24/2020] [Indexed: 10/24/2022]
Abstract
Human neuroimaging research suggests the existence of one core network for the subjective valuation of rewards, including the striatum and orbitofrontal cortex. However, there is little research on the neural representation of subjective reward values of visual sexual stimuli (VSS) and on the role of these subjective valuations in the development of related addictive behaviors. Here, we investigate how neural reactivity to VSS is connected to individual preference using functional magnetic resonance imaging (fMRI). During the fMRI scan, 72 men viewed different VSS film clips. Ratings regarding valence and sexual arousal were collected and used as parametric modulators in the fMRI analysis. Subjects also filled out questionnaires on self-reported symptoms of problematic pornography use (PPU). Firstly, we found that neural reactivity towards VSS clips in the nucleus accumbens, caudate nucleus and orbitofrontal cortex was positively correlated with individual ratings of the respective VSS in all subjects. Second, the strength of the association between neural activity and sexual arousal ratings was positively correlated with self-reported symptoms of PPU. The first result suggests a precise appraisal of VSS according to individual preferences in established reward valuation regions. Secondly, stronger neural differentiation based on preference in participants with more PPU symptoms indicates an increased importance of VSS/preference fit in these individuals. This heightened correspondence between individual liking and neural activity may facilitate PPU development by increased signaling of incentive salience, thus boosting motivation to seek out and respond to these preferred stimuli.
Collapse
Affiliation(s)
- Sanja Klein
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany; Center of Mind, Brain and Behavior, Universities of Marburg and Giessen, 35032 Marburg, Germany.
| | - Onno Kruse
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany; Clinical Psychology, University Siegen, 57076 Siegen, Germany
| | - Charlotte Markert
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany; Center of Mind, Brain and Behavior, Universities of Marburg and Giessen, 35032 Marburg, Germany
| | - Isabell Tapia León
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany; Clinical Psychology, University Siegen, 57076 Siegen, Germany
| | - Jana Strahler
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany
| | - Rudolf Stark
- Department of Psychotherapy and Systems Neuroscience, Justus Liebig University, 35394 Giessen, Germany; Bender Institute for Neuroimaging (BION), Justus Liebig University, 35394 Giessen, Germany; Center of Mind, Brain and Behavior, Universities of Marburg and Giessen, 35032 Marburg, Germany
| |
Collapse
|
39
|
González-Torres R, Flores J, Orduña V. Suboptimal choice by pigeons is eliminated when key-pecking behavior is replaced by treadle-pressing. Behav Processes 2020; 178:104157. [PMID: 32497555 DOI: 10.1016/j.beproc.2020.104157] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2020] [Revised: 05/29/2020] [Accepted: 05/29/2020] [Indexed: 11/25/2022]
Abstract
In the study of suboptimal choice, a reliable result is that pigeons strongly prefer an alternative that signals whether a reinforcer will be delivered or not over another alternative without that information even if the first provides a lower probability of reinforcement. In the aforementioned research, key pecking has been the operant response and illuminated keys the discriminative stimuli. In the present study we modified both of these aspects of the procedure in order to analyze the generality of suboptimal preferences of pigeons and to investigate the effect of changes in the incentive salience of the discriminative stimuli. To accomplish this, we presented pigeons a choice situation with the same parameters of reinforcement than previous research, but with treadle pressing as the choice response and ambient lights as discriminative stimuli. Under these conditions, most of the pigeons showed optimal behavior and a high degree of discrimination of the stimuli associated with the discriminative alternative. A control condition with key pecking as choice response and keylights as discriminative stimuli showed that the same pigeons turned to be suboptimal, a result that discards the possibility that the optimality found in the main condition was a consequence of a particular characteristic of our sample of subjects or of our procedure. We discuss the influence that the attribution of incentive salience to the discriminative stimuli has on suboptimal choice in both pigeons and rats.
Collapse
Affiliation(s)
| | - Julio Flores
- Facultad de Psicología, Universidad Nacional Autónoma de México, México, DF, 04510, Mexico
| | - Vladimir Orduña
- Facultad de Psicología, Universidad Nacional Autónoma de México, México, DF, 04510, Mexico.
| |
Collapse
|
40
|
Warlow SM, Naffziger EE, Berridge KC. The central amygdala recruits mesocorticolimbic circuitry for pursuit of reward or pain. Nat Commun 2020; 11:2716. [PMID: 32483118 PMCID: PMC7264246 DOI: 10.1038/s41467-020-16407-1] [Citation(s) in RCA: 35] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2019] [Accepted: 05/01/2020] [Indexed: 02/06/2023] Open
Abstract
How do brain mechanisms create maladaptive attractions? Here intense maladaptive attractions are created in laboratory rats by pairing optogenetic channelrhodopsin (ChR2) stimulation of central nucleus of amygdala (CeA) in rats with encountering either sucrose, cocaine, or a painful shock-delivering object. We find that pairings make the respective rats pursue either sucrose exclusively, or cocaine exclusively, or repeatedly self-inflict shocks. CeA-induced maladaptive attractions, even to the painful shock-rod, recruit mesocorticolimbic incentive-related circuitry. Shock-associated cues also gain positive incentive value and are pursued. Yet the motivational effects of paired CeA stimulation can be reversed to negative valence in a Pavlovian fear learning situation, where CeA ChR2 pairing increases defensive reactions. Finally, CeA ChR2 valence can be switched to neutral by pairing with innocuous stimuli. These results reveal valence plasticity and multiple modes for motivation via mesocorticolimbic circuitry under the control of CeA activation.
Collapse
Affiliation(s)
- Shelley M Warlow
- Department of Psychology, University of Michigan, 530 Church St., Ann Arbor, MI, 48109, USA.
- Department of Neurosciences, University of California San Diego, La Jolla, CA, 92093, USA.
| | - Erin E Naffziger
- Department of Psychology, University of Michigan, 530 Church St., Ann Arbor, MI, 48109, USA
| | - Kent C Berridge
- Department of Psychology, University of Michigan, 530 Church St., Ann Arbor, MI, 48109, USA
| |
Collapse
|
41
|
van Swieten MMH, Bogacz R. Modeling the effects of motivation on choice and learning in the basal ganglia. PLoS Comput Biol 2020; 16:e1007465. [PMID: 32453725 PMCID: PMC7274475 DOI: 10.1371/journal.pcbi.1007465] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2019] [Revised: 06/05/2020] [Accepted: 04/03/2020] [Indexed: 01/08/2023] Open
Abstract
Decision making relies on adequately evaluating the consequences of actions on the basis of past experience and the current physiological state. A key role in this process is played by the basal ganglia, where neural activity and plasticity are modulated by dopaminergic input from the midbrain. Internal physiological factors, such as hunger, scale signals encoded by dopaminergic neurons and thus they alter the motivation for taking actions and learning. However, to our knowledge, no formal mathematical formulation exists for how a physiological state affects learning and action selection in the basal ganglia. We developed a framework for modelling the effect of motivation on choice and learning. The framework defines the motivation to obtain a particular resource as the difference between the desired and the current level of this resource, and proposes how the utility of reinforcements depends on the motivation. To account for dopaminergic activity previously recorded in different physiological states, the paper argues that the prediction error encoded in the dopaminergic activity needs to be redefined as the difference between utility and expected utility, which depends on both the objective reinforcement and the motivation. We also demonstrate a possible mechanism by which the evaluation and learning of utility of actions can be implemented in the basal ganglia network. The presented theory brings together models of learning in the basal ganglia with the incentive salience theory in a single simple framework, and it provides a mechanistic insight into how decision processes and learning in the basal ganglia are modulated by the motivation. Moreover, this theory is also consistent with data on neural underpinnings of overeating and obesity, and makes further experimental predictions.
Collapse
Affiliation(s)
| | - Rafal Bogacz
- MRC Brain Network Dynamics Unit, University of Oxford, Oxford, United Kingdom
| |
Collapse
|
42
|
Abstract
Although the gut and brain are separate organs, they communicate with each other via trillions of intestinal bacteria that collectively make up one's gut microbiome. Findings from both humans and animals support a critical role of gut microbes in regulating brain function, mood, and behavior. Gut bacteria influence neural circuits that are notably affected in addiction-related behaviors. These include circuits involved in stress, reward, and motivation, with substance use influencing gut microbial abnormalities, suggesting significant gut-brain interactions in drug addiction. Given the overwhelming rates of opioid overdose deaths driven by abuse and addiction, it is essential to characterize mechanisms mediating the abuse potential of opioids. We discuss in this review the role of gut microbiota in factors that influence opioid addiction, including incentive salience, reward, tolerance, withdrawal, stress, and compromised executive function. We present clinical and preclinical evidence supporting a bidirectional relationship between gut microbiota and opioid-related behaviors by highlighting the effects of opioid use on gut bacteria, and the effects of gut bacteria on behavioral responses to opioids. Further, we discuss possible mechanisms of this gut-brain communication influencing opioid use. By clarifying the relationship between the gut microbiome and opioid-related behaviors, we improve understanding on mechanisms mediating reward-, motivation-, and stress-related behaviors and disorders, which may contribute to the development of effective, targeted therapeutic interventions in opioid dependence and addiction.
Collapse
Affiliation(s)
- Michelle Ren
- Department of Pharmaceutical Sciences, University of California, Irvine, Irvine, CA, USA,
| | - Shahrdad Lotfipour
- Department of Pharmaceutical Sciences, University of California, Irvine, Irvine, CA, USA,
- Department of Emergency Medicine, School of Medicine, University of California, Irvine, Irvine, CA, USA,
| |
Collapse
|
43
|
|
44
|
Hulme OJ, Morville T, Gutkin B. Neurocomputational theories of homeostatic control. Phys Life Rev 2019; 31:214-232. [DOI: 10.1016/j.plrev.2019.07.005] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2018] [Revised: 06/05/2019] [Accepted: 07/06/2019] [Indexed: 01/19/2023]
|
45
|
Ahrens AM, Ahmed OJ. Neural circuits linking sleep and addiction: Animal models to understand why select individuals are more vulnerable to substance use disorders after sleep deprivation. Neurosci Biobehav Rev 2019; 108:435-444. [PMID: 31756346 DOI: 10.1016/j.neubiorev.2019.11.007] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2019] [Revised: 10/26/2019] [Accepted: 11/12/2019] [Indexed: 12/12/2022]
Abstract
Individuals differ widely in their drug-craving behaviors. One reason for these differences involves sleep. Sleep disturbances lead to an increased risk of substance use disorders and relapse in only some individuals. While animal studies have examined the impact of sleep on reward circuitry, few have addressed the role of individual differences in the effects of altered sleep. There does, however, exist a rodent model of individual differences in reward-seeking behavior: the sign/goal-tracker model of Pavlovian conditioned approach. In this model, only some rats show the key behavioral traits associated with addiction, including impulsivity and poor attentional control, making this an ideal model system to examine individually distinct sleep-reward interactions. Here, we describe how the limbic neural circuits responsible for individual differences in incentive motivation overlap with those involved in sleep-wake regulation, and how this model can elucidate the common underlying mechanisms. Consideration of individual differences in preclinical models would improve our understanding of how sleep interacts with motivational systems, and why sleep deprivation contributes to addiction in only select individuals.
Collapse
Affiliation(s)
| | - Omar J Ahmed
- Dept. of Psychology, United States; Neuroscience Graduate Program, United States; Michigan Center for Integrative Research in Critical Care, United States; Kresge Hearing Research Institute, United States; Dept. of Biomedical Engineering, University of Michigan, Ann Arbor, MI, 48109, United States.
| |
Collapse
|
46
|
Meissner F, Grigutsch LA, Koranyi N, Müller F, Rothermund K. Predicting Behavior With Implicit Measures: Disillusioning Findings, Reasonable Explanations, and Sophisticated Solutions. Front Psychol 2019; 10:2483. [PMID: 31787912 PMCID: PMC6856205 DOI: 10.3389/fpsyg.2019.02483] [Citation(s) in RCA: 53] [Impact Index Per Article: 10.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2019] [Accepted: 10/21/2019] [Indexed: 11/13/2022] Open
Abstract
Two decades ago, the introduction of the Implicit Association Test (IAT) sparked enthusiastic reactions. With implicit measures like the IAT, researchers hoped to finally be able to bridge the gap between self-reported attitudes on one hand and behavior on the other. Twenty years of research and several meta-analyses later, however, we have to conclude that neither the IAT nor its derivatives have fulfilled these expectations. Their predictive value for behavioral criteria is weak and their incremental validity over and above self-report measures is negligible. In our review, we present an overview of explanations for these unsatisfactory findings and delineate promising ways forward. Over the years, several reasons for the IAT's weak predictive validity have been proposed. They point to four potentially problematic features: First, the IAT is by no means a pure measure of individual differences in associations but suffers from extraneous influences like recoding. Hence, the predictive validity of IAT-scores should not be confused with the predictive validity of associations. Second, with the IAT, we usually aim to measure evaluation ("liking") instead of motivation ("wanting"). Yet, behavior might be determined much more often by the latter than the former. Third, the IAT focuses on measuring associations instead of propositional beliefs and thus taps into a construct that might be too unspecific to account for behavior. Finally, studies on predictive validity are often characterized by a mismatch between predictor and criterion (e.g., while behavior is highly context-specific, the IAT usually takes into account neither the situation nor the domain). Recent research, however, also revealed advances addressing each of these problems, namely (1) procedural and analytical advances to control for recoding in the IAT, (2) measurement procedures to assess implicit wanting, (3) measurement procedures to assess implicit beliefs, and (4) approaches to increase the fit between implicit measures and behavioral criteria (e.g., by incorporating contextual information). Implicit measures like the IAT hold an enormous potential. In order to allow them to fulfill this potential, however, we have to refine our understanding of these measures, and we should incorporate recent conceptual and methodological advancements. This review provides specific recommendations on how to do so.
Collapse
Affiliation(s)
- Franziska Meissner
- General Psychology II, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| | - Laura Anne Grigutsch
- General Psychology II, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| | - Nicolas Koranyi
- General Psychology II, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| | - Florian Müller
- Department for the Psychology of Human Movement and Sport, Institute for Sports Science, Friedrich Schiller University Jena, Jena, Germany
| | - Klaus Rothermund
- General Psychology II, Institute of Psychology, Friedrich Schiller University Jena, Jena, Germany
| |
Collapse
|
47
|
Girard R, Obeso I, Thobois S, Park SA, Vidal T, Favre E, Ulla M, Broussolle E, Krack P, Durif F, Dreher JC. Wait and you shall see: sexual delay discounting in hypersexual Parkinson's disease. Brain 2019; 142:146-162. [PMID: 30590514 DOI: 10.1093/brain/awy298] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2018] [Accepted: 10/04/2018] [Indexed: 02/02/2023] Open
Abstract
Patients with Parkinson's disease may develop impulse control disorders under dopaminergic treatments. Impulse control disorders include a wide spectrum of behaviours, such as hypersexuality, pathological gambling or compulsive shopping. Yet, the neural systems engaged in specific impulse control disorders remain poorly characterized. Here, using model-based functional MRI, we aimed to determine the brain systems involved during delay-discounting of erotic rewards in hypersexual patients with Parkinson's disease (PD+HS), patients with Parkinson's disease without hypersexuality (PD - HS) and controls. Patients with Parkinson's disease were evaluated ON and OFF levodopa (counterbalanced). Participants had to decide between two options: (i) wait for 1.5 s to briefly view an erotic image; or (ii) wait longer to see the erotic image for a longer period of time. At the time of decision-making, we investigated which brain regions were engaged with the subjective valuation of the delayed erotic reward. At the time of the rewarded outcome, we searched for the brain regions responding more robustly after waiting longer to view the erotic image. PD+HS patients showed reduced discounting of erotic delayed rewards, compared to both patients with Parkinson's disease and controls, suggesting that they accepted waiting longer to view erotic images for a longer period of time. Thus, when using erotic stimuli that motivate PD+HS, these patients were less impulsive for the immediate reward. At the brain system level, this effect was paralleled by the fact that PD+HS, as compared to controls and PD - HS, showed a negative correlation between subjective value of the delayed reward and activity of medial prefrontal cortex and ventral striatum. Consistent with the incentive salience hypothesis combining learned cue-reward associations with current relevant physiological state, dopaminergic treatment in PD+HS boosted excessive 'wanting' of rewards and heightened activity in the anterior medial prefrontal cortex and the posterior cingulate cortex, as reflected by higher correlation with subjective value of the option associated to the delayed reward when ON medication as compared to the OFF medication state. At the time of outcome, the anterior medial prefrontal/rostral anterior cingulate cortex showed an interaction between group (PD+HS versus PD - HS) and medication (ON versus OFF), suggesting that dopaminergic treatment boosted activity of this brain region in PD+HS when viewing erotic images after waiting for longer periods of time. Our findings point to reduced delay discounting of erotic rewards in PD+HS, both at the behavioural and brain system levels, and abnormal reinforcing effect of levodopa when PD+HS patients are confronted with erotic stimuli.10.1093/brain/awy298_video1awy298media15983845074001.
Collapse
Affiliation(s)
- Romuald Girard
- Neuroeconomics, Reward and Decision-making Team, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France.,University Claude Bernard Lyon, Lyon 1, Villeurbanne, France.,Section of Neurosurgery, The University of Chicago Medicine and Biological Sciences, Chicago, IL, USA
| | - Ignacio Obeso
- Neuroeconomics, Reward and Decision-making Team, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France.,University Claude Bernard Lyon, Lyon 1, Villeurbanne, France.,HM Hospitales - Centro Integral en Neurociencias HM CINAC, Móstoles, Madrid, Spain
| | - Stéphane Thobois
- Université de Lyon, Université Claude Bernard Lyon 1, Faculté de Médecine Lyon Sud Charles Mérieux, Oullins, France.,Neurologie C, Hôpital Neurologique Pierre Wertheimer, Hospices Civils de Lyon, Lyon, France.,Physiopathology of basal ganglia, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France
| | - Seongmin A Park
- Neuroeconomics, Reward and Decision-making Team, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France.,University Claude Bernard Lyon, Lyon 1, Villeurbanne, France
| | - Tiphaine Vidal
- Neurology Department, CHU de Clermont-Ferrand, Clermont-Ferrand, France.,University Clermont Auvergne, Clermont Ferrand, France
| | - Emilie Favre
- Neurologie C, Hôpital Neurologique Pierre Wertheimer, Hospices Civils de Lyon, Lyon, France
| | - Miguel Ulla
- Neurology Department, CHU de Clermont-Ferrand, Clermont-Ferrand, France
| | - Emmanuel Broussolle
- University Claude Bernard Lyon, Lyon 1, Villeurbanne, France.,Neurologie C, Hôpital Neurologique Pierre Wertheimer, Hospices Civils de Lyon, Lyon, France.,Physiopathology of basal ganglia, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France
| | - Paul Krack
- University Grenoble Alpes, Grenoble Institut des Neurosciences, Grenoble, France, Inserm, Grenoble, France.,Movement Disorders Unit, Neurology Department, CHU de Grenoble, Grenoble, France
| | - Franck Durif
- Neurology Department, CHU de Clermont-Ferrand, Clermont-Ferrand, France.,University Clermont Auvergne, Clermont Ferrand, France
| | - Jean-Claude Dreher
- Neuroeconomics, Reward and Decision-making Team, Institut des Sciences Cognitives Marc Jeannerod, Centre National de la Recherche Scientifique, UMR 5229, Bron, France.,University Claude Bernard Lyon, Lyon 1, Villeurbanne, France
| |
Collapse
|
48
|
Badiani A, Caprioli D, De Pirro S. Opposite environmental gating of the experienced utility ('liking') and decision utility ('wanting') of heroin versus cocaine in animals and humans: implications for computational neuroscience. Psychopharmacology (Berl) 2019; 236:2451-2471. [PMID: 31289884 PMCID: PMC6695361 DOI: 10.1007/s00213-019-05318-9] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/19/2019] [Accepted: 06/30/2019] [Indexed: 12/21/2022]
Abstract
BACKGROUND In this paper, we reviewed translational studies concerned with environmental influences on the rewarding effects of heroin versus cocaine in rats and humans with substance use disorder. These studies show that both experienced utility ('liking') and decision utility ('wanting') of heroin and cocaine shift in opposite directions as a function of the setting in which these drugs were used. Briefly, rats and humans prefer using heroin at home but cocaine outside the home. These findings appear to challenge prevailing theories of drug reward, which focus on the notion of shared substrate of action for drug of abuse, and in particular on their shared ability to facilitate dopaminergic transmission. AIMS Thus, in the second part of the paper, we verified whether our findings could be accounted for by available computational models of reward. To account for our findings, a model must include a component that could mediate the substance-specific influence of setting on drug reward RESULTS: It appears of the extant models that none is fully compatible with the results of our studies. CONCLUSIONS We hope that this paper will serve as stimulus to design computational models more attuned to the complex mechanisms responsible for the rewarding effects of drugs in real-world contexts.
Collapse
Affiliation(s)
- Aldo Badiani
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy.
- Sussex Addiction Research & Intervention Centre (SARIC) and School of Psychology, University of Sussex, Brighton, UK.
| | - Daniele Caprioli
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy
- Santa Lucia Foundation (IRCCS Fondazione Santa Lucia), Rome, Italy
| | - Silvana De Pirro
- Department of Physiology and Pharmacology, Sapienza University of Rome, Rome, Italy
- Sussex Addiction Research & Intervention Centre (SARIC) and School of Psychology, University of Sussex, Brighton, UK
| |
Collapse
|
49
|
Roux-en-Y gastric bypass in rat reduces mu-opioid receptor levels in brain regions associated with stress and energy regulation. PLoS One 2019; 14:e0218680. [PMID: 31220174 PMCID: PMC6586324 DOI: 10.1371/journal.pone.0218680] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2019] [Accepted: 06/06/2019] [Indexed: 12/23/2022] Open
Abstract
Roux-en-Y gastric bypass surgery (RYGB) is the most common and effective weight loss procedure for severe obesity. However, a significant increase in addictive behaviors and new-onset substance use disorder (SUD) are sometimes observed post-surgery. The endogenous opioid system is known to play a major role in motivated behavior and reward, as well as the abuse of substances, including alcohol, tobacco, opioids and highly palatable foods. Here, we examined the effects of RYGB on mu-opioid receptor levels in the brain. Male Sprague-Dawley rats were assigned to one of four groups: standard diet with sham surgery (control), ad libitum high-energy high-fat (HF) diet with sham surgery, calorie restricted HF diet with sham surgery (Sham-FR), or HF diet with RYGB surgery. Control and HF groups were fed their respective diets for 8 weeks, with surgery performed on the eighth week. After 9 weeks on their respective diets post-surgery, animals were sacrificed for mu-opioid receptor autoradiography using the [3H] [D-Ala2,N-Me-Phe4-Gly5-ol]- enkephalin (DAMGO) ligand. Rats with RYGB showed reduced DAMGO binding in the central amygdala compared to sham-operated HF diet controls, and in the hypothalamus compared to high-fat fed Sham-FR. Diet alone did not change [3H] DAMGO binding in any region. These findings show that RYGB surgery, independent of diet or caloric restriction, decreases mu opioid signaling in specific regions important for stress and energy regulation. Thus, RYGB surgery may lead to greater stress sensitivity via downregulated mu opioid signaling in the central amygdala, which may contribute to the observed increased risk in some subjects for addictive behavior.
Collapse
|
50
|
Hellberg SN, Russell TI, Robinson MJF. Cued for risk: Evidence for an incentive sensitization framework to explain the interplay between stress and anxiety, substance abuse, and reward uncertainty in disordered gambling behavior. COGNITIVE, AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2019; 19:737-758. [PMID: 30357661 PMCID: PMC6482104 DOI: 10.3758/s13415-018-00662-3] [Citation(s) in RCA: 29] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
Gambling disorder is an impairing condition confounded by psychiatric co-morbidity, particularly with substance use and anxiety disorders. Yet, our knowledge of the mechanisms that cause these disorders to coalesce remains limited. The Incentive Sensitization Theory suggests that sensitization of neural "wanting" pathways, which attribute incentive salience to rewards and their cues, is responsible for the excessive desire for drugs and cue-triggered craving. The resulting hyper-reactivity of the "wanting' system is believed to heavily influence compulsive drug use and relapse. Notably, evidence for sensitization of the mesolimbic dopamine pathway has been seen across gambling and substance use, as well as anxiety and stress-related pathology, with stress playing a major role in relapse. Together, this evidence highlights a phenomenon known as cross-sensitization, whereby sensitization to stress, drugs, or gambling behaviors enhance the sensitivity and dopaminergic response to any of those stimuli. Here, we review the literature on how cue attraction and reward uncertainty may underlie gambling pathology, and examine how this framework may advance our understanding of co-mordidity with substance-use disorders (e.g., alcohol, nicotine) and anxiety disorders. We argue that reward uncertainty, as seen in slot machines and games of chance, increases dopaminergic activity in the mesolimbic pathway and enhances the incentive value of reward cues. We propose that incentive sensitization by reward uncertainty may interact with and predispose individuals to drug abuse and stress, creating a mechanism through which co-mordidity of these disorders may emerge.
Collapse
Affiliation(s)
- Samantha N Hellberg
- Psychology Department and the Neuroscience and Behavior Program, Wesleyan University, 207 High Street, Middletown, CT, 06457, USA
- University of North Carolina Chapel Hill, Chapel Hill, NC, USA
| | - Trinity I Russell
- Psychology Department and the Neuroscience and Behavior Program, Wesleyan University, 207 High Street, Middletown, CT, 06457, USA
- National Institutes on Drug Abuse, Baltimore, MD, USA
| | - Mike J F Robinson
- Psychology Department and the Neuroscience and Behavior Program, Wesleyan University, 207 High Street, Middletown, CT, 06457, USA.
| |
Collapse
|