1
|
Maroto-Gómez M, Burguete-Alventosa J, Álvarez-Arias S, Malfaz M, Salichs MÁ. A Bio-Inspired Dopamine Model for Robots with Autonomous Decision-Making. Biomimetics (Basel) 2024; 9:504. [PMID: 39194483 DOI: 10.3390/biomimetics9080504] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2024] [Revised: 08/18/2024] [Accepted: 08/19/2024] [Indexed: 08/29/2024] Open
Abstract
Decision-making systems allow artificial agents to adapt their behaviours, depending on the information they perceive from the environment and internal processes. Human beings possess unique decision-making capabilities, adapting to current situations and anticipating future challenges. Autonomous robots with adaptive and anticipatory decision-making emulating humans can bring robots with skills that users can understand more easily. Human decisions highly depend on dopamine, a brain substance that regulates motivation and reward, acknowledging positive and negative situations. Considering recent neuroscience studies about the dopamine role in the human brain and its influence on decision-making and motivated behaviour, this paper proposes a model based on how dopamine drives human motivation and decision-making. The model allows robots to behave autonomously in dynamic environments, learning the best action selection strategy and anticipating future rewards. The results show the model's performance in five scenarios, emphasising how dopamine levels vary depending on the robot's situation and stimuli perception. Moreover, we show the model's integration into the Mini social robot to provide insights into how dopamine levels drive motivated autonomous behaviour regulating biologically inspired internal processes emulated in the robot.
Collapse
Affiliation(s)
- Marcos Maroto-Gómez
- Department of Systems Engineering and Automation, University Carlos III of Madrid, Av. de la Universidad, 30, 28911 Leganes, Madrid, Spain
| | - Javier Burguete-Alventosa
- Department of Systems Engineering and Automation, University Carlos III of Madrid, Av. de la Universidad, 30, 28911 Leganes, Madrid, Spain
| | - Sofía Álvarez-Arias
- Department of Systems Engineering and Automation, University Carlos III of Madrid, Av. de la Universidad, 30, 28911 Leganes, Madrid, Spain
| | - María Malfaz
- Department of Systems Engineering and Automation, University Carlos III of Madrid, Av. de la Universidad, 30, 28911 Leganes, Madrid, Spain
| | - Miguel Ángel Salichs
- Department of Systems Engineering and Automation, University Carlos III of Madrid, Av. de la Universidad, 30, 28911 Leganes, Madrid, Spain
| |
Collapse
|
2
|
Bacharach SZ, Martin DA, Stapf CA, Sun F, Li Y, Cheer JF, Calu DJ. Decreased Ventral Tegmental Area CB1R Signaling Reduces Sign Tracking and Shifts Cue-Outcome Dynamics in Rat Nucleus Accumbens. J Neurosci 2023; 43:4684-4696. [PMID: 37208179 PMCID: PMC10286939 DOI: 10.1523/jneurosci.1486-22.2023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 05/10/2023] [Accepted: 05/15/2023] [Indexed: 05/21/2023] Open
Abstract
Sign-tracking (ST) rats show enhanced cue sensitivity before drug experience that predicts greater discrete cue-induced drug seeking compared with goal-tracking or intermediate rats. Cue-evoked dopamine in the nucleus accumbens (NAc) is a neurobiological signature of sign-tracking behaviors. Here, we examine a critical regulator of the dopamine system, endocannabinoids, which bind the cannabinoid receptor-1 (CB1R) in the ventral tegmental area (VTA) to control cue-evoked striatal dopamine levels. We use cell type-specific optogenetics, intra-VTA pharmacology, and fiber photometry to test the hypothesis that VTA CB1R receptor signaling regulates NAc dopamine levels to control sign tracking. We trained male and female rats in a Pavlovian lever autoshaping (PLA) task to determine their tracking groups before testing the effect of VTA → NAc dopamine inhibition. We found that this circuit is critical for mediating the vigor of the ST response. Upstream of this circuit, intra-VTA infusions of rimonabant, a CB1R inverse agonist, during PLA decrease lever and increase food cup approach in sign-trackers. Using fiber photometry to measure fluorescent signals from a dopamine sensor, GRABDA (AAV9-hSyn-DA2m), we tested the effects of intra-VTA rimonabant on NAc dopamine dynamics during autoshaping in female rats. We found that intra-VTA rimonabant decreased sign-tracking behaviors, which was associated with increases in NAc shell, but not core, dopamine levels during reward delivery [unconditioned stimulus (US)]. Our results suggest that CB1R signaling in the VTA influences the balance between the conditioned stimulus-evoked and US-evoked dopamine responses in the NAc shell and biases behavioral responding to cues in sign-tracking rats.SIGNIFICANCE STATEMENT Substance use disorder (SUD) is a chronically relapsing psychological disorder that affects a subset of individuals who engage in drug use. Recent research suggests that there are individual behavioral and neurobiological differences before drug experience that predict SUD and relapse vulnerabilities. Here, we investigate how midbrain endocannabinoids regulate a brain pathway that is exclusively involved in driving cue-motivated behaviors of sign-tracking rats. This work contributes to our mechanistic understanding of individual vulnerabilities to cue-triggered natural reward seeking that have relevance for drug-motivated behaviors.
Collapse
Affiliation(s)
- Sam Z Bacharach
- Department of Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - David A Martin
- Department of Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Cassie A Stapf
- Department of Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Fangmiao Sun
- State Key Laboratory of Membrane Biology, School of Life Sciences, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research at PKU, Peking University, Beijing 100871, People's Republic of China
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing 100871, People's Republic of China
| | - Yulong Li
- State Key Laboratory of Membrane Biology, School of Life Sciences, Peking University, Beijing 100871, People's Republic of China
- IDG/McGovern Institute for Brain Research at PKU, Peking University, Beijing 100871, People's Republic of China
- Peking-Tsinghua Center for Life Sciences, Peking University, Beijing 100871, People's Republic of China
| | - Joseph F Cheer
- Department of Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| | - Donna J Calu
- Department of Neurobiology, University of Maryland School of Medicine, Baltimore, Maryland 21201
- Program in Neuroscience, University of Maryland School of Medicine, Baltimore, Maryland 21201
| |
Collapse
|
3
|
Mesolimbic dopamine adapts the rate of learning from action. Nature 2023; 614:294-302. [PMID: 36653450 PMCID: PMC9908546 DOI: 10.1038/s41586-022-05614-z] [Citation(s) in RCA: 27] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2021] [Accepted: 11/30/2022] [Indexed: 01/20/2023]
Abstract
Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction4; however, so far there has been little consideration of how direct policy learning might inform our understanding5. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning6.
Collapse
|
4
|
Lefner MJ, Stelly CE, Fonzi KM, Zurita H, Wanat MJ. Critical periods when dopamine controls behavioral responding during Pavlovian learning. Psychopharmacology (Berl) 2022; 239:2985-2996. [PMID: 35796814 PMCID: PMC10039755 DOI: 10.1007/s00213-022-06182-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/01/2022] [Accepted: 06/17/2022] [Indexed: 10/17/2022]
Abstract
RATIONALE Learning the association between rewards and predictive cues is critical for appetitive behavioral responding. The mesolimbic dopamine system is thought to play an integral role in establishing these cue-reward associations. The dopamine response to cues can signal differences in reward value, though this emerges only after significant training. This suggests that the dopamine system may differentially regulate behavioral responding depending on the phase of training. OBJECTIVES The purpose of this study was to determine whether antagonizing dopamine receptors elicited different effects on behavior depending on the phase of training or the type of Pavlovian task. METHODS Separate groups of male rats were trained on Pavlovian tasks in which distinct audio cues signaled either differences in reward size or differences in reward rate. The dopamine receptor antagonist flupenthixol was systemically administered prior to either the first ten sessions of training (acquisition phase) or the second ten sessions of training (expression phase), and we monitored the effect of these manipulations for an additional ten training sessions. RESULTS We identified acute effects of dopamine receptor antagonism on conditioned responding, the latency to respond, and post-reward head entries in both Pavlovian tasks. Interestingly, dopamine receptor antagonism during the expression phase produced persistent deficits in behavioral responding only in rats trained on the reward size Pavlovian task. CONCLUSIONS Together, our results illustrate that dopamine's control over behavior in Pavlovian tasks depends upon one's prior training experience and the information signaled by the cues.
Collapse
Affiliation(s)
- Merridee J Lefner
- Department of Neuroscience Developmental and Regenerative Biology, Neurosciences Institute, Brain Health Consortium, University of Texas at San Antonio, San Antonio, TX, USA
| | - Claire E Stelly
- Department of Neuroscience Developmental and Regenerative Biology, Neurosciences Institute, Brain Health Consortium, University of Texas at San Antonio, San Antonio, TX, USA
| | - Kaitlyn M Fonzi
- Department of Neuroscience Developmental and Regenerative Biology, Neurosciences Institute, Brain Health Consortium, University of Texas at San Antonio, San Antonio, TX, USA
| | - Hector Zurita
- Department of Neuroscience Developmental and Regenerative Biology, Neurosciences Institute, Brain Health Consortium, University of Texas at San Antonio, San Antonio, TX, USA
| | - Matthew J Wanat
- Department of Neuroscience Developmental and Regenerative Biology, Neurosciences Institute, Brain Health Consortium, University of Texas at San Antonio, San Antonio, TX, USA.
| |
Collapse
|
5
|
Laing PAF, Steward T, Davey CG, Felmingham KL, Fullana MA, Vervliet B, Greaves MD, Moffat B, Glarin RK, Harrison BJ. Cortico-Striatal Activity Characterizes Human Safety Learning via Pavlovian Conditioned Inhibition. J Neurosci 2022; 42:5047-5057. [PMID: 35577553 PMCID: PMC9233447 DOI: 10.1523/jneurosci.2181-21.2022] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Revised: 05/03/2022] [Accepted: 05/06/2022] [Indexed: 12/24/2022] Open
Abstract
Safety learning generates associative links between neutral stimuli and the absence of threat, promoting the inhibition of fear and security-seeking behaviors. Precisely how safety learning is mediated at the level of underlying brain systems, particularly in humans, remains unclear. Here, we integrated a novel Pavlovian conditioned inhibition task with ultra-high field (7 Tesla) fMRI to examine the neural basis of safety learning in 49 healthy participants. In our task, participants were conditioned to two safety signals: a conditioned inhibitor that predicted threat omission when paired with a known threat signal (A+/AX-), and a standard safety signal that generally predicted threat omission (BC-). Both safety signals evoked equivalent autonomic and subjective learning responses but diverged strongly in terms of underlying brain activation (PFDR whole-brain corrected). The conditioned inhibitor was characterized by more prominent activation of the dorsal striatum, anterior insular, and dorsolateral PFC compared with the standard safety signal, whereas the latter evoked greater activation of the ventromedial PFC, posterior cingulate, and hippocampus, among other regions. Further analyses of the conditioned inhibitor indicated that its initial learning was characterized by consistent engagement of dorsal striatal, midbrain, thalamic, premotor, and prefrontal subregions. These findings suggest that safety learning via conditioned inhibition involves a distributed cortico-striatal circuitry, separable from broader cortical regions involved with processing standard safety signals (e.g., CS-). This cortico-striatal system could represent a novel neural substrate of safety learning, underlying the initial generation of "stimulus-safety" associations, distinct from wider cortical correlates of safety processing, which facilitate the behavioral outcomes of learning.SIGNIFICANCE STATEMENT Identifying safety is critical for maintaining adaptive levels of anxiety, but the neural mechanisms of human safety learning remain unclear. Using 7 Tesla fMRI, we compared learning-related brain activity for a conditioned inhibitor, which actively predicted threat omission, and a standard safety signal (CS-), which was passively unpaired with threat. The inhibitor engaged an extended circuitry primarily featuring the dorsal striatum, along with thalamic, midbrain, and premotor/PFC regions. The CS- exclusively involved cortical safety-related regions observed in basic safety conditioning, such as the vmPFC. These findings extend current models to include learning-specific mechanisms for encoding stimulus-safety associations, which might be distinguished from expression-related cortical mechanisms. These insights may suggest novel avenues for targeting dysfunctional safety learning in psychopathology.
Collapse
Affiliation(s)
- Patrick A F Laing
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia
| | - Trevor Steward
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Victoria 3052, Australia
| | - Christopher G Davey
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia
| | - Kim L Felmingham
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Victoria 3052, Australia
| | - Miguel Angel Fullana
- Adult Psychiatry and Psychology Department, Institute of Neurosciences, Hospital Clinic, Barcelona 08001, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Centro de Investigación Biomédia en Red de Salud Mental, Barcelona 08036, Spain
| | - Bram Vervliet
- Laboratory of Biological Psychology, Faculty of Psychology and Educational Sciences, KU Leuven 3000, Belgium
- Leuven Brain Institute, KU Leuven 3000, Belgium
| | - Matthew D Greaves
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia
| | - Bradford Moffat
- The Melbourne Brain Centre Imaging Unit, Department of Radiology, University of Melbourne, Melbourne, Victoria 3052, Australia
| | - Rebecca K Glarin
- The Melbourne Brain Centre Imaging Unit, Department of Radiology, University of Melbourne, Melbourne, Victoria 3052, Australia
| | - Ben J Harrison
- Melbourne Neuropsychiatry Centre, Department of Psychiatry, University of Melbourne, Melbourne, Victoria 3053, Australia
| |
Collapse
|
6
|
Shen Q, Xie B, Galaj E, Yu H, Li X, Lu Y, Zhang M, Wen D, Ma C. CircTmeff-1 in the nucleus accumbens regulates the reconsolidation of cocaine-associated memory. Brain Res Bull 2022; 185:64-73. [PMID: 35489671 DOI: 10.1016/j.brainresbull.2022.04.010] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2022] [Revised: 04/22/2022] [Accepted: 04/23/2022] [Indexed: 12/27/2022]
Abstract
Reconsolidation of drug memories is the process of restoring unstable memories after unconditioned (UCS; e.g., drugs) or conditioned stimulus (CS; e.g., drug-paired contexts), and provides promise for prevention of drug relapse. Circular RNAs (circRNAs) have important effects on the transcription and post-transcriptional regulation of gene expression. However, the role of circRNAs in the reconsolidation of drug memories is unclear. Here, we observed that cocaine-induced memory retrieval significantly increased circTmeff-1 level in the nucleus accumbens (NAc) core but not shell. Importantly, the disrupted expression of circTmeff-1 using virus in the NAc core damaged the reconsolidation of cocaine-associated memories. The knockdown of circTmeff-1 in the NAc shell or without UCS retrieval or 9 h after UCS retrieval had no such effects. Mechanistically, using bioinformatic analysis and loss- or gain- of function assays, we revealed that antagomiR-206 reversed the inhibitory effect of circTmeff-1 knockdown on the expression of brain-derived neurotrophic factor (BDNF) during the reconsolidation of cocaine-associated memories. Taken together, these results demonstrate the role of circTmeff-1 in the reconsolidation of cocaine-associated memory and that circTmeff-1 may function as a decoy for miR-206 to regulate the expression of BDNF.
Collapse
Affiliation(s)
- Qianchao Shen
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Bing Xie
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Ewa Galaj
- Department of Psychological and Brain Sciences, Colgate University, 13 Oak Drive, Hamilton, NY 13346, USA
| | - Hailei Yu
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Xiaojie Li
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Yun Lu
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Minglong Zhang
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China
| | - Di Wen
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China.
| | - Chunling Ma
- College of Forensic Medicine, Hebei Medical University, Hebei Key Laboratory of Forensic Medicine, Collaborative Innovation Center of Forensic Medical Molecular Identification, Shijiazhuang 050017, Hebei, PR China; Research Unit of Digestive Tract Microecosystem Pharmacology and Toxicology, Chinese Academy of Medical Sciences, Shijiazhuang 050017, Hebei, PR China.
| |
Collapse
|
7
|
Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning. eNeuro 2022; 9:ENEURO.0050-22.2022. [PMID: 35264461 PMCID: PMC8941639 DOI: 10.1523/eneuro.0050-22.2022] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2022] [Revised: 02/24/2022] [Accepted: 03/02/2022] [Indexed: 12/15/2022] Open
Abstract
Learning associations between cues and rewards require the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well trained animals. However, these value-related dopamine responses are absent during early training sessions when cues signal differences in the reward rate. These findings suggest cue-evoked dopamine release conveys differences between outcomes only after extensive training, though it is unclear whether this is unique to when cues signal differences in reward rate, or whether this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we used a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout learning. While female rats exhibited higher levels of conditioned responding, a faster latency to respond, and elevated post-reward head entries relative to male rats, there were no sex differences in the dopamine response to cues. Multiple training sessions were required before cue-evoked dopamine release signaled differences in reward size. Reward-evoked dopamine release scaled with reward size, though females displayed lower reward-evoked dopamine responses relative to males. Conditioned responding related to the decrease in the peak reward-evoked dopamine response and not to cue-evoked dopamine release. Collectively, these data illustrate sex differences in behavioral responding as well as in reward-evoked dopamine release during Pavlovian learning.
Collapse
|