Dopamine release and its control over early Pavlovian learning differs between the NAc core and medial NAc shell

A Bio-Inspired Dopamine Model for Robots with Autonomous Decision-Making

Decision-making systems allow artificial agents to adapt their behaviours, depending on the information they perceive from the environment and internal processes. Human beings possess unique decision-making capabilities, adapting to current situations and anticipating future challenges. Autonomous robots with adaptive and anticipatory decision-making emulating humans can bring robots with skills that users can understand more easily. Human decisions highly depend on dopamine, a brain substance that regulates motivation and reward, acknowledging positive and negative situations. Considering recent neuroscience studies about the dopamine role in the human brain and its influence on decision-making and motivated behaviour, this paper proposes a model based on how dopamine drives human motivation and decision-making. The model allows robots to behave autonomously in dynamic environments, learning the best action selection strategy and anticipating future rewards. The results show the model's performance in five scenarios, emphasising how dopamine levels vary depending on the robot's situation and stimuli perception. Moreover, we show the model's integration into the Mini social robot to provide insights into how dopamine levels drive motivated autonomous behaviour regulating biologically inspired internal processes emulated in the robot.

Decreased Ventral Tegmental Area CB1R Signaling Reduces Sign Tracking and Shifts Cue-Outcome Dynamics in Rat Nucleus Accumbens

Sign-tracking (ST) rats show enhanced cue sensitivity before drug experience that predicts greater discrete cue-induced drug seeking compared with goal-tracking or intermediate rats. Cue-evoked dopamine in the nucleus accumbens (NAc) is a neurobiological signature of sign-tracking behaviors. Here, we examine a critical regulator of the dopamine system, endocannabinoids, which bind the cannabinoid receptor-1 (CB1R) in the ventral tegmental area (VTA) to control cue-evoked striatal dopamine levels. We use cell type-specific optogenetics, intra-VTA pharmacology, and fiber photometry to test the hypothesis that VTA CB1R receptor signaling regulates NAc dopamine levels to control sign tracking. We trained male and female rats in a Pavlovian lever autoshaping (PLA) task to determine their tracking groups before testing the effect of VTA → NAc dopamine inhibition. We found that this circuit is critical for mediating the vigor of the ST response. Upstream of this circuit, intra-VTA infusions of rimonabant, a CB1R inverse agonist, during PLA decrease lever and increase food cup approach in sign-trackers. Using fiber photometry to measure fluorescent signals from a dopamine sensor, GRABDA (AAV9-hSyn-DA2m), we tested the effects of intra-VTA rimonabant on NAc dopamine dynamics during autoshaping in female rats. We found that intra-VTA rimonabant decreased sign-tracking behaviors, which was associated with increases in NAc shell, but not core, dopamine levels during reward delivery [unconditioned stimulus (US)]. Our results suggest that CB1R signaling in the VTA influences the balance between the conditioned stimulus-evoked and US-evoked dopamine responses in the NAc shell and biases behavioral responding to cues in sign-tracking rats.SIGNIFICANCE STATEMENT Substance use disorder (SUD) is a chronically relapsing psychological disorder that affects a subset of individuals who engage in drug use. Recent research suggests that there are individual behavioral and neurobiological differences before drug experience that predict SUD and relapse vulnerabilities. Here, we investigate how midbrain endocannabinoids regulate a brain pathway that is exclusively involved in driving cue-motivated behaviors of sign-tracking rats. This work contributes to our mechanistic understanding of individual vulnerabilities to cue-triggered natural reward seeking that have relevance for drug-motivated behaviors.

Mesolimbic dopamine adapts the rate of learning from action

Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions^1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction⁴; however, so far there has been little consideration of how direct policy learning might inform our understanding⁵. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning⁶.

Critical periods when dopamine controls behavioral responding during Pavlovian learning

RATIONALE

Learning the association between rewards and predictive cues is critical for appetitive behavioral responding. The mesolimbic dopamine system is thought to play an integral role in establishing these cue-reward associations. The dopamine response to cues can signal differences in reward value, though this emerges only after significant training. This suggests that the dopamine system may differentially regulate behavioral responding depending on the phase of training.

OBJECTIVES

The purpose of this study was to determine whether antagonizing dopamine receptors elicited different effects on behavior depending on the phase of training or the type of Pavlovian task.

METHODS

Separate groups of male rats were trained on Pavlovian tasks in which distinct audio cues signaled either differences in reward size or differences in reward rate. The dopamine receptor antagonist flupenthixol was systemically administered prior to either the first ten sessions of training (acquisition phase) or the second ten sessions of training (expression phase), and we monitored the effect of these manipulations for an additional ten training sessions.

RESULTS

We identified acute effects of dopamine receptor antagonism on conditioned responding, the latency to respond, and post-reward head entries in both Pavlovian tasks. Interestingly, dopamine receptor antagonism during the expression phase produced persistent deficits in behavioral responding only in rats trained on the reward size Pavlovian task.

CONCLUSIONS

Together, our results illustrate that dopamine's control over behavior in Pavlovian tasks depends upon one's prior training experience and the information signaled by the cues.

Cortico-Striatal Activity Characterizes Human Safety Learning via Pavlovian Conditioned Inhibition

Safety learning generates associative links between neutral stimuli and the absence of threat, promoting the inhibition of fear and security-seeking behaviors. Precisely how safety learning is mediated at the level of underlying brain systems, particularly in humans, remains unclear. Here, we integrated a novel Pavlovian conditioned inhibition task with ultra-high field (7 Tesla) fMRI to examine the neural basis of safety learning in 49 healthy participants. In our task, participants were conditioned to two safety signals: a conditioned inhibitor that predicted threat omission when paired with a known threat signal (A+/AX-), and a standard safety signal that generally predicted threat omission (BC-). Both safety signals evoked equivalent autonomic and subjective learning responses but diverged strongly in terms of underlying brain activation (PFDR whole-brain corrected). The conditioned inhibitor was characterized by more prominent activation of the dorsal striatum, anterior insular, and dorsolateral PFC compared with the standard safety signal, whereas the latter evoked greater activation of the ventromedial PFC, posterior cingulate, and hippocampus, among other regions. Further analyses of the conditioned inhibitor indicated that its initial learning was characterized by consistent engagement of dorsal striatal, midbrain, thalamic, premotor, and prefrontal subregions. These findings suggest that safety learning via conditioned inhibition involves a distributed cortico-striatal circuitry, separable from broader cortical regions involved with processing standard safety signals (e.g., CS-). This cortico-striatal system could represent a novel neural substrate of safety learning, underlying the initial generation of "stimulus-safety" associations, distinct from wider cortical correlates of safety processing, which facilitate the behavioral outcomes of learning.SIGNIFICANCE STATEMENT Identifying safety is critical for maintaining adaptive levels of anxiety, but the neural mechanisms of human safety learning remain unclear. Using 7 Tesla fMRI, we compared learning-related brain activity for a conditioned inhibitor, which actively predicted threat omission, and a standard safety signal (CS-), which was passively unpaired with threat. The inhibitor engaged an extended circuitry primarily featuring the dorsal striatum, along with thalamic, midbrain, and premotor/PFC regions. The CS- exclusively involved cortical safety-related regions observed in basic safety conditioning, such as the vmPFC. These findings extend current models to include learning-specific mechanisms for encoding stimulus-safety associations, which might be distinguished from expression-related cortical mechanisms. These insights may suggest novel avenues for targeting dysfunctional safety learning in psychopathology.

CircTmeff-1 in the nucleus accumbens regulates the reconsolidation of cocaine-associated memory

Reconsolidation of drug memories is the process of restoring unstable memories after unconditioned (UCS; e.g., drugs) or conditioned stimulus (CS; e.g., drug-paired contexts), and provides promise for prevention of drug relapse. Circular RNAs (circRNAs) have important effects on the transcription and post-transcriptional regulation of gene expression. However, the role of circRNAs in the reconsolidation of drug memories is unclear. Here, we observed that cocaine-induced memory retrieval significantly increased circTmeff-1 level in the nucleus accumbens (NAc) core but not shell. Importantly, the disrupted expression of circTmeff-1 using virus in the NAc core damaged the reconsolidation of cocaine-associated memories. The knockdown of circTmeff-1 in the NAc shell or without UCS retrieval or 9 h after UCS retrieval had no such effects. Mechanistically, using bioinformatic analysis and loss- or gain- of function assays, we revealed that antagomiR-206 reversed the inhibitory effect of circTmeff-1 knockdown on the expression of brain-derived neurotrophic factor (BDNF) during the reconsolidation of cocaine-associated memories. Taken together, these results demonstrate the role of circTmeff-1 in the reconsolidation of cocaine-associated memory and that circTmeff-1 may function as a decoy for miR-206 to regulate the expression of BDNF.

Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning

Learning associations between cues and rewards require the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well trained animals. However, these value-related dopamine responses are absent during early training sessions when cues signal differences in the reward rate. These findings suggest cue-evoked dopamine release conveys differences between outcomes only after extensive training, though it is unclear whether this is unique to when cues signal differences in reward rate, or whether this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we used a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout learning. While female rats exhibited higher levels of conditioned responding, a faster latency to respond, and elevated post-reward head entries relative to male rats, there were no sex differences in the dopamine response to cues. Multiple training sessions were required before cue-evoked dopamine release signaled differences in reward size. Reward-evoked dopamine release scaled with reward size, though females displayed lower reward-evoked dopamine responses relative to males. Conditioned responding related to the decrease in the peak reward-evoked dopamine response and not to cue-evoked dopamine release. Collectively, these data illustrate sex differences in behavioral responding as well as in reward-evoked dopamine release during Pavlovian learning.