Dopaminergic modulation of appetitive and aversive predictive learning

The Role of the Endogenous Opioid System in the Vocal Behavior of Songbirds and Its Possible Role in Vocal Learning

The opioid system in the brain is responsible for processing affective states such as pain, pleasure, and reward. It consists of three main receptors, mu- (μ-ORs), delta- (δ-ORs), and kappa- (κ-ORs), and their ligands – the endogenous opioid peptides. Despite their involvement in the reward pathway, and a signaling mechanism operating in synergy with the dopaminergic system, fewer reports focus on the role of these receptors in higher cognitive processes. Whereas research on opioids is predominated by studies on their addictive properties and role in pain pathways, recent studies suggest that these receptors may be involved in learning. Rodents deficient in δ-ORs were poor at recognizing the location of novel objects in their surroundings. Furthermore, in chicken, learning to avoid beads coated with a bitter chemical from those without the coating was modulated by δ-ORs. Similarly, μ-ORs facilitate long term potentiation in hippocampal CA3 neurons in mammals, thereby having a positive impact on spatial learning. Whereas these studies have explored the role of opioid receptors on learning using reward/punishment-based paradigms, the role of these receptors in natural learning processes, such as vocal learning, are yet unexplored. In this review, we explore studies that have established the expression pattern of these receptors in different brain regions of birds, with an emphasis on songbirds which are model systems for vocal learning. We also review the role of opioid receptors in modulating the cognitive processes associated with vocalizations in birds. Finally, we discuss the role of these receptors in regulating the motivation to vocalize, and a possible role in modulating vocal learning.

Dopamine release in the nucleus accumbens core signals perceived saliency

A large body of work has aimed to define the precise information encoded by dopaminergic projections innervating the nucleus accumbens (NAc). Prevailing models are based on reward prediction error (RPE) theory, in which dopamine updates associations between rewards and predictive cues by encoding perceived errors between predictions and outcomes. However, RPE cannot describe multiple phenomena to which dopamine is inextricably linked, such as behavior driven by aversive and neutral stimuli. We combined a series of behavioral tasks with direct, subsecond dopamine monitoring in the NAc of mice, machine learning, computational modeling, and optogenetic manipulations to describe behavior and related dopamine release patterns across multiple contingencies reinforced by differentially valenced outcomes. We show that dopamine release only conforms to RPE predictions in a subset of learning scenarios but fits valence-independent perceived saliency encoding across conditions. Here, we provide an extended, comprehensive framework for accumbal dopamine release in behavioral control.

Retrorubral field is a hub for diverse threat and aversive outcome signals

Adaptive fear scales to the degree of threat and requires diverse neural signals for threat and aversive outcome. We propose that the retrorubral field (RRF), a midbrain region containing A8 dopamine, is a neural origin of such signals. To reveal these signals, we recorded RRF single-unit activity while male rats discriminated danger, uncertainty, and safety. Many RRF neurons showed firing extremes to danger and safety that framed intermediate firing to uncertainty. The remaining neurons showed unique, threat-selective cue firing patterns. Diversity in firing direction, magnitude, and temporal characteristics led to the detection of at least eight functional neuron types. Neuron types defined with respect to threat showed unique firing patterns following aversive outcome. The result was RRF signals for foot shock receipt, positive prediction error, anti-positive prediction error, persistent safety, and persistent threat. The diversity of threat and aversive outcome signals points to a key role for the RRF in adaptive fear.

The Nucleus Accumbens Core is Necessary to Scale Fear to Degree of Threat

Fear is adaptive when the level of the response rapidly scales to degree of threat. Using a discrimination procedure consisting of danger, uncertainty, and safety cues, we have found rapid fear scaling (within 2 s of cue presentation) in male rats. Here, we examined a possible role for the nucleus accumbens core (NAcc) in the acquisition and expression of fear scaling. In experiment 1, male Long-Evans rats received bilateral sham or neurotoxic NAcc lesions, recovered, and underwent fear discrimination. NAcc-lesioned rats were generally impaired in scaling fear to degree of threat, and specifically impaired in rapid uncertainty-safety discrimination. In experiment 2, male Long-Evans rats received NAcc transduction with halorhodopsin (Halo) or a control fluorophore. After fear scaling was established, the NAcc was illuminated during cue or control periods. NAcc-Halo rats receiving cue illumination were specifically impaired in rapid uncertainty-safety discrimination. The results reveal a general role for the NAcc in scaling fear to degree of threat, and a specific role in rapid discrimination of uncertain threat and safety.SIGNIFICANCE STATEMENT Rapidly discriminating cues for threat and safety is essential for survival and impaired threat-safety discrimination is a hallmark of stress and anxiety disorders. In two experiments, we induced nucleus accumbens core (NAcc) dysfunction in rats receiving fear discrimination consisting of cues for danger, uncertainty, and safety. Permanent NAcc dysfunction, via neurotoxic lesion, generally disrupted the ability to scale fear to degree of threat, and specifically impaired one component of scaling: rapid discrimination of uncertain threat and safety. Reversible NAcc dysfunction, via optogenetic inhibition, specifically impaired rapid discrimination of uncertain threat and safety. The results reveal that the NAcc is essential to scale fear to degree of threat, and is a plausible source of dysfunction in stress and anxiety disorders.

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto’s (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. This is argued to endow the reward-predictive cue with the value inherent in the reward, motivating behavior toward cues signaling the presence of reward. Yet theoretical and empirical research has implicated prediction-error signaling in learning that extends far beyond a transfer of quantitative value to a reward-predictive cue. Here, we review the research which demonstrates the complexity of how dopaminergic prediction errors facilitate learning. After briefly discussing the literature demonstrating that phasic dopaminergic signals can act in the manner described by Sutton and Barto (1981), we consider how these signals may also influence attentional processing across multiple attentional systems in distinct brain circuits. Then, we discuss how prediction errors encode and promote the development of context-specific associations between cues and rewards. Finally, we consider recent evidence that shows dopaminergic activity contains information about causal relationships between cues and rewards that reflect information garnered from rich associative models of the world that can be adapted in the absence of direct experience. In discussing this research we hope to support the expansion of how dopaminergic prediction errors are thought to contribute to the learning process beyond the traditional concept of transferring quantitative value.

Striatum on the anxiety map: Small detours into adolescence

Adolescence is the most sensitive period for the development of pathological anxiety. Moreover, specific neural changes associated with the striatum might be related to adolescent vulnerability to anxiety. Up to now, the study of anxiety has primarily focused on the amygdala, bed nucleus of the stria terminalis (BNST), hippocampus and ventromedial prefrontal cortex (vmPFC), while the striatum has typically not been considered as part of the anxiety system. This review proposes the addition of the striatum, a complex, multi-component structure, to the anxiety network by underscoring two lines of research. First, the co-occurrence of the adolescent striatal development with the peak vulnerability of adolescents to anxiety disorders might potentially reflect a causal relationship. Second, the recognition of the role of the striatum in fundamental behavioral processes that do affect anxiety supports the putative importance of the striatum in anxiety. These behavioral processes include (1) attention, (2) conditioning/prediction error, and (3) motivation. This review proposes a simplistic schematic representation of the anxiety circuitry that includes the striatum, and aims to promote further work in this direction, as the role of the striatum in shaping an anxiety phenotype during adolescence could have critical implications for understanding and preventing the peak onset of anxiety disorders during this period. This article is part of a Special Issue entitled SI: Adolescent plasticity.

The dopamine D1 receptor agonist SKF81297 has dose-related effects on locomotor activity but is without effect in a CER trace conditioning procedure conducted with two versus four trials

In an appetitively motivated procedure, we have previously reported that systemic treatment with the dopamine (DA) D1 receptor agonist SKF81297 (0.4 and 0.8 mg/kg) depressed acquisition at a 2 s inter-stimulus-interval (ISI), suitable to detect trace conditioning impairment. However since DA is involved in reinforcement processes, the generality of effects across appetitively- and aversively-motivated trace conditioning procedures cannot be assumed. The present study tested the effects of SKF81297 (0.4 and 0.8 mg/kg) in an established conditioned emotional response (CER) procedure. Trace-dependent conditioning was clearly shown in two experiments: while conditioning was relatively strong at a 3-s ISI, it was attenuated at a 30-s ISI. This was shown after two (Experiment 1) or four (Experiment 2) conditioning trials conducted in – as far as possible – the same CER procedure. Contrary to prediction, in neither experiment was there any indication that trace conditioning was attenuated by treatment with 0.4 or 0.8 mg/kg SKF81297. In the same rats, locomotor activity was significantly enhanced at the 0.8 mg/kg dose of SKF81297. These results suggest that procedural details of the trace conditioning variant in use are an important determinant of the profile of dopaminergic modulation.

Activation of D1/5 Dopamine Receptors: A Common Mechanism for Enhancing Extinction of Fear and Reward-Seeking Behaviors

Dopamine is critical for many processes that drive learning and memory, including motivation, prediction error, incentive salience, memory consolidation, and response output. Theories of dopamine's function in these processes have, for the most part, been developed from behavioral approaches that examine learning mechanisms in appetitive tasks. A parallel and growing literature indicates that dopamine signaling is involved in consolidation of memories into stable representations in aversive tasks such as fear conditioning. Relatively little is known about how dopamine may modulate memories that form during extinction, when organisms learn that the relation between previously associated events is severed. We investigated whether fear and reward extinction share common mechanisms that could be enhanced with dopamine D1/5 receptor activation. Pharmacological activation of dopamine D1/5 receptors (with SKF 81297) enhanced extinction of both cued and contextual fear. These effects also occurred in the extinction of cocaine-induced conditioned place preference, suggesting that the observed effects on extinction were not specific to a particular type of procedure (aversive or appetitive). A cAMP/PKA biased D1 agonist (SKF 83959) did not affect fear extinction, whereas a broadly efficacious D1 agonist (SKF 83822) promoted fear extinction. Together, these findings show that dopamine D1/5 receptor activation is a target for the enhancement of fear or reward extinction.

Dopamine D1-like receptor signalling in the hippocampus and amygdala modulates the acquisition of contextual fear conditioning

RATIONALE

Dopamine D1-like receptor signalling is involved in contextual fear conditioning, but the brain regions involved and its role in other contextual fear memory processes remain unclear.

OBJECTIVES

The objective of this study was to investigate (1) the effects of SCH 23390, a dopamine D1/D5 receptor antagonist, on contextual fear memory encoding, retrieval and reconsolidation, and (2) if the effects of SCH 23390 on conditioning involve the dorsal hippocampus (DH) and/or basolateral amygdala (BLA).

METHODS

Rats were used to examine the effects of systemically administering SCH 23390 on the acquisition, consolidation, retrieval and reconsolidation of contextual fear memory, and on locomotor activity and shock sensitivity. We also determined the effects of MK-801, an NMDA receptor antagonist, on contextual fear memory reconsolidation. The effects of infusing SCH 23390 locally into DH or BLA on contextual fear conditioning and locomotor activity were also examined.

RESULTS

Systemic administration of SCH 23390 impaired contextual fear conditioning but had no effects on fear memory consolidation, retrieval or reconsolidation. MK-801 was found to impair reconsolidation, suggesting that the behavioural parameters used allowed for the pharmacological disruption of memory reconsolidation. The effects of SCH 23390 on conditioning were unlikely the result of any lasting drug effects on locomotor activity at memory test or any acute drug effects on shock sensitivity during conditioning. SCH 23390 infused into either DH or BLA impaired contextual fear conditioning and decreased locomotor activity.

CONCLUSIONS

These findings suggest that dopamine D1-like receptor signalling in DH and BLA contributes to the acquisition of contextual fear memory.

The dorsal raphe nucleus is integral to negative prediction errors in Pavlovian fear

Prediction errors are central to modern learning theories. While brain regions contributing to reward prediction errors have been uncovered, the sources of aversive prediction errors remain largely unknown. Here we used probabilistic and deterministic reinforcement procedures, followed by extinction, to examine the contribution of the dorsal raphe nucleus to negative, aversive prediction errors in Pavlovian fear. Rats with dorsal raphe lesions were able to acquire fear and reduce fear to a non-reinforced deterministic cue. However, dorsal raphe lesions impaired the reduction of fear to a probabilistic cue and fear extinction to a deterministic cue, both of which involve the use of negative prediction errors. The results point to an integral role for the dorsal raphe nucleus in negative prediction error signaling in Pavlovian fear.

Dopamine and extinction: a convergence of theory with fear and reward circuitry

Research on dopamine lies at the intersection of sophisticated theoretical and neurobiological approaches to learning and memory. Dopamine has been shown to be critical for many processes that drive learning and memory, including motivation, prediction error, incentive salience, memory consolidation, and response output. Theories of dopamine's function in these processes have, for the most part, been developed from behavioral approaches that examine learning mechanisms in reward-related tasks. A parallel and growing literature indicates that dopamine is involved in fear conditioning and extinction. These studies are consistent with long-standing ideas about appetitive-aversive interactions in learning theory and they speak to the general nature of cellular and molecular processes that underlie behavior. We review the behavioral and neurobiological literature showing a role for dopamine in fear conditioning and extinction. At a cellular level, we review dopamine signaling and receptor pharmacology, cellular and molecular events that follow dopamine receptor activation, and brain systems in which dopamine functions. At a behavioral level, we describe theories of learning and dopamine function that could describe the fundamental rules underlying how dopamine modulates different aspects of learning and memory processes.

Accumbal opioid receptors modulate cue competition in one-trial overshadowing

The contribution of opioid receptors in the nucleus accumbens to contextual and auditory fear conditioning was examined. Impairment in contextual fear conditioning was found when training occurred under accumbal infusions of the opioid receptor agonist morphine in a dose-dependent and receptor specific fashion, only when shock onset coincided with auditory stimulus offset. Contextual fear conditioning was spared, however when the delivery of shock was not signalled by an auditory stimulus, the auditory stimulus was of low intensity (70dB), or an interval (10s or 30s) was interpolated between auditory stimulus offset and shock onset. These results provide evidence that opioid receptors in the nucleus accumbens regulate competition between contextual and discrete auditory stimuli for association formation.

Disambiguating ventral striatum fMRI-related BOLD signal during reward prediction in schizophrenia

Reward detection, surprise detection and prediction-error signaling have all been proposed as roles for the ventral striatum (vStr). Previous neuroimaging studies of striatal function in schizophrenia have found attenuated neural responses to reward-related prediction errors; however, as prediction errors represent a discrepancy in mesolimbic neural activity between expected and actual events, it is critical to examine responses to both expected and unexpected rewards (URs) in conjunction with expected and UR omissions in order to clarify the nature of ventral striatal dysfunction in schizophrenia. In the present study, healthy adults and people with schizophrenia were tested with a reward-related prediction-error task during functional magnetic resonance imaging to determine whether schizophrenia is associated with altered neural responses in the vStr to rewards, surprise prediction errors or all three factors. In healthy adults, we found neural responses in the vStr were correlated more specifically with prediction errors than to surprising events or reward stimuli alone. People with schizophrenia did not display the normal differential activation between expected and URs, which was partially due to exaggerated ventral striatal responses to expected rewards (right vStr) but also included blunted responses to unexpected outcomes (left vStr). This finding shows that neural responses, which typically are elicited by surprise, can also occur to well-predicted events in schizophrenia and identifies aberrant activity in the vStr as a key node of dysfunction in the neural circuitry used to differentiate expected and unexpected feedback in schizophrenia.

Using computational patients to evaluate illness mechanisms in schizophrenia

BACKGROUND

Various malfunctions involving working memory, semantics, prediction error, and dopamine neuromodulation have been hypothesized to cause disorganized speech and delusions in schizophrenia. Computational models may provide insights into why some mechanisms are unlikely, suggest alternative mechanisms, and tie together explanations of seemingly disparate symptoms and experimental findings.

METHODS

Eight corresponding illness mechanisms were simulated in DISCERN, an artificial neural network model of narrative understanding and recall. For this study, DISCERN learned sets of autobiographical and impersonal crime stories with associated emotion coding. In addition, 20 healthy control subjects and 37 patients with schizophrenia or schizoaffective disorder matched for age, gender, and parental education were studied using a delayed story recall task. A goodness-of-fit analysis was performed to determine the mechanism best reproducing narrative breakdown profiles generated by healthy control subjects and patients with schizophrenia. Evidence of delusion-like narratives was sought in simulations best matching the narrative breakdown profile of patients.

RESULTS

All mechanisms were equivalent in matching the narrative breakdown profile of healthy control subjects. However, exaggerated prediction-error signaling during consolidation of episodic memories, termed hyperlearning, was statistically superior to other mechanisms in matching the narrative breakdown profile of patients. These simulations also systematically confused autobiographical agents with impersonal crime story agents to model fixed, self-referential delusions.

CONCLUSIONS

Findings suggest that exaggerated prediction-error signaling in schizophrenia intermingles and corrupts narrative memories when incorporated into long-term storage, thereby disrupting narrative language and producing fixed delusional narratives. If further validated by clinical studies, these computational patients could provide a platform for developing and testing novel treatments.

Opponency revisited: competition and cooperation between dopamine and serotonin

Affective valence lies on a spectrum ranging from punishment to reward. The coding of such spectra in the brain almost always involves opponency between pairs of systems or structures. There is ample evidence for the role of dopamine in the appetitive half of this spectrum, but little agreement about the existence, nature, or role of putative aversive opponents such as serotonin. In this review, we consider the structure of opponency in terms of previous biases about the nature of the decision problems that animals face, the conflicts that may thus arise between Pavlovian and instrumental responses, and an additional spectrum joining invigoration to inhibition. We use this analysis to shed light on aspects of the role of serotonin and its interactions with dopamine.