The basolateral amygdala is necessary for negative prediction errors to enhance cue salience, but not to produce conditioned inhibition

The Recruitment of a Neuronal Ensemble in the Central Nucleus of the Amygdala During the First Extinction Episode Has Persistent Effects on Extinction Expression

BACKGROUND

Adaptive behavior depends on the delicate and dynamic balance between acquisition and extinction memories. Disruption of this balance, particularly when the extinction of memory loses control over behavior, is the root of treatment failure of maladaptive behaviors such as substance abuse or anxiety disorders. Understanding this balance requires a better understanding of the underlying neurobiology and its contribution to behavioral regulation.

METHODS

We microinjected Daun02 in Fos-lacZ transgenic rats following a single extinction training episode to delete extinction-recruited neuronal ensembles in the basolateral amygdala (BLA) and central nucleus of the amygdala (CN) and examined their contribution to behavior in an appetitive Pavlovian task. In addition, we used immunohistochemistry and neuronal staining methods to identify the molecular markers of activated neurons in the BLA and CN during extinction learning or retrieval.

RESULTS

CN neurons were preferentially engaged following extinction, and deletion of these extinction-activated ensembles in the CN but not the BLA impaired the retrieval of extinction despite additional extinction training and promoted greater levels of behavioral restoration in spontaneous recovery and reinstatement. Disrupting extinction processing in the CN in turn increased activity in the BLA. Our results also show a specific role for CN PKCδ+ neurons in behavioral inhibition but not during initial extinction learning.

CONCLUSIONS

We showed that the initial extinction-recruited CN ensemble is critical to the acquisition-extinction balance and that greater behavioral restoration does not mean weaker extinction contribution. These findings provide a novel avenue for thinking about the neural mechanisms of extinction and for developing treatments for cue-triggered appetitive behaviors.

Necessity and recruitment of cue-specific neuronal ensembles within the basolateral amygdala during appetitive reversal learning

Through Pavlovian appetitive conditioning, environmental cues can become predictors of food availability. Over time, however, the food, and thus the value of the associated cues, can change based on environmental variations. This change in outcome necessitates updating of the value of the cue to appropriately alter behavioral responses to these cues. The basolateral amygdala (BLA) is critical in updating the outcomes of learned cues. However, it is unknown if the same BLA neuronal ensembles that are recruited in the initial associative memory are required when the new cue-outcome association is formed during reversal learning. The current study used the Daun02 inactivation method that enables selective targeting and disruption of activated neuronal ensembles in Fos-lacZ transgenic rats. Rats were implanted with bilateral cannulas that target the BLA and underwent appetitive discriminative conditioning in which rats had to discriminate between two auditory stimuli. One stimulus (CS+) co-terminated with food delivery, and the other stimulus was unrewarded (CS-; counterbalanced). Rats were then tested for CS+ or CS- memory retrieval and infused with either Daun02 or a vehicle solution into the BLA to inactivate either CS+ or CS- neuronal ensembles that were activated during that test. To assess if the same neuronal ensembles are necessary to update the value of the new association when the outcomes are changed, rats underwent reversal learning: the CS+ was no longer followed by food (reversal CS-, rCS-), and the CS- was now followed by food (reversal CS+; rCS+). The group that received Daun02 following CS+ session showed a decrease in conditioned responding and increased latency to the rCS- (previously CS+) during the first session of reversal learning, specifically during the first trial. This indicates that the neuronal ensemble that was activated during the recall of the CS+ memory was the same neuronal ensemble needed for learning the new outcome of the same CS, now rCS-. Additionally, the group that received Daun02 following CS- session was slower to respond to the rCS+ (previously CS-) during reversal learning. This indicates that the neuronal ensemble that was activated during the recall of the CS- memory was the same neuronal ensemble needed for learning the new outcome of the same CS. These results demonstrate that different neuronal ensembles within the BLA mediate memory recall of CS+ and CS- cues and reactivation of each cue-specific neuronal ensemble is necessary to update the value of that specific cue to respond appropriately during reversal learning. These results also indicate substantial plasticity within the BLA for behavioral flexibility as both groups eventually showed similar terminal levels of reversal learning.

Amygdala-cortical collaboration in reward learning and decision making

Adaptive reward-related decision making requires accurate prospective consideration of the specific outcome of each option and its current desirability. These mental simulations are informed by stored memories of the associative relationships that exist within an environment. In this review, I discuss recent investigations of the function of circuitry between the basolateral amygdala (BLA) and lateral (lOFC) and medial (mOFC) orbitofrontal cortex in the learning and use of associative reward memories. I draw conclusions from data collected using sophisticated behavioral approaches to diagnose the content of appetitive memory in combination with modern circuit dissection tools. I propose that, via their direct bidirectional connections, the BLA and OFC collaborate to help us encode detailed, outcome-specific, state-dependent reward memories and to use those memories to enable the predictions and inferences that support adaptive decision making. Whereas lOFC→BLA projections mediate the encoding of outcome-specific reward memories, mOFC→BLA projections regulate the ability to use these memories to inform reward pursuit decisions. BLA projections to lOFC and mOFC both contribute to using reward memories to guide decision making. The BLA→lOFC pathway mediates the ability to represent the identity of a specific predicted reward and the BLA→mOFC pathway facilitates understanding of the value of predicted events. Thus, I outline a neuronal circuit architecture for reward learning and decision making and provide new testable hypotheses as well as implications for both adaptive and maladaptive decision making.

Valence encoding in the amygdala influences motivated behavior

The amygdala is critical for emotional processing and motivated behavior. Its role in these functions is due to its processing of the valence of environmental stimuli. The amygdala receives direct sensory input from sensory thalamus and cortical regions to integrate sensory information from the environment with aversive and/or appetitive outcomes. As many reviews have discussed the amygdala's role in threat processing and fear conditioning, this review will focus on how the amygdala encodes positive valence and the mechanisms that allow it to distinguish between stimuli of positive and negative valence. These findings are also extended to consider how valence encoding populations in the amygdala contribute to local and long-range circuits including those that integrate environmental cues and positive valence. Understanding the complexity of valence encoding in the amygdala is crucial as these mechanisms are implicated in a variety of disease states including anxiety disorders and substance use disorders.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Off-Target Influences of Arch-Mediated Axon Terminal Inhibition on Network Activity and Behavior

Archaerhodopsin (ArchT)-mediated photoinhibition of axon terminals is commonly used to test the involvement of specific long-range neural projections in behavior. Although sustained activation of this opsin in axon terminals has the unintended consequence of enhancing spontaneous vesicle release, it is unclear whether this desynchronized signaling is consequential for ArchT’s behavioral effects. Here, we compare axon terminal and cell body photoinhibition of nucleus accumbens (NAc) afferents to test the utility of these approaches for uncovering pathway-specific contributions of neural circuits to behavior. First, in brain slice recordings we confirmed that ArchT photoinhibition of glutamatergic axons reduces evoked synaptic currents and increases spontaneous transmitter release. A further consequence was increased interneuron activity, which served to broadly suppress glutamate input via presynaptic GABA_B receptors. In vivo, axon terminal photoinhibition increased feeding and reward-seeking behavior irrespective of the afferent pathway targeted. These behavioral effects are comparable to those obtained with broad inhibition of NAc neurons. In contrast, cell body inhibition of excitatory NAc afferents revealed a pathway-specific contribution of thalamic input to feeding behavior and amygdala input to reward-seeking under extinction conditions. These findings underscore the off-target behavioral consequences of ArchT-mediated axon terminal inhibition while highlighting cell body inhibition as a valuable alternative for pathway-specific optogenetic silencing.

Complementary contributions of basolateral amygdala and orbitofrontal cortex to value learning under uncertainty

We make choices based on the values of expected outcomes, informed by previous experience in similar settings. When the outcomes of our decisions consistently violate expectations, new learning is needed to maximize rewards. Yet not every surprising event indicates a meaningful change in the environment. Even when conditions are stable overall, outcomes of a single experience can still be unpredictable due to small fluctuations (i.e., expected uncertainty) in reward or costs. In the present work, we investigate causal contributions of the basolateral amygdala (BLA) and orbitofrontal cortex (OFC) in rats to learning under expected outcome uncertainty in a novel delay-based task that incorporates both predictable fluctuations and directional shifts in outcome values. We demonstrate that OFC is required to accurately represent the distribution of wait times to stabilize choice preferences despite trial-by-trial fluctuations in outcomes, whereas BLA is necessary for the facilitation of learning in response to surprising events.

DOI:http://dx.doi.org/10.7554/eLife.27483.001

Nobody likes waiting – we opt for online shopping to avoid standing in lines, grow impatient in traffic, and often prefer restaurants that serve food quickly. When making decisions, humans and other animals try to maximize the benefits by weighing up the costs and rewards associated with a situation. Many regions in the brain help us choose the best options based on quality and size of rewards, and required waiting times. Even before we make decisions, the activity in these brain regions predicts what we will choose.

Sometimes, however, unexpected changes can lead to longer waiting times and our preferences suddenly become less desirable. The brain can detect such changes by comparing the outcomes we anticipate to those we experience. When the outcomes are surprising, specific areas in the brain such as the amygdala and the orbitofrontal cortex help us learn to make better choices. However, as surprising events can occur purely by chance, we need to be able to ignore irrelevant surprises and only learn from meaningful ones. Until now, it was not clear whether the amygdala and orbitofrontal cortex play specific roles in successfully learning under such conditions.

Stolyarova and Izquierdo trained rats to select between two images and rewarded them with sugar pellets after different delays. If rats chose one of these images they received the rewards after a predictable delay that was about 10 seconds, while choosing the other one produced variable delays – sometimes the time intervals were either very short or very long. Then, the waiting times for one of the alternatives changed unexpectedly. Rats with healthy brains quickly learned to choose the option with the shorter waiting time.

Stolyarova and Izquierdo repeated the experiments with rats that had damage in a part of the amygdala. These rats learned more slowly, particularly when the variable option changed for the better. Rats with damage to the orbitofrontal cortex failed to learn at all. Stolyarova and Izquierdo then examined the rats’ behavior during delays. Rats with damage to the orbitofrontal cortex could not distinguish between meaningful and irrelevant surprises and always looked for the food pellet (i.e. anticipated a reward) at the average delay interval.

These findings highlight two brain regions that help us distinguish meaningful surprises from irrelevant ones. A next step will be to examine how the amygdala and orbitofrontal cortex interact during learning and see if changes to the activity of these brain regions may affect responses. Advanced methods to non-invasively manipulate brain activity in humans may help people who find it hard to cope with changes; or individuals suffering from substance use disorders, who often struggle to give up drugs that provide them immediate and predictable rewards.

DOI:http://dx.doi.org/10.7554/eLife.27483.002

The Dopamine Prediction Error: Contributions to Associative Models of Reward Learning

Phasic activity of midbrain dopamine neurons is currently thought to encapsulate the prediction-error signal described in Sutton and Barto’s (1981) model-free reinforcement learning algorithm. This phasic signal is thought to contain information about the quantitative value of reward, which transfers to the reward-predictive cue after learning. This is argued to endow the reward-predictive cue with the value inherent in the reward, motivating behavior toward cues signaling the presence of reward. Yet theoretical and empirical research has implicated prediction-error signaling in learning that extends far beyond a transfer of quantitative value to a reward-predictive cue. Here, we review the research which demonstrates the complexity of how dopaminergic prediction errors facilitate learning. After briefly discussing the literature demonstrating that phasic dopaminergic signals can act in the manner described by Sutton and Barto (1981), we consider how these signals may also influence attentional processing across multiple attentional systems in distinct brain circuits. Then, we discuss how prediction errors encode and promote the development of context-specific associations between cues and rewards. Finally, we consider recent evidence that shows dopaminergic activity contains information about causal relationships between cues and rewards that reflect information garnered from rich associative models of the world that can be adapted in the absence of direct experience. In discussing this research we hope to support the expansion of how dopaminergic prediction errors are thought to contribute to the learning process beyond the traditional concept of transferring quantitative value.

Consolidation of altered associability information by amygdala central nucleus

The surprising omission of a reinforcer can enhance the associability of the stimuli that were present when the reward prediction error was induced, so that they more readily enter into new associations in the future. Previous research from this laboratory identified brain circuit elements critical to the enhancement of stimulus associability by the omission of an expected event and to the subsequent expression of that altered associability in more rapid learning. These elements include the amygdala, the midbrain substantia nigra, the basal forebrain substantia innominata, the dorsolateral striatum, the secondary visual cortex, and the posterior parietal cortex. Here, we found that consolidation of a surprise-enhanced associability memory in a serial prediction task depends on processing in the amygdala central nucleus (CeA) after completion of sessions that included the surprising omission of an expected event. Post-surprise infusions of anisomycin, lidocaine, or muscimol prevented subsequent display of surprise-enhanced associability. Because previous studies indicated that CeA function is unnecessary for the expression of associability enhancements that were induced previously when CeA function was intact (Holland & Gallagher, 2006), we interpreted these results as indicating that post-surprise activity of CeA ("surprise replay") is necessary for the consolidation of altered associability memories elsewhere in the brain, such as the posterior parietal cortex (Schiffino et al., 2014a).

Effects of amygdala lesions on overexpectation phenomena in food cup approach and autoshaping procedures

Prediction error (PE) plays a critical role in most modern theories of associative learning, by determining the effectiveness of conditioned stimuli (CS) or unconditioned stimuli (US). Here, we examined the effects of lesions of central (CeA) or basolateral (BLA) amygdala on performance in overexpectation tasks. In 2 experiments, after 2 CSs were separately paired with the US, they were combined and followed by the same US. In a subsequent test, we observed losses in strength of both CSs, as expected if the negative PE generated on reinforced compound trials encouraged inhibitory learning. CeA lesions, known to interfere with PE-induced enhancements in CS effectiveness, reduced those losses, suggesting that normally the negative PE also enhances cue associability in this task. BLA lesions had no effect. When a novel cue accompanied the reinforced compound, it acquired net conditioned inhibition, despite its consistent pairings with the US, consonant with US effectiveness models. That acquisition was unaffected by either CeA or BLA lesions, suggesting different rules for assignment of credit of changes in cue strength and cue associability. Finally, we examined a puzzling autoshaping phenomenon previously attributed to overexpectation effects. When a previously food-paired auditory cue was combined with the insertion of a lever and paired with the same food US, the auditory cue not only failed to block conditioning to the lever, but also lost strength, as in an overexpectation experiment. This effect was abolished by BLA lesions but unaffected by CeA lesions, suggesting it was unrelated to other overexpectation effects. (PsycINFO Database Record

Mini-review: Prediction errors, attention and associative learning

Most modern theories of associative learning emphasize a critical role for prediction error (PE, the difference between received and expected events). One class of theories, exemplified by the Rescorla-Wagner (1972) model, asserts that PE determines the effectiveness of the reinforcer or unconditioned stimulus (US): surprising reinforcers are more effective than expected ones. A second class, represented by the Pearce-Hall (1980) model, argues that PE determines the associability of conditioned stimuli (CSs), the rate at which they may enter into new learning: the surprising delivery or omission of a reinforcer enhances subsequent processing of the CSs that were present when PE was induced. In this mini-review we describe evidence, mostly from our laboratory, for PE-induced changes in the associability of both CSs and USs, and the brain systems involved in the coding, storage and retrieval of these altered associability values. This evidence favors a number of modifications to behavioral models of how PE influences event processing, and suggests the involvement of widespread brain systems in animals' responses to PE.

Lateral orbitofrontal neurons acquire responses to upshifted, downshifted, or blocked cues during unblocking

The lateral orbitofrontal cortex (lOFC) has been described as signaling either outcome expectancies or value. Previously, we used unblocking to show that lOFC neurons respond to a predictive cue signaling a ‘valueless’ change in outcome features (McDannald, 2014). However, many lOFC neurons also fired to a cue that simply signaled more reward. Here, we recorded lOFC neurons in a variant of this task in which rats learned about cues that signaled either more (upshift), less (downshift) or the same (blocked) amount of reward. We found that neurons acquired responses specifically to one of the three cues and did not fire to the other two. These results show that, at least early in learning, lOFC neurons fire to valued cues in a way that is more consistent with signaling of the predicted outcome’s features than with signaling of a general, abstract or cached value that is independent of the outcome.

DOI:http://dx.doi.org/10.7554/eLife.11299.001

The basolateral amygdala in reward learning and addiction

Sophisticated behavioral paradigms partnered with the emergence of increasingly selective techniques to target the basolateral amygdala (BLA) have resulted in an enhanced understanding of the role of this nucleus in learning and using reward information. Due to the wide variety of behavioral approaches many questions remain on the circumscribed role of BLA in appetitive behavior. In this review, we integrate conclusions of BLA function in reward-related behavior using traditional interference techniques (lesion, pharmacological inactivation) with those using newer methodological approaches in experimental animals that allow in vivo manipulation of cell type-specific populations and neural recordings. Secondly, from a review of appetitive behavioral tasks in rodents and monkeys and recent computational models of reward procurement, we derive evidence for BLA as a neural integrator of reward value, history, and cost parameters. Taken together, BLA codes specific and temporally dynamic outcome representations in a distributed network to orchestrate adaptive responses. We provide evidence that experiences with opiates and psychostimulants alter these outcome representations in BLA, resulting in long-term modified action.