Conditioned reinforcement can be mediated by either outcome-specific or general affective representations

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Dopamine in the dorsal bed nucleus of stria terminalis signals Pavlovian sign-tracking and reward violations

Midbrain and striatal dopamine signals have been extremely well characterized over the past several decades, yet novel dopamine signals and functions in reward learning and motivation continue to emerge. A similar characterization of real-time sub-second dopamine signals in areas outside of the striatum has been limited. Recent advances in fluorescent sensor technology and fiber photometry permit the measurement of dopamine binding correlates, which can divulge basic functions of dopamine signaling in non-striatal dopamine terminal regions, like the dorsal bed nucleus of the stria terminalis (dBNST). Here, we record GRABDA signals in the dBNST during a Pavlovian lever autoshaping task. We observe greater Pavlovian cue-evoked dBNST GRABDA signals in sign-tracking (ST) compared to goal-tracking/intermediate (GT/INT) rats and the magnitude of cue-evoked dBNST GRABDA signals decreases immediately following reinforcer-specific satiety. When we deliver unexpected rewards or omit expected rewards, we find that dBNST dopamine signals encode bidirectional reward prediction errors in GT/INT rats, but only positive prediction errors in ST rats. Since sign- and goal-tracking approach strategies are associated with distinct drug relapse vulnerabilities, we examined the effects of experimenter-administered fentanyl on dBNST dopamine associative encoding. Systemic fentanyl injections do not disrupt cue discrimination but generally potentiate dBNST dopamine signals. These results reveal multiple dBNST dopamine correlates of learning and motivation that depend on the Pavlovian approach strategy employed.

Reinstatement of Pavlovian responses to alcohol cues by stress

RATIONALE

Stress may contribute to relapse to alcohol use in part by enhancing reactivity to cues previously paired with alcohol. Yet, standard models of stress-induced reinstatement generally use contingent presentations of alcohol-paired cues to reinforce instrumental behaviors, making it difficult to isolate the ability of cues to invigorate alcohol-seeking.

OBJECTIVE

Here we sought to test the impact of stress on behavioral responses to alcohol-paired cues, using a model of stress-induced reinstatement of Pavlovian conditioned approach, inspired by Nadia Chaudhri's work on context-induced reinstatement.

METHODS

Long Evans rats were trained to associate one auditory cue with delivery of alcohol or sucrose and an alternative auditory cue with no reward. Following extinction training, rats were exposed to a stressor prior to being re-exposed to the cues under extinction conditions. We assessed the effects of yohimbine, intermittent footshock and olfactory cues paired with social defeat on responses to alcohol-paired cues and the effects of yohimbine on responses to sucrose-paired cues.

RESULTS

The pharmacological stressor, yohimbine, enhanced alcohol seeking in a Pavlovian setting, but not in a cue-selective manner. Intermittent footshock and social defeat cues did not enhance alcohol seeking in this paradigm.

CONCLUSIONS

While yohimbine elicited reinstatement of reward-seeking in a Pavlovian setting, these effects may be unrelated to activation of stress systems or to interactions with specific cues.

Inactivation of the Basolateral Amygdala to Insular Cortex Pathway Makes Sign-Tracking Sensitive to Outcome Devaluation

Goal-tracking (GT) rats are sensitive to Pavlovian outcome devaluation while sign-tracking (ST) rats are devaluation insensitive. During outcome devaluation, GT rats flexibly modify responding to cues based on the current value of the associated outcome. However, ST rats rigidly respond to cues regardless of the current outcome value. Prior work demonstrated disconnection of the basolateral amygdala (BLA) and anterior insular cortex (aIC) decreased both GT and ST behaviors. Given the role of these regions in appetitive motivation and behavioral flexibility, we predicted that disrupting BLA to aIC pathway during outcome devaluation would reduce flexibility in GT rats and reduce rigid appetitive motivation in ST rats. We inhibited the BLA to aIC pathway by infusing inhibitory DREADDs (hM4Di-mcherry) or control (mCherry) virus into the BLA and implanted cannulae into the aIC to inhibit BLA terminals using intracranial injections of clozapine N-oxide (CNO). After training, we used a within-subject satiety-induced outcome devaluation procedure in which we sated rats on training pellets (devalued condition) or homecage chow (valued condition). All rats received bilateral CNO infusions into the aIC before brief nonreinforced test sessions. Contrary to our hypothesis, BLA-IC inhibition did not interfere with devaluation sensitivity in GT rats but did make ST behaviors sensitive to devaluation. Intermediate rats showed the opposite effect, showing rigid responding to cues with BLA-aIC pathway inactivation. Together, these results demonstrate BLA-IC projections mediate tracking-specific Pavlovian devaluation sensitivity and highlights the importance of considering individual differences in Pavlovian approach when evaluating circuitry contributions to behavioral flexibility.

Choose your path: Divergent basolateral amygdala efferents differentially mediate incentive motivation, flexibility and decision-making

To survive in a complex environment, individuals form associations between environmental stimuli and rewards to organize and optimize reward seeking behaviors. The basolateral amygdala (BLA) uses these learned associations to inform decision-making processes. In this review, we describe functional projections between BLA and its cortical and striatal targets that promote learning and motivational processes central to decision-making. Specifically, we compare and contrast divergent projections from the BLA to the orbitofrontal (OFC) and to the nucleus accumbens (NAc) and examine the roles of these pathways in associative learning, value-guided decision-making, choice behaviors, as well as cue and context-driven drug seeking. Finally, we consider how these projections are involved in disorders of motivation, with a focus on Substance Use Disorder.

Associative processes in addiction relapse models: A review of their Pavlovian and instrumental mechanisms, history, and terminology

Animal models of relapse to drug-seeking have borrowed heavily from associative learning approaches. In studies of relapse-like behaviour, animals learn to self-administer drugs then receive a period of extinction during which they learn to inhibit the operant response. Several triggers can produce a recovery of responding which form the basis of a variety of models. These include the passage of time (spontaneous recovery), drug availability (rapid reacquisition), extinction of an alternative response (resurgence), context change (renewal), drug priming, stress, and cues (reinstatement). In most cases, the behavioural processes driving extinction and recovery in operant drug self-administration studies are similar to those in the Pavlovian and behavioural literature, such as context effects. However, reinstatement in addiction studies have several differences with Pavlovian reinstatement, which have emerged over several decades, in experimental procedures, associative mechanisms, and terminology. Interestingly, in cue-induced reinstatement, drug-paired cues that are present during acquisition are omitted during lever extinction. The unextinguished drug-paired cue may limit the model’s translational relevance to cue exposure therapy and renders its underlying associative mechanisms ambiguous. We review major behavioural theories that explain recovery phenomena, with a particular focus on cue-induced reinstatement because it is a widely used model in addiction. We argue that cue-induced reinstatement may be explained by a combination of behavioural processes, including reacquisition of conditioned reinforcement and Pavlovian to Instrumental Transfer. While there are important differences between addiction studies and the behavioural literature in terminology and procedures, it is clear that understanding associative learning processes is essential for studying relapse.

The Role of the Rodent Lateral Orbitofrontal Cortex in Simple Pavlovian Cue-Outcome Learning Depends on Training Experience

The orbitofrontal cortex (OFC) is a critical structure in the flexible control of value-based behaviors. OFC dysfunction is typically only detected when task or environmental contingencies change, against a backdrop of apparently intact initial acquisition and behavior. While intact acquisition following OFC lesions in simple Pavlovian cue-outcome conditioning is often predicted by models of OFC function, this predicted null effect has not been thoroughly investigated. Here, we test the effects of lesions and temporary muscimol inactivation of the rodent lateral OFC on the acquisition of a simple single cue-outcome relationship. Surprisingly, pretraining lesions significantly enhanced acquisition after overtraining, whereas post-training lesions and inactivation significantly impaired acquisition. This impaired acquisition to the cue reflects a disruption of behavioral control and not learning since the cue could also act as an effective blocking stimulus in an associative blocking procedure. These findings suggest that even simple cue-outcome representations acquired in the absence of OFC function are impoverished. Therefore, while OFC function is often associated with flexible behavioral control in complex environments, it is also involved in very simple Pavlovian acquisition where complex cue-outcome relationships are irrelevant to task performance.

Behavioral determinants in the expression of the Kamin blocking effect: Implications for associative learning theory

Associative learning makes important contributions to our behavior and decisions. The Kamin blocking effect is an associative learning phenomenon that plays a central role in understanding of the psychological principles underlying associative learning. However, several recent failures to replicate the blocking effect suggest that the conditions necessary for blocking are poorly understood. To understand the conditions necessary for blocking, here we review studies into the expression of blocking in subjects that either approach and interact with the conditioned cue (sign trackers) or approach and interact with the reward location (goal trackers) during appetitive classical conditioning. Psychological theory and the neurophysiological correlates of appetitive classical conditioning make opposing predictions regarding the expression of blocking in sign and goal trackers. We reconcile these opposing predictions in a qualitative model using two parallel learning processes. Such models offer a better framework for understanding the psychological associative structures acquired during learning, their interactions contributing to the conditioned response, and how they affect subsequent learning and the expression of the Kamin blocking effect.

Occasion setters attain incentive motivational value: implications for contextual influences on reward-seeking

The context in which reward-paired cues are encountered can resolve ambiguity and set the occasion for appropriate reward-seeking. The psychological processes by which contexts regulate reward-seeking remain unclear as contexts are diffuse and difficult to isolate from other stimuli. To overcome this, we modeled a context as a phasic and discrete event—an occasion setter (OS)—which allowed for control over its presentation and influence on cue-driven reward-seeking. This allowed us to directly assess how OSs regulate the predictive and motivational significance of Pavlovian cues. Male rats (n = 50) were trained in a Pavlovian procedure where the presentation of an ambiguous conditioned stimulus (CS) was reinforced only if preceded by an occasion setting cue. We assessed the motivational value of the OS and CS alone or in combination using tests of conditioned reinforcement. Rats enhanced conditioned approach to the reward port during the CS when it was preceded by the OS. When allowed the opportunity, rats responded more to obtain presentations of the CS in combination with the OS than the CS alone. Critically, rats also worked to obtain presentations of the OS alone more than the CS alone, and this was resistant to manipulations of the value of the OS. We conclude that occasion setting can act via incentive motivational mechanisms and that, apart from resolving predictive information about ambiguous reward-paired cues, OSs themselves generate states of appetitive motivation that can facilitate reward-seeking.

An Integrative Way for Studying Neural Basis of Basic Emotions With fMRI

How emotions are represented in the nervous system is a crucial unsolved problem in the affective neuroscience. Many studies are striving to find the localization of basic emotions in the brain but failed. Thus, many psychologists suspect the specific neural loci for basic emotions, but instead, some proposed that there are specific neural structures for the core affects, such as arousal and hedonic value. The reason for this widespread difference might be that basic emotions used previously can be further divided into more “basic” emotions. Here we review brain imaging data and neuropsychological data, and try to address this question with an integrative model. In this model, we argue that basic emotions are not contrary to the dimensional studies of emotions (core affects). We propose that basic emotion should locate on the axis in the dimensions of emotion, and only represent one typical core affect (arousal or valence). Therefore, we propose four basic emotions: joy-on positive axis of hedonic dimension, sadness-on negative axis of hedonic dimension, fear, and anger-on the top of vertical dimensions. This new model about basic emotions and construction model of emotions is promising to improve and reformulate neurobiological models of basic emotions.

Preconditioned cues have no value

Sensory preconditioning has been used to implicate midbrain dopamine in model-based learning, contradicting the view that dopamine transients reflect model-free value. However, it has been suggested that model-free value might accrue directly to the preconditioned cue through mediated learning. Here, building on previous work (Sadacca et al., 2016), we address this question by testing whether a preconditioned cue will support conditioned reinforcement in rats. We found that while both directly conditioned and second-order conditioned cues supported robust conditioned reinforcement, a preconditioned cue did not. These data show that the preconditioned cue in our procedure does not directly accrue model-free value and further suggest that the cue may not necessarily access value even indirectly in a model-based manner. If so, then phasic response of dopamine neurons to cues in this setting cannot be described as signaling errors in predicting value.

Effects of sex and remifentanil dose on rats' acquisition of responding for a remifentanil-conditioned reinforcer

Opioid-conditioned reinforcement is thought to exacerbate opioid abuse and dependence. Sex/gender can influence opioid abuse behaviors, but the effects of sex/gender on opioid-conditioned reinforcement, specifically, are unclear. In this study, we compared new-response acquisition with opioid-conditioned reinforcement in male and female rats. First, separate groups received response-independent remifentanil injections (0.0-32.0 μg/kg, intravenous) and presentations of a light-noise stimulus. In the experimental groups, injections and stimulus presentations always co-occurred [paired Pavlovian conditioning (PAV)]; in the control groups, the two occurred independently of each other (random PAV). Next, in the instrumental acquisition (ACQ) sessions, two novel nose-poke manipulanda were introduced. All animals (regardless of sex, dose, and PAV type) could respond in the active nose-poke, which produced the stimulus alone, or in the inactive nose-poke. Both males and females dose-dependently acquired nose-poke responding (active>inactive) after paired PAV, but not after random PAV. Therefore, the stimulus was a conditioned reinforcer. We identified three sex differences. First, only females acquired responding after paired PAV with 32.0 μg/kg remifentanil. Second, using a progressive ratio schedule for ACQ, both sexes acquired responding, but females made significantly more active responses. Third, when a single session of PAV was conducted, only males acquired responding. Thus, rats' sex interacts with pharmacological and environmental factors to determine opioid-conditioned reinforcement.

Dopamine D2/3- and μ-opioid receptor antagonists reduce cue-induced responding and reward impulsivity in humans

Increased responding to drug-associated stimuli (cue reactivity) and an inability to tolerate delayed gratification (reward impulsivity) have been implicated in the development and maintenance of drug addiction. Whereas data from animal studies suggest that both the dopamine and opioid system are involved in these two reward-related processes, their role in humans is less clear. Moreover, dopaminergic and opioidergic drugs have not been directly compared with regard to these functions, even though a deeper understanding of the underlying mechanisms might inform the development of specific treatments for elevated cue reactivity and reward impulsivity. In a randomized, double-blind, between-subject design we administered the selective dopamine D2/D3 receptor antagonist amisulpride (400 mg, n=41), the unspecific opioid receptor antagonist naltrexone (50 mg, n=40) or placebo (n=40) to healthy humans and measured cue-induced responding with a Pavlovian-instrumental transfer task and reward impulsivity with a delay discounting task. Mood was assessed using a visual analogue scale. Compared with placebo, amisulpride significantly suppressed cue-induced responding and reward impulsivity. The effects of naltrexone were similar, although less pronounced. Both amisulpride and naltrexone decreased average mood ratings compared with placebo. Our results demonstrate that a selective blockade of dopamine D2/D3 receptors reduces cue-induced responding and reward impulsivity in healthy humans. Antagonizing μ-opioid receptors has similar effects for cue-induced responding and to a lesser extent for reward impulsivity.

Post-training depletions of basolateral amygdala serotonin fail to disrupt discrimination, retention, or reversal learning

In goal-directed pursuits, the basolateral amygdala (BLA) is critical in learning about changes in the value of rewards. BLA-lesioned rats show enhanced reversal learning, a task employed to measure the flexibility of response to changes in reward. Similarly, there is a trend for enhanced discrimination learning, suggesting that BLA may modulate formation of stimulus-reward associations. There is a parallel literature on the importance of serotonin (5HT) in new stimulus-reward and reversal learning. Recent postulations implicate 5HT in learning from punishment. Whereas, dopaminergic involvement is critical in behavioral activation and reinforcement, 5HT may be most critical for aversive processing and behavioral inhibition, complementary cognitive processes. Given these findings, a 5HT-mediated mechanism in BLA may mediate the facilitated learning observed previously. The present study investigated the effects of selective 5HT lesions in BLA using 5,7-dihydroxytryptamine (5,7-DHT) vs. infusions of saline (Sham) on discrimination, retention, and deterministic reversal learning. Rats were required to reach an 85% correct pairwise discrimination and single reversal criterion prior to surgery. Postoperatively, rats were then tested on the (1) retention of the pretreatment discrimination pair, (2) discrimination of a novel pair, and (3) reversal learning performance. We found statistically comparable preoperative learning rates between groups, intact postoperative retention, and unaltered novel discrimination and reversal learning in 5,7-DHT rats. These findings suggest that 5HT in BLA is not required for formation and flexible adjustment of new stimulus-reward associations when the strategy to efficiently solve the task has already been learned. Given the complementary role of orbitofrontal cortex in reward learning and its interconnectivity with BLA, these findings add to the list of dissociable mechanisms for BLA and orbitofrontal cortex in reward learning.

Effects of pramipexole on the acquisition of responding with opioid-conditioned reinforcement in the rat

RATIONALE

Dopamine D3 receptor-preferring ligands may be able to modify the conditioned reinforcing effects of drug-associated stimuli. In evaluating the effects of these compounds, it is important to clarify the extent to which responding depends on (1) conditioned reinforcement vs. other behavioral mechanisms and (2) dopamine D3 vs. D2 receptor activity.

OBJECTIVES

To use behaviorally stringent new-response acquisition procedures to characterize the effects of the D3-preferring agonist, pramipexole, on the conditioned reinforcing effects of a stimulus paired with the opioid agonist, remifentanil.

METHODS

First, in Pavlovian conditioning (PAV) sessions, rats received response-independent IV injections of remifentanil and presentations of a light-noise stimulus. In separate groups, injections and stimuli either always co-occurred ("paired PAV") or occurred with no consistent relationship ("random PAV" control). Next, in instrumental acquisition (ACQ) sessions, all animals could respond in two nose-poke manipulanda: an active nose-poke, which produced the stimulus alone, or an inactive nose-poke. Pramipexole was injected SC prior to ACQ sessions with or without pretreatments of the D3-preferring antagonist, SB-277011A, or the D2-preferring antagonist, L-741,626.

RESULTS

After paired PAV, but not random PAV, rats acquired nose-poke responding during ACQ (i.e., active > inactive). Pramipexole dose-dependently increased active responding without changing inactive responding. Pramipexole-induced increases in responding were blocked by pretreatment with L-741,626, but not SB-277011A.

CONCLUSIONS

Pramipexole specifically enhanced remifentanil-conditioned reinforcement: active responding was selectively increased only after the stimulus was paired with remifentanil. Although pramipexole is D3-preferring, the antagonist effects obtained presently suggest an important role for the D2 receptor in opioid-conditioned reinforcement.

The amygdala: securing pleasure and avoiding pain

The amygdala has traditionally been associated with fear, mediating the impact of negative emotions on memory. However, this view does not fully encapsulate the function of the amygdala, nor the impact that processing in this structure has on the motivational limbic corticostriatal circuitry of which it is an important structure. Here we discuss the interactions between different amygdala nuclei with cortical and striatal regions involved in motivation; interconnections and parallel circuitries that have become increasingly understood in recent years. We review the evidence that the amygdala stores memories that allow initially motivationally neutral stimuli to become associated through pavlovian conditioning with motivationally relevant outcomes which, importantly, can be either appetitive (e.g. food) or aversive (e.g. electric shock). We also consider how different psychological processes supported by the amygdala such as conditioned reinforcement and punishment, conditioned motivation and suppression, and conditioned approach and avoidance behavior, are not only psychologically but also neurobiologically dissociable, being mediated by distinct yet overlapping neural circuits within the limbic corticostriatal circuitry. Clearly the role of the amygdala goes beyond encoding aversive stimuli to also encode the appetitive, requiring an appreciation of the amygdala's mediation of both appetitive and fearful behavior through diverse psychological processes.

Individual variation in resisting temptation: implications for addiction

When exposed to the sights, sounds, smells and/or places that have been associated with rewards, such as food or drugs, some individuals have difficulty resisting the temptation to seek out and consume them. Others have less difficulty restraining themselves. Thus, Pavlovian reward cues may motivate maladaptive patterns of behavior to a greater extent in some individuals than in others. We are just beginning to understand the factors underlying individual differences in the extent to which reward cues acquire powerful motivational properties, and therefore, the ability to act as incentive stimuli. Here we review converging evidence from studies in both human and non-human animals suggesting that a subset of individuals are more "cue reactive", in that certain reward cues are more likely to attract these individuals to them and motivate actions to get them. We suggest that those individuals for whom Pavlovian reward cues become especially powerful incentives may be more vulnerable to impulse control disorders, such as binge eating and addiction.

Acquisition of responding with a remifentanil-associated conditioned reinforcer in the rat

RATIONALE

Drug-associated environmental stimuli may serve as conditioned reinforcers to enhance drug self-administration behaviors in humans and laboratory animals. However, it can be difficult to distinguish experimentally the conditioned reinforcing effects of a stimulus from other behavioral processes that can change rates of responding.

OBJECTIVES

To characterize the conditioned reinforcing effects of a stimulus paired with the μ-opioid agonist, remifentanil, using a new-response acquisition procedure in the rat.

METHODS

First, in Pavlovian conditioning (PAV) sessions, rats received response-independent IV injections of remifentanil and presentations of a light-noise compound stimulus. In paired PAV groups, injections and stimulus presentations always co-occurred. In random PAV control groups, injections and stimulus presentations occurred with no consistent relationship. Second, in instrumental acquisition (ACQ) sessions, all animals could respond in an active nose-poke that produced the stimulus alone or in an inactive nose-poke that had no scheduled consequences.

RESULTS

During ACQ, rats made significantly more active nose-pokes than inactive nose-pokes after paired PAV, but not after random PAV. Between groups, rats also made more active nose-pokes after paired PAV than after random PAV. After paired PAV, increased active responding was obtained under different schedules of reinforcement, persisted across multiple ACQ sessions, and depended on the number of PAV sessions conducted.

CONCLUSIONS

The remifentanil-paired stimulus served as a conditioned reinforcer for nose-poking: responding depended on both the contingency between the stimulus and remifentanil and the contingency between the nose-poke and the stimulus. Generally, new-response acquisition procedures may provide valid, flexible models for studying opioid-based conditioned reinforcement.

Mechanisms of reward circuit dysfunction in psychiatric illness: prefrontal-striatal interactions

The brain's "reward circuit" has been widely implicated in the pathophysiology of mental illness. Although there has been significant progress in identifying the functional characteristics of individual nodes within the circuit and linking dysfunction of these brain areas to various forms of psychopathology, there remains a substantial gap in understanding how the nodes of the circuit interact with one another, and how the growing neurobiological knowledge may be applied to improve psychiatric patient care. In this article, we summarize what is currently known about the functions and interactions of two key nodes of this circuit-the ventral striatum and the ventromedial prefrontal/orbital frontal cortex-in relation to mental illness.

Perseverative instrumental and Pavlovian responding to conditioned stimuli in serotonin transporter knockout rats

Environmental stimuli can influence behavior via the process of Pavlovian conditioning. Recent genetic research suggests that some individuals are more sensitive to environmental stimuli for behavioral guidance than others. One important mediator of this effect is serotonin transporter (5-HTT) genetic variance, which increases sensitivity to Pavlovian conditioned stimuli through changes in the build-up of corticolimbic circuits. As these stimuli can have reinforcing effects on instrumental responding, we here investigated their effects on instrumental behavior in 5-HTT knockout rats and their wild-type counterparts by means of the signal attenuation paradigm. In this paradigm animals acquired a Pavlovian association between a stimulus and food reward, and subsequently they had to lever press in order to gain access to this food reward-associated stimulus. Thereafter, half of the animals underwent extinction training during which extinction of the primary Pavlovian association was induced via non-reinforced stimulus presentations, whereas the other half did not receive this training. During a final test session all animals were tested for instrumental responding for the non-reinforced Pavlovian conditioned stimulus, as well as instrumental and Pavlovian responding to the stimulus after an initial lever-press. No genotype differences were observed during the training and extinction sessions. However, during the test session 5-HTT knockout rats that had not received prior extinction training displayed excessive instrumental responding. This was specifically observed during presentation of the stimulus (induced by the first lever press) and was accompanied by an increased number of feeder visits after termination of the stimulus presentation. An additionally performed c-Fos immunohistochemistry study revealed that the behaviors in these animals were associated with abnormal c-Fos immunoreactivity in the orbitofrontal cortex and basolateral amygdala, regions important for the acquisition and maintenance of Pavlovian conditioned stimuli. These findings complement earlier findings showing that 5-HTT knockout animals' behavior is heavily influenced by environmental stimuli and indicate that this extends to the instrumental domain.

Does the orbitofrontal cortex signal value?

The orbitofrontal cortex (OFC) has long been implicated in associative learning. Early work by Mishkin and Rolls showed that the OFC was critical for rapid changes in learned behavior, a role that was reflected in the encoding of associative information by orbitofrontal neurons. Over the years, new data-particularly neurophysiological data-have increasingly emphasized the OFC in signaling actual value. These signals have been reported to vary according to internal preferences and judgments and to even be completely independent of the sensory qualities of predictive cues, the actual rewards, and the responses required to obtain them. At the same time, increasingly sophisticated behavioral studies have shown that the OFC is often unnecessary for simple value-based behavior and instead seems critical when information about specific outcomes must be used to guide behavior and learning. Here, we review these data and suggest a theory that potentially reconciles these two ideas, value versus specific outcomes, and bodies of work on the OFC.

Representations of appetitive and aversive information in the primate orbitofrontal cortex

Individuals weigh information about both rewarding and aversive stimuli to make adaptive decisions. Most studies of the orbitofrontal cortex (OFC), an area where appetitive and aversive neural subsystems might interact, have focused only on reward. Using a classical conditioning task where novel stimuli are paired with a reward or an aversive air puff, we discovered that two groups of orbitofrontal neurons respond preferentially to conditioned stimuli associated with rewarding and aversive outcomes; however, information about appetitive and aversive stimuli converges on individual neurons from both populations. Therefore, neurons in the OFC might participate in appetitive and aversive networks that track the motivational significance of stimuli even when they vary in valence and sensory modality. Further, we show that these networks, which also extend to the amygdala, exhibit different rates of change during reversal learning. Thus, although both networks represent appetitive and aversive associations, their distinct temporal dynamics might indicate different roles in learning processes.

The impact of orbitofrontal dysfunction on cocaine addiction

Cocaine addiction is characterized by poor judgment and maladaptive decision-making. Here we review evidence implicating the orbitofrontal cortex in such behavior. This evidence suggests that cocaine-induced changes in orbitofrontal cortex disrupt the representation of states and transition functions that form the basis of flexible and adaptive 'model-based' behavioral control. By impairing this function, cocaine exposure leads to an overemphasis on less flexible, maladaptive 'model-free' control systems. We propose that such an effect accounts for the complex pattern of maladaptive behaviors associated with cocaine addiction.

Orbitofrontal cortex and basolateral amygdala lesions result in suboptimal and dissociable reward choices on cue-guided effort in rats

The orbitofrontal cortex (OFC) and basolateral nucleus of the amygdala (BLA) are important neural regions in responding adaptively to changes in the incentive value of reward. Recent evidence suggests these structures may be differentially engaged in effort and cue-guided choice behavior. In 2 T-maze experiments, we examined the effects of bilateral lesions of either BLA or OFC on (1) effortful choices in which rats could climb a barrier for a high reward or select a low reward with no effort and (2) effortful choices when a visual cue signaled changes in reward magnitude. In both experiments, BLA rats displayed transient work aversion, choosing the effortless low reward option. OFC rats were work averse only in the no cue conditions, displaying a pattern of attenuated recovery from the cue conditions signaling reward unavailability in the effortful arm. Control measures rule out an inability to discriminate the cue in either lesion group.

Experimental medicine in drug addiction: towards behavioral, cognitive and neurobiological biomarkers

Several theoretical frameworks have been developed to understand putative processes and mechanisms involved in addiction. Whilst these 'theories of addiction' disagree about importance and/or nature of a number of key psychological processes (e.g. the necessity of craving and/or the involvement of drug-value representations), a number of commonalities exist. For instance, it is widely accepted that Pavlovian associations between cues and environmental contexts and the drug effects acquired over the course of addiction play a critical role, especially in relapse vulnerability in detoxified addicts. Additionally, all theories of addiction (explicitly or implicitly) propose that chronic drug exposure produces persistent neuroplastic changes in neurobiological circuitries underlying critical emotional, cognitive and motivational processes, although disagreement exists as to the precise nature of these neurobiological changes and/or their psychological consequences. The present review, rather than limiting itself to any particular theoretical stance, considers various candidate psychological, neurobiological and/or behavioral processes in addiction and outlines conceptual and procedural approaches for the experimental medicine laboratory. The review discusses (1) extinction, renewal and (re)consolidation of learned associations between cues and drugs, (2) the drug reward value, (3) motivational states contributing to drug seeking and (4) reflective (top-down) and sensory (bottom-up) driven decision-making. In evaluating these psychological and/or behavioral processes and their relationship to addiction we make reference to putative underlying brain structures identified by basic animal studies and/or imaging studies with humans.

Contribution of the amygdala, but not orbitofrontal or medial prefrontal cortices, to the expression of flavour preferences in marmoset monkeys

The development of food preferences contributes to a balanced diet, and involves both innate and learnt factors. By associating flavour cues with the reinforcing properties of the food (i.e. postingestive nutrient cues and innately preferred tastes, such as sweetness), animals acquire individual preferences. How the brain codes and guides selection when the subject has to choose between different palatable foods is little understood. To investigate this issue, we trained common marmoset monkeys (Callithrix jacchus) to respond to abstract visual patterns on a touch-sensitive computer screen to gain access to four different flavoured juices. After preferences were stable, animals received excitotoxic lesions of either the amygdala, the orbitofrontal cortex or the medial prefrontal cortex. Neither the orbitofrontal nor the medial prefrontal cortex lesions affected pre-surgery-expressed flavour preferences or the expression of preferences for novel flavours post-surgery. In contrast, amygdala lesions caused a shift in the preferences for juices expressed pre-surgery such that, post-surgery, juices were chosen according to their overall carbohydrate (simple sugars) content or 'sweetness'. Subsequent tests revealed that amygdala-lesioned animals only expressed juice preferences if they differed in 'sweetness'. Unlike controls, orbitofrontal cortex-lesioned and medial prefrontal cortex-lesioned animals, they were unable to display preferences between juices matched for 'sweetness' i.e. 5% sucrose solutions aromatised with different essential oils. The most parsimonious explanation is that the amygdala contributes to the expression of food preferences based on learnt cues but not those based on an innate preference for sweetness.

Modeling appetitive Pavlovian-instrumental interactions in mice

In appetitive Pavlovian associative learning, a stimulus (conditioned stimulus, CS) that has been associated with the delivery of a reinforcing event (unconditioned stimulus, US; e.g., food) can subsequently elicit or modulate goal-directed instrumental behaviors. For example, a Pavlovian CS can serve to reinforce (novel) instrumental behavior (conditioned reinforcement or CRf), or it can energize and potentiate ongoing instrumental responses when presented non-contingently (Pavlovian-instrumental transfer or PIT). Notably, these different effects of a Pavlovian CS on instrumental behavior are mediated by dissociable psychological and neurobiological mechanisms. Given the critical role that Pavlovian-instrumental interactions play in regulating motivated behavior and maladaptive manifestations of motivation such as eating disorders and addictions, understanding the underlying psychological and neurobiological mechanisms will be important. This unit describes behavioral protocols that produce robust and reliable PIT and CRf in mice and that open the door for future studies using transgenic approaches into the molecular mechanisms underlying associative learning and motivation.

Considering PTSD from the perspective of brain processes: a psychological construction approach

Posttraumatic stress disorder (PTSD) is a complex psychiatric disorder that involves symptoms from various domains that appear to be produced by the combination of several mechanisms. The authors contend that existing neural accounts fail to provide a viable model that explains the emergence and maintenance of PTSD and the associated heterogeneity in the expression of this disorder (cf. Garfinkel & Liberzon, 2009). They introduce a psychological construction approach as a novel framework to probe the brain basis of PTSD, where distributed networks within the human brain are thought to correspond to the basic psychological ingredients of the mind. The authors posit that it is the combination of these ingredients that produces the heterogeneous symptom clusters in PTSD. Their goal is show that a constructionist approach has significant heuristic value in understanding the emergence and maintenance of PTSD symptoms, and leads to different and perhaps more useful conjectures about the origins and maintenance of the syndrome than the traditional hyperreactive fear account.

Nucleus Accumbens Core and Shell are Necessary for Reinforcer Devaluation Effects on Pavlovian Conditioned Responding

The nucleus accumbens (NA) has been hypothesized to be part of a circuit in which cue-evoked information about expected outcomes is mobilized to guide behavior. Here we tested this hypothesis using a Pavlovian reinforcer devaluation task, previously applied to assess outcome-guided behavior after damage to regions such as the orbitofrontal cortex and amygdala that send projections to NA. Rats with sham lesions or neurotoxic lesions of either the core or shell subdivision of NA were trained to associate a 10-s CS+ with delivery of three food pellets. After training, half of the rats in each lesion group received food paired with illness induced by LiCl injections; the remaining rats received food and illness unpaired. Subsequently, responding to the CS+ was assessed in an extinction probe test. Both sham and lesioned rats conditioned to the CS+ and formed a conditioned taste aversion. However only sham rats reduced their conditioned responding as a result of reinforcer devaluation; devalued rats with lesions of either core or shell showed levels of responding that were similar to lesioned, non-devalued rats. This impairment was not due to the loss of motivational salience conferred to the CS+ in lesioned rats as both groups responded similarly for the cue in conditioned reinforcement testing. These data suggest that NA core and shell are part of a circuit necessary for the use of cue-evoked information about expected outcomes to guide behavior.

The mGluR5 antagonist MTEP dissociates the acquisition of predictive and incentive motivational properties of reward-paired stimuli in mice

An environmental stimulus paired with reward (a conditioned stimulus; CS) can acquire predictive properties that signal reward availability and may also acquire incentive motivational properties that enable the CS to influence appetitive behaviors. The neural mechanisms involved in the acquisition and expression of these CS properties are not fully understood. The metabotropic glutamate receptor, mGluR5, contributes to synaptic plasticity underlying learning and memory processes. We examined the role of mGluR5 in the acquisition and expression of learning that enables a CS to predict reward (goal-tracking) and acquire incentive properties (conditioned reinforcement). Mice were injected with vehicle or the mGluR5 antagonist, MTEP (3 or 10 mg/kg), before each Pavlovian conditioning session in which a stimulus (CS+) was paired with food delivery. Subsequently, in the absence of the primary food reward, we determined whether the CS+ could reinforce a novel instrumental response (conditioned reinforcement) and direct behavior toward the place of reward delivery (goal-tracking). MTEP did not affect performance during the conditioning phase, or the ability of the CS+ to elicit a goal-tracking response. In contrast, 10 mg/kg MTEP given before each conditioning session prevented the subsequent expression of conditioned reinforcement. This dose of MTEP did not affect conditioned reinforcement when administered before the test, in mice that had received vehicle before conditioning sessions. Thus, mGluR5 has a critical role in the acquisition of incentive properties by a CS, but is not required for the expression of incentive learning, or for the CS to acquire predictive properties that signal reward availability.

Role of the mesoamygdaloid dopamine projection in emotional learning

RATIONALE

Amygdala dopamine is crucially involved in the acquisition of Pavlovian associations, as measured via conditioned approach to the location of the unconditioned stimulus (US). However, learning begins before skeletomotor output, so this study assessed whether amygdala dopamine is also involved in earlier 'emotional' learning.

OBJECTIVES

A variant of the conditioned reinforcement (CR) procedure was validated where training was restricted to curtail the development of selective conditioned approach to the US location, and effects of amygdala dopamine manipulations before training or later CR testing assessed.

METHODS

Experiment 1a presented a light paired (CS+ group) or unpaired (CS- group) with a US. There were 1, 2 or 10 sessions, 4 trials per session. Then, the US was removed, and two novel levers presented. One lever (CR+) presented the light, and lever pressing was recorded. Experiment 1b also included a tone stimulus. Experiment 2 applied intra-amygdala R(+) 7-OH-DPAT (10 nmol/1.0 microl/side) before two training sessions (Experiment 2a) or a CR session (Experiment 2b).

RESULTS

For Experiments 1a and 1b, the CS+ group preferred the CR+ lever across all sessions. Conditioned alcove approach during 1 or 2 training sessions or associated CR tests was low and nonspecific. In Experiment 2a, R(+) 7-OH-DPAT before training greatly diminished lever pressing during a subsequent CR test, preferentially on the CR+ lever. For Experiment 2b, R(+) 7-OH-DPAT infusions before the CR test also reduced lever pressing.

CONCLUSIONS

Manipulations of amygdala dopamine impact the earliest stage of learning in which emotional reactions may be most prevalent.

The psychological and neurochemical mechanisms of drug memory reconsolidation: implications for the treatment of addiction

Memory reconsolidation is the process by which memories, destabilised at retrieval, require restabilisation to persist in the brain. It has been demonstrated that even old, well-established memories require reconsolidation following retrieval; therefore, memory reconsolidation could potentially be exploited to disrupt, or even erase, aberrant memories that underlie psychiatric disorders, thereby providing a novel therapeutic target. Drug addiction is one such disorder; it is both chronic and relapsing, and one prominent risk factor for a relapse episode is the presentation of environmental cues that have previously been associated with drugs of abuse. This 'cue-induced relapse' can be accounted for in psychological terms by reinforcing memories of the pavlovian association between the cue and the drug, which can thus influence behaviour through at least three psychologically and neurobiologically dissociable mechanisms: conditioned reinforcement, conditioned approach and conditioned motivation. As each of these psychological processes could contribute to the resumption of drug-seeking following abstinence, it is important to develop treatments that can reduce drug-seeking re-established via influences on each or all of these pavlovian processes, in order to minimise the risk of a subsequent relapse. Investigation of the memory reconsolidation mechanisms of the memories underlying conditioned reinforcement, conditioned approach and conditioned motivation indicate that they depend upon different neurochemical systems, including the glutamatergic and adrenergic systems within limbic corticostriatal circuitry. We also discuss here the subsequent translation to the clinic of this preclinical work.

Reward expectation alters learning and memory: the impact of the amygdala on appetitive-driven behaviors

The capacity to seek and obtain rewards is essential for survival. Pavlovian conditioning is one mechanism by which organisms develop predictions about rewards and such anticipatory or expectancy states enable successful behavioral adaptations to environmental demands. Reward expectancies have both affective/motivational and discriminative properties that allow for the modulation of instrumental goal-directed behavior. Recent data provide evidence that different cognitive strategies (cue-outcome associations) and neural systems (amygdala) are used when subjects are trained under conditions that allow Pavlovian-induced reward expectancies to guide instrumental behavioral choices. Furthermore, it has been demonstrated that impairments typically observed in a number of brain-damaged models are alleviated or eliminated by embedding unique reward expectancies into learning/memory tasks. These results suggest that Pavlovian-induced reward expectancies can change both behavioral and brain processes.