From prediction error to incentive salience: mesolimbic computation of reward motivation

Dopamine or opioid stimulation of nucleus accumbens similarly amplify cue-triggered 'wanting' for reward: entire core and medial shell mapped as substrates for PIT enhancement

Pavlovian cues [conditioned stimulus (CS+)] often trigger intense motivation to pursue and consume related reward [unconditioned stimulus (UCS)]. But cues do not always trigger the same intensity of motivation. Encountering a reward cue can be more tempting on some occasions than on others. What makes the same cue trigger more intense motivation to pursue reward on a particular encounter? The answer may be the level of incentive salience ('wanting') that is dynamically generated by mesocorticolimbic brain systems, influenced especially by dopamine and opioid neurotransmission in the nucleus accumbens (NAc) at that moment. We tested the ability of dopamine stimulation (by amphetamine microinjection) vs. mu opioid stimulation [by d-Ala, nMe-Phe, Glyol-enkephalin (DAMGO) microinjection] of either the core or shell of the NAc to amplify cue-triggered levels of motivation to pursue sucrose reward, measured with a Pavlovian-Instrumental Transfer (PIT) procedure, a relatively pure assay of incentive salience. Cue-triggered 'wanting' in PIT was enhanced by amphetamine or DAMGO microinjections equally, and also equally at nearly all sites throughout the entire core and medial shell (except for a small far-rostral strip of shell). NAc dopamine/opioid stimulations specifically enhanced CS+ ability to trigger phasic peaks of 'wanting' to obtain UCS, without altering baseline efforts when CS+ was absent. We conclude that dopamine/opioid stimulation throughout nearly the entire NAc can causally amplify the reactivity of mesocorticolimbic circuits, and so magnify incentive salience or phasic UCS 'wanting' peaks triggered by a CS+. Mesolimbic amplification of incentive salience may explain why a particular cue encounter can become irresistibly tempting, even when previous encounters were successfully resisted before.

Presynaptic leptin action suppresses excitatory synaptic transmission onto ventral tegmental area dopamine neurons

BACKGROUND

Leptin is an adipocyte-derived cytokine that can act in the brain to suppress feeding and maintain energy homeostasis. Additionally, leptin activates its receptors in the ventral tegmental area (VTA), a critical site for neuroadaptations to rewarding stimuli, to modulate reward-seeking behaviors. Although leptin can decrease intrinsic excitability of dopamine neurons in the VTA, it is unknown whether leptin can modulate excitatory synaptic transmission in this brain region. Because plasticity of glutamatergic synapses onto VTA neurons can encode predictive information about reward, we hypothesized that leptin can decrease excitatory synaptic transmission onto dopamine neurons.

METHODS

Using whole-cell patch clamp electrophysiology in mouse midbrain slices, we tested the effects of leptin on evoked α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptor (AMPAR) or N-methyl-D-aspartate receptor (NMDAR)-mediated excitatory postsynaptic currents (EPSCs) onto VTA dopamine neurons.

RESULTS

Leptin depressed both AMPAR and NMDAR EPSCs in VTA dopamine neurons and reduced frequency but not amplitude of mini EPSCs. Bath application of the MEK1/2 inhibitor U0126 did not alter leptin-induced suppression of AMPAR EPSCs. However, external, but not internal, application of the phosphoinositol 3-kinase (PI3K) or Janus kinase 2 (Jak2) tyrosine kinase inhibitors abolished leptin-induced synaptic depression.

CONCLUSIONS

This study demonstrates that leptin causes a presynaptic inhibition of the probability of glutamate release onto VTA dopamine neurons. This synaptic inhibition requires Jak2 and PI3K activation. Leptin-induced weakening of synaptic strength onto dopamine cells may underlie its inhibitory effects on appetitive behavior for rewarding stimuli.

Substance-specific environmental influences on drug use and drug preference in animals and humans

Epidemiological, clinical, and preclinical evidence indicate that the setting of drug use can exert a powerful modulatory influence on drug reward and that this influence is substance-specific. When heroin and cocaine co-abusers, for example, report on the circumstances of drug use, they indicate distinct settings for the two drugs: heroin being used preferentially at home and cocaine being used preferentially outside the home. Similar results were obtained in laboratory rats. These findings will be interpreted in the light of a novel model of drug reward, based on the emotional appraisal of central and peripheral drug effects as a function of environmental context. I argue here that drug addiction research has not paid sufficient attention to the substance-specific aspects of drug abuse and this may have contributed to the present dearth of effective treatments. Pharmacological and cognitive-behavioral therapy, for example, should be tailored so as to allow the addict to anticipate, and cope with, the risks associated, in a substance-specific manner, to the different settings of drug use.

A value-driven mechanism of attentional selection

Attention selects stimuli for cognitive processing, and the mechanisms that underlie the process of attentional selection have been a major topic of psychological research for over 30 years. From this research, it has been well documented that attentional selection can proceed both voluntarily, driven by visual search goals, and involuntarily, driven by the physical salience of stimuli. In this review, I provide a conceptual framework for attentional control that emphasizes the need for stimulus selection to promote the survival and wellbeing of an organism. I argue that although goal-driven and salience-driven mechanisms of attentional selection fit within this framework, a central component that is missing is a mechanism of attentional selection that is uniquely driven by learned associations between stimuli and rewards. I go on to review recent evidence for such a value-driven mechanism of attentional selection, and describe how this mechanism functions independently of the well-documented salience-driven and goal-driven mechanisms. I conclude by arguing that reward learning modifies the attentional priority of stimuli, allowing them to compete more effectively for selection even when nonsalient and task-irrelevant.

Prediction error in reinforcement learning: a meta-analysis of neuroimaging studies

Activation likelihood estimation (ALE) meta-analyses were used to examine the neural correlates of prediction error in reinforcement learning. The findings are interpreted in the light of current computational models of learning and action selection. In this context, particular consideration is given to the comparison of activation patterns from studies using instrumental and Pavlovian conditioning, and where reinforcement involved rewarding or punishing feedback. The striatum was the key brain area encoding for prediction error, with activity encompassing dorsal and ventral regions for instrumental and Pavlovian reinforcement alike, a finding which challenges the functional separation of the striatum into a dorsal 'actor' and a ventral 'critic'. Prediction error activity was further observed in diverse areas of predominantly anterior cerebral cortex including medial prefrontal cortex and anterior cingulate cortex. Distinct patterns of prediction error activity were found for studies using rewarding and aversive reinforcers; reward prediction errors were observed primarily in the striatum while aversive prediction errors were found more widely including insula and habenula.

A classically conditioned cocaine cue acquires greater control over motivated behavior in rats prone to attribute incentive salience to a food cue

RATIONALE

Cues associated with rewards bias attention towards them and can motivate drug-seeking and drug-taking behavior. There is, however, considerable individual variation in the extent to which cues associated with rewards acquire motivational properties. For example, only in some rats does a localizable food cue become attractive, eliciting approach towards it, and "wanted", in the sense that it serves as an effective conditioned reinforcer.

OBJECTIVES

We asked whether the propensity of animals to attribute incentive salience to a food cue predicts the extent to which a classically conditioned cocaine cue acquires incentive motivational properties.

METHODS

First, a Pavlovian conditioned approach procedure was used to identify rats prone to attribute incentive salience to a food cue. We then measured the extent to which a classically conditioned cocaine cue acquired two properties of an incentive stimulus: (1) the ability to elicit approach towards it, and (2) the ability to reinstate drug-seeking behavior, using an extinction-reinstatement procedure (i.e., to act as a conditioned reinforcer).

RESULTS

We found that a classically conditioned cocaine cue became more attractive, in that it elicited greater approach toward it, and more desired, in that it supported more drug-seeking behavior under extinction conditions, in individuals prone to attribute incentive salience to a food cue.

CONCLUSIONS

We conclude that rats vary in their propensity to attribute incentive salience to both food and cocaine cues, and it is possible to predict, prior to any drug experience, in which rats a cocaine cue will acquire the strongest motivational control over behavior.

Beyond simple reinforcement learning: the computational neurobiology of reward-learning and valuation

Neural computational accounts of reward-learning have been dominated by the hypothesis that dopamine neurons behave like a reward-prediction error and thus facilitate reinforcement learning in striatal target neurons. While this framework is consistent with a lot of behavioral and neural evidence, this theory fails to account for a number of behavioral and neurobiological observations. In this special issue of EJN we feature a combination of theoretical and experimental papers highlighting some of the explanatory challenges faced by simple reinforcement-learning models and describing some of the ways in which the framework is being extended in order to address these challenges.

Instant transformation of learned repulsion into motivational "wanting"

BACKGROUND

Learned cues for pleasant reward often elicit desire, which, in addicts, may become compulsive. According to the dominant view in addiction neuroscience and reinforcement modeling, such desires are the simple products of learning, coming from a past association with reward outcome.

RESULTS

We demonstrate that cravings are more than merely the products of accumulated pleasure memories-even a repulsive learned cue for unpleasantness can become suddenly desired via the activation of mesocorticolimbic circuitry. Rats learned repulsion toward a Pavlovian cue (a briefly-inserted metal lever) that always predicted an unpleasant Dead Sea saltiness sensation. Yet, upon first reencounter in a novel sodium-depletion state to promote mesocorticolimbic reactivity (reflected by elevated Fos activation in ventral tegmentum, nucleus accumbens, ventral pallidum, and the orbitofrontal prefrontal cortex), the learned cue was instantly transformed into an attractive and powerful motivational magnet. Rats jumped and gnawed on the suddenly attractive Pavlovian lever cue, despite never having tasted intense saltiness as anything other than disgusting.

CONCLUSIONS

Instant desire transformation of a learned cue contradicts views that Pavlovian desires are essentially based on previously learned values (e.g., prediction error or temporal difference models). Instead desire is recomputed at reencounter by integrating Pavlovian information with the current brain/physiological state. This powerful brain transformation reverses strong learned revulsion into avid attraction. When applied to addiction, related mesocorticolimbic transformations (e.g., drugs or neural sensitization) of cues for already-pleasant drug experiences could create even more intense cravings. This cue/state transformation helps define what it means to say that addiction hijacks brain limbic circuits of natural reward.

Neural response to visual sexual cues in dopamine treatment-linked hypersexuality in Parkinson's disease

Hypersexuality with compulsive sexual behaviour is a significant source of morbidity for patients with Parkinson's disease receiving dopamine replacement therapies. We know relatively little about the pathophysiology of hypersexuality in Parkinson's disease, and it is unknown how visual sexual stimuli, similar to the portrayals of sexuality in the mainstream mass media may affect the brain and behaviour in such susceptible individuals. Here, we have studied a group of 12 patients with Parkinson's disease with hypersexuality using a functional magnetic resonance imaging block design exposing participants to both sexual, other reward-related and neutral visual cues. We hypothesized that exposure to visual sexual cues would trigger increased sexual desire in patients with Parkinson's disease with hypersexuality that would correspond to changes in brain activity in regions linked to dopaminergically stimulated sexual motivation. Patients with Parkinson's disease with hypersexuality were scanned ON and OFF dopamine drugs, and their results were compared with a group of 12 Parkinson's disease control patients without hypersexuality or other impulse control disorders. Exposure to sexual cues significantly increased sexual desire and hedonic responses in the Parkinson's disease hypersexuality group compared with the Parkinson's disease control patients. These behavioural changes corresponded to significant blood oxygen level-dependent signal changes in regions within limbic, paralimbic, temporal, occipital, somatosensory and prefrontal cortices that correspond to emotional, cognitive, autonomic, visual and motivational processes. The functional imaging data showed that the hypersexuality patients' increased sexual desire correlated with enhanced activations in the ventral striatum, and cingulate and orbitofrontal cortices. When the patients with Parkinson's disease with hypersexuality were OFF medication, the functional imaging data showed decreases in activation during the presentation of sexual cues relative to rest. These deactivations were not observed when the patients were ON medication, suggesting that dopamine drugs may release inhibition within local neuronal circuits in the cerebral cortex that may contribute to compulsive sexual behaviour. The findings of this study have implications with respect to the potential influence of cue exposure via exposure to mass media in enhancing libido, which in this group of vulnerable patients can lead to devastating social consequences and occasionally, custodial sentences. Stimulation through exposure to sexual visual cues in patients with Parkinson's disease with hypersexuality provides a motivational impetus for seeking this reward behaviour through activations and deactivations of cerebral cortex.

Neuroscience of affect: brain mechanisms of pleasure and displeasure

Affective neuroscience aims to understand how affect (pleasure or displeasure) is created by brains. Progress is aided by recognizing that affect has both objective and subjective features. Those dual aspects reflect that affective reactions are generated by neural mechanisms, selected in evolution based on their real (objective) consequences for genetic fitness. We review evidence for neural representation of pleasure in the brain (gained largely from neuroimaging studies), and evidence for the causal generation of pleasure (gained largely from brain manipulation studies). We suggest that representation and causation may actually reflect somewhat separable neuropsychological functions. Representation reaches an apex in limbic regions of prefrontal cortex, especially orbitofrontal cortex, influencing decisions and affective regulation. Causation of core pleasure or 'liking' reactions is much more subcortically weighted, and sometimes surprisingly localized. Pleasure 'liking' is especially generated by restricted hedonic hotspot circuits in nucleus accumbens (NAc) and ventral pallidum. Another example of localized valence generation, beyond hedonic hotspots, is an affective keyboard mechanism in NAc for releasing intense motivations such as either positively valenced desire and/or negatively valenced dread.

Persistence of value-driven attentional capture

Stimuli that have previously been associated with the delivery of reward involuntarily capture attention when presented as unrewarded and task-irrelevant distractors in a subsequent visual search task. It is unknown how long such effects of reward learning on attention persist. One possibility is that value-driven attentional biases are plastic and constantly evolve to reflect only recent reward history. According to such a mechanism of attentional control, only consistently reinforced patterns of attention allocation persist for extended periods of time. Another possibility is that reward learning creates enduring changes in attentional priority that can persist indefinitely without further learning. Here we provide evidence for an enduring effect of reward learning on attentional priority: stimuli previously associated with reward in a training phase capture attention when presented as irrelevant distractors over half a year later, without the need for further reward learning.

Reward uncertainty enhances incentive salience attribution as sign-tracking

Conditioned stimuli (CSs) come to act as motivational magnets following repeated association with unconditioned stimuli (UCSs) such as sucrose rewards. By traditional views, the more reliably predictive a Pavlovian CS-UCS association, the more the CS becomes attractive. However, in some cases, less predictability might equal more motivation. Here we examined the effect of introducing uncertainty in CS-UCS association on CS strength as an attractive motivation magnet. In the present study, Experiment 1 assessed the effects of Pavlovian predictability versus uncertainty about reward probability and/or reward magnitude on the acquisition and expression of sign-tracking (ST) and goal-tracking (GT) responses in an autoshaping procedure. Results suggested that uncertainty produced strongest incentive salience expressed as sign-tracking. Experiment 2 examined whether a within-individual temporal shift from certainty to uncertainty conditions could produce a stronger CS motivational magnet when uncertainty began, and found that sign-tracking still increased after the shift. Overall, our results support earlier reports that ST responses become more pronounced in the presence of uncertainty regarding CS-UCS associations, especially when uncertainty combines both probability and magnitude. These results suggest that Pavlovian uncertainty, although diluting predictability, is still able to enhance the incentive motivational power of particular CSs.

Conscious motor intention emerges in the inferior parietal lobule

Willed actions are characterized by the subjective experience of 'conscious intention'. During the last decade the neural bases of this experience have been widely investigated. Previous findings have suggested that conscious intention emerges in the mesial precentral area (MPA) including the supplementary and pre-supplementary motor area. However, this assumption was later challenged by evidence indicating a key contribution of the inferior parietal lobule (IPL) in this process. In this review we show that this second hypothesis is the most plausible. We provide behavioral, clinical and electrophysiological evidence that IPL, rather than MPA, mediates the early subjective experience of 'wanting to move'. We argue that MPA generates a feeling of 'motor urge', near movement onset, when the inhibitory control exerted on the low-level motor areas is released. We propose that goal directed actions trigger two forms of motor intention: the first, 'wanting to move' intention is under the control of the inferior parietal regions and specifies a general goal to be reached before movement planning; the second, the 'urge to move' intention, is controlled by mesial precentral areas and signals the time when the planned movement is about to start. The time locked activation of these two forms of intention is at the basis of movement control.

The neuroscience of learning: beyond the Hebbian synapse

From the traditional perspective of associative learning theory, the hypothesis linking modifications of synaptic transmission to learning and memory is plausible. It is less so from an information-processing perspective, in which learning is mediated by computations that make implicit commitments to physical and mathematical principles governing the domains where domain-specific cognitive mechanisms operate. We compare the properties of associative learning and memory to the properties of long-term potentiation, concluding that the properties of the latter do not explain the fundamental properties of the former. We briefly review the neuroscience of reinforcement learning, emphasizing the representational implications of the neuroscientific findings. We then review more extensively findings that confirm the existence of complex computations in three information-processing domains: probabilistic inference, the representation of uncertainty, and the representation of space. We argue for a change in the conceptual framework within which neuroscientists approach the study of learning mechanisms in the brain.

The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses

The role of dopamine in reward is a topic of debate. For example, some have argued that phasic dopamine signaling provides a prediction-error signal necessary for stimulus-reward learning, whereas others have hypothesized that dopamine is not necessary for learning per se, but for attributing incentive motivational value ('incentive salience') to reward cues. These psychological processes are difficult to tease apart, because they tend to change together. To disentangle them we took advantage of natural individual variation in the extent to which reward cues are attributed with incentive salience, and asked whether dopamine (specifically in the core of the nucleus accumbens) is necessary for the expression of two forms of pavlovian-conditioned approach behavior--one in which the cue acquires powerful motivational properties (sign-tracking) and another closely related one in which it does not (goal-tracking). After acquisition of these conditioned responses (CRs), intra-accumbens injection of the dopamine receptor antagonist flupenthixol markedly impaired the expression of a sign-tracking CR, but not a goal-tracking CR. Furthermore, dopamine antagonism did not produce a gradual extinction-like decline in behavior, but maximally impaired expression of a sign-tracking CR on the very first trial, indicating the effect was not due to new learning (i.e. it occurred in the absence of new prediction-error computations). The data support the view that dopamine in the accumbens core is not necessary for learning stimulus-reward associations, but for attributing incentive salience to reward cues, transforming predictive conditional stimuli into incentive stimuli with powerful motivational properties.

Quantifying individual variation in the propensity to attribute incentive salience to reward cues

If reward-associated cues acquire the properties of incentive stimuli they can come to powerfully control behavior, and potentially promote maladaptive behavior. Pavlovian incentive stimuli are defined as stimuli that have three fundamental properties: they are attractive, they are themselves desired, and they can spur instrumental actions. We have found, however, that there is considerable individual variation in the extent to which animals attribute Pavlovian incentive motivational properties ("incentive salience") to reward cues. The purpose of this paper was to develop criteria for identifying and classifying individuals based on their propensity to attribute incentive salience to reward cues. To do this, we conducted a meta-analysis of a large sample of rats (N = 1,878) subjected to a classic Pavlovian conditioning procedure. We then used the propensity of animals to approach a cue predictive of reward (one index of the extent to which the cue was attributed with incentive salience), to characterize two behavioral phenotypes in this population: animals that approached the cue ("sign-trackers") vs. others that approached the location of reward delivery ("goal-trackers"). This variation in Pavlovian approach behavior predicted other behavioral indices of the propensity to attribute incentive salience to reward cues. Thus, the procedures reported here should be useful for making comparisons across studies and for assessing individual variation in incentive salience attribution in small samples of the population, or even for classifying single animals.

An imaging genetics approach to understanding social influence

Normative social influences shape nearly every aspect of our lives, yet the biological processes mediating the impact of these social influences on behavior remain incompletely understood. In this Hypothesis, we outline a theoretical framework and an integrative research approach to the study of social influences on the brain and genetic moderators of such effects. First, we review neuroimaging evidence linking social influence and conformity to the brain's reward system. We next review neuroimaging evidence linking social punishment (exclusion) to brain systems involved in the experience of pain, as well as evidence linking exclusion to conformity. We suggest that genetic variants that increase sensitivity to social cues may predispose individuals to be more sensitive to either social rewards or punishments (or potentially both), which in turn increases conformity and susceptibility to normative social influences more broadly. To this end, we review evidence for genetic moderators of neurochemical responses in the brain, and suggest ways in which genes and pharmacology may modulate sensitivity to social influences. We conclude by proposing an integrative imaging genetics approach to the study of brain mediators and genetic modulators of a variety of social influences on human attitudes, beliefs, and actions.

The human sexual response cycle: brain imaging evidence linking sex to other pleasures

Sexual behavior is critical to species survival, yet comparatively little is known about the neural mechanisms in the human brain. Here we systematically review the existing human brain imaging literature on sexual behavior and show that the functional neuroanatomy of sexual behavior is comparable to that involved in processing other rewarding stimuli. Sexual behavior clearly follows the established principles and phases for wanting, liking and satiety involved in the pleasure cycle of other rewards. The studies have uncovered the brain networks involved in sexual wanting or motivation/anticipation, as well as sexual liking or arousal/consummation, while there is very little data on sexual satiety or post-orgasmic refractory period. Human sexual behavior also interacts with other pleasures, most notably social interaction and high arousal states. We discuss the changes in the underlying brain networks supporting sexual behavior in the context of the pleasure cycle, the changes to this cycle over the individual's life-time and the interactions between them. Overall, it is clear from the data that the functional neuroanatomy of sex is very similar to that of other pleasures and that it is unlikely that there is anything special about the brain mechanisms and networks underlying sex.