Instant transformation of learned repulsion into motivational "wanting"

Treatment for Anhedonia: A Neuroscience Driven Approach

Anhedonia, or loss of interest or pleasure in usual activities, is characteristic of depression, some types of anxiety, as well as substance abuse and schizophrenia. Anhedonia is a predictor of poor long-term outcomes, including suicide, and poor treatment response. Because extant psychological and pharmacological treatments are relatively ineffective for anhedonia, there is an unmet therapeutic need for this high-risk symptom. Current psychological and drug treatments for anxiety and depression focus largely on reducing excesses in negative affect rather than improving deficits in positive affect. Recent advances in affective neuroscience posit that anhedonia is associated with deficits in the appetitive reward system, specifically the anticipation, consumption, and learning of reward. In this paper, we review the evidence for positive affect as a symptom cluster, and its neural underpinnings, and introduce a novel psychological treatment for anxiety and depression that targets appetitive responding. First, we review anhedonia in relation to positive and negative valence systems and current treatment approaches. Second, we discuss the evidence linking anhedonia to biological, experiential, and behavioral deficits in the reward subsystems. Third, we describe the therapeutic approach for Positive Affect Treatment (PAT), an intervention designed to specifically target deficits in reward sensitivity.

Input-specific contributions to valence processing in the amygdala

Reward and punishment are often thought of as opposing processes: rewards and the environmental cues that predict them elicit approach and consummatory behaviors, while punishments drive aversion and avoidance behaviors. This framework suggests that there may be segregated brain circuits for these valenced behaviors. The basolateral amygdala (BLA) is one brain region that contributes to both types of motivated behavior. Individual neurons in the BLA can favor positive over negative valence, or vice versa, but these neurons are intermingled, showing no anatomical segregation. The amygdala receives inputs from many brain areas and current theories posit that encoding of positive versus negative valence by BLA neurons is determined by the wiring of each neuron. Specifically, many projections from other brain areas that respond to positive and negative valence stimuli and predictive cues project strongly to the BLA and likely contribute to valence processing within the BLA. Here we review three of these areas, the basal forebrain, the dorsal raphe nucleus and the ventral tegmental area, and discuss how these may promote encoding of positive and negative valence within the BLA.

Toward isolating the role of dopamine in the acquisition of incentive salience attribution

Stimulus-reward learning has been heavily linked to the reward-prediction error learning hypothesis and dopaminergic function. However, some evidence suggests dopaminergic function may not strictly underlie reward-prediction error learning, but may be specific to incentive salience attribution. Utilizing a Pavlovian conditioned approach procedure consisting of two stimuli that were equally reward-predictive (both undergoing reward-prediction error learning) but functionally distinct in regard to incentive salience (levers that elicited sign-tracking and tones that elicited goal-tracking), we tested the differential role of D1 and D2 dopamine receptors and nucleus accumbens dopamine in the acquisition of sign- and goal-tracking behavior and their associated conditioned reinforcing value within individuals. Overall, the results revealed that both D1 and D2 inhibition disrupted performance of sign- and goal-tracking. However, D1 inhibition specifically prevented the acquisition of sign-tracking to a lever, instead promoting goal-tracking and decreasing its conditioned reinforcing value, while neither D1 nor D2 signaling was required for goal-tracking in response to a tone. Likewise, nucleus accumbens dopaminergic lesions disrupted acquisition of sign-tracking to a lever, while leaving goal-tracking in response to a tone unaffected. Collectively, these results are the first evidence of an intraindividual dissociation of dopaminergic function in incentive salience attribution from reward-prediction error learning, indicating that incentive salience, reward-prediction error, and their associated dopaminergic signaling exist within individuals and are stimulus-specific. Thus, individual differences in incentive salience attribution may be reflective of a differential balance in dopaminergic function that may bias toward the attribution of incentive salience, relative to reward-prediction error learning only.

Dorsolateral neostriatum contribution to incentive salience: opioid or dopamine stimulation makes one reward cue more motivationally attractive than another

Pavlovian cues for rewards can become attractive incentives: approached and 'wanted' as the rewards themselves. The motivational attractiveness of a previously learned cue is not fixed, but can be dynamically amplified during re-encounter by simultaneous activation of brain limbic circuitry. Here it was reported that opioid or dopamine microinjections in the dorsolateral quadrant of the neostriatum (DLS) of rats selectively amplify attraction toward a previously learned Pavlovian cue in an individualized fashion, at the expense of a competing cue. In an autoshaping (sign-tracking vs. goal-tracking) paradigm, microinjection of the mu opioid receptor agonist (DAMGO) or dopamine indirect agonist (amphetamine) in the DLS of sign-tracker individuals selectively enhanced their sign-tracking attraction toward the reward-predictive lever cue. By contrast, DAMGO or amphetamine in the DLS of goal-trackers selectively enhanced prepotent attraction toward the reward-proximal cue of sucrose dish. Amphetamine also enhanced goal-tracking in some sign-tracker individuals (if they ever defected to the dish even once). That DLS enhancement of cue attraction was due to stronger motivation, not stronger habits, was suggested by: (i) sign-trackers flexibly followed their cue to a new location when the lever was suddenly moved after DLS DAMGO microinjection; and (ii) DAMGO in the DLS also made sign-trackers work harder on a new instrumental nose-poke response required to earn presentations of their Pavlovian lever cue (instrumental conditioned reinforcement). Altogether, the current results suggest that DLS circuitry can enhance the incentive salience of a Pavlovian reward cue, selectively making that cue a stronger motivational magnet.

Two types of psychological hedonism

I develop a distinction between two types of psychological hedonism. Inferential hedonism (or "I-hedonism") holds that each person only has ultimate desires regarding his or her own hedonic states (pleasure and pain). Reinforcement hedonism (or "R-hedonism") holds that each person's ultimate desires, whatever their contents are, are differentially reinforced in that person's cognitive system only by virtue of their association with hedonic states. I'll argue that accepting R-hedonism and rejecting I-hedonism provides a conciliatory position on the traditional altruism debate, and that it coheres well with the neuroscientist Anthony Dickinson's theory about the evolutionary function of hedonic states, the "hedonic interface theory." Finally, I'll defend R-hedonism from potential objections.

An omission procedure reorganizes the microstructure of sign-tracking while preserving incentive salience

Appetitive sign-tracking, in which reward-paired cues elicit approach that can result in cue interaction, demonstrates how cues acquire motivational value. For example, rats will approach and subsequently interact with a lever insertion cue that signals food delivery upon its retraction. However, lever deflections are rapidly reduced once rats are trained on an omission schedule in which lever interactions cancel food delivery. Here we evaluated the change in sign-tracking response topography in rats exposed to such an omission procedure. Lever deflections dropped precipitously when they canceled reward. However, rats that were on an omission schedule continued to approach, sniff, and contact the lever without pressing it, and did so at comparable rates to rats that were not under an omission schedule. Thus, sign-tracking was maintained, albeit in a different manner, following omission. Such findings show that the motivational attraction to reward cues can be expressed with remarkable persistence and flexibility.

Cocaine can generate a stronger conditioned reinforcer than food despite being a weaker primary reinforcer

The present study aimed to test the hypothesis that cues associated with drug-taking behavior become extra strong motivators of behavior compared with cues paired with non-drug reinforcers. In experiment 1, rats were trained to lever press for intravenous cocaine infusions and grain pellets. Each reinforcer was paired with a distinct audiovisual cue. When allowed to choose between these alternatives, rats chose grain on ~70-80 percent of trials. However, after extinguishing lever pressing, reintroduction of press-contingent cues during a test for cue-induced reinstatement generated more cocaine seeking than grain seeking (also observed on 3- and 8-week follow-up tests). To examine whether the same pattern of results would occur with two non-drug reinforcers, experiment 2 replicated experiment 1 using grain and sucrose as reinforcement alternatives. Rats chose sucrose over grain on ~70-80 percent of choice trials and also responded more for the sucrose cue than for the grain cue on the reinstatement test. The disconnect between primary and conditioned reinforcements in experiment 1 but not in experiment 2 suggests that drug cues may become exceptionally strong motivators of drug seeking. These results are consistent with cue-focused theories of addiction and may offer insight into the persistent cue-driven drug-seeking behavior observed in addiction.

Remembering Components of Food in Drosophila

Remembering features of past feeding experience can refine foraging and food choice. Insects can learn to associate sensory cues with components of food, such as sugars, amino acids, water, salt, alcohol, toxins and pathogens. In the fruit fly Drosophila some food components activate unique subsets of dopaminergic neurons (DANs) that innervate distinct functional zones on the mushroom bodies (MBs). This architecture suggests that the overall dopaminergic neuron population could provide a potential cellular substrate through which the fly might learn to value a variety of food components. In addition, such an arrangement predicts that individual component memories reside in unique locations. DANs are also critical for food memory consolidation and deprivation-state dependent motivational control of the expression of food-relevant memories. Here, we review our current knowledge of how nutrient-specific memories are formed, consolidated and specifically retrieved in insects, with a particular emphasis on Drosophila.

Physiological state gates acquisition and expression of mesolimbic reward prediction signals

Phasic dopamine signaling participates in associative learning by reinforcing associations between outcomes (unconditioned stimulus; US) and their predictors (conditioned stimulus; CS). However, prior work has always engendered these associations with innately rewarding stimuli. Thus, whether dopamine neurons can acquire prediction signals in the absence of appetitive experience and update them when the value of the outcome changes remains unknown. Here, we used sodium depletion to reversibly manipulate the appetitive value of a hypertonic sodium solution while measuring phasic dopamine signaling in rat nucleus accumbens. Dopamine responses to the NaCl US following sodium depletion updated independent of prior experience. In contrast, prediction signals were only acquired through extensive experience with a US that had positive affective value. Once learned, dopamine prediction signals were flexibly expressed in a state-dependent manner. Our results reveal striking differences with respect to how physiological state shapes dopamine signals evoked by outcomes and their predictors.

Developmental Cognitive Neuroscience of Adolescent Sexual Risk and Alcohol Use

Human adolescents engage in very high rates of unprotected sex. This behavior has a high potential for unintended, serious, and sustained health consequences including HIV/AIDS. Despite these serious health consequences, we know little about the neural and cognitive factors that influence adolescents' decision-making around sex, and their potential overlap with behaviorally co-occurring risk behaviors, including alcohol use. Thus, in this review, we evaluate the developmental neuroscience of sexual risk and alcohol use for human adolescents with an eye to relevant prevention and intervention implications.

Thirst and the state-dependent representation of incentive stimulus value in human motive circuitry

Depletion imposes both need and desire to drink, and potentiates the response to need-relevant cues in the environment. The present fMRI study aimed to determine which neural structures selectively increase the incentive value of need-relevant stimuli in a thirst state. Towards this end, participants were scanned twice--either in a thirst or no-thirst state--while viewing pictures of beverages and chairs. As expected, thirst led to a selective increase in self-reported pleasantness and arousal by beverages. Increased responses to beverage when compared with chair stimuli were observed in the cingulate cortex, insular cortex and the amygdala in the thirst state, which were absent in the no-thirst condition. Enhancing the incentive value of need-relevant cues in a thirst state is a key mechanism for motivating drinking behavior. Overall, distributed regions of the motive circuitry, which are also implicated in salience processing, craving and interoception, provide a dynamic body-state dependent representation of stimulus value.

The attention habit: how reward learning shapes attentional selection

There is growing consensus that reward plays an important role in the control of attention. Until recently, reward was thought to influence attention indirectly by modulating task-specific motivation and its effects on voluntary control over selection. Such an account was consistent with the goal-directed (endogenous) versus stimulus-driven (exogenous) framework that had long dominated the field of attention research. Now, a different perspective is emerging. Demonstrations that previously reward-associated stimuli can automatically capture attention even when physically inconspicuous and task-irrelevant challenge previously held assumptions about attentional control. The idea that attentional selection can be value driven, reflecting a distinct and previously unrecognized control mechanism, has gained traction. Since these early demonstrations, the influence of reward learning on attention has rapidly become an area of intense investigation, sparking many new insights. The result is an emerging picture of how the reward system of the brain automatically biases information processing. Here, I review the progress that has been made in this area, synthesizing a wealth of recent evidence to provide an integrated, up-to-date account of value-driven attention and some of its broader implications.

The Origins and Organization of Vertebrate Pavlovian Conditioning

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

Neuroscience of Internet Pornography Addiction: A Review and Update

Many recognize that several behaviors potentially affecting the reward circuitry in human brains lead to a loss of control and other symptoms of addiction in at least some individuals. Regarding Internet addiction, neuroscientific research supports the assumption that underlying neural processes are similar to substance addiction. The American Psychiatric Association (APA) has recognized one such Internet related behavior, Internet gaming, as a potential addictive disorder warranting further study, in the 2013 revision of their Diagnostic and Statistical Manual. Other Internet related behaviors, e.g., Internet pornography use, were not covered. Within this review, we give a summary of the concepts proposed underlying addiction and give an overview about neuroscientific studies on Internet addiction and Internet gaming disorder. Moreover, we reviewed available neuroscientific literature on Internet pornography addiction and connect the results to the addiction model. The review leads to the conclusion that Internet pornography addiction fits into the addiction framework and shares similar basic mechanisms with substance addiction. Together with studies on Internet addiction and Internet Gaming Disorder we see strong evidence for considering addictive Internet behaviors as behavioral addiction. Future research needs to address whether or not there are specific differences between substance and behavioral addiction.

Components and characteristics of the dopamine reward utility signal

Rewards are defined by their behavioral functions in learning (positive reinforcement), approach behavior, economic choices, and emotions. Dopamine neurons respond to rewards with two components, similar to higher order sensory and cognitive neurons. The initial, rapid, unselective dopamine detection component reports all salient environmental events irrespective of their reward association. It is highly sensitive to factors related to reward and thus detects a maximal number of potential rewards. It also senses aversive stimuli but reports their physical impact rather than their aversiveness. The second response component processes reward value accurately and starts early enough to prevent confusion with unrewarded stimuli and objects. It codes reward value as a numeric, quantitative utility prediction error, consistent with formal concepts of economic decision theory. Thus, the dopamine reward signal is fast, highly sensitive and appropriate for driving and updating economic decisions.

Obesity and the neurocognitive basis of food reward and the control of intake

With the rising prevalence of obesity, hedonic eating has become an important theme in obesity research. Hedonic eating is thought to be that driven by the reward of food consumption and not metabolic need, and this has focused attention on the brain reward system and how its dysregulation may cause overeating and obesity. Here, we begin by examining the brain reward system and the evidence for its dysregulation in human obesity. We then consider the issue of how individuals are able to control their hedonic eating in the present obesogenic environment and compare 2 contrasting perspectives on the control of hedonic eating, specifically, enhanced control of intake via higher cognitive control and loss of control over intake as captured by the food addiction model. We conclude by considering what these perspectives offer in terms of directions for future research and for potential interventions to improve control over food intake at the population and the individual levels.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

The algorithmic anatomy of model-based evaluation

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

Lateral hypothalamus, nucleus accumbens, and ventral pallidum roles in eating and hunger: interactions between homeostatic and reward circuitry

The study of the neural bases of eating behavior, hunger, and reward has consistently implicated the lateral hypothalamus (LH) and its interactions with mesocorticolimbic circuitry, such as mesolimbic dopamine projections to nucleus accumbens (NAc) and ventral pallidum (VP), in controlling motivation to eat. The NAc and VP play special roles in mediating the hedonic impact (“liking”) and motivational incentive salience (“wanting”) of food rewards, and their interactions with LH help permit regulatory hunger/satiety modulation of food motivation and reward. Here, we review some progress that has been made regarding this circuitry and its functions: the identification of localized anatomical hedonic hotspots within NAc and VP for enhancing hedonic impact; interactions of NAc/VP hedonic hotspots with specific LH signals such as orexin; an anterior-posterior gradient of sites in NAc shell for producing intense appetitive eating vs. intense fearful reactions; and anatomically distributed appetitive functions of dopamine and mu opioid signals in NAc shell and related structures. Such findings help improve our understanding of NAc, VP, and LH interactions in mediating affective and motivation functions, including “liking” and “wanting” for food rewards.

Goal-directed, habitual and Pavlovian prosocial behavior

Although prosocial behaviors have been widely studied across disciplines, the mechanisms underlying them are not fully understood. Evidence from psychology, biology and economics suggests that prosocial behaviors can be driven by a variety of seemingly opposing factors: altruism or egoism, intuition or deliberation, inborn instincts or learned dispositions, and utility derived from actions or their outcomes. Here we propose a framework inspired by research on reinforcement learning and decision making that links these processes and explains characteristics of prosocial behaviors in different contexts. More specifically, we suggest that prosocial behaviors inherit features of up to three decision-making systems employed to choose between self- and other- regarding acts: a goal-directed system that selects actions based on their predicted consequences, a habitual system that selects actions based on their reinforcement history, and a Pavlovian system that emits reflexive responses based on evolutionarily prescribed priors. This framework, initially described in the field of cognitive neuroscience and machine learning, provides insight into the potential neural circuits and computations shaping prosocial behaviors. Furthermore, it identifies specific conditions in which each of these three systems should dominate and promote other- or self- regarding behavior.

Pleasure systems in the brain

Pleasure is mediated by well-developed mesocorticolimbic circuitry and serves adaptive functions. In affective disorders, anhedonia (lack of pleasure) or dysphoria (negative affect) can result from breakdowns of that hedonic system. Human neuroimaging studies indicate that surprisingly similar circuitry is activated by quite diverse pleasures, suggesting a common neural currency shared by all. Wanting for reward is generated by a large and distributed brain system. Liking, or pleasure itself, is generated by a smaller set of hedonic hot spots within limbic circuitry. Those hot spots also can be embedded in broader anatomical patterns of valence organization, such as in a keyboard pattern of nucleus accumbens generators for desire versus dread. In contrast, some of the best known textbook candidates for pleasure generators, including classic pleasure electrodes and the mesolimbic dopamine system, may not generate pleasure after all. These emerging insights into brain pleasure mechanisms may eventually facilitate better treatments for affective disorders.

Drug versus sweet reward: greater attraction to and preference for sweet versus drug cues

Despite the unique ability of addictive drugs to directly activate brain reward circuits, recent evidence suggests that drugs induce reinforcing and incentive effects that are comparable to, or even lower than some nondrug rewards. In particular, when rats have a choice between pressing a lever associated with intravenous cocaine or heroin delivery and another lever associated with sweet water delivery, most respond on the latter. This outcome suggests that sweet water is more reinforcing and attractive than either drug. However, this outcome may also be due to the differential ability of sweet versus drug levers to elicit Pavlovian feeding-like conditioned responses that can cause involuntary lever pressing, such as pawing and biting the lever. To test this hypothesis, rats first underwent Pavlovian conditioning to associate one lever with sweet water (0.2% saccharin) and a different lever with intravenous cocaine (0.25 mg) or heroin (0.01 mg). Choice between these two levers was then assessed under two operant choice procedures: one that permitted the expression of Pavlovian-conditioned lever press responses during choice, the other not. During conditioning, Pavlovian-conditioned lever press responses were considerably higher on the sweet lever than on either drug lever, and slightly greater on the heroin lever than on the cocaine lever. Importantly, though these differences in Pavlovian-conditioned behavior predicted subsequent preference for sweet water during choice, they were not required for its expression. Overall, this study confirms that rats prefer the sweet lever because sweet water is more reinforcing and attractive than cocaine or heroin.

Neurons for hunger and thirst transmit a negative-valence teaching signal

Homeostasis is a biological principle for regulation of essential physiological parameters within a set range. Behavioural responses due to deviation from homeostasis are critical for survival, but motivational processes engaged by physiological need states are incompletely understood. We examined motivational characteristics and dynamics of two separate neuron populations that regulate energy and fluid homeostasis by using cell type-specific activity manipulations in mice. We found that starvation-sensitive AGRP neurons exhibit properties consistent with a negative-valence teaching signal. Mice avoided activation of AGRP neurons, indicating that AGRP neuron activity has negative valence. AGRP neuron inhibition conditioned preference for flavours and places. Correspondingly, deep-brain calcium imaging revealed that AGRP neuron activity rapidly reduced in response to food-related cues. Complementary experiments activating thirst-promoting neurons also conditioned avoidance. Therefore, these need-sensing neurons condition preference for environmental cues associated with nutrient or water ingestion, which is learned through reduction of negative-valence signals during restoration of homeostasis.

Reconceptualizing anhedonia: novel perspectives on balancing the pleasure networks in the human brain

Anhedonia, the lack of pleasure, has been shown to be a critical feature of a range of psychiatric disorders. Yet, it is currently measured primarily through subjective self-reports and as such has been difficult to submit to rigorous scientific analysis. New insights from affective neuroscience hold considerable promise in improving our understanding of anhedonia and for providing useful objective behavioral measures to complement traditional self-report measures, potentially leading to better diagnoses and novel treatments. Here, we review the state-of-the-art of hedonia research and specifically the established mechanisms of wanting, liking, and learning. Based on this framework we propose to conceptualize anhedonia as impairments in some or all of these processes, thereby departing from the longstanding view of anhedonia as solely reduced subjective experience of pleasure. We discuss how deficits in each of the reward components can lead to different expressions, or subtypes, of anhedonia affording novel ways of measurement. Specifically, we review evidence suggesting that patients suffering from depression and schizophrenia show impairments in wanting and learning, while some aspects of conscious liking seem surprisingly intact. Furthermore, the evidence suggests that anhedonia is heterogeneous across psychiatric disorders, depending on which parts of the pleasure networks are most affected. This in turn has implications for diagnosis and treatment of anhedonia.

The biopsychology of salt hunger and sodium deficiency

Sodium is a necessary dietary macromineral that tended to be sparsely distributed in mankind's environment in the past. Evolutionary selection pressure shaped physiological mechanisms including hormonal systems and neural circuits that serve to promote sodium ingestion. Sodium deficiency triggers the activation of these hormonal systems and neural circuits to engage motivational processes that elicit a craving for salty substances and a state of reward when salty foods are consumed. Sodium deficiency also appears to be associated with aversive psychological states including anhedonia, impaired cognition, and fatigue. Under certain circumstances the psychological processes that promote salt intake can become powerful enough to cause "salt gluttony," or salt intake far in excess of physiological need. The present review discusses three aspects of the biopsychology of salt hunger and sodium deficiency: (1) the psychological processes that promote salt intake during sodium deficiency, (2) the effects of sodium deficiency on mood and cognition, and (3) the sensitization of sodium appetite as a possible cause of salt gluttony.

Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation

Evidence supports at least two methods for learning about reward and punishment and making predictions for guiding actions. One method, called model-free, progressively acquires cached estimates of the long-run values of circumstances and actions from retrospective experience. The other method, called model-based, uses representations of the environment, expectations, and prospective calculations to make cognitive predictions of future value. Extensive attention has been paid to both methods in computational analyses of instrumental learning. By contrast, although a full computational analysis has been lacking, Pavlovian learning and prediction has typically been presumed to be solely model-free. Here, we revise that presumption and review compelling evidence from Pavlovian revaluation experiments showing that Pavlovian predictions can involve their own form of model-based evaluation. In model-based Pavlovian evaluation, prevailing states of the body and brain influence value computations, and thereby produce powerful incentive motivations that can sometimes be quite new. We consider the consequences of this revised Pavlovian view for the computational landscape of prediction, response, and choice. We also revisit differences between Pavlovian and instrumental learning in the control of incentive motivation.

The Brain Basis of Positive and Negative Affect: Evidence from a Meta-Analysis of the Human Neuroimaging Literature

The ability to experience pleasant or unpleasant feelings or to represent objects as "positive" or "negative" is known as representing hedonic "valence." Although scientists overwhelmingly agree that valence is a basic psychological phenomenon, debate continues about how to best conceptualize it scientifically. We used a meta-analysis of 397 functional magnetic resonance imaging (fMRI) and positron emission tomography studies (containing 914 experimental contrasts and 6827 participants) to test 3 competing hypotheses about the brain basis of valence: the bipolarity hypothesis that positive and negative affect are supported by a brain system that monotonically increases and/or decreases along the valence dimension, the bivalent hypothesis that positive and negative affect are supported by independent brain systems, and the affective workspace hypothesis that positive and negative affect are supported by a flexible set of valence-general regions. We found little evidence for the bipolar or bivalent hypotheses. Findings instead supported the hypothesis that, at the level of brain activity measurable by fMRI, valence is flexibly implemented across instances by a set of valence-general limbic and paralimbic brain regions.

Reconsolidation allows fear memory to be updated to a less aversive level through the incorporation of appetitive information

The capacity to adapt to new situations is one of the most important features of memory. When retrieved, memories may undergo a labile state that is sensitive to modification. This process, called reconsolidation, can lead to memory updating through the integration of new information into a previously consolidated memory background. Thus reconsolidation provides the opportunity to modify an undesired fear memory by updating its emotional valence to a less aversive level. Here we evaluated whether a fear memory can be reinterpreted by the concomitant presentation of an appetitive stimulus during its reactivation, hindering fear expression. We found that memory reactivation in the presence of appetitive stimuli resulted in the suppression of a fear response. In addition, fear expression was not amenable to reinstatement, spontaneous recovery, or rapid reacquisition. Such effect was prevented by either systemic injection of nimodipine or intra-hippocampal infusion of ifenprodil, indicating that memory updating was mediated by a reconsolidation mechanism relying on hippocampal neuronal plasticity. Taken together, this study shows that reconsolidation allows for a 're-signification' of unwanted fear memories through the incorporation of appetitive information. It brings a new promising cognitive approach to treat fear-related disorders.

The role of the lateral hypothalamus and orexin in ingestive behavior: a model for the translation of past experience and sensed deficits into motivated behaviors

The hypothalamus has been recognized for its involvement in both maintaining homeostasis and mediating motivated behaviors. The present article discusses a region of the hypothalamus known as the lateral hypothalamic area (LHA). It is proposed that brain nuclei within the LHA including the dorsal region of the lateral hypothalamus (LHAd) and perifornical area (PeF) provide a link between neural systems that regulate homeostasis and those that mediate appetitive motivated behaviors. Functional and immunohistochemical data indicate that the LHA promotes many motivated behaviors including food intake, water intake, salt intake, and sexual behavior. Anatomical tracing experiments demonstrate that the LHA is positioned to receive inputs from brain areas involved in regulating body fluid and energy homeostasis. Regions within the LHA send dense projections to the ventral tegmental area (VTA), providing a pathway for the LHA to influence dopaminergic systems generally recognized to be involved in motivated behaviors and their reinforcement. Furthermore, the LHA contains neurons that synthesize orexin/hypocretin, a neuropeptide that promotes many appetitive motivated behaviors. The LHA also receives inputs from brain areas involved in reward-related learning and orexin neuron activation can become conditioned to environmental stimuli that are associated with rewards. Therefore, it is hypothesized that the LHA integrates signaling from areas that regulate body fluid and energy balance and reward-related learning. In turn, this information is “fed into” mesolimbic circuitry to influence the performance of motivated behaviors. This hypothesis may foster experiments that will result in an improved understanding of LHA function. An improved understanding of LHA function may aid in treating disorders that are associated with an excess or impairment in the expression of ingestive behavior including obesity, anorexia, impairments in thirst, salt gluttony, and salt deficiency.

Incentive salience attribution under reward uncertainty: A Pavlovian model

There is a vast literature on the behavioural effects of partial reinforcement in Pavlovian conditioning. Compared with animals receiving continuous reinforcement, partially rewarded animals typically show (a) a slower development of the conditioned response (CR) early in training and (b) a higher asymptotic level of the CR later in training. This phenomenon is known as the partial reinforcement acquisition effect (PRAE). Learning models of Pavlovian conditioning fail to account for it. In accordance with the incentive salience hypothesis, it is here argued that incentive motivation (or 'wanting') plays a more direct role in controlling behaviour than does learning, and reward uncertainty is shown to have an excitatory effect on incentive motivation. The psychological origin of that effect is discussed and a computational model integrating this new interpretation is developed. Many features of CRs under partial reinforcement emerge from this model.

Attribution and expression of incentive salience are differentially signaled by ultrasonic vocalizations in rats

During Pavlovian incentive learning, the affective properties of rewards are thought to be transferred to their predicting cues. However, how rewards are represented emotionally in animals is widely unknown. This study sought to determine whether 50-kHz ultrasonic vocalizations (USVs) in rats may signal such a state of incentive motivation to natural, nutritional rewards. To this end, rats learned to anticipate food rewards and, across experiments, the current physiological state (deprived vs. sated), the type of learning mechanism recruited (Pavlovian vs. instrumental), the hedonic properties of UCS (low vs. high palatable food), and the availability of food reward (continued vs. discontinued) were manipulated. Overall, we found that reward-cues elicited 50-kHz calls as they were signaling a putative affective state indicative of incentive motivation in the rat. Attribution and expression of incentive salience, however, seemed not to be an unified process, and could be teased apart in two different ways: 1) under high motivational state (i.e., hunger), the attribution of incentive salience to cues occurred without being expressed at the USVs level, if reward expectations were higher than the outcome; 2) in all experiments when food rewards were devalued by satiation, reward cues were still able to elicit USVs and conditioned anticipatory activity although reward seeking and consumption were drastically weakened. Our results suggest that rats are capable of representing rewards emotionally beyond apparent, immediate physiological demands. These findings may have translational potential in uncovering mechanisms underlying aberrant and persistent motivation as observed in drug addiction, gambling, and eating disorders.

Applying incentive sensitization models to behavioral addiction

The incentive sensitization theory is a promising model for understanding the mechanisms underlying drug addiction, and has received support in animal and human studies. So far the theory has not been applied to the case of behavioral addictions like Gambling Disorder, despite sharing clinical symptoms and underlying neurobiology. We examine the relevance of this theory for Gambling Disorder and point to predictions for future studies. The theory promises a significant contribution to the understanding of behavioral addiction and opens new avenues for treatment.

Advances in the neurobiological bases for food 'liking' versus 'wanting'

The neural basis of food sensory pleasure has become an increasingly studied topic in neuroscience and psychology. Progress has been aided by the discovery of localized brain subregions called hedonic hotspots in the early 2000s, which are able to causally amplify positive affective reactions to palatable tastes ('liking') in response to particular neurochemical or neurobiological stimulations. Those hedonic mechanisms are at least partly distinct from larger mesocorticolimbic circuitry that generates the incentive motivation to eat ('wanting'). In this review, we aim to describe findings on these brain hedonic hotspots, especially in the nucleus accumbens and ventral pallidum, and discuss their role in generating food pleasure and appetite.

Food cravings, appetite, and snack-food consumption in response to a psychomotor stimulant drug: the moderating effect of "food-addiction"

There is mounting evidence that many highly processed foods have addictive properties, and that some cases of compulsive overeating resemble an addiction disorder. While support for the Yale Food Addiction Scale (YFAS) as a valid diagnostic tool has been impressive and continues to increase, to date, no research has examined the food-addiction construct in response to an actual food stimulus, and in relation to direct measures of appetite and food consumption. As part of a larger community-based study of overeating in healthy adults who were predominately overweight and obese (aged 25-50 years), 136 participants completed the YFAS, of whom 23 met the diagnostic criteria for food-addiction. They took part in a 2-day, double-blind, cross-over, single-dose drug challenge using a psychomotor stimulant (methylphenidate) and placebo. Participants were first assessed on ratings of appetite and food cravings after holding and tasting their favorite snack food, after which they were able to eat all or part of the snack, as they wished. Three separate repeated-measures analysis-of-variance procedures were carried out, each with two between-subjects factors (Diagnosis: food-addiction vs. non-food addiction) and (Sex: male vs. female) and 1 within-subjects factor (Days: drug vs. placebo). As anticipated, for all three dependent variables, there was a significant main effect for Days with a response decrease from placebo to the drug condition. With respect to food cravings and appetite ratings, results indicated that the food addiction group had significantly higher scores on both variables. For food consumption, there was a significant Days × Diagnosis interaction whereby the food-addiction group showed no food-intake suppression across days compared to the non-food-addiction group who demonstrated a significant decrease in snack-food consumption with methylphenidate. The finding that the food-addiction group was resistant to the food-intake suppression typically induced by a dopamine agonist supports evidence of dopamine signaling-strength differences in individuals with compulsive overeating compared to those without this disorder. This represents the first demonstration that individuals defined by their food-addiction status have a unique pattern of food-intake following a pharmacologic challenge with such agents.

Initial uncertainty in Pavlovian reward prediction persistently elevates incentive salience and extends sign-tracking to normally unattractive cues

Uncertainty is a component of many gambling games and may play a role in incentive motivation and cue attraction. Uncertainty can increase the attractiveness for predictors of reward in the Pavlovian procedure of autoshaping, visible as enhanced sign-tracking (or approach and nibbles) by rats of a metal lever whose sudden appearance acts as a conditioned stimulus (CS+) to predict sucrose pellets as an unconditioned stimulus (UCS). Here we examined how reward uncertainty might enhance incentive salience as sign-tracking both in intensity and by broadening the range of attractive CS+s. We also examined whether initially induced uncertainty enhancements of CS+ attraction can endure beyond uncertainty itself, and persist even when Pavlovian prediction becomes 100% certain. Our results show that uncertainty can broaden incentive salience attribution to make CS cues attractive that would otherwise not be (either because they are too distal from reward or too risky to normally attract sign-tracking). In addition, uncertainty enhancement of CS+ incentive salience, once induced by initial exposure, persisted even when Pavlovian CS-UCS correlations later rose toward 100% certainty in prediction. Persistence suggests an enduring incentive motivation enhancement potentially relevant to gambling, which in some ways resembles incentive-sensitization. Higher motivation to uncertain CS+s leads to more potent attraction to these cues when they predict the delivery of uncertain rewards. In humans, those cues might possibly include the sights and sounds associated with gambling, which contribute a major component of the play immersion experienced by problematic gamblers.

Designer receptors show role for ventral pallidum input to ventral tegmental area in cocaine seeking

Ventral pallidum (VP) is centrally positioned within mesocorticolimbic reward circuits, and its dense projection to ventral tegmental area (VTA) regulates neuronal activity there. However, VP is a heterogeneous structure, and how this complexity affects its role within wider reward circuits is unclear. Here we demonstrate that projections to VTA from rostral (RVP), but not caudal VP (CVP) are robustly Fos-activated during cue-induced reinstatement of cocaine seeking—a rat model of relapse in addiction. Moreover, designer receptor-mediated transient inactivation of RVP neurons, their terminals in VTA, or functional connectivity between RVP and VTA dopamine neurons blocks the ability of drug-associated cues (but not a cocaine prime) to reinstate cocaine seeking. In contrast, CVP neuronal inhibition instead blocked cocaine-primed, but not cue-induced reinstatement. This novel double dissociation in VP sub-regional roles in drug seeking is likely important for understanding mesocorticolimbic circuits underlying reward seeking and addiction.

Investigating habits: strategies, technologies and models

Understanding habits at a biological level requires a combination of behavioral observations and measures of ongoing neural activity. Theoretical frameworks as well as definitions of habitual behaviors emerging from classic behavioral research have been enriched by new approaches taking account of the identification of brain regions and circuits related to habitual behavior. Together, this combination of experimental and theoretical work has provided key insights into how brain circuits underlying action-learning and action-selection are organized, and how a balance between behavioral flexibility and fixity is achieved. New methods to monitor and manipulate neural activity in real time are allowing us to have a first look “under the hood” of a habit as it is formed and expressed. Here we discuss ideas emerging from such approaches. We pay special attention to the unexpected findings that have arisen from our own experiments suggesting that habitual behaviors likely require the simultaneous activity of multiple distinct components, or operators, seen as responsible for the contrasting dynamics of neural activity in both cortico-limbic and sensorimotor circuits recorded concurrently during different stages of habit learning. The neural dynamics identified thus far do not fully meet expectations derived from traditional models of the structure of habits, and the behavioral measures of habits that we have made also are not fully aligned with these models. We explore these new clues as opportunities to refine an understanding of habits.

Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself - a lever - more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful.

Goals and habits in the brain

An enduring and richly elaborated dichotomy in cognitive neuroscience is that of reflective versus reflexive decision making and choice. Other literatures refer to the two ends of what is likely to be a spectrum with terms such as goal-directed versus habitual, model-based versus model-free or prospective versus retrospective. One of the most rigorous traditions of experimental work in the field started with studies in rodents and graduated via human versions and enrichments of those experiments to a current state in which new paradigms are probing and challenging the very heart of the distinction. We review four generations of work in this tradition and provide pointers to the forefront of the field's fifth generation.