Relations Between Pavlovian-Instrumental Transfer and Reinforcer Devaluation

Unraveling the influence of Pavlovian cues on decision-making: A pre-registered meta-analysis on Pavlovian-to-instrumental transfer

Amidst the replicability crisis, promoting transparency and rigor in research becomes imperative. The Pavlovian-to-Instrumental Transfer (PIT) paradigm is increasingly used in human studies to offer insights into how Pavlovian cues, by anticipating rewards or punishments, influence decision-making and potentially contribute to the development of clinical conditions. However, research on this topic faces challenges, including methodological variability and the need for standardized approaches, which can undermine the quality and robustness of experimental findings. Hence, we conducted a meta-analysis to unravel the methodological, task-related, individual, training, and learning factors that can modulate PIT. By scrutinizing these factors, the present meta-analysis reviews the current literature on human PIT, provides practical guidelines for future research to enhance study outcomes and refine methodologies, and identifies knowledge gaps that can serve as a direction for future studies aiming to advance the comprehension of how Pavlovian cues shape decision-making.

The consequences of AI training on human decision-making

AI is now an integral part of everyday decision-making, assisting us in both routine and high-stakes choices. These AI models often learn from human behavior, assuming this training data is unbiased. However, we report five studies that show that people change their behavior to instill desired routines into AI, indicating this assumption is invalid. To show this behavioral shift, we recruited participants to play the ultimatum game, where they were asked to decide whether to accept proposals of monetary splits made by either other human participants or AI. Some participants were informed their choices would be used to train an AI proposer, while others did not receive this information. Across five experiments, we found that people modified their behavior to train AI to make fair proposals, regardless of whether they could directly benefit from the AI training. After completing this task once, participants were invited to complete this task again but were told their responses would not be used for AI training. People who had previously trained AI persisted with this behavioral shift, indicating that the new behavioral routine had become habitual. This work demonstrates that using human behavior as training data has more consequences than previously thought since it can engender AI to perpetuate human biases and cause people to form habits that deviate from how they would normally act. Therefore, this work underscores a problem for AI algorithms that aim to learn unbiased representations of human preferences.

Punishment resistance for cocaine is associated with inflexible habits in rats

Addiction is characterized by continued drug use despite negative consequences. In an animal model, a subset of rats continues to self-administer cocaine despite footshock consequences, showing punishment resistance. We sought to test the hypothesis that punishment resistance arises from failure to exert goal-directed control over habitual cocaine seeking. While habits are not inherently permanent or maladaptive, continued use of habits under conditions that should encourage goal-directed control makes them maladaptive and inflexible. We trained male and female Sprague Dawley rats on a seeking-taking chained schedule of cocaine self-administration. We then exposed them to four days of punishment testing in which footshock was delivered randomly on one-third of trials. Before and after punishment testing (four days pre-punishment and ≥ four days post-punishment), we assessed whether cocaine seeking was goal-directed or habitual using outcome devaluation via cocaine satiety. We found that punishment resistance was associated with continued use of habits, whereas punishment sensitivity was associated with increased goal-directed control. Although punishment resistance for cocaine was not predicted by habitual responding pre-punishment, it was associated with habitual responding post-punishment. In parallel studies of food self-administration, we similarly observed that punishment resistance was associated with habitual responding post-punishment but not pre-punishment in males, although it was associated with habitual responding both pre- and post-punishment in females, indicating that punishment resistance was predicted by habitual responding in food-seeking females. These findings indicate that punishment resistance is related to habits that have become inflexible and persist under conditions that should encourage a transition to goal-directed behavior.

Effects of an ethanol-paired conditioned stimulus on responding for ethanol suppressed by a conditioned-taste-aversion

Ethanol-Paired Conditioned Stimuli (CS) can increase ethanol-responding either in extinction or occurring at low rates late in a session. To examine the generality of CS-induced increases in ethanol-responding, we examined whether a CS could increase responding suppressed by Conditioned-Taste-Aversion (CTA), which presumably suppresses responding by changing ethanol's valence from positive to negative. Rats were trained to respond for ethanol under a Random Interval (RI) schedule. We then removed the lever and paired Random-Time ethanol deliveries with illumination of a stimulus light (i.e., CS) for 10 sessions. Results were compared with a Truly Random Control group, in which the light and ethanol deliveries occurred independently. In a subsequent experiment, rats were treated similarly, except the light served as a discriminative stimulus, as the lever was extended and ethanol deliveries were available under a RI during light presentations. After this training, the lever was returned and rats again responded for ethanol. Subsequently, sessions were followed by LiCl administration. When responding reached low levels, LiCl administration stopped and the light was occasionally illuminated during the session. Responding during the light presentation was compared to responding during the period preceding light presentation. Responding partially recovered across 10 sessions and was greater during light presentations than in the period before it in all three groups. Increases were not reliably different between the groups, indicating that explanations for these increases such as CS-induced increases in motivation or approach toward the light are unlikely to be correct. The most likely explanation for these light-induced increases is that during sessions in which the light had been presented previously, LiCl had never been presented and thus, the light had come to signal that ethanol was safe to drink.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Resurgence of goal-directed actions and habits

This study investigated how goal-directed and habitual behaviors recover after extinction within the context of the resurgence effect, a form of relapse induced by the removal or worsening of alternative reinforcement. Rats were trained to press a target lever with one reinforcer (O1) for either minimal (4) or extended (16) sessions. An extinction test after the completion of O1 devaluation confirmed that minimal and extended training formed goal-directed and habitual behaviors, respectively. Then, pressing an alternative lever was reinforced with a second reinforcer (O2) while the target response was placed on extinction. When O2 was discontinued, the minimally trained target response resurged with goal-directed status as in the extinction test. However, the extinguished habitual behavior in the extensively trained rats did not recover as a habit but instead with goal-directed status, possibly due to the context specificity of habits or the introduction of a new response-reinforcer contingency. The critical finding that reinforcer devaluation consistently led to less resurgence regardless of the amount of acquisition training provides a clinical implication that coupling differential-reinforcement-of-alternative-behavior (DRA) treatments with the devaluation of the associated reinforcer of problematic behavior could effectively diminish its recurrence.

Action control and selection in social disinhibition following severe TBI: a pavlovian-to-instrumental transfer and outcome devaluation study

INTRODUCTION

Social disinhibition is a significant sequela of severe traumatic brain injury (TBI). Some research suggests that it could reflect a deficiency in goal-directed behavior. The current study aimed to test whether these inappropriate behaviors tend to be deficient in goal-directed control, that is, triggered more by environmental stimuli than by the known consequences of their actions.

METHOD

We used a between-group design with 25 adult participants with severe TBI, and 27 control participants. Social disinhibition was measured using Frontal Systems Behavior Scale and Social Disinhibition Interview. Changes in reward-related goal-directed behavior were evaluated using a computer-based task in which we assessed the influence of cues predicting reward and of reward devaluation on choice performance.

RESULTS

We found no difference in the levels of social disinhibition between the TBI and control groups and, using mixed two-way ANCOVAs, no overall effect of the stimuli or outcome devaluation. However, after combing these groups and splitting them based on their disinhibition levels, a significant interaction between group (High vs Low disinhibition) and reward type (Valued vs Devalued) in sensitivity to outcome devaluation test (F = 5.99, p = .01, ηp2 = .13) appeared. Comparing with the baseline rate of responding, the Low disinhibition group decreased their responding to devalued and increased their responding to still-valued outcomes. In contrast, the High disinhibition group showed the opposite pattern of choice performance.

CONCLUSIONS

It appears that people with clinical levels of social disinhibition are both prone to outcome-response priming effects and insensitive to changes in the value of the consequences of their actions, that is, despite evidence they were aware of the reduction in the value of their actions's outcomes, people with high-level disinhibition kept performing those actions. This pattern has the hallmarks of a habit suggesting their disinhibition reflects a loss of executive control.

Differential effects of acute and prolonged morphine withdrawal on motivational and goal-directed control over reward-seeking behavior

Opioid addiction is a relapsing disorder marked by uncontrolled drug use and reduced interest in normally rewarding activities. The current study investigated the impact of spontaneous withdrawal from chronic morphine exposure on emotional, motivational, and cognitive processes involved in regulating the pursuit and consumption of natural food rewards in male rats. In Experiment 1, rats experiencing acute morphine withdrawal lost weight and displayed somatic signs of drug dependence. However, hedonically-driven sucrose consumption was significantly elevated, suggesting intact and potentially heightened emotional reward processing. In Experiment 2, rats undergoing acute morphine withdrawal displayed reduced motivation when performing an effortful response for palatable food reward. Subsequent reward devaluation testing revealed that acute withdrawal also disrupted their ability to exert flexible goal-directed control over their reward-seeking behavior. Specifically, morphine-withdrawn rats displayed insensitivity to reward devaluation both when relying on prior action-outcome learning and when given direct feedback about the consequences of their actions. In Experiment 3, rats tested after prolonged morphine withdrawal displayed heightened rather than diminished motivation for food rewards and retained their ability to engage in flexible goal-directed action selection. However, brief re-exposure to morphine was sufficient to impair motivation and disrupt goal-directed action selection, though in this case insensitivity to reward devaluation was only observed in the presence of morphine-paired context cues and in the absence of response-contingent feedback. We suggest that these opioid-withdrawal induced deficits in motivation and goal-directed control may contribute to addiction by interfering with the pursuit of adaptive alternatives to drug use.

Unpredictable, intermittent access to sucrose or water promotes increased reward pursuit in rats

Reward uncertainty can sensitize reward pathways, promoting increased reward-seeking and -taking behaviours. This is relevant to human conditions such as pathological gambling, eating disorders and drug addiction. In the context of addiction, preclinical self-administration procedures have been developed to model the intermittency of human drug use. These intermittent-access (IntA) procedures involve intermittent but predictable access to drug during self-administration sessions. However, human drug use typically involves intermittent and unpredictable drug access. We introduce a new procedure modeling unpredictable, intermittent access (UIntA) to a reinforcer, and we compare it to procedures that provide predictable reinforcer availability; continuous (ContA) or intermittent (IntA) access. Female rats self-administered water or liquid sucrose in daily hour-long sessions. IntA and ContA rats had access to a fixed volume of water or sucrose (0.1 ml), under a fixed ratio 3 schedule of reinforcement. IntA rats had predictable 5-min reinforcer ON and 25-min reinforcer OFF periods. ContA rats had 60 min of reinforcer access during each session. For UIntA rats, variation in the length of ON and OFF periods (1, 5 or 9 min/period), response requirement (variable ratio 3 schedule of reinforcement), reinforcer probability (50%) and quantity (0, 0.1 or 0.2 ml) introduced reward uncertainty. Following 14 daily self-administration sessions, UIntA rats showed the highest levels of responding for water or sucrose under progressive ratio conditions, responding under extinction conditions, and cue-induced reinstatement of sucrose seeking. Thus, unpredictable, intermittent reward access promotes increased reward pursuit. This has implications for modeling addiction and other disorders of increased reward seeking.

Investigating habits in humans with a symmetrical outcome-revaluation task

The translation of the outcome-devaluation paradigm to study habit in humans has yielded interesting insights but proven to be challenging. We present a novel, outcome-revaluation task with a symmetrical design, in the sense that half of the available outcomes are always valuable and the other half not-valuable. In the present studies, during the instrumental learning phase, participants learned to respond (Go) to certain stimuli to collect valuable outcomes (and points) while refraining to respond (NoGo) to stimuli signaling not-valuable outcomes. Half of the stimuli were short-trained, while the other half were long-trained. Subsequently, in the test phase, the signaled outcomes were either value-congruent with training (still-valuable and still-not-valuable), or value-incongruent (devalued and upvalued). The change in outcome value on value-incongruent trials meant that participants had to flexibly adjust their behavior. At the end of the training phase, participants completed the self-report behavioral automaticity index - providing an automaticity score for each stimulus-response association. We conducted two experiments using this task, that both provided evidence for stimulus-driven habits as reflected in poorer performance on devalued and upvalued trials relative to still-not-valuable trials and still-valuable trials, respectively. While self-reported automaticity increased with longer training, behavioral flexibility was not affected. After extended training (Experiment 2), higher levels of self-reported automaticity when responding to stimuli signaling valuable outcomes were related to more 'slips of action' when the associated outcome was subsequently devalued. We conclude that the symmetrical outcome-revaluation task provides a promising paradigm for the experimental investigation of habits in humans.

Revealing abrupt transitions from goal-directed to habitual behavior

A fundamental tenet of animal behavior is that decision-making involves multiple 'controllers.' Initially, behavior is goal-directed, driven by desired outcomes, shifting later to habitual control, where cues trigger actions independent of motivational state. Clark Hull's question from 1943 still resonates today: "Is this transition abrupt, or is it gradual and progressive?"1 Despite a century-long belief in gradual transitions, this question remains unanswered2,3 as current methods cannot disambiguate goal-directed versus habitual control in real-time. Here, we introduce a novel 'volitional engagement' approach, motivating animals by palatability rather than biological need. Offering less palatable water in the home cage4,5 reduced motivation to 'work' for plain water in an auditory discrimination task when compared to water-restricted animals. Using quantitative behavior and computational modeling6, we found that palatability-driven animals learned to discriminate as quickly as water-restricted animals but exhibited state-like fluctuations when responding to the reward-predicting cue-reflecting goal-directed behavior. These fluctuations spontaneously and abruptly ceased after thousands of trials, with animals now always responding to the reward-predicting cue. In line with habitual control, post-transition behavior displayed motor automaticity, decreased error sensitivity (assessed via pupillary responses), and insensitivity to outcome devaluation. Bilateral lesions of the habit-related dorsolateral striatum7 blocked transitions to habitual behavior. Thus, 'volitional engagement' reveals spontaneous and abrupt transitions from goal-directed to habitual behavior, suggesting the involvement of a higher-level process that arbitrates between the two.

A laboratory model of canine search vigilance decrement, II: Noncontingent reward and Pavlovian appetitive stimuli

Detection dogs have demonstrated reduced performance in operational settings when required to search in an environment where few to no target odors are present. This study's purpose was to increase detection dog accuracy using noncontingent reward (NCR) and Pavlovian stimuli associated with reward. Eighteen dogs were randomly spilt into two groups and received four 40-trial sessions in an operational and training context at 90% odor prevalence (baseline). Following baseline, in the operational context (now at 10% odor prevalence), experimental dogs received an NCR schedule consisting of delivering food rewards at the end of 66% of trials. After the NCR Test, dogs returned to baseline. During baseline, the experimental dogs received 10 days of delayed Pavlovian conditioning to a tone. During the test phase, the conditioned stimulus (tone) was presented to experimental dogs on average every two trials for 30 s in the operational context (now at 10% odor prevalence). Overall, NCR showed a nonsignificant trend for increased responding in the experimental group but tended to increase false alerts; therefore, a permutation of an NCR-like reward schedule may maintain search. The Pavlovian conditioned stimulus didn't decrease timeouts or improve accuracy, but a within-session analysis indicated that the dogs were more likely to time out and less likely to false alert when the tone was on than when it was off.

Fear Learning in Genital Pain: Toward a Biopsychosocial, Ecologically Valid Research and Treatment Model

Although fear learning mechanisms are implicated in the development, maintenance, exacerbation, and reduction of genital pain, systematic research on how fear of genital pain emerges, spreads, persists, and reemerges after treatment is lacking. This paper provides an overview of the literature on pain-related fear, integrates the ideas on learning and sexual arousal responding, and specifies the pathways through which compromised learning may contribute to the development and persistence of genital pain. In order to refine theories of genital pain and optimize treatments, we need to adopt a biopsychosocial framework to pain-related fear learning and uncover potential moderators that shape individual trajectories. This involves examining the role of physiological processes, subjective experiences, as well as partner and relational cues in fear acquisition, excessive generalization and impaired safety learning, extinction of fear, counterconditioning, and return of fear. Recent methodological advances in fear conditioning and sex research are promising to enable more symptom-specific and ecologically valid experimental paradigms.

Punishment resistance for cocaine is associated with inflexible habits in rats

Addiction is characterized by continued drug use despite negative consequences. In an animal model, a subset of rats continues to self-administer cocaine despite footshock consequences, showing punishment resistance. We sought to test the hypothesis that punishment resistance arises from failure to exert goal-directed control over habitual cocaine seeking. While habits are not inherently permanent or maladaptive, continued use of habits under conditions that should encourage goal-directed control makes them maladaptive and inflexible. We trained male and female Sprague Dawley rats on a seeking-taking chained schedule of cocaine self-administration (2 h/day). We then exposed them to 4 days of punishment testing, in which footshock (0.4 mA, 0.3 s) was delivered randomly on one-third of trials, immediately following completion of seeking and prior to extension of the taking lever. Before and after punishment testing (4 days pre-punishment and ≥4 days post-punishment), we assessed whether cocaine seeking was goal-directed or habitual using outcome devaluation via cocaine satiety. We found that punishment resistance was associated with continued use of habits, whereas punishment sensitivity was associated with increased goal-directed control. Although punishment resistance was not predicted by habitual responding pre-punishment, it was associated with habitual responding post-punishment. In parallel studies of food self-administration, we similarly observed that punishment resistance was associated with habitual responding post-punishment but not pre-punishment. These findings indicate that punishment resistance is related to habits that have become inflexible and persist under conditions that should encourage a transition to goal-directed behavior.

Metabotropic group II glutamate receptors mediate cue-triggered increases in reward-seeking behaviour

RATIONALE

Reward-associated cues can invigorate reward-seeking actions via Pavlovian-to-instrumental transfer (PIT). Glutamatergic neurotransmission mediates the appetitive effects of reward-associated cues. We characterized the expression of PIT and its mediation by metabotropic group II glutamate (mGlu_2/3) receptor activity in female and male rats.

OBJECTIVES

Across the sexes, we used PIT procedures to determine (i) cue-triggered increases in instrumental responding for water reward (experiment 1) and (ii) the respective influences of the mGlu_2/3 receptor agonist LY379268 and thirst satiation on this effect (experiment 2).

METHODS

Water-restricted female and male Sprague-Dawley rats learned to lever press for water. Separately, they learned that one of two auditory stimuli predicts free water (CS + vs CS -). On PIT test days, the CS + and CS - were presented independent of instrumental responding, and we measured effects on lever pressing under extinction (no water). In experiment 1, we characterized PIT across the sexes. In experiment 2, we measured PIT after systemic LY379268 administration (0, 0.3 and 1 mg/kg) and thirst satiation, respectively.

RESULTS

Female and male rats showed similar PIT, with CS + but not CS - presentations potentiating water-seeking behaviour. LY379268 (1 mg/kg) attenuated CS + evoked increases in both water-associated lever pressing and conditioned approach to the water port. Thirst satiation attenuated both water-seeking and CS + evoked conditioned approach behaviour.

CONCLUSIONS

The sexes show similar cue-triggered increases in reward seeking, and thirst satiation suppresses both water-seeking and cue-triggered anticipation of water reward. Finally, across the sexes, mGlu_2/3 receptor activity mediates cue-triggered increases in reward seeking.

Mechanisms underlying performance in a cued go/no-go Pavlovian-to-instrumental transfer paradigm

Environmental cues that remind us of rewarding outcomes (drugs, food) play a significant role in addiction relapse. In the lab the Pavlovian-to-instrumental transfer (PIT) task has been used to formally examine how cues associated with reward or punishment can bias ongoing instrumental responding. Using a version of this paradigm that integrates PIT with a go/no-go task many studies have related stronger PIT effects (with non-drug rewards) to problematic alcohol use including risky alcohol users relative to non-risky drinkers, individuals with alcohol dependence versus healthy controls and individuals in recovery from alcohol use disorder who are more likely to relapse. However the theoretical importance of these findings and the implications for models of addiction was previously not clear. Understanding if this task indexes the general motivating effects of reward cues on instrumental responding (and whether this is sensitive to shifts in motivation for those outcomes) is critical for understanding these previous results within the context of addiction. Thus, in the current study we aimed to delineate the associative mechanisms that drive the stimulus effects observed in this PIT task. Specifically, we wished to examine whether the cueing effects observed in the cued-go/no-go task were selective in their effect on action, insofar as Pavlovian cues specifically invigorated (or suppressed) responding only if they were associated with congruent outcomes. We conclude that the PIT measured with this task is general in nature. Surprisingly however, the biasing effects of Pavlovian cues on instrumental responding did not appear to be sensitive to outcome devaluation.

Environmental control of social goals: using Pavlovian-to-instrumental transfer to test cue-based pro-self and pro-social outcome responses

A large amount of literature demonstrates that social behaviour can be triggered by environmental cues. A long-standing debate involves the question of whether such stimuli trigger behaviour directly (i.e. habits) or whether these effects mediate goals. As studies on automatic goal pursuit typically use real-world cues that are already associated with the behaviour and potentially the goal, it is impossible to make strong claims about the nature of the effects. In the present paper, we use a paradigm inspired by the Pavlovian-to-instrumental transfer (PIT) literature to examine how the environment can trigger goal-directed behaviour. Building on the essence of pro-self and pro-social motives in humans, two experiments explored the PIT effect when the outcomes were framed in terms of self- versus other-interest. Participants performed actions to earn money for themselves or a charity. Each outcome was linked to a different cue. The results showed that a cue predictive of self-interest outcomes facilitated responses instrumental in gaining the outcome, while such specific PIT effect for other-interest outcomes only emerged when participants were free to donate the money. We briefly discuss these findings reflecting on whether the PIT effect in our paradigm is indeed sensitive to the value of social goals.

Intermittent access training produces greater motivation for a non-drug reinforcer than long access training

It has recently been proposed that the intermittent access (IntA) drug self-administration procedure better produces behavioral changes relevant to addiction than the long access (LgA) procedure. In this version of the IntA procedure, the drug is made available for a 5-min period during each half hour of a 6-h session. In contrast, on the LgA procedure, the drug is available continuously for 6 h. Previous studies have found that IntA drug self-administration produces greater drug motivation, measured by increased progressive ratio breakpoints, than LgA self-administration. It has been hypothesized that this effect is due to the rapid, "spiking" brain levels of the drug, and consequent neuroadaptations, experienced by rats during IntA sessions. However, no study has compared the effects of IntA versus LgA training on reinforcer motivation when using a non-drug reinforcer. The present study compared motivation for a saccharin reinforcer after IntA or LgA training. In Experiment 1, separate groups of rats lever-pressed for saccharin on the IntA or LgA procedures. In Experiment 2, a within-subjects design was used where rats pressed one lever on the IntA procedure and another lever on the LgA procedure for saccharin. In both experiments, IntA training produced greater breakpoints than LgA training. As no drug was used here, spiking drug levels could not have been responsible for the increased saccharin motivation observed after IntA training. Instead, it is proposed that differences in stimulus-reinforcer associations learned during IntA versus LgA training may be responsible for the effect. Future research is needed to determine the extent to which such learning factors may contribute to the increased motivation observed after IntA training with drug reinforcers.

Outcome devaluation by specific satiety disrupts sensory-specific Pavlovian-to-instrumental transfer

Reward predictive cues can selectively motivate instrumental behaviors that predict the same rewarding outcomes, an effect known as specific Pavlovian-to-instrumental transfer (PIT). This selective effect is thought to be mediated by a representation of the sensory specific properties of an outcome, that has become associated with both the Pavlovian cue and the instrumental response during initial learning. Specific satiety is a common method of outcome devaluation that reduces an outcome's value but might also lead to the habituation of the outcome's sensory properties. Previous research has demonstrated that specific PIT is insensitive to changes in specific outcome value following taste aversion devaluation, as well as general satiety manipulations, and therefore specific satiety should not disrupt specific PIT by reducing outcome value. The present rodent experiments used a specific satiety devaluation procedure immediately prior to a specific PIT test to show that habituation of these outcome specific sensory representations can disrupt its efficacy as a stimulus and abolish the specific PIT effect. Experiment 1 employed a two-lever choice test to show that a non-devalued stimulus supports specific PIT, whereas a devalued stimulus abolished the specific PIT effect. Experiment 2 replicated this procedure while controlling for response competition by using a single-lever test to confirm that a devalued stimulus abolishes the specific PIT effect. These findings demonstrate that specific satiety can disrupt the ability of an outcome specific representation to support specific PIT. Given previous findings that specific PIT is insensitive to changes in outcome value by general satiety and taste aversion devaluation, this suggests that specific satiety devaluation might disrupt the use of sensory specific outcome representations to guide behavior via a mechanism that is independent of the outcome's current value.

Abnormal Brain Networks Related to Drug and Nondrug Reward Anticipation and Outcome Processing in Stimulant Use Disorder: A Functional Connectomics Approach

BACKGROUND

Drug addiction is associated with blunted neural responses to nondrug rewards, such as money, but heightened responses to drug cues that predict drug-reward outcomes. This dissociation underscores the role of incentive context in the attribution of motivational salience, which may reflect a narrowing toward drug-related goals. This hypothesis, however, has scarcely been investigated.

METHODS

To address this important scientific gap, the current study performed an empirical assessment of differences in salience attribution by comparing patients with stimulant use disorder (SUD) (n = 41) with control participants (n = 48) on network connectivity related to anticipation and outcome processing using a modified monetary incentive delay task. We hypothesized increased task-related activation and connectivity to drug rewards in patients with SUD, and reduced task-related activation and connectivity to monetary rewards during incentive processing across brain networks.

RESULTS

In the presence of behavioral and regional brain activation similarities, we found that patients with SUD showed significantly less connectivity involving three separate distributed networks during monetary reward anticipation, and drug and monetary reward outcome processing. No group connectivity differences for drug reward anticipation were identified. Additional graph theory analyses revealed that patients with SUD had longer path lengths across these networks, all of which positively correlated with the duration of stimulant drug use.

CONCLUSIONS

Specific disruptions in connectivity in networks related to the anticipation of nondrug reward together with more general dysconnectivity in the processing of rewarding outcomes suggest an insensitivity to consequences. These observations support the notion of a predominance of habitual control in patients with SUD.

General Pavlovian-to-instrumental transfer in humans: Evidence from Bayesian inference

When repeatedly paired with rewarding outcomes (i.e., Pavlovian conditioning), environmental cues may acquire predictive and motivational significance and later enhance instrumental responding for the same (i.e., outcome-specific transfer) or motivationally similar (i.e., general transfer) outcomes. Although outcome-specific and general Pavlovian-to-Instrumental Transfer (PIT) are characterized by different neural substrates and behavioral mechanisms, general transfer has never been studied in isolation from outcome-specific transfer in humans. The first aim of the present study was to test whether the general transfer effect could emerge in isolation and independently of outcome-specific transfer. Our results showed that general transfer can be elicited without the concurrent presence of outcome-specific transfer, supporting the idea that outcome-specific and general transfer can be studied independently of each other. The second aim of the present study was to clarify whether the affordance-like properties of the outcomes can affect the general transfer. In fact, a critical difference in current studies on general transfer concerns the use of cues associated with outcomes for which an action was previously learned (or not) during the instrumental training. This apparently minor difference affects the affordance-like properties of the outcome and may also be transferred to the cue, in turn impacting general transfer. Results revealed a general transfer of the same magnitude regardless of whether cues were associated with reward earned or not during instrumental conditioning. These findings increase the current knowledge on the incentive motivational mechanism behind general transfer, indicating that it is independent of the motor features of the outcome.

Sensory-Specific Satiety Dissociates General and Specific Pavlovian-Instrumental Transfer

Pavlovian conditioning enables predictive stimuli to control action performance and action selection. The present experiments used sensory-specific satiety to examine the role of outcome value in these two forms of control. Experiment 1 employed a general Pavlovian-instrumental transfer design to show that a stimulus predicting a food outcome energizes the performance of an instrumental action earning another food outcome. This energizing effect was removed when the stimulus-predicted outcome or a novel outcome was devalued by sensory-specific satiety. Experiments 2 and 3 employed a specific Pavlovian-instrumental transfer design to demonstrate that a stimulus predicting a particular food outcome promotes the selection of an instrumental action earning the same, but not a different, food outcome. Remarkably, this effect was maintained when all or just one of the stimulus-predicted outcomes were devalued by sensory-specific satiety. These results indicate that satiety alone removes the expression of general PIT. By contrast, satiety or outcome-specific devaluation does not regulate the expression of specific PIT, which is insensitive to changes in outcome value. This dissociation is consistent with the view that general and specific PIT are two separate phenomena driven by distinct psychological mechanisms.

Effects of Motivational Downshifts on Specific Pavlovian-Instrumental Transfer in Rats

BACKGROUND

Pavlovian stimuli predictive of appetitive outcomes can exert a powerful influence on the selection and initiation of action, a phenomenon termed outcome-selective Pavlovian-instrumental transfer (sPIT). Rodent studies suggest that sPIT is insensitive to motivational downshift induced by outcome devaluation, an effect that is, however, relatively underexplored.

METHODS

Here we examined in detail the effects of distinct shifts in motivation from hunger to a state of relative satiety on sPIT in rats.

RESULTS

A motivational downshift by outcome-specific devaluation immediately prior to testing markedly reduced overall lever responding and magazine entries but left intact the sPIT effect. A motivational downshift prior testing by (1) giving ad libitum rather than restricted access to maintenance diet in the home cage for 24 hours or by (2) a systemic blockade of hormone secretagogue receptor subtype 1A receptors to inhibit orexigenic actions of ghrelin both reduced overall lever responding and magazine entries. Moreover, these latter motivational downshifts reduced the sPIT effect; however, the sizes of the sPIT effects were still large.

CONCLUSIONS

Collectively, our rodent findings indicate that major effects of various motivational downshifts are overall inhibition of lever pressing and magazine approach, possibly reflecting reduced general motivation. The observed effects of motivational downshifts on sPIT have implications with regard to the role of general motivating effects in sPIT and to the contribution of Pavlovian-instrumental interactions to excessive food seeking as well as obesity in humans.

Determining the effects of training duration on the behavioral expression of habitual control in humans: a multilaboratory investigation

It has been suggested that there are two distinct and parallel mechanisms for controlling instrumental behavior in mammals: goal-directed actions and habits. To gain an understanding of how these two systems interact to control behavior, it is essential to characterize the mechanisms by which the balance between these systems is influenced by experience. Studies in rodents have shown that the amount of training governs the relative expression of these two systems: Behavior is goal-directed following moderate training, but the more extensively an instrumental action is trained, the more it becomes habitual. It is less clear whether humans exhibit similar training effects on the expression of goal-directed and habitual behavior, as human studies have reported contradictory findings. To tackle these contradictory findings, we formed a consortium, where four laboratories undertook a preregistered experimental induction of habits by manipulating the amount of training. There was no statistical evidence for a main effect of the amount of training on the formation and expression of habits. However, exploratory analyses suggest a moderating effect of the affective component of stress on the impact of training over habit expression. Participants who were lower in affective stress appeared to be initially goal-directed, but became habitual with increased training, whereas participants who were high in affective stress were already habitual even after moderate training, thereby manifesting insensitivity to overtraining effects. Our findings highlight the importance of the role of moderating variables such as individual differences in stress and anxiety when studying the experimental induction of habits in humans.

Metabotropic group II glutamate receptors in the basolateral amygdala mediate cue-triggered increases in incentive motivation

RATIONALE

Reward-associated cues can trigger incentive motivation for reward and invigorate reward-seeking behaviour via Pavlovian-to-instrumental transfer (PIT). Glutamate signaling within the basolateral amygdala (BLA) modulates cue-triggered increases in incentive motivation. However, the role of BLA metabotropic group II glutamate (mGlu_2/3) receptors is largely unknown.

OBJECTIVES

In Experiment 1, we characterized cue-triggered increases in incentive motivation for water reward using the PIT paradigm. In Experiment 2, we assessed the influence of intra-BLA microinjections of the mGlu_2/3 receptor agonist LY379268 on this effect.

METHODS

Water-restricted male Sprague-Dawley rats learned to press a lever for water. Separately, they learned to associate one of two auditory cues with free water. On test days, rats could lever press under extinction conditions (no water), with intermittent, non-contingent CS+ and CS- presentations. In Experiment 1, rats were tested under baseline conditions. In Experiment 2, rats received intra-BLA microinjections of LY379268 (0, 3 and 6 [Formula: see text]g/hemisphere) before testing.

RESULTS

Across experiments, CS+, but not CS-, presentations increased water-associated lever pressing during testing, even though responding was reinforced neither by water nor the CS+. Intra-BLA LY379268 abolished both CS+ potentiated pressing on the water-associated lever and CS+ evoked conditioned approach to the site of water delivery. LY379268 did not influence locomotion or instrumental and Pavlovian response rates during intervals between CS presentations or during the CS-, indicating no motor effects.

CONCLUSIONS

mGlu_2/3 receptor activity in the BLA mediates cue-triggered potentiation of incentive motivation for reward, suppressing both cue-induced increases in instrumental pursuit of the reward and anticipatory approach behaviour.

Problematic eating as an issue of habitual control

Obesity has reached alarming rates worldwide. Although many people attempt to control weight by modifying their food-related behaviours, this typically only has short-term effects and most dieters regain the weight that was lost. Why do so many people struggle to regulate their food-related behaviours? One possible explanation is that these behaviours have become habits that are not immediately sensitive to their consequences. Here we review experimental evidence for a shift to habitual control over food-related behaviours and the neural systems that control them and how this relates to difficulty changing ones' eating behavior.

Is the habit system altered in individuals with obesity? A systematic review

Habit-like eating behavior is repeatedly pointed to as a key cognitive mechanism contributing to the emergence and maintenance of obesity. Here, we conducted a systematic review of the literature to assess the existent behavioral evidence for the Habit Hypothesis for Overeating (HHO) which states that obesity is the consequence of an imbalance between the habit and goal-directed reward learning systems, leading to overconsumption of food. We found a total of 19 studies implementing a variety of experimental protocols (i.e., free operant paradigm, slips-of-action test, two-step task, Pavlovian-to-Instrumental paradigm, probabilistic learning task) and manipulations. Taken together, the studies on clinical (binge eating disorder) and non-clinical individuals with overweight or obesity do not support the HHO conclusively. While the scientific literature on HHO is still in its infancy, the heterogeneity of the extant studies makes it difficult to evaluate the degree of convergence of these findings. Uncovering the role of reward learning systems in eating behaviors might have a transformative impact on public health.

Helpless against food cues: the influence of pro- and anti-sugar videos on instrumental food-seeking behaviour in a Pavlovian-to-instrumental transfer paradigm

OBJECTIVE

External food cues can trigger food seeking by means of associative Stimulus-Outcome-Response learning mechanisms. These mechanisms can contribute to cued overeating. The present study aims at investigating if (cued) food-seeking behaviour can be influenced by pro- and anti-sugar videos.

DESIGN

Participants (N = 81) completed a Pavlovian-to-instrumental transfer (PIT) task: in an instrumental training, they learned associations between button presses and resulting sugary or sugar-free snacks. In a subsequent Pavlovian training, the snacks were paired with different cues. During the following transfer test, participants performed free button presses to win snacks while the cues were present or not.

MAIN OUTCOME MEASURES

The number of button presses for the different snacks in the transfer test was analysed.

RESULTS

We observed an outcome specific PIT effect, i.e. higher response rates for cued snacks. The videos did not affect the PIT effect. However, exploratory analyses revealed that the anti-sugar video led to fewer button presses for sugary snacks (compared to the pro-sugar video).

CONCLUSION

While snack-seeking behaviour was unaffected by the video's messages in the presence of food cues, in the absence of food cues there was evidence for a reduction of sugary snack choices by the anti-sugar video.

The Differential Effects of the Amount of Training on Sensitivity of Distinct Actions to Reward Devaluation

Shifting between goal-directed and habitual behaviors is essential for daily functioning. An inability to do so is associated with various clinical conditions, such as obsessive-compulsive disorder (OCD). Here we developed a new behavioral model in mice allowing us to produce and examine the development of different behaviors under goal-directed or habitual control. By using overtraining of instrumental associations between two levers and two rewards, and later devaluating one of the rewards, we differentiate and explore the motivational control of behaviors within the task which consequentially promotes what seems like excessive irrational behavior. Using our model, we found that the ability of instrumental behavior, to adapt to a change in the value of a known reward, is a function of practice. Once an instrumental action was practiced extensively it becomes habitual and, thus, under S-R control and could not be amended, not even when resulting in a noxious outcome. However, direct consummatory or Pavlovian actions, such as licking or checking, responds immediately to the change in value. This imbalance could render an instrumental behavior excessive and unresponsive to changes in outcome while the direct change in consumption implies that the change was in fact registered. This could suggest a system that, when out of balance, can create excessive behaviors, not adapting to an acknowledged change.

Probing the decision-making mechanisms underlying choice between drug and nondrug rewards in rats

Delineating the decision-making mechanisms underlying choice between drug and nondrug rewards remains a challenge. This study adopts an original approach to probe these mechanisms by comparing response latencies during sampling versus choice trials. While lengthening of latencies during choice is predicted in a deliberative choice model (DCM), the race-like response competition mechanism postulated by the Sequential choice model (SCM) predicts a shortening of latencies during choice compared to sampling. Here, we tested these predictions by conducting a retrospective analysis of cocaine-versus-saccharin choice experiments conducted in our laboratory. We found that rats engage deliberative decision-making mechanisms after limited training, but adopt a SCM-like response selection mechanism after more extended training, while their behavior is presumably habitual. Thus, the DCM and SCM may not be general models of choice, as initially formulated, but could be dynamically engaged to control choice behavior across early and extended training.

How predictive learning influences choice: Evidence for a GPCR-based memory process necessary for Pavlovian-instrumental transfer

Predictive learning endows stimuli with the capacity to signal both the sensory-specific and general motivational properties of their associated rewards or outcomes. These two signals can be distinguished behaviorally by their influence on the selection and performance of instrumental actions, respectively. This review focuses on how sensory-specific predictive learning guides choice between actions that earn otherwise equally desirable outcomes. We describe evidence that outcome-specific predictive learning is encoded in the basolateral amygdala and drives the accumulation of delta-opioid receptors on the surface of cholinergic interneurons located in the nucleus accumbens shell. This accumulation constitutes a novel form of cellular memory, not for outcome-specific predictive learning per se but for the selection of, and choice between, future instrumental actions. We describe recent evidence regarding the cascade of events necessary for the formation and expression of this cellular memory and point to open questions for future research into this process. Beyond these mechanistic considerations, the discovery of this new form of memory is consistent with recent evidence suggesting that intracellular rather than synaptic changes can mediate learning-related plasticity to modify brain circuitry to prepare for future significant events.

Habit, choice, and addiction

Addiction was suggested to emerge from the progressive dominance of habits over goal-directed behaviors. However, it is generally assumed that habits do not persist in choice settings. Therefore, it is unclear how drug habits may persist in real-world scenarios where this factor predominates. Here, we discuss the poor translational validity of the habit construct, which impedes our ability to determine its role in addiction. New evidence of habitual behavior in a drug choice setting are then described and discussed. Interestingly, habitual preference did not promote drug choice but instead favored abstinence. Here, we propose several clues to reconcile these unexpected results with the habit theory of addiction, and we highlight the need in experimental research to face the complexity of drug addicts' decision-making environments by investigating drug habits in the context of choice and in the presence of cues. On a theoretical level, we need to consider more complex frameworks, taking into account continuous interactions between goal-directed and habitual systems, and alternative decision-making models more representative of real-world conditions.

Associative processes in addiction relapse models: A review of their Pavlovian and instrumental mechanisms, history, and terminology

Animal models of relapse to drug-seeking have borrowed heavily from associative learning approaches. In studies of relapse-like behaviour, animals learn to self-administer drugs then receive a period of extinction during which they learn to inhibit the operant response. Several triggers can produce a recovery of responding which form the basis of a variety of models. These include the passage of time (spontaneous recovery), drug availability (rapid reacquisition), extinction of an alternative response (resurgence), context change (renewal), drug priming, stress, and cues (reinstatement). In most cases, the behavioural processes driving extinction and recovery in operant drug self-administration studies are similar to those in the Pavlovian and behavioural literature, such as context effects. However, reinstatement in addiction studies have several differences with Pavlovian reinstatement, which have emerged over several decades, in experimental procedures, associative mechanisms, and terminology. Interestingly, in cue-induced reinstatement, drug-paired cues that are present during acquisition are omitted during lever extinction. The unextinguished drug-paired cue may limit the model’s translational relevance to cue exposure therapy and renders its underlying associative mechanisms ambiguous. We review major behavioural theories that explain recovery phenomena, with a particular focus on cue-induced reinstatement because it is a widely used model in addiction. We argue that cue-induced reinstatement may be explained by a combination of behavioural processes, including reacquisition of conditioned reinforcement and Pavlovian to Instrumental Transfer. While there are important differences between addiction studies and the behavioural literature in terminology and procedures, it is clear that understanding associative learning processes is essential for studying relapse.

Motivational competition and the paraventricular thalamus

Although significant progress has been made in understanding the behavioral and brain mechanisms for motivational systems, much less is known about competition between motivational states or motivational conflict (e.g., approach - avoidance conflict). Despite being produced under diverse conditions, behavior during motivational competition has two signatures: bistability and metastability. These signatures reveal the operation of positive feedback mechanisms in behavioral selection. Different neuronal architectures can instantiate this selection to achieve bistability and metastability in behavior, but each relies on circuit-level inhibition to achieve rapid and stable selection between competing tendencies. Paraventricular thalamus (PVT) is identified as critical to this circuit level inhibition, resolving motivational competition via its extensive projections to local inhibitory networks in the ventral striatum and extended amygdala, enabling adaptive responding under motivational conflict.

Modulation of cue-guided choices by transcranial direct current stimulation

Environmental cues may anticipate the availability of rewards, thus acting as a guide towards a specific choice (i.e., cue-guided choices). Despite the lateral prefrontal cortex having a critical role in using past learning and flexibly selecting relevant information to guide behavior, the literature on the neural basis of human cue-guided choice mainly focused on the subcortical brain structures implicated, while the specific role of cortical areas remained unclear. The present study aimed to provide causal evidence for the involvement of the lateral prefrontal cortex in two forms of human cue-guided choice, namely outcome-specific and general. To do this, 2 mA cathodal, anodal or sham transcranial direct current stimulation was applied over the lateral prefrontal cortex (with the posterior parietal cortex serving as control region) in three separate groups performing a Pavlovian-to-Instrumental Transfer task. Results showed, for the first time, a dissociation in the cortical structures involved in human cue-guided choice. Cathodal stimulation of the lateral prefrontal cortex reduced the outcome-specific transfer. In striking contrast, there was no influence on the general transfer. These results argue in favor of the presence of at least two possible neural pathways underlying cue-guided choices.

Les addictions sous l’angle neurocognitif

Les principaux modèles neurocognitifs conçoivent l’addiction comme le résultat d’un déséquilibre de trois systèmes cérébraux en constante interaction : un circuit striatoamygdalien qui favorise les comportements automatiques, habituels et saillants, le cortex préfrontal impliqué dans la prise de décision et dans le contrôle inhibiteur. Enfin, la région insulaire permet un traitement proprioceptif à l’origine d’états émotionnels conscients, ce qui exerce une influence sur la prise de décision. L’influence du stress ainsi que les perspectives cliniques sont discutées.

Motivational factors underlying aversive Pavlovian-instrumental transfer

While interest in active avoidance has recently been resurgent, many concerns relating to the nature of this form of learning remain unresolved. By separating stimulus and response acquisition, aversive Pavlovian-instrumental transfer can be used to measure the effect of avoidance learning on threat processing with more control than typical avoidance procedures. However, the motivational substrates that contribute to the aversive transfer effect have not been thoroughly examined. In three studies using rodents, the impact of a variety of aversive signals on shock-avoidance responding (i.e., two-way shuttling) was evaluated. Fox urine, as well as a tone paired with the delivery of the predator odor were insufficient modulatory stimuli for the avoidance response. Similarly, a signal for the absence of food did not generate appropriate aversive motivation to enhance shuttling. Only conditioned Pavlovian stimuli that had been paired with unconditioned threats were capable of augmenting shock-avoidance responding. This was true whether the signaled outcome was the same (e.g., shock) or different (e.g., klaxon) from the avoidance outcome (i.e., shock). These findings help to characterize the aversive transfer effect and provide a more thorough analysis of its generalization to warning signals for different kinds of threats. This feature of aversive motivation has not been demonstrated using conventional avoidance procedures and could be potentially useful for applying avoidance in treatment settings.

Reward-predictive cues elicit excessive reward seeking in adolescent rats

Impulsive behavior during adolescence may stem from developmental imbalances between motivational and cognitive-control systems, producing greater urges to pursue reward and weakened capacities to inhibit such actions. Here, we developed a Pavlovian-instrumental transfer (PIT) protocol to assay rats' ability to suppress cue-motivated reward seeking based on changes in reward expectancy. Traditionally, PIT studies focus on how reward-predictive cues motivate instrumental reward-seeking behavior (lever pressing). However, cues signaling imminent reward delivery also elicit countervailing focal-search responses (food-port entry). We first examined how reward expectancy (cue-reward probability) influences expression of these competing behaviors. Adult male rats increased rates of lever pressing when presented with cues signaling lower probabilities of reward but focused their activity at the food cup on trials with cues that signaled higher probabilities of reward. We then compared adolescent and adult male rats in their responsivity to cues signaling different reward probabilities. In contrast to adults, adolescent rats did not flexibly adjust patterns of responding based on the expected likelihood of reward delivery but increased their rate of lever pressing for both weak and strong cues. These findings indicate that control over cue-motivated behavior is fundamentally dysregulated during adolescence, providing a model for studying neurobiological mechanisms of adolescent impulsivity.

Adolescents exhibit reduced Pavlovian biases on instrumental learning

Multiple learning systems allow individuals to flexibly respond to opportunities and challenges present in the environment. An evolutionarily conserved "Pavlovian" learning mechanism couples valence and action, promoting a tendency to approach cues associated with reward and to inhibit action in the face of anticipated punishment. Although this default response system may be adaptive, these hard-wired reactions can hinder the ability to learn flexible "instrumental" actions in pursuit of a goal. Such constraints on behavioral flexibility have been studied extensively in adults. However, the extent to which these valence-specific response tendencies bias instrumental learning across development remains poorly characterized. Here, we show that while Pavlovian response biases constrain flexible action learning in children and adults, these biases are attenuated in adolescents. This adolescent-specific reduction in Pavlovian bias may promote unbiased exploration of approach and avoidance responses, facilitating the discovery of rewarding behavior in the many novel contexts that adolescents encounter.

Computational models of drug use and addiction: A review

In this brief review, we describe current computational models of drug-use and addiction that fall into 2 broad categories: mathematically based models that rely on computational theories, and brain-based models that link computations to brain areas or circuits. Across categories, many are models of learning and decision-making, which may be compromised in addiction. Several mathematical models take predictive coding approaches, focusing on Bayesian prediction error. Other models focus on learning processes and (traditional) prediction error. Brain-based models have incorporated prefrontal cortex, basal ganglia, and the dopamine system, based on the effects of drugs on dopamine, motivation, and executive control circuits. Several models specifically describe how behavioral control may transition from habitual to goal-directed systems, consistent with computational accounts of compromised "model-based" control. Some brain-based models have linked this to the transition of behavioral control from ventral to dorsal striatum. Overall, we propose that while computational models capture some aspects of addiction and have advanced our thinking, most have focused on the effects of drug use rather than addiction per se, most have not been tested on and/or supported by human data, and few capture multiple stages and symptoms of addiction. We conclude by suggesting a path forward for computational models of addiction. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Subliminal determinants of cue-guided choice

By anticipating potential rewards, external cues can guide behavior to achieve a goal. Whether the conscious elaboration of these cues is necessary to elicit cue-guided choices is still unknown. The goal of the present study is to test whether the subliminal presentation of a visual cue previously paired with a reward is sufficient to bias responses that can lead to the same or a similar reward. To this aim, three experiments compared the subliminal and supraliminal presentation of reward-associated cues during a Pavlovian-to-Instrumental Transfer task. In line with previous evidence, results showed that the supraliminal presentation of reward-associated Pavlovian cues biased participant’s choice towards motivationally similar rewards (general transfer) as well as towards rewards sharing the precise sensory-specific properties of the cue (outcome-specific transfer). In striking contrast, subliminal cues biased choice only towards motivationally similar rewards (general transfer). Taken together, these findings suggest that cue-guided choices are modulated by the level of perceptual threshold (i.e., subliminal vs supraliminal) of reward-associated cues. Although conscious elaboration of the cue is necessary to guide choice towards a specific reward, subliminal processing is still sufficient to push towards choices sharing the motivational properties of the cue. Implications for everyday life, clinical conditions, and theoretical accounts of cue-guided choices are discussed.

Habitual Behavior Is Goal-Driven

We address the relation between goal-driven and habitual behaviors. Whereas in recent years the two have been juxtaposed, we suggest that habitual behavior is in fact goal-driven. To support this view, we show that habitual behavior is sensitive to changes in goal properties (reward contingencies), namely goal value and its expectancy of attainment. Whereas adjustment to these properties may be slower for habitual (or overlearned) than for nonhabitual behavior, this is likely due to the routinized (or automatic) nature of such behavior, characterized as it is by reduced attention to its consequences. Furthermore, we show that habitual behavior’s prolonged persistence despite its manifest detachment from the original goal likely stems from its attachment to a different goal. Thus, there is no need to postulate purposeless behavior. The view that habitual behavior is goal-driven offers an integrative account of a considerable body of evidence and is consistent with a functional account of psychological processes.

Habitual Preference for the Nondrug Reward in a Drug Choice Setting

For adaptive and efficient decision making, it must be possible to select between habitual alternative courses of action. However, research in rodents suggests that, even in the context of simple decision-making, choice behavior remains goal-directed. In contrast, we recently found that during discrete trial choice between cocaine and water, water-restricted rats preferred water and this preference was habitual and inflexible (i.e., resistant to water devaluation by satiation). Here we sought to test the reproducibility and generality of this surprising finding by assessing habitual control of preference for saccharin over cocaine in non-restricted rats. Specifically, after the acquisition of preference for saccharin, saccharin was devalued and concurrent responding for both options was measured under extinction. As expected, rats responded more for saccharin than for cocaine during extinction, but this difference was unaffected by saccharin devaluation. Together with our previous research, this result indicates that preference for nondrug alternatives over cocaine is under habitual control, even under conditions that normally support goal-directed control of choice between nondrug options. The possible reasons for this difference are discussed.

The transition to compulsion in addiction

Compulsion is a cardinal symptom of drug addiction (severe substance use disorder). However, compulsion is observed in only a small proportion of individuals who repeatedly seek and use addictive substances. Here, we integrate accounts of the neuropharmacological mechanisms that underlie the transition to compulsion with overarching learning theories, to outline how compulsion develops in addiction. Importantly, we emphasize the conceptual distinctions between compulsive drug-seeking behaviour and compulsive drug-taking behaviour (that is, use). In the latter, an individual cannot stop using a drug despite major negative consequences, possibly reflecting an imbalance in frontostriatal circuits that encode reward and aversion. By contrast, an individual may compulsively seek drugs (that is, persist in seeking drugs despite the negative consequences of doing so) when the neural systems that underlie habitual behaviour dominate goal-directed behavioural systems, and when executive control over this maladaptive behaviour is diminished. This distinction between different aspects of addiction may help to identify its neural substrates and new treatment strategies.

Unexpected food outcomes can return a habit to goal-directed action

Three experiments examined the return of a habitual instrumental response to the status of goal-directed action. In all experiments, rats received extensive training in which lever pressing was reinforced with food pellets on a random-interval schedule of reinforcement. In Experiment 1, the extensively-trained response was not affected by conditioning a taste aversion to the reinforcer, and was therefore considered a habit. However, if the response had earned a new and unexpected food pellet during the final training session, the response was affected by taste aversion conditioning to the (first) reinforcer, and had thus been converted to a goal-directed action. In Experiment 3, 30 min of prefeeding with an irrelevant food pellet immediately before the test also converted a habit back to action, as judged by the taste-aversion devaluation method. That result was consistent with difficulty in finding evidence of habit with the sensory-specific satiety method after extensive instrumental training (Experiment 2). The results suggest that an instrumental behavior's status as a habit is not permanent, and that a habit can be returned to action status by associating it with a surprising reinforcer (Experiment 1) or by giving the animal an unexpected prefeeding immediately prior to the action/habit test (Experiment 3).

Some factors that restore goal-direction to a habitual behavior

Recent findings from our laboratory suggest that an extensively-practiced instrumental behavior can appear to be a goal-directed action (rather than a habit) when a second behavior is added and reinforced during intermixed final sessions (Shipman et al., 2018). The present experiments were designed to explore and understand this finding. All used the taste aversion method of devaluing the reinforcer to distinguish between goal-directed actions and habits. Experiment 1 confirmed that reinforcing a second response in a separate context (but not mere exposure to that context) can return an extensively-trained habit to the status of goal-directed action. Experiment 2 showed that training of the second response needs to be intermixed with training of the first response to produce this effect; training the second response after the first-response training was complete preserved the first response as a habit. Experiment 3 demonstrated that reinforcing the second response with a different reinforcer breaks the habit status of the first response. Experiment 4 found that free reinforcers (that were not response-contingent) were sufficient to restore goal-directed performance. Together, the results suggest that unexpected reinforcer delivery can render a habitual response goal-directed again.

Dysfunctional approach behavior triggered by alcohol-unrelated Pavlovian cues predicts long-term relapse in alcohol dependence

We demonstrated that alcohol-dependent patients who relapsed within 1 year after detoxification showed stronger PIT effects compared with abstainers and controls. Relapsers particularly failed to correctly perform in trials where an instrumental stimulus required inhibition while a Pavlovian background cue indicated a monetary gain. Under that condition, relapsers approached the instrumental stimulus, independent of the expected punishment. The failure of inhibiting an aversive stimulus in favor of approaching an appetitive context cue reflects dysfunctional altered learning mechanisms in relapsers.