Relations Between Pavlovian-Instrumental Transfer and Reinforcer Devaluation

Predictive but not emotional value of Pavlovian stimuli leads to pavlovian-to-instrumental transfer

•

Pavlovian stimuli (CSs) of different reward value induce outcome specific transfer.

•

Knowledge of the predicted outcome appears necessary for the transfer to occur.

•

Emotional response to CSs occurs without knowledge of predicted outcome.

•

Emotional response to CSs is not sufficient to elicit transfer.

Reward-predictive stimuli augment instrumental reward-seeking in humans, an effect denoted Pavlovian-to-instrumental transfer (PIT), but the role of differential reward value of these stimuli and of emotional conditioned responses in PIT remains unknown.

Fifty one participants experienced a Pavlovian phase that associated two stimuli with either 10p (CS10) or 50p (CS50). Next, participants underwent instrumental training for two responses reinforced with either 10p or 50p. Finally, the transfer phase continued as had instrumental training, now in the presence of the Pavlovian stimuli.

Participants were dichotomised as aware/unaware according to their expectancy awareness of the CS → outcome associations. Only aware participants demonstrated PIT (increased choice and number of responses on the 10p and 50p response key in the presence of CS10 and CS50 respectively), yet both aware and unaware groups rated the 50p stimulus as more pleasant than the 10p stimulus.

These findings suggest that expectancy of reward is necessary for PIT; however, emotional conditioned responses appear not sufficient to influence PIT. Future research should attempt to manipulate emotional conditioned responses in a PIT context, to test the sufficiency of reward expectancy in PIT.

Effects of an ethanol-paired CS on responding for ethanol and food: Comparisons with a stimulus in a Truly-Random-Control group and to a food-paired CS on responding for food

Motivational increases due to exposure to alcohol-paired Conditioned Stimuli (CS) are central to some accounts of alcoholism. However, few studies isolate a stimulus's function as a CS from its other potential functions. Pavlovian-Instrumental-Transfer (PIT) procedures isolate a stimulus's function as a CS from its other functions. Though there are several relevant studies using PIT, knowledge gaps exist. Particularly, it is not clear that an alcohol-paired CS will increase alcohol seeking compared to the same stimulus in a Truly-Random-Control group, nor whether such increases are specific to alcohol seeking. To address these knowledge gaps in Experiment 1, rats responded for ethanol (0.1 ml 8% w/v) under an RI 30-sec schedule, then the lever was removed and half the rats had ethanol delivered during occasional 120-sec light presentations, while the remainder had ethanol and the light presented under independent RT schedules. Later the lever was returned and the light was presented during responding in extinction (PIT test). Following this test, levers were again removed and the light was presented without ethanol (light extinction), following again by a PIT test. Responding in the two groups during light presentations did not differ in either PIT test. Experiment 2 repeated Experiment 1 using food instead of ethanol. In Experiment 2, responding during light presentations increased in the paired group. In Experiment 3, rats were trained on a concurrent FR schedule of food and ethanol delivery. Ethanol was delivered following 5 responses and the response requirement for food adjusted so that similar numbers of food and ethanol deliveries were obtained. Subsequently, rats underwent conditioning, control and testing procedures identical to those in Experiment 1. In Experiment 3, the ethanol-paired CS increased ethanol-responding, but not food-responding. These results are most easily interpreted as changes in responding resulting from CS-elicited behavior rather than motivational changes. This interpretation is more compatible with some descriptions of the role of an alcohol-paired CS in alcoholism than others.

The propositional basis of cue-controlled reward seeking

Two experiments examined the role of propositional and automatic (ideomotor) processes in cue-elicited responding for rewarding outcomes (beer and chocolate). In a training phase, participants earned either chocolate or beer points by making one of two button-press responses. Rewards were indicated by the presentation of chocolate and beer pictures. On test, each trial began with a picture of beer or chocolate, or a blank screen, and choice of the beer versus chocolate response was assessed in the presence of these three pictures. Participants tended to choose the beer and chocolate response in the presence of the beer and chocolate pictures, respectively. In Experiment 1, instructions signalling that the pictures did not indicate which response would be rewarded significantly reduced the priming effect. In Experiment 2, instructions indicating that the pictures signified which response would not be rewarded resulted in a reversed priming effect. Finally, in both experiments, the priming effect correlated with self-reported beliefs that the cues signalled which response was more likely to be reinforced. These results suggest that cue-elicited response selection is mediated by a propositional belief regarding the efficacy of the response–outcome relationship, rather than an automatic ideomotor mechanism.

Instrumental Divergence and the Value of Control

A critical aspect of flexible choice is that alternative actions yield distinct consequences: Only when available action alternatives produce distinct outcome states does discrimination and selection between actions allow an agent to flexibly obtain the currently most desired outcome. Here, we use instrumental divergence - the degree to which alternative actions differ with respect to their outcome probability distributions - as an index of flexible instrumental control, and assess the influence of this novel decision variable on choice preference. In Experiment 1, when other decision variables, such as expected value and outcome entropy, were held constant, we found a significant preference for high instrumental divergence. In Experiment 2, we used an "auto- vs. self-play" manipulation to eliminate outcome diversity as a source of behavioral preferences, and to contrast flexible instrumental control with the complete absence of voluntary choice. Our results suggest that flexible instrumental control over decision outcomes may have intrinsic value.

Reassessing wanting and liking in the study of mesolimbic influence on food intake

Humans and animals such as rats and mice tend to overconsume calorie-dense foods, a phenomenon that likely contributes to obesity. One often-advanced explanation for why we preferentially consume sweet and fatty foods is that they are more "rewarding" than low-calorie foods. "Reward" has been subdivided into three interdependent psychological processes: hedonia (liking a food), reinforcement (formation of associations among stimuli, actions, and/or the food), and motivation (wanting the food). Research into these processes has focused on the mesolimbic system, which comprises both dopamine neurons in the ventral tegmental area and neurons in their major projection target, the nucleus accumbens. The mesolimbic system and closely connected structures are commonly referred to as the brain's "reward circuit." Implicit in this title is the assumption that "rewarding" experiences are generally the result of activity in this circuit. In this review, I argue that food intake and the preference for calorie-dense foods can be explained without reference to subjective emotions. Furthermore, the contribution of mesolimbic dopamine to food intake and preference may not be a general one of promoting or coordinating behaviors that result in the most reward or caloric intake but may instead be limited to the facilitation of a specific form of neural computation that results in conditioned approach behavior. Studies on the neural mechanisms of caloric intake regulation must address how sensory information about calorie intake affects not just the mesolimbic system but also many other forms of computation that govern other types of food-seeking and food-oriented behaviors.

Habitual Alcohol Seeking: Neural Bases and Possible Relations to Alcohol Use Disorders

Loss of flexible control over alcohol use may contribute to the development of alcohol use disorders. An increased contribution of response habits to alcohol-related behaviors may help explain this loss of control. Focusing on data from outcome devaluation and Pavlovian-instrumental transfer procedures, we review evidence for loss of goal-directed control over alcohol seeking and consumption drawing from both preclinical findings and clinical data where they exist. Over the course of extended alcohol self-administration and exposure, the performance of alcohol-seeking responses becomes less sensitive to reduction in the value of alcohol and more vulnerable to the influences of alcohol-predictive stimuli. These behavioral changes are accompanied by a shift in the corticostriatal circuits that control responding from circuits centered on the dorsomedial to those centered on the dorsolateral striatum. These changes in behavioral and neural control could help explain failures to abstain from alcohol despite intention to do so. Understanding and ultimately ameliorating these changes will aid development of more effective treatment interventions.

Safety out of control: dopamine and defence

We enjoy a sophisticated understanding of how animals learn to predict appetitive outcomes and direct their behaviour accordingly. This encompasses well-defined learning algorithms and details of how these might be implemented in the brain. Dopamine has played an important part in this unfolding story, appearing to embody a learning signal for predicting rewards and stamping in useful actions, while also being a modulator of behavioural vigour. By contrast, although choosing correct actions and executing them vigorously in the face of adversity is at least as important, our understanding of learning and behaviour in aversive settings is less well developed. We examine aversive processing through the medium of the role of dopamine and targets such as D2 receptors in the striatum. We consider critical factors such as the degree of control that an animal believes it exerts over key aspects of its environment, the distinction between 'better' and 'good' actual or predicted future states, and the potential requirement for a particular form of opponent to dopamine to ensure proper calibration of state values.

Motivational state, reward value, and Pavlovian cues differentially affect skilled forelimb grasping in rats

Motor skills represent high-precision movements performed at optimal speed and accuracy. Such motor skills are learned with practice over time. Besides practice, effects of motivation have also been shown to influence speed and accuracy of movements, suggesting that fast movements are performed to maximize gained reward over time as noted in previous studies. In rodents, skilled motor performance has been successfully modeled with the skilled grasping task, in which animals use their forepaw to grasp for sugar pellet rewards through a narrow window. Using sugar pellets, the skilled grasping task is inherently tied to motivation processes. In the present study, we performed three experiments modulating animals’ motivation during skilled grasping by changing the motivational state, presenting different reward value ratios, and displaying Pavlovian stimuli. We found in all three studies that motivation affected the speed of skilled grasping movements, with the strongest effects seen due to motivational state and reward value. Furthermore, accuracy of the movement, measured in success rate, showed a strong dependence on motivational state as well. Pavlovian cues had only minor effects on skilled grasping, but results indicate an inverse Pavlovian-instrumental transfer effect on movement speed. These findings have broad implications considering the increasing use of skilled grasping in studies of motor system structure, function, and recovery after injuries.

Nucleus accumbens core and shell are differentially involved in general and outcome-specific forms of Pavlovian-instrumental transfer with alcohol and sucrose rewards

Alcohol-associated stimuli contribute to relapse risk. Therefore, understanding the behavioural and neural mechanisms underlying the ability of such stimuli to promote alcohol-seeking is important for developing effective treatments for alcohol-use disorders. The Pavlovian-instrumental transfer (PIT) paradigm can be used to study the influence of Pavlovian cues on independently-trained instrumental responses earning reward. The effects can be either general, increasing the vigour of reward-related behaviours, or specific to responses that earn a common outcome. These different forms of PIT are mediated by distinct neural circuits involving the nucleus accumbens (NAC) core and shell, respectively. Here we examined the effects of pharmacological inactivation of either the NAC core or shell on PIT generated by alcohol-predictive and sucrose-predictive stimuli in rats. We found that presentations of a stimulus predicting sucrose enhanced responding for sucrose but not alcohol, suggesting an outcome-specific effect. In contrast, presentations of an alcohol-predictive stimulus enhanced responding for both alcohol and sucrose, suggesting a generally arousing effect. Inactivation of the NAC core reduced PIT and, in particular, the effect of the alcohol stimulus. Inactivation of the NAC shell reduced the specificity of the stimulus effects but left the ability of the stimuli to non-specifically invigorate responding intact, consistent with a role in mediating the specificity of PIT. Together, these results suggest that the NAC core plays a particularly important role in mediating the influence of alcohol-predictive cues on reward-seeking behaviours.

Don't Think, Just Feel the Music: Individuals with Strong Pavlovian-to-Instrumental Transfer Effects Rely Less on Model-based Reinforcement Learning

Behavioral choice can be characterized along two axes. One axis distinguishes reflexive, model-free systems that slowly accumulate values through experience and a model-based system that uses knowledge to reason prospectively. The second axis distinguishes Pavlovian valuation of stimuli from instrumental valuation of actions or stimulus-action pairs. This results in four values and many possible interactions between them, with important consequences for accounts of individual variation. We here explored whether individual variation along one axis was related to individual variation along the other. Specifically, we asked whether individuals' balance between model-based and model-free learning was related to their tendency to show Pavlovian interferences with instrumental decisions. In two independent samples with a total of 243 participants, Pavlovian-instrumental transfer effects were negatively correlated with the strength of model-based reasoning in a two-step task. This suggests a potential common underlying substrate predisposing individuals to both have strong Pavlovian interference and be less model-based and provides a framework within which to interpret the observation of both effects in addiction.

Cue-elicited food seeking is eliminated with aversive outcomes following outcome devaluation

In outcome-selective Pavlovian-to-instrumental transfer (PIT), stimuli that are predictive of specific outcomes prime instrumental responses that are associated with these outcomes. Previous human studies yielded mixed evidence in respect to whether the PIT effect is affected by a posttraining devaluation of an outcome, with the PIT effect being preserved after a devaluation of a primary reinforcer (food, drugs) but not following the devaluation of a secondary reinforcer (money). The present research examined whether outcome-selective transfer is eliminated when the devaluation of a primary (liquid) reinforcer is strong and aversive. Experiment 1 confirmed these expectations following a devaluation with bad tasting Tween 20. However, outcome-selective transfer was still observed when the earned (devalued) outcome was not consumed immediately after each test (Experiment 2). These results suggest that the capacity of a Pavlovian cue to motivate a specific response is affected by the incentive value of the shared outcome only when the devaluation yields an aversive outcome that is consumed immediately.

The antagonism of ghrelin alters the appetitive response to learned cues associated with food

The rapid increase in obesity may be partly mediated by an increase in the exposure to cues for food. Food-paired cues play a role in food procurement and intake under conditions of satiety. The mechanism by which this occurs requires characterization, but may involve ghrelin. This orexigenic peptide alters the response to food-paired conditioned stimuli, and neural responses to food images in reward nuclei. Therefore, we tested whether a ghrelin receptor antagonist alters the influence of food-paired cues on the performance of instrumental responses that earn food and the consumption of food itself using tests of Pavlovian-to-instrumental transfer (PIT) and cue potentiated feeding (CPF), respectively. Food-deprived rats received Pavlovian conditioning where an auditory cue was paired with delivery of sucrose solution followed by instrumental conditioning to lever press for sucrose. Following training, rats were given ad libitum access to chow. On test day, rats were injected with the ghrelin receptor antagonist GHRP-6 [D-Lys3] and then tested for PIT or CPF. Disrupting ghrelin signaling enhanced expression of PIT. In addition, GHRP-6 [D-Lys3] impaired the initiation of feeding behavior in CPF without influencing overall intake of sucrose. Finally, in PIT tested rats, enhanced FOS immunoreactivity was revealed following the antagonist in regions thought to underlie PIT; however, the antagonist had no effect on FOS immunoreactivity in CPF tested rats.

A Closer Look at the Effects of Repeated Cocaine Exposure on Adaptive Decision-Making under Conditions That Promote Goal-Directed Control

It has been proposed that compulsive drug seeking reflects an underlying dysregulation in adaptive behavior that favors habitual (automatic and inflexible) over goal-directed (deliberative and highly flexible) action selection. Rodent studies have established that repeated exposure to cocaine or amphetamine facilitates the development of habits, producing behavior that becomes unusually insensitive to a reduction in the value of its outcome. The current study more directly investigated the effects of cocaine pre-exposure on goal-directed learning and action selection using an approach that discourages habitual performance. After undergoing a 15-day series of cocaine (15 or 30 mg/kg, i.p.) or saline injections and a drug withdrawal period, rats were trained to perform two different lever-press actions for distinct reward options. During a subsequent outcome devaluation test, both cocaine- and saline-treated rats showed a robust bias in their choice between the two actions, preferring whichever action had been trained with the reward that retained its value. Thus, it appears that the tendency for repeated cocaine exposure to promote habit formation does not extend to a more complex behavioral scenario that encourages goal-directed control. To further explore this issue, we assessed how prior cocaine treatment would affect the rats' ability to learn about a selective reduction in the predictive relationship between one of the two actions and its outcome, which is another fundamental feature of goal-directed behavior. Interestingly, we found that cocaine-treated rats showed enhanced, rather than diminished, sensitivity to this action-outcome contingency degradation manipulation. Given their mutual dependence on striatal dopamine signaling, we suggest that cocaine's effects on habit formation and contingency learning may stem from a common adaptation in this neurochemical system.

Changes in the Influence of Alcohol-Paired Stimuli on Alcohol Seeking across Extended Training

Previous work has demonstrated that goal-directed control of alcohol-seeking and other drug-related behaviors is reduced following extended self-administration and drug exposure. Here, we examined how the magnitude of stimulus influences on responding changes across similar training and drug exposure. Rats self-administered alcohol or sucrose for 2 or 8 weeks. Previous work has shown that 8 weeks, but not 2 weeks of self-administration produces habitual alcohol seeking. Next, all animals received equivalent Pavlovian conditioning sessions where a discrete stimulus predicted the delivery of alcohol or sucrose. Finally, the impact of the stimuli on ongoing instrumental responding was examined in a Pavlovian-instrumental transfer (PIT) test. While a significant PIT effect was observed following 2 weeks of either alcohol or sucrose self-administration, the magnitude of this effect was greater following 8 weeks of training. The specificity of the PIT effect appeared unchanged by extended training. While it is well established that evaluation of the outcome of responding contributes less to behavioral control following extended training and/or drug exposure, our data indicate that reward-predictive stimuli have a stronger contribution to responding after extended training. Together, these findings provide insight into the factors that control behavior after extended drug use, which will be important for developing effective methods for controlling and ideally reducing these behaviors.

Prospective and Pavlovian mechanisms in aversive behaviour

Studying aversive behaviour is critical for understanding negative emotions and associated psychopathologies. However a comprehensive picture of the mechanisms underlying aversion is lacking, with associative learning theories focusing on Pavlovian reactions and decision-making theoretic approaches on prospective functions. We propose a computational model of aversion that combines goal-directed and Pavlovian forms of control into a unifying framework in which their relative importance is regulated by factors such as threat distance and controllability. Using simulations, we test whether the model can reproduce available empirical findings and discuss its relevance to understanding factors underlying negative emotions such as fear and anxiety. Furthermore, the specific method used to construct the model permits a natural mapping from its components to brain structure and function. Our model provides a basis for a unifying account of aversion that can guide empirical and interventional study contexts.

N-methyl-D-aspartate receptors in the ventral tegmental area mediate the excitatory influence of Pavlovian stimuli on instrumental performance

Pavlovian stimuli predictive of food can markedly amplify instrumental responding for food. This effect is termed Pavlovian-instrumental transfer (PIT). The ventral tegmental area (VTA) plays a key role in mediating PIT, however, it is yet unknown whether N-methyl-D-aspartate (NMDA)-type glutamate receptors in the VTA are involved in PIT. Here, we examined the effects of an NMDA-receptor blockade in the VTA on PIT. Immediately prior to PIT testing, rats were subjected to intra-VTA infusions of vehicle or of the NMDA-receptor antagonist 2-amino-5-phosphonopentanoic acid (AP-5) (1, 5 µg/side). In rats that received AP-5 at the lower dose, the PIT effect was intact, i.e. presentation of the Pavlovian stimulus enhanced instrumental responding. By contrast, in rats that received AP-5 at the higher dose, the PIT effect was blocked. The data suggest that NMDA receptors in the VTA mediate the activating effects of Pavlovian stimuli on instrumental responding.

The Relationship Between Specific Pavlovian Instrumental Transfer and Instrumental Reward Probability

Goal-directed behavior is influenced by environmental cues: in particular, cues associated with a reward can bias action choice toward actions directed to that same reward. This effect is studied experimentally as specific Pavlovian-instrumental transfer (specific PIT). We have investigated the hypothesis that cues associated to an outcome elicit specific PIT by rising the estimates of reward probability of actions associated to that same outcome. In other words, cues reduce the uncertainty on the efficacy of instrumental actions. We used a human PIT experimental paradigm to test the effects of two different instrumental contingencies: one group of participants had a 33% chance of being rewarded for each button press, while another had a 100% chance. The group trained with 33% reward probability showed a stronger PIT effect than the 100% group, in line with the hypothesis that Pavlovian cues linked to an outcome work by reducing the uncertainty of receiving it. The 100% group also showed a significant specific PIT effect, highlighting additional factors that could contribute to specific PIT beyond the instrumental training contingency. We hypothesize that the uncertainty about reward delivery due to testing in extinction might be one of these factors. These results add knowledge on how goal-directed behavior is influenced by the presence of environmental cues associated with a reward: such influence depends on the probability that we have to reach a reward, namely when there is less chance of getting a reward we are more influenced by cues associated with it, and vice versa.

The Origins and Organization of Vertebrate Pavlovian Conditioning

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.

Individual differences in the influence of task-irrelevant Pavlovian cues on human behavior

Pavlovian-to-instrumental transfer (PIT) refers to the process of a Pavlovian reward-paired cue acquiring incentive motivational proprieties that drive choices. It represents a crucial phenomenon for understanding cue-controlled behavior, and it has both adaptive and maladaptive implications (i.e., drug-taking). In animals, individual differences in the degree to which such cues bias performance have been identified in two types of individuals that exhibit distinct Conditioned Responses (CR) during Pavlovian conditioning: Sign-Trackers (ST) and Goal-Trackers (GT). Using an appetitive PIT procedure with a monetary reward, the present study investigated, for the first time, the extent to which such individual differences might affect the influence of reward-paired cues in humans. In a first task, participants learned an instrumental response leading to reward; then, in a second task, a visual Pavlovian cue was associated with the same reward; finally, in a third task, PIT was tested by measuring the preference for the reward-paired instrumental response when the task-irrelevant reward-paired cue was presented, in the absence of the reward itself. In ST individuals, but not in GT individuals, reward-related cues biased behavior, resulting in an increased likelihood to perform the instrumental response independently paired with the same reward when presented with the task-irrelevant reward-paired cue, even if the reward itself was no longer available (i.e., stronger PIT effect). This finding has important implications for developing individualized treatment for maladaptive behaviors, such as addiction.

The algorithmic anatomy of model-based evaluation

Despite many debates in the first half of the twentieth century, it is now largely a truism that humans and other animals build models of their environments and use them for prediction and control. However, model-based (MB) reasoning presents severe computational challenges. Alternative, computationally simpler, model-free (MF) schemes have been suggested in the reinforcement learning literature, and have afforded influential accounts of behavioural and neural data. Here, we study the realization of MB calculations, and the ways that this might be woven together with MF values and evaluation methods. There are as yet mostly only hints in the literature as to the resulting tapestry, so we offer more preview than review.

Differentiating neural systems mediating the acquisition vs. expression of goal-directed and habitual behavioral control

Considerable behavioral data indicate that operant actions can become habitual, as demonstrated by insensitivity to changes in the action-outcome contingency and in subjective outcome values. Notably, although several studies have investigated the neural substrates of habits, none has clearly differentiated the areas of the human brain that support habit formation from those that implement habitual control. We scanned participants with functional magnetic resonance imaging as they learned and performed an operant task in which the conditional structure of the environment encouraged either goal-directed encoding of the consequences of actions, or a habit-like mapping of actions to antecedent cues. Participants were also scanned during a subsequent assessment of insensitivity to outcome devaluation. We identified dissociable roles of the cerebellum and ventral striatum, across learning and test performance, in behavioral insensitivity to outcome devaluation. We also showed that the inferior parietal lobule (an area previously implicated in several aspects of goal-directed action selection, including the attribution of intent and awareness of agency) predicted sensitivity to outcome devaluation. Finally, we revealed a potential functional homology between the human subgenual cortex and rodent infralimbic cortex in the implementation of habitual control. In summary, our findings suggested a broad systems division, at the cortical and subcortical levels, between brain areas mediating the encoding and expression of action-outcome and stimulus-response associations.

Model-based and model-free Pavlovian reward learning: revaluation, revision, and revelation

Evidence supports at least two methods for learning about reward and punishment and making predictions for guiding actions. One method, called model-free, progressively acquires cached estimates of the long-run values of circumstances and actions from retrospective experience. The other method, called model-based, uses representations of the environment, expectations, and prospective calculations to make cognitive predictions of future value. Extensive attention has been paid to both methods in computational analyses of instrumental learning. By contrast, although a full computational analysis has been lacking, Pavlovian learning and prediction has typically been presumed to be solely model-free. Here, we revise that presumption and review compelling evidence from Pavlovian revaluation experiments showing that Pavlovian predictions can involve their own form of model-based evaluation. In model-based Pavlovian evaluation, prevailing states of the body and brain influence value computations, and thereby produce powerful incentive motivations that can sometimes be quite new. We consider the consequences of this revised Pavlovian view for the computational landscape of prediction, response, and choice. We also revisit differences between Pavlovian and instrumental learning in the control of incentive motivation.

Mechanisms of motivation-cognition interaction: challenges and opportunities

Recent years have seen a rejuvenation of interest in studies of motivation-cognition interactions arising from many different areas of psychology and neuroscience. The present issue of Cognitive, Affective, & Behavioral Neuroscience provides a sampling of some of the latest research from a number of these different areas. In this introductory article, we provide an overview of the current state of the field, in terms of key research developments and candidate neural mechanisms receiving focused investigation as potential sources of motivation-cognition interaction. However, our primary goal is conceptual: to highlight the distinct perspectives taken by different research areas, in terms of how motivation is defined, the relevant dimensions and dissociations that are emphasized, and the theoretical questions being targeted. Together, these distinctions present both challenges and opportunities for efforts aiming toward a more unified and cross-disciplinary approach. We identify a set of pressing research questions calling for this sort of cross-disciplinary approach, with the explicit goal of encouraging integrative and collaborative investigations directed toward them.

Differential vulnerability to the punishment of cocaine related behaviours: effects of locus of punishment, cocaine taking history and alternative reinforcer availability

BACKGROUND

The availability of alternative reinforcement has been shown to reduce drug use, but it remains unclear whether it facilitates a reduction or cessation of drug seeking or taking.

OBJECTIVES

We compared the effects of punishment of cocaine seeking or taking behaviour after brief or extended cocaine-taking histories when behavioural reallocation was facilitated or not by making available an alternative ingestive reinforcer (sucrose).

METHODS

In the first experiment, punishment of either seeking or taking responses was introduced immediately after training on the seeking-taking chained schedule. In the second experiment, punishment of cocaine seeking was introduced after 12 additional days of either 1 or 6 h daily access to cocaine self-administration. In both experiments, beginning 1 week before the introduction of punishment, a subset of rats had concurrent nose poke access to sucrose while seeking or taking cocaine.

RESULTS

The presence of an alternative source of reinforcement markedly facilitated behavioural reallocation from punished cocaine taking after acquisition. It also facilitated punishment-induced suppression of cocaine seeking after an extensive cocaine self-administration history likely by prompting goal-directed motivational control over drug use. However, a significant proportion of rats were deemed compulsive-maintaining drug use after an extensive cocaine history despite the presence of abstinence-promoting positive and negative incentives.

CONCLUSION

Making available an alternative reinforcer facilitates disengagement from punished cocaine use through at least two different processes but remains ineffective in a subpopulation of vulnerable animals, which continued to seek cocaine despite the aversive consequence of punishment and the presence of the alternative positive reinforcer.

Serotonin and dopamine differentially affect appetitive and aversive general Pavlovian-to-instrumental transfer

RATIONALE

Human motivation and decision-making is influenced by the interaction of Pavlovian and instrumental systems. The neurotransmitters dopamine and serotonin have been suggested to play a major role in motivation and decision-making, but how they affect this interaction in humans is largely unknown.

OBJECTIVE

We investigated the effect of these neurotransmitters in a general Pavlovian-to-instrumental transfer (PIT) task which measured the nonspecific effect of appetitive and aversive Pavlovian cues on instrumental responses.

METHODS

For that purpose, we used selective dietary depletion of the amino acid precursors of serotonin and dopamine: tryptophan (n = 34) and tyrosine/phenylalanine (n = 35), respectively, and compared the performance of these groups to a control group (n = 34) receiving a nondepleted (balanced) amino acid drink.

RESULTS

We found that PIT differed between groups: Relative to the control group that exhibited only appetitive PIT, we found reduced appetitive PIT in the tyrosine/phenylalanine-depleted group and enhanced aversive PIT in the tryptophan-depleted group.

CONCLUSIONS

These results demonstrate a differential involvement of serotonin and dopamine in motivated behavior. They suggest that reductions in serotonin enhance the motivational influence of aversive stimuli on instrumental behavior and do not affect the influence of appetitive stimuli, while reductions in dopamine diminish the influence of appetitive stimuli. No conclusions could be drawn about how dopamine affects the influence of aversive stimuli. The interplay of both neurotransmitter systems allows for flexible and adaptive responses depending on the behavioral context.

Learning and Motivational Processes Contributing to Pavlovian-Instrumental Transfer and Their Neural Bases: Dopamine and Beyond

Pavlovian stimuli exert a range of effects on behavior from simple conditioned reflexes, such as salivation, to altering the vigor and direction of instrumental actions. It is currently accepted that these distinct behavioral effects stem from two sources (i) the various associative connections between predictive stimuli and the component features of the events that these stimuli predict and (ii) the distinct motivational and cognitive functions served by cues, particularly their arousing and informational effects on the selection and performance of specific actions. Here, we describe studies that have assessed these latter phenomena using a paradigm that has come to be called Pavlovian-instrumental transfer. We focus first on behavioral experiments that have described distinct sources of stimulus control derived from the general affective and outcome-specific predictions of conditioned stimuli, referred to as general transfer and specific transfer, respectively. Subsequently, we describe research efforts attempting to establish the neural bases of these transfer effects, largely in the afferent and efferent connections of the nucleus accumbens (NAc) core and shell. Finally, we examine the role of predictive cues in examples of aberrant stimulus control associated with psychiatric disorders and addiction.

Plain cigarette packs do not exert Pavlovian to instrumental transfer of control over tobacco-seeking

AIMS

To gain insight into the potential impact of plain tobacco packaging policy, two experiments were undertaken to test whether 'prototype' plain compared with branded UK cigarette pack stimuli would differentially elicit instrumental tobacco-seeking in a nominal Pavlovian to instrumental transfer (PIT) procedure.

DESIGN, SETTING AND PARTICIPANTS

Two experiments were undertaken at the University of Bristol UK, with a convenience sample of adult smokers (experiment 1, n = 23, experiment 2, n = 121).

MEASUREMENT

In both experiments, smokers were trained on a concurrent choice procedure in which two responses earned points for cigarettes and chocolate, respectively, before images of branded and plain packs were tested for capacity to elicit the tobacco-seeking response in extinction. The primary outcome was percentage choice of the tobacco- over the chocolate-seeking response in plain pack, branded pack and no-stimulus conditions.

FINDINGS

Both experiments found that branded packs primed a greater percentage of tobacco-seeking (overall mean = 62%) than plain packs (overall mean = 53%) and the no-stimulus condition (overall mean = 52%; Ps ≤ 0.01, ŋp (2) s ≥ 0.16), and that there was no difference in percentage tobacco-seeking between plain packs and the no-stimulus condition (Ps ≥ 0.17, ŋp (2) s ≤ 0.04). Plain tobacco packs showed an overall 9% reduction in the priming of a tobacco choice response compared to branded tobacco packs.

CONCLUSIONS

Plain packaging may reduce smoking in current smokers by degrading cue-elicited tobacco-seeking.

Multiple Systems for the Motivational Control of Behavior and Associated Neural Substrates in Humans

In this chapter, we will review evidence about the role of multiple distinct systems in driving the motivation to perform actions in humans. Specifically, we will consider the contribution of goal-directed action selection mechanisms, habitual action selection mechanisms and the influence of Pavlovian predictors on instrumental action selection. We will further evaluate evidence for the contribution of multiple brain areas including ventral frontal and dorsal cortical areas and several distinct parts of the striatum in these processes. Furthermore, we will consider circumstances in which adverse interactions between these systems can result in the decoupling of motivation from incentive valuation and performance.

Repeated exposure of adult rats to transient oxidative stress induces various long-lasting alterations in cognitive and behavioral functions

Exposure of neonates to oxidative stress may increase the risk of psychiatric disorders such as schizophrenia in adulthood. However, the effects of moderate oxidative stress on the adult brain are not completely understood. To address this issue, we systemically administrated 2-cyclohexen-1-one (CHX) to adult rats to transiently reduce glutathione levels. Repeated administration of CHX did not affect the acquisition or motivation of an appetitive instrumental behavior (lever pressing) rewarded by a food outcome under a progressive ratio schedule. In addition, response discrimination and reversal learning were not affected. However, acute CHX administration blunted the sensitivity of the instrumental performance to outcome devaluation, and this effect was prolonged in rats with a history of repeated CHX exposure, representing pro-depression-like phenotypes. On the other hand, repeated CHX administration reduced immobility in forced swimming tests and blunted acute cocaine-induced behaviors, implicating antidepressant-like effects. Multivariate analyses segregated a characteristic group of behavioral variables influenced by repeated CHX administration. Taken together, these findings suggest that repeated administration of CHX to adult rats did not cause a specific mental disorder, but it induced long-term alterations in behavioral and cognitive functions, possibly related to specific neural correlates.

Stimuli associated with the cancellation of food and its cues enhance eating but display negative incentive value

Initially neutral conditioned stimuli paired with food often acquire motivating properties, including serving as secondary reinforcers, enhancing instrumental responding in Pavlovian-instrumental transfer procedures, and potentiating food consumption under conditions of food satiation. Interestingly, cues associated with the cancellation of food and food cues may also potentiate food consumption (e.g., Galarce and Holland, 2009), despite their apparent negative correlations with food delivery. In three experiments with rats, we investigated conditions under which potentiation of feeding by such "interruption stimuIi" (ISs) develops, and some aspects of the content of that learning. Although in all three experiments ISs enhanced food consumption beyond control levels, they were found to act as conditioned inhibitors for anticipatory food cup entry (Experiment 1), to serve as conditioned punishers of instrumental responding (Experiment 2), and to suppress instrumental lever press responding in a Pavlovian instrumental transfer procedure (Experiment 3). Furthermore, when given concurrent choice between different foods, an IS enhanced consumption of the food whose interruption it had previously signaled, but when given a choice between performing two instrumental responses, the IS shifted rats' choice away from the response that had previously yielded the food whose interruption had been signaled by IS (Experiment 3). Thus, the effects of an IS on appetitive responses were opposite to its effects on consummatory responding. Implications for our understanding of learned incentive motivation and the control of overeating are discussed.

How food cues can enhance and inhibit motivation to obtain and consume food

Learning may play an important role in over-eating. One example is Pavlovian-to-instrumental transfer (PIT), whereby reward cues facilitate responding to obtain that reward. Whilst there is increasing research indicating PIT for food in humans, these studies have exclusively tested PIT under instrumental extinction (i.e. when the food is no longer available), which may reduce their ecological validity. To address this, we conducted two experiments exploring PIT for food in humans when tested under instrumental reinforcement. Participants first underwent Pavlovian discrimination training with an auditory cue paired with a chocolate reward (CS+) and another auditory cue unpaired (CS-). In instrumental training participants learnt to press a button to receive the chocolate reward on a VR10 schedule. In the test phase, each CS was presented whilst participants maintained the opportunity to press the button to receive chocolate. In Experiment 1, the PIT test was implemented after up to 20 min of instrumental training (satiation) whereas in Experiment 2 it was implemented after only 4 min of instrumental training. In both experiments there was evidence for differential PIT, but the pattern differed according to the rate of responding at the time of the PIT test. In low baseline responders the CS+ facilitated both button press responding and consumption, whereas in high baseline responders the CS- suppressed responding. These findings suggest that both excitatory and inhibitory associations may be learnt during PIT training and that the expression of these associations depends on motivation levels at the time the cues are encountered. Particularly concerning is that a food-paired cue can elicit increased motivation to obtain and consume food even when the participant is highly satiated and no longer actively seeking food, as this may be one mechanism by which over-consumption is maintained.

Role of amygdala central nucleus in the potentiation of consuming and instrumental lever-pressing for sucrose by cues for the presentation or interruption of sucrose delivery in rats

Initially neutral conditioned stimuli (CSs) paired with food often acquire motivating properties. For example, CS presentations may enhance the rate of instrumental responding that normally earns that food reward (Pavlovian-instrumental transfer), or potentiate consumption of that food when the animal is food-sated. Recent evidence suggests that cues associated with the withdrawal of food and food cues (interruption stimuli or ISs) may also potentiate feeding, despite exhibiting some characteristics of conditioned inhibition. Here, we compared the ability of CSs and ISs to modulate both eating food and working for it. If CSs and ISs potentiate eating food by controlling a similar incentive state, both types of cues might also be expected to enhance instrumental responding for food. Although we found substantial potentiation of feeding by both CSs and ISs, and powerful enhancement of instrumental responding by a CS, we found no evidence for such instrumental enhancement by an IS. Furthermore, although an IS produced more FOS expression in the amygdala central nucleus (CeA) than either a previously reinforced CS or a control stimulus after a test for potentiated feeding, an intact CeA was unnecessary for potentiation of feeding by either a CS or an IS. Nevertheless, as in previous studies, CeA was critical to the ability of a CS to enhance instrumental responding. Implications for understanding the nature and basis for incentive learning are discussed.

Phasic mesolimbic dopamine signaling encodes the facilitation of incentive motivation produced by repeated cocaine exposure

Drug addiction is marked by pathological drug seeking and intense drug craving, particularly in response to drug-related stimuli. Repeated psychostimulant administration is known to induce long-term alterations in mesolimbic dopamine (DA) signaling that are hypothesized to mediate this heightened sensitivity to environmental stimuli. However, there is little direct evidence that drug-induced alteration in mesolimbic DA function underlies this hypersensitivity to motivational cues. In the current study, we tested this hypothesis using fast-scan cyclic voltammetry to monitor phasic DA signaling in the nucleus accumbens core of cocaine-pretreated (6 once-daily injections of 15 mg/kg, i.p.) and drug-naive rats during a test of cue-evoked incentive motivation for food-the Pavlovian-to-instrumental transfer task. We found that prior cocaine exposure augmented both reward seeking and DA release triggered by the presentation of a reward-paired cue. Furthermore, cue-evoked DA signaling positively correlated with cue-evoked food seeking and was found to be a statistical mediator of this behavioral effect of cocaine. Taken together, these findings provide support for the hypothesis that repeated cocaine exposure enhances cue-evoked incentive motivation through augmented phasic mesolimbic DA signaling. This work sheds new light on a fundamental neurobiological mechanism underlying motivated behavior and its role in the expression of compulsive reward seeking.

Reconsolidation of a well-learned instrumental memory

Once consolidated, memories are dynamic entities that go through phases of instability in order to be updated with new information, via a process of reconsolidation. The phenomenon of reconsolidation has been demonstrated in a wide variety of experimental paradigms. However, the memories underpinning instrumental behaviors are currently not believed to reconsolidate. We show that well-learned lever pressing in rats does undergo reconsolidation, which can be disrupted by systemic administration of the noncompetitive N-methyl-D-aspartate receptor (NMDAR) antagonist (+)-5-methyl-10,11-dihydro-SH-dibenzo[a,d]cyclohepten-5,10-imine maleate (MK-801) when administered prior to a switch to a variable, but not fixed, ratio schedule. Disruption of reconsolidation resulted in a reduction in long-term lever pressing performance and diminished the sensitivity of behavior to contingency change. Further investigation demonstrated that expression of the reconsolidation impairment was not affected by outcome value, implying a deficit in a stimulus–response (S–R) process. The ability to disrupt the performance of well-learned instrumental behaviors is potentially of great importance in the development of reconsolidation-based clinical treatments for conditions that involve compulsive seeking behaviors.

Extinction of cue-evoked drug-seeking relies on degrading hierarchical instrumental expectancies

There has long been need for a behavioural intervention that attenuates cue-evoked drug-seeking, but the optimal method remains obscure. To address this, we report three approaches to extinguish cue-evoked drug-seeking measured in a Pavlovian to instrumental transfer design, in non-treatment seeking adult smokers and alcohol drinkers. The results showed that the ability of a drug stimulus to transfer control over a separately trained drug-seeking response was not affected by the stimulus undergoing Pavlovian extinction training in experiment 1, but was abolished by the stimulus undergoing discriminative extinction training in experiment 2, and was abolished by explicit verbal instructions stating that the stimulus did not signal a more effective response-drug contingency in experiment 3. These data suggest that cue-evoked drug-seeking is mediated by a propositional hierarchical instrumental expectancy that the drug-seeking response is more likely to be rewarded in that stimulus. Methods which degraded this hierarchical expectancy were effective in the laboratory, and so may have therapeutic potential.

Orbitofrontal neurons acquire responses to 'valueless' Pavlovian cues during unblocking

The orbitofrontal cortex (OFC) has been described as signaling outcome expectancies or value. Evidence for the latter comes from the studies showing that neural signals in the OFC correlate with value across features. Yet features can co-vary with value, and individual units may participate in multiple ensembles coding different features. Here we used unblocking to test whether OFC neurons would respond to a predictive cue signaling a ‘valueless’ change in outcome flavor. Neurons were recorded as the rats learned about cues that signaled either an increase in reward number or a valueless change in flavor. We found that OFC neurons acquired responses to both predictive cues. This activity exceeded that exhibited to a ‘blocked’ cue and was correlated with activity to the actual outcome. These results show that OFC neurons fire to cues with no value independent of what can be inferred through features of the predicted outcome.

DOI:http://dx.doi.org/10.7554/eLife.02653.001

Imagine you are at a restaurant and the waiter offers you a choice of cheesecake or fruit salad for dessert. When making your choice it is likely that you will consider the features of these desserts, such as their taste, their sweetness or how healthy they are. However, when you decide which dessert to have, you will pick the one that you judge to have the highest value for you at that moment in time. In this sense, ‘value’ is a subjective concept that varies from person to person, while ‘features’ remain relatively static.

It is generally agreed that the orbitofrontal cortex (OFC) is involved in making these sorts of decisions, but its role is still a topic of debate. According to one theory the neurons in the OFC signal the subjective value of an outcome, whereas a rival theory suggests that they signal the features of the expected outcome. However, it has proved challenging to test these theories in experiments because it is difficult to say for certain that a given decision was clearly due to the value or a feature.

Now, McDannald et al. have devised an approach that can tell the difference between neurons signaling value and neurons signaling features. They trained thirsty rats to associate different odours with either an increase in the amount of milk they were given (a change in both value and a feature), or a change in the flavor of the milk (a change in a feature without a change in value). Extensive testing showed that the rats did not value one flavor over the other.

McDannald et al. then examined how the neurons in the OFC responded. If these neurons signal only value, they should only fire when the value of the outcome changes. On the other hand, if they signal features, they should fire when a feature changes, even if the value does not. It turned out that the neurons in the OFC responded whenever the features changed, irrespective of whether or not the value changed. These findings present a challenge to popular conceptions of the role of the neurons in the OFC.

DOI:http://dx.doi.org/10.7554/eLife.02653.002

Unconditioned stimulus revaluation to promote conditioned fear extinction in the memory reconsolidation window

The retrieval-extinction paradigm, which disrupts the reconsolidation of fear memories in humans, is a non-invasive technique that can be used to prevent the return of fear in humans. In the present study, unconditioned stimulus revaluation was applied in the retrieval-extinction paradigm to investigate its promotion of conditioned fear extinction in the memory reconsolidation window after participants acquired conditioned fear. This experiment comprised three stages (acquisition, unconditioned stimulus revaluation, retrieval-extinction) and three methods for indexing fear (unconditioned stimulus expectancy, skin conductance response, conditioned stimulus pleasure rating). After the acquisition phase, we decreased the intensity of the unconditioned stimulus in one group (devaluation) and maintained constant for the other group (control). The results indicated that both groups exhibited similar levels of unconditioned stimulus expectancy, but the devaluation group had significantly smaller skin conductance responses and exhibited a growth in conditioned stimulus + pleasure. Thus, our findings indicate unconditioned stimulus revaluation effectively promoted the extinction of conditioned fear within the memory reconsolidation window.

Individual differences in bodily freezing predict emotional biases in decision making

Instrumental decision making has long been argued to be vulnerable to emotional responses. Literature on multiple decision making systems suggests that this emotional biasing might reflect effects of a system that regulates innately specified, evolutionarily preprogrammed responses. To test this hypothesis directly, we investigated whether effects of emotional faces on instrumental action can be predicted by effects of emotional faces on bodily freezing, an innately specified response to aversive relative to appetitive cues. We tested 43 women using a novel emotional decision making task combined with posturography, which involves a force platform to detect small oscillations of the body to accurately quantify postural control in upright stance. On the platform, participants learned whole body approach-avoidance actions based on monetary feedback, while being primed by emotional faces (angry/happy). Our data evidence an emotional biasing of instrumental action. Thus, angry relative to happy faces slowed instrumental approach relative to avoidance responses. Critically, individual differences in this emotional biasing effect were predicted by individual differences in bodily freezing. This result suggests that emotional biasing of instrumental action involves interaction with a system that controls innately specified responses. Furthermore, our findings help bridge (animal and human) decision making and emotion research to advance our mechanistic understanding of decision making anomalies in daily encounters as well as in a wide range of psychopathology.

Habit formation: implications for alcoholism research

Characteristics of individuals with severe alcohol use disorders include heightened cue sensitivity, compulsive seeking, craving, and continued alcohol use in the face of negative consequences. Animal models are useful for understanding behavioral and neurological mechanisms underlying problematic alcohol use. Seeking of operant reinforcers including alcohol is processed by two mechanisms, commonly referred to as "goal-directed" (action-outcome) and "habitual" (stimulus-response). As substance use disorders are characterized by continued use regardless of unfavorable outcomes, it is plausible that drug use causes an unnatural disruption of these mechanisms. We present a critical analysis of literature pertaining to behavioral neuroscience alcoholism research involving habit formation. Traditionally, when operant behavior is unaffected by a loss of subjective value of a reinforcer (devaluation), the behavior is considered habitual. Acquisition of instrumental behavior requires corticostriatal mechanisms that depend heavily on the prefrontal cortex and ventral striatum, whereas practiced behavior is more predominantly controlled by the dorsal striatum. Dopaminergic signaling is necessary for the neurological adaptations involved in stimulus-response action, and drugs of abuse appear to facilitate habitual behavior through high levels of dopamine release. Evidence suggests that the use of alcohol as a reinforcer expedites habit formation, and that a history of alcohol use produces alterations in striatal morphology, aids habit learning for non-psychoactive reinforcers, and promotes alcohol drinking despite aversive adulterants. In this review, we suggest directions for future alcoholism research that seeks to measure action made despite a devalued outcome, including procedural modifications and genotypic, pharmacological, or neurological manipulations. Most alcoholism models currently in use fail to reach substantial blood ethanol concentrations, a shortcoming that may be alleviated through the use of high-drinking rodent lines. Additionally, satiety, one common mechanism of devaluing reinforcers, is not recommended for alcohol research because the psychoactive effects of alcohol depress response rates, mimicking devaluation effects. Overall, further research of habit formation and potentially related perseverative behaviors could be invaluable in discovering genetic variance, traits that correlate with persistent alcohol seeking, implicated neural structures and processes of alcohol use, and eventually novel pharmacological treatment for alcoholism.

The motivational drive to natural rewards is modulated by prenatal glucocorticoid exposure

Exposure to elevated levels of glucocorticoids (GCs) during neurodevelopment has been identified as a triggering factor for the development of reward-associated disorders in adulthood. Disturbances in the neural networks responsible for the complex processes that assign value to rewards and associated stimuli are critical for disorders such as depression, obsessive–compulsive disorders, obesity and addiction. Essential in the understanding on how cues influence behavior is the Pavlovian–instrumental transfer (PIT), a phenomenon that refers to the capacity of a Pavlovian stimulus that predicts a reward to elicit instrumental responses for that same reward. Here, we demonstrate that in utero exposure to GCs (iuGC) impairs both general and selective versions of the PIT paradigm, suggestive of deficits in motivational drive. The iuGC animals presented impaired neuronal activation pattern upon PIT performance in cortical and limbic regions, as well as morphometric changes and reduced levels of dopamine in prefrontal and orbitofrontal cortices, key regions involved in the integration of Pavlovian and instrumental stimuli. Normalization of dopamine levels rescued this behavior, a process that relied on D2/D3, but not D1, dopamine receptor activation. In summary, iuGC exposure programs the mesocorticolimbic dopaminergic circuitry, leading to a reduction in the attribution of the incentive salience to cues, in a dopamine-D2/D3-dependent manner. Ultimately, these results are important to understand how GCs bias incentive processes, a fact that is particularly relevant for disorders where differential attribution of incentive salience is critical.

Differential effects of systemic cholinergic receptor blockade on Pavlovian incentive motivation and goal-directed action selection

Reward-seeking actions can be guided by external cues that signal reward availability. For instance, when confronted with a stimulus that signals sugar, rats will prefer an action that produces sugar over a second action that produces grain pellets. Action selection is also sensitive to changes in the incentive value of potential rewards. Thus, rats that have been prefed a large meal of sucrose will prefer a grain-seeking action to a sucrose-seeking action. The current study investigated the dependence of these different aspects of action selection on cholinergic transmission. Hungry rats were given differential training with two unique stimulus-outcome (S1-O1 and S2-O2) and action-outcome (A1-O1 and A2-O2) contingencies during separate training phases. Rats were then given a series of Pavlovian-to-instrumental transfer tests, an assay of cue-triggered responding. Before each test, rats were injected with scopolamine (0, 0.03, or 0.1 mg/kg, intraperitoneally), a muscarinic receptor antagonist, or mecamylamine (0, 0.75, or 2.25 mg/kg, intraperitoneally), a nicotinic receptor antagonist. Although the reward-paired cues were capable of biasing action selection when rats were tested off-drug, both anticholinergic treatments were effective in disrupting this effect. During a subsequent round of outcome devaluation testing-used to assess the sensitivity of action selection to a change in reward value--we found no effect of either scopolamine or mecamylamine. These results reveal that cholinergic signaling at both muscarinic and nicotinic receptors mediates action selection based on Pavlovian reward expectations, but is not critical for flexibly selecting actions using current reward values.

Investigating habits: strategies, technologies and models

Understanding habits at a biological level requires a combination of behavioral observations and measures of ongoing neural activity. Theoretical frameworks as well as definitions of habitual behaviors emerging from classic behavioral research have been enriched by new approaches taking account of the identification of brain regions and circuits related to habitual behavior. Together, this combination of experimental and theoretical work has provided key insights into how brain circuits underlying action-learning and action-selection are organized, and how a balance between behavioral flexibility and fixity is achieved. New methods to monitor and manipulate neural activity in real time are allowing us to have a first look “under the hood” of a habit as it is formed and expressed. Here we discuss ideas emerging from such approaches. We pay special attention to the unexpected findings that have arisen from our own experiments suggesting that habitual behaviors likely require the simultaneous activity of multiple distinct components, or operators, seen as responsible for the contrasting dynamics of neural activity in both cortico-limbic and sensorimotor circuits recorded concurrently during different stages of habit learning. The neural dynamics identified thus far do not fully meet expectations derived from traditional models of the structure of habits, and the behavioral measures of habits that we have made also are not fully aligned with these models. We explore these new clues as opportunities to refine an understanding of habits.

Dorsal and ventral streams: the distinct role of striatal subregions in the acquisition and performance of goal-directed actions

Considerable evidence suggests that distinct neural processes mediate the acquisition and performance of goal-directed instrumental actions. Whereas a cortical-dorsomedial striatal circuit appears critical for the acquisition of goal-directed actions, a cortical-ventral striatal circuit appears to mediate instrumental performance, particularly the motivational control of performance. Here we review evidence that these distinct mechanisms of learning and performance constitute two distinct 'streams' controlling instrumental conditioning. From this perspective, the regulation of the interaction between these 'streams' becomes a matter of considerable importance. We describe evidence that the basolateral amygdala, which is heavily interconnected with both the dorsal and ventral subregions of the striatum, coordinates this interaction providing input to the final common path to action as a critical component of the limbic-motor interface.

Goals and habits in the brain

An enduring and richly elaborated dichotomy in cognitive neuroscience is that of reflective versus reflexive decision making and choice. Other literatures refer to the two ends of what is likely to be a spectrum with terms such as goal-directed versus habitual, model-based versus model-free or prospective versus retrospective. One of the most rigorous traditions of experimental work in the field started with studies in rodents and graduated via human versions and enrichments of those experiments to a current state in which new paradigms are probing and challenging the very heart of the distinction. We review four generations of work in this tradition and provide pointers to the forefront of the field's fifth generation.