The form of a conditioned stimulus can influence the degree to which it acquires incentive motivational properties

Relative reward effects on operant behavior: Incentive contrast, induction and variety effects

Comparing different rewards automatically produces dynamic relative outcome effects on behavior. Each new outcome exposure is to an updated version evaluated relative to alternatives. Relative reward effects include incentive contrast, positive induction and variety effects. The present study utilized a novel behavioral design to examine relative reward effects on a chain of operant behavior using auditory cues. Incentive contrast is the most often examined effect and focuses on increases or decreases in behavioral performance after value upshifts (positive) or downshifts (negative) relative to another outcome. We examined the impact of comparing two reward outcomes in a repeated measures design with three sessions: a single outcome and a mixed outcome and a final single outcome session. Relative reward effects should be apparent when comparing trials for the identical outcome between the single and mixed session types. An auditory cue triggered a series of operant responses (nosepoke-leverpress-food retrieval), and we measured possible contrast effects for different reward magnitude combinations. We found positive contrast for trials with the greatest magnitude differential but positive induction or variety effects in other combinations. This behavioral task could be useful for analyzing environmental or neurobiological factors involved in reward comparisons, decision-making and choice during instrumental, goal-directed action.

The attribution of incentive salience to Pavlovian alcohol cues: a shift from goal-tracking to sign-tracking

Environmental stimuli that are reliably paired with alcohol may acquire incentive salience, a property that can operate in the use and abuse of alcohol. Here we investigated the incentive salience of Pavlovian alcohol cues using a preclinical animal model. Male, Long-Evans rats (Harlan) with unrestricted access to food and water were acclimated to drinking 15% ethanol (v/v) in their home-cages. Rats then received Pavlovian autoshaping training in which the 10 s presentation of a retractable lever served as the conditioned stimulus (CS) and 15% ethanol served as the unconditioned stimulus (US) (0.2 ml/CS; 12 CS presentations/session; 27 sessions). Next, in an operant test of conditioned reinforcement, nose pokes into an active aperture delivered presentations of the lever-CS, whereas nose pokes into an inactive aperture had no consequences. Across initial autoshaping sessions, goal-tracking behavior, as measured by entries into the fluid port where ethanol was delivered, developed rapidly. However, with extended training goal-tracking diminished, and sign-tracking responses, as measured by lever-CS activations, emerged. Control rats that received explicitly unpaired CS and US presentations did not show goal-tracking or sign-tracking responses. In the test for conditioned reinforcement, rats with CS-US pairings during autoshaping training made more active relative to inactive nose pokes, whereas rats in the unpaired control group did not. Moreover, active nose pokes were positively correlated with sign-tracking behavior during autoshaping. Extended training may produce a shift in the learned properties of Pavlovian alcohol cues, such that after initially predicting alcohol availability they acquire robust incentive salience.

Isolating the incentive salience of reward-associated stimuli: value, choice, and persistence

Sign- and goal-tracking are differentially associated with drug abuse-related behavior. Recently, it has been hypothesized that sign- and goal-tracking behavior are mediated by different neurobehavioral valuation systems, including differential incentive salience attribution. Herein, we used different conditioned stimuli to preferentially elicit different response types to study the different incentive valuation characteristics of stimuli associated with sign- and goal-tracking within individuals. The results demonstrate that all stimuli used were equally effective conditioned stimuli; however, only a lever stimulus associated with sign-tracking behavior served as a robust conditioned reinforcer and was preferred over a tone associated with goal-tracking. Moreover, the incentive value attributed to the lever stimulus was capable of promoting suboptimal choice, leading to a significant reduction in reinforcers (food) earned. Furthermore, sign-tracking to a lever was more persistent than goal-tracking to a tone under omission and extinction contingencies. Finally, a conditional discrimination procedure demonstrated that sign-tracking to a lever and goal-tracking to a tone were dependent on learned stimulus–reinforcer relations. Collectively, these results suggest that the different neurobehavioral valuation processes proposed to govern sign- and goal-tracking behavior are independent but parallel processes within individuals. Examining these systems within individuals will provide a better understanding of how one system comes to dominate stimulus–reward learning, thus leading to the differential role these systems play in abuse-related behavior.

Sign-tracking predicts increased choice of cocaine over food in rats

The purpose of this study was to determine whether the tendency to sign-track to a food cue was predictive of rats' choice of cocaine over food. First, rats were trained on a procedure where insertion of a retractable lever was paired with food. A sub-group of rats - sign-trackers - primarily approached and contacted the lever, while another sub-group - goal-trackers - approached the site of food delivery. Rats were then trained on a choice task where they could choose between an infusion of cocaine (1.0 mg/kg) and a food pellet (45 mg). Sign-trackers chose cocaine over food significantly more often than did goal-trackers. These results support the incentive-salience theory of addiction and add to a growing number of studies which suggest that sign-trackers may model an addiction-prone phenotype.

Incentive salience attribution under reward uncertainty: A Pavlovian model

There is a vast literature on the behavioural effects of partial reinforcement in Pavlovian conditioning. Compared with animals receiving continuous reinforcement, partially rewarded animals typically show (a) a slower development of the conditioned response (CR) early in training and (b) a higher asymptotic level of the CR later in training. This phenomenon is known as the partial reinforcement acquisition effect (PRAE). Learning models of Pavlovian conditioning fail to account for it. In accordance with the incentive salience hypothesis, it is here argued that incentive motivation (or 'wanting') plays a more direct role in controlling behaviour than does learning, and reward uncertainty is shown to have an excitatory effect on incentive motivation. The psychological origin of that effect is discussed and a computational model integrating this new interpretation is developed. Many features of CRs under partial reinforcement emerge from this model.

A cocaine context renews drug seeking preferentially in a subset of individuals

Addiction is characterized by a high propensity for relapse, in part because cues associated with drugs can acquire Pavlovian incentive motivational properties, and acting as incentive stimuli, such cues can instigate and invigorate drug-seeking behavior. There is, however, considerable individual variation in the propensity to attribute incentive salience to reward cues. Discrete and localizable reward cues act as much more effective incentive stimuli in some rats ('sign-trackers', STs), than others ('goal-trackers', GTs). We asked whether similar individual variation exists for contextual cues associated with cocaine. Cocaine context conditioned motivation was quantified in two ways: (1) the ability of a cocaine context to evoke conditioned hyperactivity and (2) the ability of a context in which cocaine was previously self-administered to renew cocaine-seeking behavior. Finally, we assessed the effects of intra-accumbens core flupenthixol, a nonselective dopamine receptor antagonist, on context renewal. In contrast to studies using discrete cues, a cocaine context spurred greater conditioned hyperactivity, and more robustly renewed extinguished cocaine seeking in GTs than STs. In addition, cocaine context renewal was blocked by antagonism of dopamine receptors in the accumbens core. Thus, contextual cues associated with cocaine preferentially acquire motivational control over behavior in different individuals than do discrete cues, and in these individuals the ability of a cocaine context to create conditioned motivation for cocaine requires dopamine in the core of the nucleus accumbens. We speculate that different individuals may be preferentially sensitive to different 'triggers' of relapse.

Neuromodulatory adaptive combination of correlation-based learning in cerebellum and reward-based learning in basal ganglia for goal-directed behavior control

Goal-directed decision making in biological systems is broadly based on associations between conditional and unconditional stimuli. This can be further classified as classical conditioning (correlation-based learning) and operant conditioning (reward-based learning). A number of computational and experimental studies have well established the role of the basal ganglia in reward-based learning, where as the cerebellum plays an important role in developing specific conditioned responses. Although viewed as distinct learning systems, recent animal experiments point toward their complementary role in behavioral learning, and also show the existence of substantial two-way communication between these two brain structures. Based on this notion of co-operative learning, in this paper we hypothesize that the basal ganglia and cerebellar learning systems work in parallel and interact with each other. We envision that such an interaction is influenced by reward modulated heterosynaptic plasticity (RMHP) rule at the thalamus, guiding the overall goal directed behavior. Using a recurrent neural network actor-critic model of the basal ganglia and a feed-forward correlation-based learning model of the cerebellum, we demonstrate that the RMHP rule can effectively balance the outcomes of the two learning systems. This is tested using simulated environments of increasing complexity with a four-wheeled robot in a foraging task in both static and dynamic configurations. Although modeled with a simplified level of biological abstraction, we clearly demonstrate that such a RMHP induced combinatorial learning mechanism, leads to stabler and faster learning of goal-directed behaviors, in comparison to the individual systems. Thus, in this paper we provide a computational model for adaptive combination of the basal ganglia and cerebellum learning systems by way of neuromodulated plasticity for goal-directed decision making in biological and bio-mimetic organisms.

Accounting for negative automaintenance in pigeons: a dual learning systems approach and factored representations

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.