Effects of cytotoxic nucleus accumbens lesions on instrumental conditioning in rats

Human pavlovian-instrumental transfer

The vigor with which a participant performs actions that produce valuable outcomes is subject to a complex set of motivational influences. Many of these are believed to involve the amygdala and the nucleus accumbens, which act as an interface between limbic and motor systems. One prominent class of influences is called pavlovian-instrumental transfer (PIT), in which the motivational characteristics of a predictor influence the vigor of an action with respect to which it is formally completely independent. We provide a demonstration of behavioral PIT in humans, with an audiovisual predictor of the noncontingent delivery of money inducing participants to perform more avidly an action involving squeezing a handgrip to earn money. Furthermore, using functional magnetic resonance imaging, we show that this enhanced motivation was associated with a trial-by-trial correlation with the blood oxygenation level-dependent (BOLD) signal in the nucleus accumbens and a subject-by-subject correlation with the BOLD signal in the amygdala. Our data dovetails well with the animal literature and sheds light on the neural control of vigor.

Differential involvement of the basolateral amygdala, orbitofrontal cortex, and nucleus accumbens core in the acquisition and use of reward expectancies

In this study, the authors tested the hypothesis that the basolateral amygdala (BLA), orbitofrontal cortex (OFC), nucleus accumbens core (NA-core), and the extended hippocampus mediate different aspects of the development-maintenance of unique reward expectancies produced by the differential outcomes procedure (DOP). Rats were trained with either DOP or a nondifferential outcomes procedure (NOP) on a simple discrimination task. Fornix lesions did not affect either version of the task, demonstrating that the extended hippocampal system has no role in stimulus-outcome (S-O) associations. In contrast, in the DOP condition, BLA lesions impaired performance throughout training, OFC lesions impaired choice accuracy only in the later maintenance phase, and NA-core lesions resulted in enhanced learning. These results suggest that BLA and OFC are important for establishment (BLA) and behavioral maintenance (OFC) of S-O associations, whereas the NA-core is not needed and can in fact impede using multiple S-O associations. No impairments were observed in the NOP condition, demonstrating that these structures are not critical to stimulus-response learning.

Conditioned reinforcement can be mediated by either outcome-specific or general affective representations

Conditioned reinforcers are Pavlovian cues that support the acquisition and maintenance of new instrumental responses. Responding on the basis of conditioned rather than primary reinforcers is a pervasive part of modern life, yet we have a remarkably limited understanding of what underlying associative information is triggered by these cues to guide responding. Specifically, it is not certain whether conditioned reinforcers are effective because they evoke representations of specific outcomes or because they trigger general affective states that are independent of any specific outcome. This question has important implications for how different brain circuits might be involved in conditioned reinforcement. Here, we use specialized Pavlovian training procedures, reinforcer devaluation and transreinforcer blocking, to create cues that were biased to preferentially evoke either devaluation-insensitive, general affect representations or, devaluation-sensitive, outcome-specific representations. Subsequently, these cues, along with normally conditioned control cues, were presented contingent on lever pressing. We found that intact rats learned to lever press for either the outcome or the affect cues to the same extent as for a normally conditioned cue. These results demonstrate that conditioned reinforcers can guide responding through either type of associative information. Interestingly, conditioned reinforcement was abolished in rats with basolateral amygdala lesions. Consistent with the extant literature, this result suggests a general role for basolateral amygdala in conditioned reinforcement. The implications of these data, combined with recent reports from our laboratory of a more specialized role of orbitofrontal cortex in conditioned reinforcement, will be discussed.

Still at the Choice-Point: Action Selection and Initiation in Instrumental Conditioning

Contrary to classic stimulus-response (S-R) theory, recent evidence suggests that, in instrumental conditioning, rats encode the relationship between their actions and the specific consequences that these actions produce. It has remained unclear, however, how encoding this relationship acts to control instrumental performance. Although S-R theories were able to give a clear account of how learning translates into performance, the argument that instrumental learning constitutes the acquisition of information of the form "response R leads to outcome O" does not directly imply a particular performance rule or policy; this information can be used both to perform R and to avoid performing R. Recognition of this problem has forced the development of accounts that allow the O and stimuli that predict the O (i.e., S-O) to play a role in the initiation of specific Rs. In recent experiments, we have used a variety of behavioral procedures in an attempt to isolate the processes that contribute to instrumental performance, including outcome devaluation, reinstatement, and Pavlovian-instrumental transfer. Our results, particularly from experiments assessing outcome-selective reinstatement, suggest that both "feed-forward" (O-R) and "feed-back" (R-O) associations are critical and that although the former appear to be important to response selection, the latter-together with processes that determine outcome value-mediate response initiation. We discuss a conceptual model that integrates these processes and its neural implementation.

Striatal Contributions to Reward and Decision Making: Making Sense of Regional Variations in a Reiterated Processing Matrix

The striatum is the major input nucleus of the basal ganglia. It is thought to play a key role in learning on the basis of positive reinforcement and in action selection. One view of the striatum conceives it as comprising a reiterated matrix of processing units that perform common operations in different striatal regions, namely synaptic plasticity according to a three-factor rule, and lateral inhibition. These operations are required for reinforcement learning and selection of previously reinforced actions. Analysis of the behavioral effects of circumscribed lesions of the striatum, however, suggests regional specialization of learning and decision-making operations. We consider how a basic processing unit may be modified by regional variations in neurochemical parameters, for example, by the gradient in density of dopamine terminals from dorsal to ventral striatum. These variations suggest subtle differences between dorsolateral and ventromedial striatal regions in the temporal properties of dopamine signaling, which are superimposed on regional differences in connectivity. We propose that these variations make sense in relation to the temporal structure of activity in striatal inputs from different regions, and the requirements of different learning operations. Dorsolateral striatal (DLS) regions may be subject to brief, precisely timed pulses of dopamine, whereas ventromedial striatal regions integrate dopamine signals over a longer time course. These differences may be important for understanding regional variations in the contribution to reinforcement of habits, versus incentive processes that are sensitive to the value of expected rewards.

Dopamine efflux in the nucleus accumbens during within-session extinction, outcome-dependent, and habit-based instrumental responding for food reward

RATIONALE

Dopamine (DA) activity in the nucleus accumbens (NAc) is related to the general motivational effects of rewarding stimuli. Dickinson and colleagues have shown that initial acquisition of instrumental responding reflects action-outcome relationships based on instrumental incentive learning, which establishes the value of an outcome. Given that the sensitivity of responding to outcome devaluation is not affected by NAc lesions, it is unlikely that incentive learning during the action-outcome phase is mediated by DA activity in the NAc.

OBJECTIVES

DA efflux in the NAc after limited and extended training was compared on the assumption that comparable changes would be observed during both action-outcome- and habit-based phases of instrumental responding for food. This study also tested the hypothesis that increase in NAc DA activity is correlated with instrumental responding during extinction maintained by a conditioned stimulus paired with food.

METHODS

Rats were trained to lever press for food (random-interval 30 s schedule). On the 5th and 16th day of training, microdialysis samples were collected from the NAc or mediodorsal striatum (a control site for generalized activity) during instrumental responding in extinction and then for food reward, and analyzed for DA content using high performance liquid chromatography.

RESULTS

Increase in DA efflux in the NAc accompanied responding for food pellets on both days 5 and 16, with the magnitude of increase significantly enhanced on day 16. DA efflux was also significantly elevated during responding in extinction only on day 16.

CONCLUSIONS

These results support a role for NAc DA activity in Pavlovian, but not instrumental, incentive learning.

Intact discrimination reversal learning but slowed responding to reward-predictive cues after dopamine D1 and D2 receptor blockade in the nucleus accumbens of rats

RATIONALE

The prediction error hypothesis of dopamine action states that dopamine signals are necessary for the brain to update the predictive significance of cues. Yet, little is known whether D1 or D2 receptor-mediated signals in the nucleus accumbens core (AcbC) are required to learn a reversal of the predictive significance of cues.

OBJECTIVE

Here we examined the effects of a selective D1 or D2 receptor blockade in the AcbC on learning a reversal of previously acquired cue-reward magnitude contingencies.

MATERIALS AND METHODS

Rats were trained on a reaction time (RT) task demanding conditioned lever release with discriminative visual cues signalling in advance the upcoming reward magnitude (one or five food pellets). After acquisition, RTs were guided by cue-associated reward magnitudes, i.e. RTs of responses were significantly shorter for expected high vs low reward. Thereafter, cue-reward magnitude contingencies were reversed. Reversal learning was tested for 12 daily sessions with intra-AcbC micro-infusions being given on sessions 1-6. Subjects received pre-trial infusions of the selective D1 or D2 receptor antagonists, SCH23390 (0.5, 2 microg per side) or raclopride (1, 4 microg per side), or vehicle (0.5 microl).

RESULTS

Intra-AcbC infusion of SCH23390 (0.5, 2 microg) or raclopride (1, 4 microg) did not inhibit discrimination reversal learning, but the higher dose of each drug increased RTs of instrumental responses.

CONCLUSIONS

In a visual discrimination task as used here, D1 and D2 receptor-mediated signals in the AcbC seem to be unnecessary in updating the reward-predictive significance of cues, rather, they serve to activate instrumental behaviour.

The nucleus accumbens as part of a basal ganglia action selection circuit

BACKGROUND

The nucleus accumbens is the ventral extent of the striatum, the main input nucleus of the basal ganglia. Recent hypotheses propose that the accumbens and its dopamine projection from the midbrain contribute to appetitive behaviors required to obtain reward. However, the specific nature of this contribution is unclear. In contrast, significant advances have been made in understanding the role of the dorsal striatum in action selection and decision making.

OBJECTIVE

In order to develop a hypothesis of the role of nucleus accumbens dopamine in action selection, the physiology and behavioral pharmacology of the nucleus accumbens are compared to those of the dorsal striatum.

HYPOTHESES

Three hypotheses concerning the role of dopamine in these structures are proposed: (1) that dopamine release in the dorsal striatum serves to facilitate the ability to respond appropriately to temporally predictable stimuli (that is, stimuli that are so predictable that animals engage in anticipatory behavior just prior to the stimulus); (2) that dopamine in the nucleus accumbens facilitates the ability to respond to temporally unpredictable stimuli (which require interruption of ongoing behavior); and (3) that accumbens neurons participate in action selection in response to such stimuli by virtue of their direct (monosynaptic inhibitory) and indirect (polysynaptic excitatory) projections to basal ganglia output nuclei.

DeltaFosB in the nucleus accumbens regulates food-reinforced instrumental behavior and motivation

Alterations in motivation have been implicated in the pathophysiology of several psychiatric disorders, including substance abuse and depression. Repeated exposure to drugs of abuse or stress is known to persistently induce the transcription factor deltaFosB in the nucleus accumbens (NAc) and dorsal striatum, effects hypothesized to contribute to neuroadaptations in dopamine-regulated signaling. Little is known, however, about the specific involvement of deltaFosB in dysregulation of appetitively motivated behaviors. We show here that inducible overexpression of deltaFosB in NAc and dorsal striatum of bitransgenic mice, or specifically in the NAc core of rats by use of viral-mediated gene transfer, enhanced food-reinforced instrumental performance and progressive ratio responding. Very similar behavioral effects were found after previous repeated exposure to cocaine, amphetamine, MDMA [(+)-3,4-methylenedioxymethamphetamine], or nicotine in rats. These results reveal the powerful regulation of motivational processes by deltaFosB, and provide evidence that drug-induced alterations in gene expression via induction of deltaFosB within the NAc core may play a critical role in the impact of motivational influences on instrumental behavior.

Neural systems implicated in delayed and probabilistic reinforcement

This review considers the theoretical problems facing agents that must learn and choose on the basis of reward or reinforcement that is uncertain or delayed, in implicit or procedural (stimulus-response) representational systems and in explicit or declarative (action-outcome-value) representational systems. Individual differences in sensitivity to delays and uncertainty may contribute to impulsivity and risk taking. Learning and choice with delayed and uncertain reinforcement are related but in some cases dissociable processes. The contributions to delay and uncertainty discounting of neuromodulators including serotonin, dopamine, and noradrenaline, and of specific neural structures including the nucleus accumbens core, nucleus accumbens shell, orbitofrontal cortex, basolateral amygdala, anterior cingulate cortex, medial prefrontal (prelimbic/infralimbic) cortex, insula, subthalamic nucleus, and hippocampus are examined.

Firing Patterns of Accumbal Neurons During a Pavlovian-Conditioned Approach Task

The nucleus accumbens (NAc) is necessary for the expression of Pavlovian-conditioned approach behavior but not for the expression of instrumental behavior conditioned in sessions that set a low response requirement. Although numerous studies have characterized firing patterns of NAc neurons in relation to instrumental behavior, very little is known about how NAc neurons encode information in Pavlovian tasks. In the present study, recordings of accumbal firing patterns were made during sessions in which rats performed a Pavlovian-conditioned approach task. Most of the recorded neurons (74/83, 89%) exhibited significant responses during the conditioned stimulus (CS) presentation and/or the reward exposure. The reward responses were prevalent, predominantly inhibitory, and comparable to reward responses observed in various types of behavioral paradigms, including instrumental tasks. The CS responses could be segregated into multiple subtypes on the basis of directionality, onset latency, and duration. Several characteristics of the CS firing patterns were unique relative to cue responses observed previously during alternative types of conditioning sessions. It is possible that the novel firing patterns correspond to the differential contributions of the accumbens to Pavlovian-conditioned approach behavior and instrumentally conditioned behavior. Regardless, the novel patterns of firing add to existing evidence that characterization of accumbal firing patterns in Pavlovian tasks may provide additional information about the neurophysiological mechanisms that mediate accumbal contributions to behavior.

Facilitated extinction of appetitive instrumental conditioning following excitotoxic lesions of the core or the medial shell subregion of the nucleus accumbens in rats

The nucleus accumbens has been implicated in the control of goal-directed behaviour, including instrumental conditioning. Here, we evaluated the effect of N-methyl-D: -aspartate (NMDA)-induced excitotoxic lesions restricted to either the core or the medial shell subregions of the nucleus accumbens (NAC) on extinction in rats using a trial discrete fixed ratio-5 (FR-5) appetitive operant procedure. Neither core nor shell lesions of the NAC affected the acquisition of instrumental responding. Both lesions facilitated the cessation of responding when the instrumental act no longer yielded reinforcement. Our results suggest that both the NAC core and medial shell contribute to the control of extinction learning of appetitively motivated instrumental behaviour.

Effects of a systemic AMPA/KA and NMDA receptor blockade on pavlovian-instrumental transfer

RATIONALE

Pavlovian stimuli can markedly elevate instrumental responding directed toward a common reward, an effect known as pavlovian-instrumental transfer (PIT). PIT critically depends on the amygdala and nucleus accumbens (ACB); however, little is known yet about its neurochemical basis.

OBJECTIVE

Here we examined the role of ionotropic glutamate receptors in PIT.

METHODS

The effects of a systemic blockade of alpha-amino-3-hydroxy-5-methyl-4-isoxazole propionate/kainate (AMPA/KA) and N-methyl-D: -aspartate (NMDA) receptors on PIT and locomotor activity were investigated in rats.

RESULTS

The competitive AMPA/KA receptor antagonist 6-cyano-7-nitroquinoxaline-2,3-dione disodium salt (CNQX) (1.5 mg/kg i.p.) did not alter the overall rate of lever pressing and left PIT intact, i.e. presentation of a pavlovian stimulus significantly enhanced instrumental responding. Furthermore, CNQX did not affect horizontal and vertical activity in an open field. The non-competitive NMDA receptor antagonist (+)-5-methyl-10,11-dihydro-5H-dibenzo[a,d]cyclohepten-5,10-imine hydrogen maleate (dizocilpine) (0.08 mg/kg i.p.) elevated the overall rate of lever pressing but did not affect PIT. In addition, dizocilpine increased horizontal and decreased vertical activity in an open field.

CONCLUSIONS

Previous studies imply that the training protocol used here induced a general, not outcome-specific, form of PIT which is mediated by the central nucleus of the amygdala (CeN) through modulation of mesoaccumbens dopamine transmission. Thus, we suggest that an AMPA/KA and NMDA receptor blockade did not affect PIT here because the general motivational influence of pavlovian stimuli to induce PIT is conveyed by GABAergic projections from the CeN to midbrain dopaminergic neurons.

Blockade of NMDA receptors in the dorsomedial striatum prevents action-outcome learning in instrumental conditioning

Although there is consensus that instrumental conditioning depends on the encoding of action-outcome associations, it is not known where this learning process is localized in the brain. Recent research suggests that the posterior dorsomedial striatum (pDMS) may be the critical locus of these associations. We tested this hypothesis by examining the contribution of N-methyl-D-aspartate receptors (NMDARs) in the pDMS to action-outcome learning. Rats with bilateral cannulae in the pDMS were first trained to perform two actions (left and right lever presses), for sucrose solution. After the pre-training phase, they were given an infusion of the NMDA antagonist 2-amino-5-phosphonopentanoic acid (APV, 1 mg/mL) or artificial cerebral spinal fluid (ACSF) before a 30-min session in which pressing one lever delivered food pellets and pressing the other delivered fruit punch. Learning during this session was tested the next day by sating the animals on either the pellets or fruit punch before assessing their performance on the two levers in extinction. The ACSF group selectively reduced responding on the lever that, in training, had earned the now devalued outcome, whereas the APV group did not. Experiment 2 replicated the effect of APV during the critical training session but found no effect of APV given after acquisition and before test. Furthermore, Experiment 3 showed that the effect of APV on instrumental learning was restricted to the pDMS; infusion into the dorsolateral striatum did not prevent learning. These experiments provide the first direct evidence that, in instrumental conditioning, NMDARs in the dorsomedial striatum are involved in encoding action-outcome associations.

The role of the dorsomedial striatum in instrumental conditioning

Considerable evidence suggests that, in instrumental conditioning, rats learn the relationship between actions and their specific consequences or outcomes. The present study examined the role of the dorsomedial striatum (DMS) in this type of learning after excitotoxic lesions and reversible, muscimol-induced inactivation. In three experiments, rats were first trained to press two levers for distinct outcomes, and then tested after training using a variety of behavioural assays that have been established to detect action-outcome learning. In Experiment 1, pre-training lesions of the posterior DMS abolished the sensitivity of rats' instrumental performance to both outcome devaluation and contingency degradation when tested in extinction, whereas lesions of the anterior DMS had no effect. In Experiment 2, both pre-training and post-training lesions of the posterior DMS were equally effective in reducing the sensitivity of performance both to devaluation and degradation treatments. In Experiment 3, the infusion of muscimol into the posterior DMS selectively abolished sensitivity of performance to devaluation and contingency degradation without impairing the ability of rats to discriminate either the instrumental actions performed or the identity of the earned outcomes. Taken together, these results suggest that the posterior region of the DMS is a crucial neural substrate for the acquisition and expression of action-outcome associations in instrumental conditioning.

Involvement of NMDA and AMPA/KA receptors in the nucleus accumbens core in instrumental learning guided by reward-predictive cues

The use of reward-predictive cues to guide behavior critically involves the nucleus accumbens. However, little is known regarding the role of ionotropic glutamate receptors in the core subregion of the nucleus accumbens (AcbC) in instrumental learning guided by reward-predictive cues. Here we examined the effects of an intra-AcbC blockade of NMDA and AMPA/KA receptors on the acquisition of an instrumental response in a reaction time (RT) task in rats. In this task, discriminative cues signaled in advance the upcoming reward magnitude (5 or 1 food pellet) associated with a lever release. During early acquisition (days 1-6) rats received daily bilateral injections of either the NMDA receptor antagonist AP5 (5.0 microg per side, n = 14), the AMPA/KA receptor antagonist CNQX (2.5 microg per side, n = 14) or vehicle (0.5 microL per side, n = 19). No treatment was given during late acquisition (days 7-12). The main result was that rats which received intra-AcbC injections of AP5 or CNQX during early acquisition exhibited a general RT increase of responses to high and low reward. However, treatment with AP5 and CNQX did not interfere with discriminative guidance of RTs by cue-associated reward magnitudes, i.e. during acquisition RTs of responses to expected high reward became significantly faster than RTs of responses to expected low reward. Our findings suggest that NMDA and AMPA/KA receptors in the AcbC play a critical role in invigorating responding during instrumental learning, but seem less important in guiding responding according to reward-predictive cues.

Ratio and time requirements on operant schedules: effort-related effects of nucleus accumbens dopamine depletions

Accumbens dopamine (DA) depletions produce deficits that are related to the ratio requirement of the operant schedule; however, it is also possible that time without reinforcement is a factor. The present study examined the effects of accumbens DA depletions in rats using variable interval (VI) schedules with additional fixed ratio (FR) requirements. Four VI schedules were used (VI 60/FR 1, VI 120/FR 1, VI 60/FR 10, VI 120/FR 10). Attachment of the additional work requirement increased response rates under control conditions. After surgery, there was no interaction between interval level (i.e. 60 vs. 120 s) and DA depletion, but there was a significant interaction between ratio requirement (i.e. 1 vs. 10) and DA depletion within the first week after surgery. DA depletions substantially impaired performance on the schedules with added FR 10 requirements, an effect that was largely dependent upon a reduction in fast responses (i.e. inter-response times less than 1.0 s). There was little effect of DA depletion on overall responding on VI 60/FR 1 and VI 120/FR 1 schedules. DA depletions also increased the tendency to take long pauses in responding (i.e. > 20.0 s), and this effect was evident across all schedules tested. Thus, accumbens DA depletions interact with work requirements and blunt the rate-enhancing effects of moderate size ratios, and also enhance the tendency to pause. Attachment of ratio requirements to interval schedules is a work-related response cost that provides a challenge to the organism, and DA in nucleus accumbens appears to be necessary for adapting to this challenge.

Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats

Background

Animals must frequently make choices between alternative courses of action, seeking to maximize the benefit obtained. They must therefore evaluate the magnitude and the likelihood of the available outcomes. Little is known of the neural basis of this process, or what might predispose individuals to be overly conservative or to take risks excessively (avoiding or preferring uncertainty, respectively). The nucleus accumbens core (AcbC) is known to contribute to rats' ability to choose large, delayed rewards over small, immediate rewards; AcbC lesions cause impulsive choice and an impairment in learning with delayed reinforcement. However, it is not known how the AcbC contributes to choice involving probabilistic reinforcement, such as between a large, uncertain reward and a small, certain reward. We examined the effects of excitotoxic lesions of the AcbC on probabilistic choice in rats.

Results

Rats chose between a single food pellet delivered with certainty (p = 1) and four food pellets delivered with varying degrees of uncertainty (p = 1, 0.5, 0.25, 0.125, and 0.0625) in a discrete-trial task, with the large-reinforcer probability decreasing or increasing across the session. Subjects were trained on this task and then received excitotoxic or sham lesions of the AcbC before being retested. After a transient period during which AcbC-lesioned rats exhibited relative indifference between the two alternatives compared to controls, AcbC-lesioned rats came to exhibit risk-averse choice, choosing the large reinforcer less often than controls when it was uncertain, to the extent that they obtained less food as a result. Rats behaved as if indifferent between a single certain pellet and four pellets at p = 0.32 (sham-operated) or at p = 0.70 (AcbC-lesioned) by the end of testing. When the probabilities did not vary across the session, AcbC-lesioned rats and controls strongly preferred the large reinforcer when it was certain, and strongly preferred the small reinforcer when the large reinforcer was very unlikely (p = 0.0625), with no differences between AcbC-lesioned and sham-operated groups.

Conclusion

These results support the view that the AcbC contributes to action selection by promoting the choice of uncertain, as well as delayed, reinforcement.

GABAA-mediated inhibition of basolateral amygdala blocks reward devaluation in macaques

Amygdala ablation disrupts reinforcer "devaluation" in monkeys (Malkova et al., 1997). Here, we tested the hypothesis that transient inactivation of amygdala by the GABA(A) agonist muscimol (MUS), specifically during the period of reward satiation, would have a similar effect. Six pigtail macaques were trained on a visual object discrimination task in which 60 objects were associated with one of two specific food rewards. Subsequently, we evaluated the selective satiation-induced change (devaluation) in object preference in probe sessions. We also examined the effect of the amygdala inactivation during the probe sessions to determine whether the inactivation limited to the testing period (and not during the satiation period) is sufficient to impair the expression of reinforcer devaluation. MUS infusions were aimed at basolateral amygdala (BLA) in a pseudorandomized design; each monkey received MUS or saline either before or after selective satiation with each of the two food rewards (six infusions total). Under the control (saline) condition, the monkeys significantly shifted their preference from objects representing the sated food rewards to those representing the nonsated rewards (30% change). When BLA was inactivated during selective satiation (i.e., MUS infused before satiation), this devaluation effect was blocked. In contrast, MUS infusion after satiation, so that it was present just during the testing period, did not impair the shift in object preference (27% change). Thus, BLA is necessary for the appropriate registration of the change in the reinforcer value but not for the subsequent expression of the devaluation involving its transfer to secondary reinforcers.

Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output

The nucleus accumbens (NAc) is a key component of the brain's reward pathway, yet little is known of how NAc cells respond to primary rewarding or aversive stimuli. Here, naive rats received brief intraoral infusions of sucrose and quinine paired with cues in a classical conditioning paradigm while the electrophysiological activity of individual NAc neurons was recorded. NAc neurons (102) were typically inhibited by sucrose (39 of 52, 75%) or excited by quinine (30 of 40, 75%) infusions. Changes in firing rate were correlated with the oromotor response to intraoral infusions. Most taste-responsive neurons responded to only one of the stimuli. NAc neurons developed responses to the cues paired with sucrose and quinine. Thus, NAc neurons are innately tuned to rewarding and aversive stimuli and rapidly develop responses to predictive cues. The results indicate that the output of the NAc is very different when rats taste rewarding versus aversive stimuli.

Nucleus accumbens neurons in the rat exhibit differential activity to conditioned reinforcers and primary reinforcers within a second-order schedule of saccharin reinforcement

The nucleus accumbens has been associated with processing information related to primary reinforcement and reward. Most neurophysiological studies report that nucleus accumbens neurons are phasically excited in response to the onsets of salient events during the seeking of reinforcement and to the delivery of primary reinforcers. However, a minority of studies report inhibition during primary reinforcement. We recorded from 65 neurons in the nucleus accumbens whilst thirsty rats performed under a second-order schedule of saccharin reinforcement. This allowed us to analyse neural activity and behaviour during reinforcer-seeking in the presence of conditioned reinforcers (second-order stimuli, also called 'conditioned stimuli'), and during primary reinforcer consumption. Specifically, we sought to examine the valence of potential neural responses to primary reinforcement, to compare these responses to second-order stimulus-evoked responses, and to determine whether responses were differential to second-order stimuli presented at different time points within the schedule. Fifty out of 65 neurons we sampled responded to the second-order stimulus and/or consumption of the primary reinforcer. Most neurons in our sample exhibited excitation following the second-order stimulus and inhibition to the primary reinforcer, a pattern also present over the average response of the neural population. However, there was no systematic variation in neural responses evoked by second-order stimuli presented at different temporal proximities to primary reinforcement. Our results provide evidence that partially overlapping mechanisms within the nucleus accumbens differentially process conditioned reinforcers and primary reinforcers.

Conditioned activity and instrumental reinforcement following long-term oral consumption of cocaine by rats

Food-reinforced conditioned activity and instrumental responding were measured in rats after 50 weeks of continuous access to a cocaine-saccharin solution in their home cages. The elevation of conditioned activity produced by acute access to the cocaine-saccharin solution in the home cage during testing was abolished by long-term preexposure to the cocaine solution, an effect that was reversed by systemic administration of cocaine immediately prior to testing. By contrast, chronic cocaine preexposure enhanced instrumental responding for both cocaine-sucrose and pure sucrose solutions. These results support the idea that long-term cocaine exposure enhances subsequent reinforcement.

Nucleus accumbens core lesions retard instrumental learning and performance with delayed reinforcement in the rat

Background

Delays between actions and their outcomes severely hinder reinforcement learning systems, but little is known of the neural mechanism by which animals overcome this problem and bridge such delays. The nucleus accumbens core (AcbC), part of the ventral striatum, is required for normal preference for a large, delayed reward over a small, immediate reward (self-controlled choice) in rats, but the reason for this is unclear. We investigated the role of the AcbC in learning a free-operant instrumental response using delayed reinforcement, performance of a previously-learned response for delayed reinforcement, and assessment of the relative magnitudes of two different rewards.

Results

Groups of rats with excitotoxic or sham lesions of the AcbC acquired an instrumental response with different delays (0, 10, or 20 s) between the lever-press response and reinforcer delivery. A second (inactive) lever was also present, but responding on it was never reinforced. As expected, the delays retarded learning in normal rats. AcbC lesions did not hinder learning in the absence of delays, but AcbC-lesioned rats were impaired in learning when there was a delay, relative to sham-operated controls. All groups eventually acquired the response and discriminated the active lever from the inactive lever to some degree. Rats were subsequently trained to discriminate reinforcers of different magnitudes. AcbC-lesioned rats were more sensitive to differences in reinforcer magnitude than sham-operated controls, suggesting that the deficit in self-controlled choice previously observed in such rats was a consequence of reduced preference for delayed rewards relative to immediate rewards, not of reduced preference for large rewards relative to small rewards. AcbC lesions also impaired the performance of a previously-learned instrumental response in a delay-dependent fashion.

Conclusions

These results demonstrate that the AcbC contributes to instrumental learning and performance by bridging delays between subjects' actions and the ensuing outcomes that reinforce behaviour.

The nucleus accumbens is not critically involved in mediating the effects of a safety signal on behavior

Although considerable progress has been made towards understanding the neural systems mediating conditioned fear, little is known about the neural mechanisms underlying conditioned inhibitors of fear (or safety signals). The present series of experiments examined the involvement of the nucleus accumbens (NAC) in mediating the effects of safety signals on behavior using a conditioned inhibition of fear-potentiated startle paradigm. Neither increasing dopaminergic nor decreasing glutamatergic function in the NAC altered the magnitude of conditioned fear or conditioned inhibition of fear in rats. Furthermore, large pre- or post-training electrolytic lesions of the NAC did not affect acquisition or expression of fear-potentiated startle or conditioned inhibition of fear-potentiated startle. Taken together, these data suggest that the NAC is not critically involved in the acquisition or expression of fear-potentiated startle or conditioned inhibition of fear-potentiated startle. Previous research has implicated the NAC in 'reward-attenuated startle' in which presentation of a stimulus paired with food decreased startle responding. The present results, therefore, indicate important neural dissociations between the processing of appetitive and safety signals, even though behavioral studies and learning theories have suggested that these two forms of learning share some commonalities.

The induction of behavioural sensitization is associated with cocaine-induced structural plasticity in the core (but not shell) of the nucleus accumbens

Repeated exposure to cocaine increases the density of dendritic spines on medium spiny neurons in the nucleus accumbens (Acb) and pyramidal cells in the medial prefrontal cortex (mPFC). To determine if this is associated with the development of psychomotor sensitization, rats were given daily i.p. injections of 15 mg/kg of cocaine (or saline) for 8 days, either in their home cage (which failed to induce significant psychomotor sensitization) or in a distinct and relatively novel test cage (which induced robust psychomotor sensitization). Their brains were obtained 2 weeks after the last injection and processed for Golgi-Cox staining. In the Acb core (AcbC) cocaine treatment increased spine density only in the group that developed psychomotor sensitization (i.e. in the Novel but not Home group), and there was a significant positive correlation between the degree of psychomotor sensitization and spine density. In the Acb shell (AcbS) cocaine increased spine density to the same extent in both groups; i.e. independent of psychomotor sensitization. In the mPFC cocaine increased spine density in both groups, but to a significantly greater extent in the Novel group. Furthermore, when rats were treated at Home with a higher dose of cocaine (30 mg/kg), cocaine now induced psychomotor sensitization in this context, and also increased spine density in the AcbC. Thus, the context in which cocaine is experienced influences its ability to reorganize patterns of synaptic connectivity in the Acb and mPFC, and the induction of psychomotor sensitization is associated with structural plasticity in the AcbC and mPFC, but not the AcbS.

Dynamics of neural coding in the accumbens during extinction and reinstatement of rewarded behavior

Neural correlates of reward-seeking behavior are observed in the nucleus accumbens (NAC). The dependence of these correlates upon the presence of a reward was studied by comparing the behavioral correlates observed when the presence of the reward was manipulated within a single behavioral session. Rats were well-trained on a continuous reinforcement instrumental task reinforced by 0.1 ml drops of 5% sucrose. Extracellular single-unit neural activity was recorded from electrode arrays implanted into the NAC when instrumental behavior was and then was not reinforced with sucrose (within-session extinction). A variable delay between the instrumental response and the sucrose delivery allowed for separation of neural activity related to these task events. A spike activity increase around the time of the instrumental response was the most common behavioral correlate, while a decrease in spike activity upon sucrose delivery was the second most common behavioral correlate. Following removal of the reinforcer, subjects continued to perform the instrumental response, allowing for the examination of response-related spike activity under extinction conditions in which the response was no longer reinforced by sucrose. A majority of the response-related neural activity patterns were lost when sucrose was no longer available. New neural responses also were detected during this period. For some subjects, the reinforcer was again made available during the same session. Encoding of the primary behavioral events during this period of reinstated reinforcer was similar, but not identical, to that observed during the first period of reinforced responding. These findings reveal that instrumental task-associated spike activity within the NAC is partially dependent upon the presence of the reinforcer, and that encoding across the population is distinct under reinforced and extinction conditions.

Contrasting effects of dopamine and glutamate receptor antagonist injection in the nucleus accumbens suggest a neural mechanism underlying cue-evoked goal-directed behavior

Discriminative stimuli (DSs) inform animals that reward can be obtained contingent on the performance of a specific behavior. Such stimuli reinstate drug-seeking behavior, evoke dopamine release in the nucleus accumbens (NAc) and excite and inhibit specific subpopulations of NAc neurons. Here we show in rats that DSs can reinstate food-seeking behavior. In addition, we compare the effects of injecting dopamine receptor antagonists into the NAc with those of general NAc inactivation on the performance of a DS task. Selective antagonism of D1 receptors reduced responding to the DS and increased the latency to respond, whereas general inactivation of NAc neuronal activity increased the latency to respond to the DS and increased behaviors extraneous to the task, such as responding in the absence of cues and responding on the inactive lever. Based on these results and our previous findings that NAc neuronal responses to DSs are dependent on the ventral tegmental area, we propose a model for the functional role of NAc neurons in controlling behavioral responses to reward-predictive stimuli.

Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning

Habits are controlled by antecedent stimuli rather than by goal expectancy. Interval schedules of feedback have been shown to generate habits, as revealed by the insensitivity of behaviour acquired under this schedule to outcome devaluation treatments. Two experiments were conducted to assess the role of the dorsolateral striatum in habit learning. In Experiment 1, sham operated controls and rats with dorsolateral striatum lesions were trained to press a lever for sucrose under interval schedules. After training, the sucrose was devalued by inducing taste aversion to it using lithium chloride, whereas saline injections were given to the controls. Only rats given the devaluation treatment reduced their consumption of sucrose and this reduction was similar in both the sham and the lesioned groups. All rats were then returned to the instrumental chamber for an extinction test, in which the lever was extended but no sucrose was delivered. In contrast to sham operated controls, rats with dorsolateral striatum lesions refrained from pressing the lever if the outcome was devalued. To assess the specificity of the role of dorsolateral striatum in this effect a second experiment was conducted in which a group with lesions of dorsomedial striatum was added. In relation now to both the sham and the dorsomedial lesioned groups, only rats with lesions of dorsolateral striatum significantly reduced responding after outcome devaluation. In conclusion, this study provides direct evidence that the dorsolateral striatum is necessary for habit formation. Furthermore, it suggests that, when the habit system is disrupted, control over instrumental performance reverts to the system controlling the performance of goal-directed instrumental actions.

Neural and psychological mechanisms underlying appetitive learning: links to drug addiction

The complexity of drug addiction mirrors the complexity of the psychological processes that motivate animals to work for any reinforcer, be it a natural reward or a drug. Here, we review the role of the nucleus accumbens, together with its dopaminergic and cortical innervation, in responding to reinforcement. One important contribution made by the nucleus accumbens is to the process through which neutral stimuli, once paired with a reinforcer such as a drug, have the capacity to motivate behaviour. This process may be one of several contributing to addiction, and it may be amenable to pharmacological intervention.

Cue-Evoked Firing of Nucleus Accumbens Neurons Encodes Motivational Significance During a Discriminative Stimulus Task

The nucleus accumbens (NAc) has long been thought of as a limbic-motor interface. Despite behavioral and anatomical evidence in favor of this idea, little is known about how NAc neurons encode information about motivationally relevant environmental cues and use this information to affect motor action. We therefore investigated the firing of these neurons during the performance of a discriminative stimulus (DS) task using simultaneous multiple single-unit recordings in rats. In this task, two stimuli are randomly presented to the animal: a DS, which signals the availability of a sucrose reward contingent on an operant response, and a similar but nonrewarded stimulus (NS). Subpopulations of NAc neurons increased or decreased their firing in association with several distinct components of the task. In this paper, we investigate cue- and operant-responsive neurons. Neurons excited and inhibited by cues showed larger firing changes in response to the DS than the NS and larger changes when the animal made an operant response to the cue than when the animal failed to respond. Excitations during operant responding were not modulated by the information contained by the cue, whereas inhibitions during operant responding were somewhat larger if the operant response occurred during the DS and somewhat smaller if they occurred in the absence of a cue. These results are consistent with the hypothesis that the firing of subpopulations of NAc neurons encode both the predictive value of environmental stimuli and the specific motor behaviors required to respond to them.

Selective deficits in appetitive conditioning as a consequence of ethanol withdrawal

The acquisition of a conditioned response to a cue associated with a fearful event has been shown to be impaired in animals that had been repeatedly withdrawn from ethanol, but not in animals with the same chronic ethanol treatment but only a single withdrawal episode [D. N. Stephens et al. (2001) Eur. J. Neurosci., 14, 2023-2031]. Lesion studies have shown that the amygdala plays a vital role in this type of conditioning process. Here we investigate aspects of conditioning for appetitive reinforcers in operant tasks, also shown to rely on amygdala processing, in rats following repeated withdrawal from ethanol. Rats were chronically treated with either an ethanol-containing liquid diet for 24 days continuously (single withdrawal) or interspersed with 2 x 3-day withdrawal periods (repeated withdrawal), or with a control diet (control). Two weeks after the final withdrawal, operant training began. In tasks that are impaired by lesions of the basolateral amygdala, conditioned reinforcement and reinforcer devaluation, there was no effect of chronic ethanol treatment or withdrawal on acquisition or performance. However, in a task that is dependent upon functioning of the central nucleus of the amygdala, Pavlovian-to-instrumental transfer, the single and repeated withdrawal groups were significantly impaired. Therefore, chronic ethanol treatment and withdrawal resulted in deficits in behavioural tasks that are sensitive to central but not to basolateral amygdala lesions, and may reflect different sensitivities of these areas to ethanol.

Addiction

The development of addiction involves a transition from casual to compulsive patterns of drug use. This transition to addiction is accompanied by many drug-induced changes in the brain and associated changes in psychological functions. In this article we present a critical analysis of the major theoretical explanations of how drug-induced alterations in psychological function might cause a transition to addiction. These include: (a) the traditional hedonic view that drug pleasure and subsequent unpleasant withdrawal symptoms are the chief causes of addiction; (b) the view that addiction is due to aberrant learning, especially the development of strong stimulus-response habits; (c) our incentive-sensitization view, which suggests that sensitization of a neural system that attributes incentive salience causes compulsive motivation or "wanting" to take addictive drugs; and (d) the idea that dysfunction of frontal cortical systems, which normally regulate decision making and inhibitory control over behavior, leads to impaired judgment and impulsivity in addicts.

Consequences of Multiple Withdrawals From Alcohol

This article represents the proceedings of a symposium at the 2003 annual meeting of the Research Society on Alcoholism in Fort Lauderdale, FL, organized by Theodora Duka and chaired by Dai Stephens. The purpose of the symposium was to examine the effects of multiple experiences of withdrawal from alcohol in animals made dependent on alcohol and in humans who are alcohol dependent. Parallels were drawn to the effects of repeated short-lived high-content alcohol exposures in animals and in humans who are social drinkers but indulge in binge drinking. The presentations were (1) Multiple detoxifications and risk of relapse in abstinent alcoholics, by John Gentry and Robert Malcolm; (2) Emotional and cognitive impairments after long-term use of alcohol: relationship to multiple detoxifications and binge drinking, by Theodora Duka; (3) The effect of repeated withdrawal from ethanol on conditioning to appetitive stimuli, by Tamzin Ripley, Gilyanna Borlikova, and Dai Stephens; (4) Alcohol withdrawal kindling: electrographic measures in a murine model of behavioral seizure sensitization, by Lynn Veatch and Howard Becker; and (5) Binge drinking induced changes in CNS, by Fulton Crews.

Learning-related changes in response patterns of prefrontal neurons during instrumental conditioning

A crucial aspect of organizing goal-directed behavior is the ability to form neural representations of relationships between environmental stimuli, actions and reinforcement. Very little is known yet about the neural encoding of response-reward relationships, a process which is deemed essential for purposeful behavior. To investigate this, tetrode recordings were made in the medial prefrontal cortex (PFC) of rats performing a Go-NoGo task. After task acquisition, a subset of neurons showed a sustained change in firing during the rewarded action sequence that was triggered by a specific visual cue. When these changes were monitored in the course of learning, they were seen to develop in parallel with the behavioral learning curve and were highly sensitive to a switch in reward contingencies. These sustained changes correlated with the reward-associated action sequence, not with sensory or reward-predicting properties of the cue or individual motor acts per se. This novel type of neural plasticity may contribute to the formation of response-reinforcer associations and of behavioral strategies for guiding goal-directed action.

Relations Between Pavlovian-Instrumental Transfer and Reinforcer Devaluation

Relations between posttraining reinforcer devaluation and Pavlovian-instrumental transfer were examined in 2 experiments. When a single reinforcer was used, extended training of the instrumental response increased transfer but reduced devaluation effects. When multiple instrumental reinforcers were used, both reinforcer-specific transfer and devaluation effects were less influenced by the amount of instrumental training. Finally, although reinforcer devaluation decreased both Pavlovian conditioned responses and baseline instrumental responding, it had no effect on either single-reinforcer or reinforcer-specific transfer. These results indicate that transfer and reinforcer devaluation can reflect different aspects of associative learning and that the nature of associative learning can be influenced by parameters such as the amount of training and the use of multiple reinforcers.

Lesions of nucleus accumbens disrupt learning about aversive outcomes

Nucleus accumbens (NAcc) is critical for encoding and using information regarding the learned significance of cues predictive of reward. However, its role in processing information about cues predictive of aversive outcomes is less well studied. Here, we examined the effects of NAcc lesions in an odor-guided discrimination task in which rats use odor cues predictive of either appetitive or aversive outcomes to guide responding. Rats with sham or neurotoxic lesions of NAcc were trained on a series of two-odor discrimination problems. Performance on each problem was assessed by monitoring accuracy of choice behavior and by measuring latency to respond for fluid reinforcement after odor sampling. After acquisition of four problems, rats were trained on serial reversals of the final problem. Rats with NAcc lesions exhibited normal choice performance relative to controls on both acquisition and reversal of the discrimination problems (indeed, lesioned rats exhibited a mild facilitation on the first discrimination problem). Despite normal choice performance, however, lesioned rats failed to show normal changes in response latency during discrimination learning, particularly on trials involving the aversive outcome. These findings are consistent with a deficit in processing cue-outcome associations. These results are compared with those obtained from studies of basolateral amygdala and orbitofrontal cortex lesions in this task and suggest that NAcc integrates the motivational value of both appetitive and aversive cues to bias or modulate the vigor of subsequent responding.

Neural encoding in ventral striatum during olfactory discrimination learning

A growing body of evidence implicates the ventral striatum in using information acquired through associative learning. The present study examined the activity of ventral striatal neurons in awake, behaving rats during go/no-go odor discrimination learning and reversal. Many neurons fired selectively to odor cues predictive of either appetitive (sucrose) or aversive (quinine) outcomes. Few neurons were selective when first exposed to the odors, but many acquired this differential activity as rats learned the significance of the cues. A substantial proportion of these neurons encoded the cues' learned motivational significance, and these neurons tended to reverse their firing selectivity after reversal of odor-outcome contingencies. Other neurons that became selectively activated during learning did not reverse, but instead appeared to encode specific combinations of cues and associated motor responses. The results support a role for ventral striatum in using the learned significance, both appetitive and aversive, of predictive cues to guide behavior.

Reward, motivation, and reinforcement learning

There is substantial evidence that dopamine is involved in reward learning and appetitive conditioning. However, the major reinforcement learning-based theoretical models of classical conditioning (crudely, prediction learning) are actually based on rules designed to explain instrumental conditioning (action learning). Extensive anatomical, pharmacological, and psychological data, particularly concerning the impact of motivational manipulations, show that these models are unreasonable. We review the data and consider the involvement of a rich collection of different neural systems in various aspects of these forms of conditioning. Dopamine plays a pivotal, but complicated, role.