Discriminative cues indicating reward magnitude continue to determine reaction time of rats following lesions of the nucleus accumbens

Social influence and external feedback control in humans

This article aims to unravel the dynamics of social influence by examining the processes that occur when one person is the target of another's influence. We hypothesized that these processes are part of a feedback loop system in an individual. This loop involves the situation (input), a goal state (reference), a comparator, a selection mechanism, a feedback predictor, and an action (output). Each element can become the target of social influence, and different types of social influence can be classified and explained by how these elements are targeted. For instance, attempting to persuade another person with strong arguments targets the goal state of the affected individual, while obedience targets the selection mechanism, and violence targets the action. In summary, this article aims to categorize, order, and explain phenomena in social influence research using a feedback loop framework focusing on the influenced individual.

Cue-Evoked Dopamine Neuron Activity Helps Maintain but Does Not Encode Expected Value

Cue-evoked midbrain dopamine (DA) neuron activity reflects expected value, but its influence on reward assessment is unclear. In mice performing a trial-based operant task, we test if bidirectional manipulations of cue or operant-associated DA neuron activity drive learning as a result of under- or overexpectation of reward value. We target optogenetic manipulations to different components of forced trials, when only one lever is presented, and assess lever biases on choice trials in the absence of photomanipulation. Although lever biases are demonstrated to be flexible and sensitive to changes in expected value, augmentation of cue or operant-associated DA signaling does not significantly alter choice behavior, and blunting DA signaling during any component of the forced trials reduces choice trial responses on the associated lever. These data suggest cue-evoked DA helps maintain cue-value associations but does not encode expected value as to set the benchmark against which received reward is judged.

The Differential Impact of a Response's Effectiveness and its Monetary Value on Response-Selection

While known reinforcers of behavior are outcomes that are valuable to the organism, recent research has demonstrated that the mere occurrence of an own-response effect can also reinforce responding. In this paper we begin investigating whether these two types of reinforcement occur via the same mechanism. To this end, we modified two different tasks, previously established to capture the influence of a response's effectiveness on the speed of motor-responses (indexed here by participants' reaction times). Specifically, in six experiments we manipulated both a response's 'pure' effectiveness and its outcome value (e.g., substantial versus negligible monetary reward) and measured the influence of both on the speed of responding. The findings strongly suggest that post action selection, responding is influenced only by pure effectiveness, as assessed by the motor system; thus, at these stages responding is not sensitive to abstract representations of the value of a response (e.g., monetary value). We discuss the benefit of distinguishing between these two necessary aspects of adaptive behavior namely, fine-tuning of motor-control and striving for desired outcomes. Finally, we embed the findings in the recently proposed Control-based response selection (CBRS) framework and elaborate on its potential for understanding motor-learning processes in developing infants.

A possible social relative reward effect: Influences of outcome inequity between rats during operant responding

Social interactions/situations have dramatic influences on motivation. Creating animal models examining these influences promotes a better understanding of the psychological and biological underpinnings of social motivation. Rodents are sensitive to social history/experience during associative conditioning and food-sharing tasks. Would reward-oriented operant behavior be sensitive to social influences by showing a negative contrast-like effect when another organism obtains a greater value outcome? We used a side-by-side arrangement of operant response chambers wherein one animal obtained consistently high reward signaled by a discrete cue. The neighboring, experimental rat experienced different combinations of high and low reward trial sequences. Control conditions included distraction from a conspecific in the neighboring chamber (rat distractor) or cue/food dispenser operating without a conspecific (program distractor) in addition to testing subjects alone. Results support an influence of the other animal actively performing the task on the experimental subject's behavior. Primarily, responding was significantly slower for the low reward trials while the neighboring rat was receiving the higher magnitude reward. The lever-press and not food-cup retrieval latency was significantly slower during exposure to a conspecific neighbor performing the operant task. The effect was not obtained in all session sequences and was more pronounced using longer series of consecutive low reward trials. The slowing effect was also obtained with the program-distractor experience in a different trial sequence. These findings suggest a social-induced negative incentive contrast effect in rats possibly mediated by an outcome inequity process that could have key similarities to complex situational-affective effects on motivation involving frustration or jealously.

Function of the nucleus accumbens in motor control during recovery after spinal cord injury

Motivation facilitates recovery after neuronal damage, but its mechanism is elusive. It is generally thought that the nucleus accumbens (NAc) regulates motivation-driven effort but is not involved in the direct control of movement. Using causality analysis, we identified the flow of activity from the NAc to the sensorimotor cortex (SMC) during the recovery of dexterous finger movements after spinal cord injury at the cervical level in macaque monkeys. Furthermore, reversible pharmacological inactivation of the NAc during the early recovery period diminished high-frequency oscillatory activity in the SMC, which was accompanied by a transient deficit of amelioration in finger dexterity obtained by rehabilitation. These results demonstrate that during recovery after spinal damage, the NAc up-regulates the high-frequency activity of the SMC and is directly involved in the control of finger movements.

I control therefore I do: judgments of agency influence action selection

Our sense of being agents, that is of willingly controlling both our own bodies and the external environment is ubiquitous if thin. Empirical and theoretical work on this 'sense of agency' has documented motivational, cognitive and neural influences on implicit (out of awareness) and explicit (conscious) judgments of agency. For example, fluency of action selection processes has been recently shown to affect judgments of one's degree of control over an external event. However, it is an open question whether and how such judgments of agency act as input to other processes. In this study we demonstrate that the opposite relationship between action selection and judgment of agency also exists. Specifically, we show that manipulating one's objective control over the environment influences both the speed and the frequency of performing an action associated with that control. This pattern bears a striking resemblance to the effect that tangible rewards have on action selection and suggests that positive control feedback is rewarding to the organism, consequently affecting action selection. If further corroborated this 'reward from control' may explain everyday addictions such as prolonged engagement in arcade games and pathological behaviors, such as stereotypy.

Ventral striatum lesions enhance stimulus and response encoding in dorsal striatum

BACKGROUND

The development of addiction is thought to reflect a transition from goal-directed to stimulus-response driven behavior, functions attributed to ventral (VS) and dorsal striatum (DS), respectively. In line with this theory, neuroadaptations that occur during prolonged drug use progress from VS to DS. Here we ask if VS dysfunction alone, independent of drug use, can affect neural selectivity in DS.

METHODS

To address this issue, we recorded from single neurons in DS while rats performed an odor-guided choice task for differently valued rewards in rats with and without unilateral VS lesions. In a separate group of animals, we used bilateral VS lesions to determine if VS was critical for performance on this task.

RESULTS

We describe data showing that unilateral lesions of VS enhance neural representations in DS during performance of a task that is dependent on VS. Furthermore, we show that VS is critical for reward-guided decision-making initially, but that rats regain function after several days.

CONCLUSIONS

These results suggest that loss of VS function, independent of chronic drug use, can trigger stronger encoding in DS in a reward-guided decision-making task and that the transition from VS to DS governed behavior observed in addiction might be due, in part, to initial loss of VS function.

Invigoration of reward seeking by cue and proximity encoding in the nucleus accumbens

A key function of the nucleus accumbens is to promote vigorous reward seeking, but the corresponding neural mechanism has not been identified despite many years of research. Here, we study cued flexible approach behavior, a form of reward seeking that strongly depends on the accumbens, and we describe a robust, single-cell neural correlate of behavioral vigor in the excitatory response of accumbens neurons to reward-predictive cues. Well before locomotion begins, this cue-evoked excitation predicts both the movement initiation latency and the speed of subsequent flexible approach responses, but not those of stereotyped, inflexible responses. Moreover, the excitation simultaneously signals the subject's proximity to the approach target, a signal that appears to mediate greater response vigor on trials that begin with the subject closer to the target. These results demonstrate a neural mechanism for response invigoration whereby accumbens neuronal encoding of reward availability and target proximity together drive the onset and speed of reward-seeking locomotion.

A model of reward- and effort-based optimal decision making and motor control

Costs (e.g. energetic expenditure) and benefits (e.g. food) are central determinants of behavior. In ecology and economics, they are combined to form a utility function which is maximized to guide choices. This principle is widely used in neuroscience as a normative model of decision and action, but current versions of this model fail to consider how decisions are actually converted into actions (i.e. the formation of trajectories). Here, we describe an approach where decision making and motor control are optimal, iterative processes derived from the maximization of the discounted, weighted difference between expected rewards and foreseeable motor efforts. The model accounts for decision making in cost/benefit situations, and detailed characteristics of control and goal tracking in realistic motor tasks. As a normative construction, the model is relevant to address the neural bases and pathological aspects of decision making and motor control.

Instantaneous and cumulative influences of competition on impulsive choices in domestic chicks

This study examined instantaneous and cumulative effects of competitive interactions on impulsiveness in the inter-temporal choices in domestic chicks. Chicks were trained to peck colored beads to gain delayed food rewards (1 or 6 grains of millet delivered after a delay ranging between 0 and 4.5 s), and were tested in binary choices between a small–short delay option (SS) and a large–long delay alternative (LL). To examine whether competitive foraging instantaneously changes impulsiveness, we intraindividually compared choices between two consecutive tests in different contexts, one with competitors and another without. We found that (1) the number of the choice of LL was not influenced by competition in the tests, but (2) the operant peck latency was shortened by competition, suggesting a socially enhanced incentive for food. To further examine the lasting changes, two groups of chicks were consecutively trained and tested daily for 2 weeks according to a “behavioral titration” procedure, one with competitors and another without. Inter-group comparisons of the choices revealed that (3) choice impulsiveness gradually decreased along development, while (4) the chicks trained in competition maintained a higher level of impulsiveness. These results suggest that competitive foraging causes impulsive choices not by direct/contextual modification. Causal link between the instantaneous enhancement of incentive and the gradual effects on impulsiveness remains to be examined. Some (yet unspecified) factors may be indirectly involved.

Dorsolateral prefrontal cortex modulates striatal reward encoding during reappraisal of reward anticipation

Recent research showed that cognitive emotion regulation (ER) both increases activity in the dorsolateral prefrontal cortex (DLPFC) and decreases striatal responsivity to monetary rewards. Using a mixed monetary incentive delay/memory task as well as functional magnetic resonance imaging, we tested in healthy subjects whether ER effectively attenuates striatal reward encoding during the anticipation of reward (€1.00 vs. €0.05 reward cues) as well as subsequent target reaction times (RTs), which are an indicator of motivation to obtain reward. ER significantly diminished feelings of pleasant anticipation and slowed down €1.00 target RT. At the neural level, ER increased activity in the DLPFC and attenuated reward encoding in the left putamen. Analyses of psychophysiological interaction revealed that DLPFC activity correlated more positively with putamen activity during €0.05 than during €1.00 reward trials. Furthermore, parametric modulations showed that anticipatory left putamen activity correlated with target RT during nonregulation. No such correlation could be observed during ER, suggesting that ER had abolished preparatory target RT encoding. Our results provide evidence that ER can attenuate behavioral and striatal measures of reward-related motivation and motor preparation. Furthermore, the present findings suggest that the DLPFC might contribute to successful regulation of reward via increased promotion of low-reward responses.

The uncertainty processing theory of motivation

Most theories describe motivation using basic terminology (drive, 'wanting', goal, pleasure, etc.) that fails to inform well about the psychological mechanisms controlling its expression. This leads to a conception of motivation as a mere psychological state 'emerging' from neurophysiological substrates. However, the involvement of motivation in a large number of behavioural parameters (triggering, intensity, duration, and directedness) and cognitive abilities (learning, memory, decision, etc.) suggest that it should be viewed as an information processing system. The uncertainty processing theory (UPT) presented here suggests that motivation is the set of cognitive processes allowing organisms to extract information from the environment by reducing uncertainty about the occurrence of psychologically significant events. This processing of information is shown to naturally result in the highlighting of specific stimuli. The UPT attempts to solve three major problems: (i) how motivations can affect behaviour and cognition so widely, (ii) how motivational specificity for objects and events can result from nonspecific neuropharmacological causal factors (such as mesolimbic dopamine), and (iii) how motivational interactions can be conceived in psychological terms, irrespective of their biological correlates. The UPT is in keeping with the conceptual tradition of the incentive salience hypothesis while trying to overcome the shortcomings inherent to this view.

Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards

The ventral striatum (VS) is thought to serve as a gateway whereby associative information from the amygdala and prefrontal regions can influence motor output to guide behavior. If VS mediates this "limbic-motor" interface, then one might expect neural correlates in VS to reflect this information. Specifically, neural activity should reflect the integration of motivational value with subsequent behavior. To test this prediction, we recorded from single units in VS while rats performed a choice task in which different odor cues indicated that reward was available on the left or on the right. The value of reward associated with a left or rightward movement was manipulated in separate blocks of trials by either varying the delay preceding reward delivery or by changing reward size. Rats' behavior was influenced by the value of the expected reward and the response required to obtain it, and activity in the majority of cue-responsive VS neurons reflected the integration of these two variables. Unlike similar cue-evoked activity reported previously in dopamine neurons, these correlates were only observed if the directional response was subsequently executed. Furthermore, activity was correlated with the speed at which the rats' executed the response. These results are consistent with the notion that VS serves to integrate information about the value of an expected reward with motor output during decision making.

Anticipatory reward signals in ventral striatal neurons of behaving rats

It has been proposed that the striatum plays a crucial role in learning to select appropriate actions, optimizing rewards according to the principles of 'Actor-Critic' models of trial-and-error learning. The ventral striatum (VS), as Critic, would employ a temporal difference (TD) learning algorithm to predict rewards and drive dopaminergic neurons. This study examined this model's adequacy for VS responses to multiple rewards in rats. The respective arms of a plus-maze provided rewards of varying magnitudes; multiple rewards were provided at 1-s intervals while the rat stood still. Neurons discharged phasically prior to each reward, during both initial approach and immobile waiting, demonstrating that this signal is predictive and not simply motor-related. In different neurons, responses could be greater for early, middle or late droplets in the sequence. Strikingly, this activity often reappeared after the final reward, as if in anticipation of yet another. In contrast, previous TD learning models show decremental reward-prediction profiles during reward consumption due to a temporal-order signal introduced to reproduce accurate timing in dopaminergic reward-prediction error signals. To resolve this inconsistency in a biologically plausible manner, we adapted the TD learning model such that input information is nonhomogeneously distributed among different neurons. By suppressing reward temporal-order signals and varying richness of spatial and visual input information, the model reproduced the experimental data. This validates the feasibility of a TD-learning architecture where different groups of neurons participate in solving the task based on varied input information.

Neural coding of reward magnitude in the orbitofrontal cortex of the rat during a five-odor olfactory discrimination task

The orbitofrontal cortex (OBFc) has been suggested to code the motivational value of environmental stimuli and to use this information for the flexible guidance of goal-directed behavior. To examine whether information regarding reward prediction is quantitatively represented in the rat OBFc, neural activity was recorded during an olfactory discrimination "go"/"no-go" task in which five different odor stimuli were predictive for various amounts of reward or an aversive reinforcer. Neural correlates related to both actual and expected reward magnitude were observed. Responses related to reward expectation occurred during the execution of the behavioral response toward the reward site and within a waiting period prior to reinforcement delivery. About one-half of these neurons demonstrated differential firing toward the different reward sizes. These data provide new and strong evidence that reward expectancy, regardless of reward magnitude, is coded by neurons of the rat OBFc, and are indicative for representation of quantitative information concerning expected reward. Moreover, neural correlates of reward expectancy appear to be distributed across both motor and nonmotor phases of the task.

Intact discrimination reversal learning but slowed responding to reward-predictive cues after dopamine D1 and D2 receptor blockade in the nucleus accumbens of rats

RATIONALE

The prediction error hypothesis of dopamine action states that dopamine signals are necessary for the brain to update the predictive significance of cues. Yet, little is known whether D1 or D2 receptor-mediated signals in the nucleus accumbens core (AcbC) are required to learn a reversal of the predictive significance of cues.

OBJECTIVE

Here we examined the effects of a selective D1 or D2 receptor blockade in the AcbC on learning a reversal of previously acquired cue-reward magnitude contingencies.

MATERIALS AND METHODS

Rats were trained on a reaction time (RT) task demanding conditioned lever release with discriminative visual cues signalling in advance the upcoming reward magnitude (one or five food pellets). After acquisition, RTs were guided by cue-associated reward magnitudes, i.e. RTs of responses were significantly shorter for expected high vs low reward. Thereafter, cue-reward magnitude contingencies were reversed. Reversal learning was tested for 12 daily sessions with intra-AcbC micro-infusions being given on sessions 1-6. Subjects received pre-trial infusions of the selective D1 or D2 receptor antagonists, SCH23390 (0.5, 2 microg per side) or raclopride (1, 4 microg per side), or vehicle (0.5 microl).

RESULTS

Intra-AcbC infusion of SCH23390 (0.5, 2 microg) or raclopride (1, 4 microg) did not inhibit discrimination reversal learning, but the higher dose of each drug increased RTs of instrumental responses.

CONCLUSIONS

In a visual discrimination task as used here, D1 and D2 receptor-mediated signals in the AcbC seem to be unnecessary in updating the reward-predictive significance of cues, rather, they serve to activate instrumental behaviour.

The nucleus accumbens as part of a basal ganglia action selection circuit

BACKGROUND

The nucleus accumbens is the ventral extent of the striatum, the main input nucleus of the basal ganglia. Recent hypotheses propose that the accumbens and its dopamine projection from the midbrain contribute to appetitive behaviors required to obtain reward. However, the specific nature of this contribution is unclear. In contrast, significant advances have been made in understanding the role of the dorsal striatum in action selection and decision making.

OBJECTIVE

In order to develop a hypothesis of the role of nucleus accumbens dopamine in action selection, the physiology and behavioral pharmacology of the nucleus accumbens are compared to those of the dorsal striatum.

HYPOTHESES

Three hypotheses concerning the role of dopamine in these structures are proposed: (1) that dopamine release in the dorsal striatum serves to facilitate the ability to respond appropriately to temporally predictable stimuli (that is, stimuli that are so predictable that animals engage in anticipatory behavior just prior to the stimulus); (2) that dopamine in the nucleus accumbens facilitates the ability to respond to temporally unpredictable stimuli (which require interruption of ongoing behavior); and (3) that accumbens neurons participate in action selection in response to such stimuli by virtue of their direct (monosynaptic inhibitory) and indirect (polysynaptic excitatory) projections to basal ganglia output nuclei.

Neural systems implicated in delayed and probabilistic reinforcement

This review considers the theoretical problems facing agents that must learn and choose on the basis of reward or reinforcement that is uncertain or delayed, in implicit or procedural (stimulus-response) representational systems and in explicit or declarative (action-outcome-value) representational systems. Individual differences in sensitivity to delays and uncertainty may contribute to impulsivity and risk taking. Learning and choice with delayed and uncertain reinforcement are related but in some cases dissociable processes. The contributions to delay and uncertainty discounting of neuromodulators including serotonin, dopamine, and noradrenaline, and of specific neural structures including the nucleus accumbens core, nucleus accumbens shell, orbitofrontal cortex, basolateral amygdala, anterior cingulate cortex, medial prefrontal (prelimbic/infralimbic) cortex, insula, subthalamic nucleus, and hippocampus are examined.

Involvement of NMDA and AMPA/KA receptors in the nucleus accumbens core in instrumental learning guided by reward-predictive cues

The use of reward-predictive cues to guide behavior critically involves the nucleus accumbens. However, little is known regarding the role of ionotropic glutamate receptors in the core subregion of the nucleus accumbens (AcbC) in instrumental learning guided by reward-predictive cues. Here we examined the effects of an intra-AcbC blockade of NMDA and AMPA/KA receptors on the acquisition of an instrumental response in a reaction time (RT) task in rats. In this task, discriminative cues signaled in advance the upcoming reward magnitude (5 or 1 food pellet) associated with a lever release. During early acquisition (days 1-6) rats received daily bilateral injections of either the NMDA receptor antagonist AP5 (5.0 microg per side, n = 14), the AMPA/KA receptor antagonist CNQX (2.5 microg per side, n = 14) or vehicle (0.5 microL per side, n = 19). No treatment was given during late acquisition (days 7-12). The main result was that rats which received intra-AcbC injections of AP5 or CNQX during early acquisition exhibited a general RT increase of responses to high and low reward. However, treatment with AP5 and CNQX did not interfere with discriminative guidance of RTs by cue-associated reward magnitudes, i.e. during acquisition RTs of responses to expected high reward became significantly faster than RTs of responses to expected low reward. Our findings suggest that NMDA and AMPA/KA receptors in the AcbC play a critical role in invigorating responding during instrumental learning, but seem less important in guiding responding according to reward-predictive cues.

Effects of lesions of the nucleus accumbens core on choice between small certain rewards and large uncertain rewards in rats

Background

Animals must frequently make choices between alternative courses of action, seeking to maximize the benefit obtained. They must therefore evaluate the magnitude and the likelihood of the available outcomes. Little is known of the neural basis of this process, or what might predispose individuals to be overly conservative or to take risks excessively (avoiding or preferring uncertainty, respectively). The nucleus accumbens core (AcbC) is known to contribute to rats' ability to choose large, delayed rewards over small, immediate rewards; AcbC lesions cause impulsive choice and an impairment in learning with delayed reinforcement. However, it is not known how the AcbC contributes to choice involving probabilistic reinforcement, such as between a large, uncertain reward and a small, certain reward. We examined the effects of excitotoxic lesions of the AcbC on probabilistic choice in rats.

Results

Rats chose between a single food pellet delivered with certainty (p = 1) and four food pellets delivered with varying degrees of uncertainty (p = 1, 0.5, 0.25, 0.125, and 0.0625) in a discrete-trial task, with the large-reinforcer probability decreasing or increasing across the session. Subjects were trained on this task and then received excitotoxic or sham lesions of the AcbC before being retested. After a transient period during which AcbC-lesioned rats exhibited relative indifference between the two alternatives compared to controls, AcbC-lesioned rats came to exhibit risk-averse choice, choosing the large reinforcer less often than controls when it was uncertain, to the extent that they obtained less food as a result. Rats behaved as if indifferent between a single certain pellet and four pellets at p = 0.32 (sham-operated) or at p = 0.70 (AcbC-lesioned) by the end of testing. When the probabilities did not vary across the session, AcbC-lesioned rats and controls strongly preferred the large reinforcer when it was certain, and strongly preferred the small reinforcer when the large reinforcer was very unlikely (p = 0.0625), with no differences between AcbC-lesioned and sham-operated groups.

Conclusion

These results support the view that the AcbC contributes to action selection by promoting the choice of uncertain, as well as delayed, reinforcement.

Systems-level integration of interval timing and reaction time

Reaction time (RT) procedures are a prominent tool for the study of information processing by humans and other animals. The interpretation of how RT changes after manipulating the appropriate experimental variables has contributed to the contemporary understanding of a variety of cognitive constructs, including attention and memory. With the use of properly designed tasks, evaluating how RT is modified in response to various neural perturbations has become common within the realms of behavioral and cognitive neuroscience. One interesting observation made during both human and animal RT experiments is that the RT to a signal often speeds-up as more time is allotted to prepare for the signal's onset-referred to as the preparatory interval (PI) effect. In the human RT literature, the PI effect has been used as evidence for time estimation playing a fundamental role in the determination of RT. On the other hand, our theoretical understanding of time estimation remains largely divorced from the RT findings in the animal cognition literature. In order to bridge these different perspectives, we provide here a review of the behavioral parallels between RT and interval-timing experiments. Moreover, both the PI effect and interval timing are shown to be jointly influenced by neuropathologies such as Parkinson's disease in humans or dopamine-depleting brain lesions in experimental animals. The primary goal of this review is to consider human and animal RT experiments within the broader context of interval timing. This is accomplished by first integrating human RT theory with scalar timing theory-the leading model of interval timing. Following this, both RT and interval timing are discussed at a brain systems level insofar as these two processes share common neural substrates. Our conclusion is that interval timing and RT processes are in fact two sides of the same coin.

Nucleus accumbens core lesions retard instrumental learning and performance with delayed reinforcement in the rat

Background

Delays between actions and their outcomes severely hinder reinforcement learning systems, but little is known of the neural mechanism by which animals overcome this problem and bridge such delays. The nucleus accumbens core (AcbC), part of the ventral striatum, is required for normal preference for a large, delayed reward over a small, immediate reward (self-controlled choice) in rats, but the reason for this is unclear. We investigated the role of the AcbC in learning a free-operant instrumental response using delayed reinforcement, performance of a previously-learned response for delayed reinforcement, and assessment of the relative magnitudes of two different rewards.

Results

Groups of rats with excitotoxic or sham lesions of the AcbC acquired an instrumental response with different delays (0, 10, or 20 s) between the lever-press response and reinforcer delivery. A second (inactive) lever was also present, but responding on it was never reinforced. As expected, the delays retarded learning in normal rats. AcbC lesions did not hinder learning in the absence of delays, but AcbC-lesioned rats were impaired in learning when there was a delay, relative to sham-operated controls. All groups eventually acquired the response and discriminated the active lever from the inactive lever to some degree. Rats were subsequently trained to discriminate reinforcers of different magnitudes. AcbC-lesioned rats were more sensitive to differences in reinforcer magnitude than sham-operated controls, suggesting that the deficit in self-controlled choice previously observed in such rats was a consequence of reduced preference for delayed rewards relative to immediate rewards, not of reduced preference for large rewards relative to small rewards. AcbC lesions also impaired the performance of a previously-learned instrumental response in a delay-dependent fashion.

Conclusions

These results demonstrate that the AcbC contributes to instrumental learning and performance by bridging delays between subjects' actions and the ensuing outcomes that reinforce behaviour.

Contrasting effects of dopamine and glutamate receptor antagonist injection in the nucleus accumbens suggest a neural mechanism underlying cue-evoked goal-directed behavior

Discriminative stimuli (DSs) inform animals that reward can be obtained contingent on the performance of a specific behavior. Such stimuli reinstate drug-seeking behavior, evoke dopamine release in the nucleus accumbens (NAc) and excite and inhibit specific subpopulations of NAc neurons. Here we show in rats that DSs can reinstate food-seeking behavior. In addition, we compare the effects of injecting dopamine receptor antagonists into the NAc with those of general NAc inactivation on the performance of a DS task. Selective antagonism of D1 receptors reduced responding to the DS and increased the latency to respond, whereas general inactivation of NAc neuronal activity increased the latency to respond to the DS and increased behaviors extraneous to the task, such as responding in the absence of cues and responding on the inactive lever. Based on these results and our previous findings that NAc neuronal responses to DSs are dependent on the ventral tegmental area, we propose a model for the functional role of NAc neurons in controlling behavioral responses to reward-predictive stimuli.

Transient inactivation of the rat nucleus accumbens does not impair guidance of instrumental behaviour by stimuli predicting reward magnitude

The involvement of the nucleus accumbens (NAc) in the determination of reaction times (RTs) of instrumental responses by the expectancy of future reward was investigated. A simple RT task demanding conditioned lever release was used, in which the upcoming reward magnitude (5 versus 1 pellet) was signalled in advance by discriminative cues. In rats which acquired the task, RTs of instrumental responses were significantly shorter to the discriminative cue predictive of high reward magnitude. Inactivation of the NAc by lidocaine had no effect on RTs and their determination by cue-associated reward magnitudes, and did not affect the rate of correct responses. In keeping with an earlier study, intra-NAc infusion of amphetamine decreased RTs, impaired RT determination by cue-associated reward magnitudes and reduced the rate of correct responses. The unexpected finding that lidocaine inactivation of the NAc had no effect parallels previous data showing that lesions of NAc did not impair RT performance, while manipulation of intra-NAc glutamate or dopamine transmission impaired various aspects of RT performance in comparable tasks. It is suggested that experimental manipulations such as transient and permanent inactivation, which almost completely inhibit NAc neuronal output, allow alternative routes to be used to effectively control behaviour in the task employed here.

Limbic Corticostriatal Systems and Delayed Reinforcement

Impulsive choice, one aspect of impulsivity, is characterized by an abnormally high preference for small, immediate rewards over larger delayed rewards, and can be a feature of adolescence, but also attention-deficit/hyperactivity disorder (ADHD), addiction, and other neuropsychiatric disorders. Both the serotonin and dopamine neuromodulator systems are implicated in impulsivity; manipulations of these systems affect animal models of impulsive choice, though these effects may depend on the receptor subtype and whether or not the reward is signaled. These systems project to limbic cortical and striatal structures shown to be abnormal in animal models of ADHD. Damage to the nucleus accumbens core (AcbC) causes rats to exhibit impulsive choice. These rats are also hyperactive, but are unimpaired in tests of visuospatial attention; they may therefore represent an animal model of the hyperactive-impulsive subtype of ADHD. Lesions to the anterior cingulate or medial prefrontal cortex, two afferents to the AcbC, do not induce impulsive choice, but lesions of the basolateral amygdala do, while lesions to the orbitofrontal cortex have had opposite effects in different tasks measuring impulsive choice. In theory, impulsive choice may emerge as a result of abnormal processing of the magnitude of rewards, or as a result of a deficit in the effects of delayed reinforcement. Recent evidence suggests that AcbC-lesioned rats perceive reward magnitude normally, but exhibit a selective deficit in learning instrumental responses using delayed reinforcement, suggesting that the AcbC is a reinforcement learning system that mediates the effects of delayed rewards.

Neurons in monkey prefrontal cortex whose activity tracks the progress of a three-step self-ordered task

The self-ordered task is a powerful tool for the analysis of dorsal prefrontal deficits. Each trial consists of a number of steps, and subjects must remember their choices in previous steps. The task becomes more difficult as the number of objects to be remembered increases. We recorded the activity of 156 neurons in the mid-dorsal prefrontal cortex of two rhesus monkeys performing an oculomotor version of the task. Although the task requires working memory, there was no convincing evidence for activity selective for the working memory of the objects that the monkey had to remember. Instead, nearly one-half of neurons (47%, 74/156) showed activity that was modulated according to the step of the task in any one or more task periods. Although the monkey's reward also increased with step, the neurons exhibited little or no step modulation in a reward control task in which reward increased without a concurrent increase in task difficulty. The activity of some neurons was also selective for the location of saccade target that the monkey voluntarily chose. Neurons showed less step modulation in error trials, and there was no increase between the second and third step responses on trials in which the error was on the third step. These results suggest that the mid-dorsal prefrontal cortex contributes to the self-ordered task, not by providing an object working memory signal, but by regulating some general aspect of the performance in the difficult task.

The rat nucleus accumbens is involved in guiding of instrumental responses by stimuli predicting reward magnitude

The present study examined the involvement of N-methyl-d-aspartate (NMDA), alpha-amino-3-hydroxy-5-methyl-4-isoxazolpropionate/kainate (AMPA/KA) and dopamine receptors in the nucleus accumbens (ACB) in influencing reaction times of instrumental responses by the expectancy of reward. A simple reaction time task demanding conditioned lever release was used in which the upcoming reward magnitude was signalled in advance by discriminative cues. After training, in control rats with vehicle infusions (0.5 micro L) into the ACB, reaction times of responses were significantly shorter to the discriminative cue predictive of high reward magnitude. Indirect stimulation of dopamine receptors in the ACB by d-amphetamine (20 micro g/0.5 micro L) decreased reaction times, impaired their guidance by cue-associated reward magnitudes and reduced the accuracy of task performance. Blockade of AMPA/KA receptors in the ACB by 6-cyano-7-nitroquino-xaline-2,3-dione (0.75 and 2.5 micro g/0.5 micro L) or NMDA receptors by d(-)-2-amino-5-phosphonopentanoic acid (5 micro g/0.5 micro L) produced a general increase in reaction times, but left guidance of reaction times by cue-associated reward magnitudes unaffected. Thus, stimulation of intra-ACB ionotropic glutamate receptors is critically involved in modulating the speed of instrumental responding to cues predictive for reward magnitude, but is not required for intact performance of previously learned instrumental behaviour.

Lesions of nucleus accumbens disrupt learning about aversive outcomes

Nucleus accumbens (NAcc) is critical for encoding and using information regarding the learned significance of cues predictive of reward. However, its role in processing information about cues predictive of aversive outcomes is less well studied. Here, we examined the effects of NAcc lesions in an odor-guided discrimination task in which rats use odor cues predictive of either appetitive or aversive outcomes to guide responding. Rats with sham or neurotoxic lesions of NAcc were trained on a series of two-odor discrimination problems. Performance on each problem was assessed by monitoring accuracy of choice behavior and by measuring latency to respond for fluid reinforcement after odor sampling. After acquisition of four problems, rats were trained on serial reversals of the final problem. Rats with NAcc lesions exhibited normal choice performance relative to controls on both acquisition and reversal of the discrimination problems (indeed, lesioned rats exhibited a mild facilitation on the first discrimination problem). Despite normal choice performance, however, lesioned rats failed to show normal changes in response latency during discrimination learning, particularly on trials involving the aversive outcome. These findings are consistent with a deficit in processing cue-outcome associations. These results are compared with those obtained from studies of basolateral amygdala and orbitofrontal cortex lesions in this task and suggest that NAcc integrates the motivational value of both appetitive and aversive cues to bias or modulate the vigor of subsequent responding.

Orbital prefrontal cortex and guidance of instrumental behaviour in rats under reversal conditions

The orbital prefrontal cortex (OPFC) is suggested to be part of a circuitry mediating the perception of reward and the initiation of adaptive behavioural responses. In the present study, we investigated in rats changes of goal-directed behaviour after bilateral OPFC-lesions by N-methyl-D-aspartate (NMDA) in more detail. A reaction time (RT) task was used which is sensitive to subtle changes in discriminative guidance of instrumental behaviour by the anticipated reward magnitudes. The task demands conditioned lever release triggered by an imperative stimulus. The upcoming reward magnitude (five or one food pellet) for each trial was randomly chosen and signalled in advance by distinct instructive stimuli. In trained rats, RTs of instrumental responses were determined by the two distinct stimulus-reward magnitude contingencies, i.e. RTs were shorter to the instructive stimulus predictive of the higher reward magnitude. Results show that lesions of the OPFC did not impair discriminative guidance of behavioural responses according to preoperatively acquired stimulus-reward magnitude contingencies. However, guidance of instrumental behaviour was altered in lesioned rats after a reversal of the stimulus-reward magnitude contingencies. The data add further support to the hypothesis that the rat OPFC is not involved in retrieval of acquired stimulus-reward magnitude contingencies but in integration of incentive information to guide behaviour after a reversal of stimulus-reward contingencies.

Effects of expectations for different reward magnitudes on neuronal activity in primate striatum

In behavioral science, it is well known that humans and nonhuman animals are highly sensitive to differences in reward magnitude when choosing an outcome from a set of alternatives. We know that a realm of behavioral reactions is altered when animals begin to expect different levels of reward outcome. Our present aim was to investigate how the expectation for different magnitudes of reward influences behavior-related neurophysiology in the anterior striatum. In a spatial delayed response task, different instruction pictures are presented to the monkey. Each image represents a different magnitude of juice. By reaching to the spatial location where an instruction picture was presented, animals could receive the particular liquid amount designated by the stimulus. Reliable preferences in reward choice trials and differences in anticipatory licks, performance errors, and reaction times indicated that animals differentially expected the various reward amounts predicted by the instruction cues. A total of 374 of 2,000 neurons in the anterior parts of the caudate nucleus, putamen, and ventral striatum showed five forms of task-related activation during the preparation or execution of movement and activations preceding or following the liquid drop delivery. Approximately one-half of these striatal neurons showed differing response levels dependent on the magnitude of liquid to be received. Results of a linear regression analysis showed that reward magnitude and single cell discharge rate were related in a subset of neurons by a monotonic positive or negative relationship. Overall, these data support the idea that the striatum utilizes expectancies that contain precise information concerning the predicted, forthcoming level of reward in directing general behavioral reactions.

Orbital prefrontal cortex and guidance of instrumental behavior of rats by visuospatial stimuli predicting reward magnitude

The orbital prefrontal cortex (OPFC) is part of a circuitry mediating the perception of reward and the initiation of adaptive behavioral responses. We investigated whether the OPFC is involved in guidance of the speed of instrumental behavior by visuospatial stimuli predictive of different reward magnitudes. Unoperated rats, sham-lesioned rats, and rats with bilateral lesions of the OPFC by N-methyl-D-aspartate (NMDA) were trained in a visuospatial discrimination task. The task required a lever press on the illuminated lever of two available to obtain a food reward. Different reward magnitudes were permanently assigned to lever presses to respective sides of the operant chamber; that is, responses to one lever (e.g., the left one) were always rewarded with one pellet and responses to the other lever with five pellets. On each trial, the position of the illuminated lever was pseudorandomly determined in advance. Results revealed that OPFC lesions did not impair acquisition of the task, as the speed of conditioned responses was significantly shorter with expectancy of a high reward magnitude. In addition, during reversal, shift and reshift of lever position-reward magnitude contingencies and under extinction conditions, performance of the OPFC-lesioned and control groups did not differ. It is concluded that the OPFC in rats might not be critical for adapting behavioral responses to changes of stimulus-reward magnitude contingencies signaled by visuospatial cues.

Dorsal striatum responses to reward and punishment: effects of valence and magnitude manipulations

The goal of this research was to further our understanding of how the striatum responds to the delivery of affective feedback. Previously, we had found that the striatum showed a pattern of sustained activation after presentation of a monetary reward, in contrast to a decrease in the hemodynamic response after a punishment. In this study, we tested whether the activity of the striatum could be modulated by parametric variations in the amount of financial reward or punishment. We used an event-related fMRI design in which participants received large or small monetary rewards or punishments after performance in a gambling task. A parametric ordering of conditions was observed in the dorsal striatum according to both magnitude and valence. In addition, an early response to the presentation of feedback was observed and replicated in a second experiment with increased temporal resolution. This study further implicates the dorsal striatum as an integral component of a reward circuitry responsible for the control of motivated behavior, serving to code for such feedback properties as valence and magnitude.

NMDA, but not dopamine D(2), receptors in the rat nucleus accumbens areinvolved in guidance of instrumental behavior by stimuli predicting reward magnitude

Expectancy of future reward is an important factor guiding the speed of instrumental behavior. The present study sought to explore whether signals transmitted via the NMDA subtype of glutamate receptors and via dopamine D(2) receptors in the nucleus accumbens (NAc) are critical for the determination of reaction times (RTs) of instrumental responses by the expectancy of future reward. A simple RT task for rats demanding conditioned lever release was used in which the upcoming reward magnitude (5 or 1 pellet) was signaled in advance by discriminative stimuli. In trained rats, RTs of conditioned responses with expectancy of a high reward magnitude were found to be significantly shorter. The shortening of RTs by stimuli predictive of high reward to be obtained was dose-dependently impaired by bilateral intra-NAc infusion of the competitive NMDA antagonist dl-2-amino-5-phosphonovaleric acid (APV) (1, 2, or 10 microg in 0.5 microl/side), but not by infusion of the preferential dopamine D(2) antagonist haloperidol (5 and 12.5 microg in 0.5 microl/side) or by infusion of vehicle (0.5 microl/side). In conclusion, the data reveal that in well trained animals stimulation of intra-NAc NMDA, but not of dopamine D(2), receptors, is critically involved in guiding the speed of instrumental responses according to stimuli predictive of the upcoming reward magnitude.

Frontal syndrome as a consequence of lesions in the pedunculopontine tegmental nucleus: a short theoretical review

In this review, it is argued that the consequence of bilateral damage to the pedunculopontine tegmental nucleus (PPTg) in experimental animals is the production of a form of frontal syndrome. Frontal syndrome is a term used to describe the behavioural consequences of damage to the frontal lobes in human patients. These behavioural changes can be classified as disinhibition of behaviour (a release of behavioural control), the production of inappropriate behaviour (which in patients can be either inappropriate actions or verbal behaviour), and the production of perseverative behaviour (the maintenance of an action beyond the point at which it should have been terminated). The psychological changes which underlie these behavioural changes are thought to involve executive functions, which include such things as the prospective planning of sequences of actions, attentional shifting and working memory. In this review, I attempt to demonstrate two things: first, that there are significant anatomical connections from frontostriatal systems to the PPTg. The motor cortex projects directly to the PPTg while the prefrontal cortex contacts it via striatal circuitry, forming clear routes by which the frontal lobes can communicate with the PPTg. Second, having established the existence of connections between frontostriatal systems and the PPTg, behavioural data are described. Experimental animals bearing bilateral lesions of the PPTg have been examined in a wide variety of tasks. Animals bearing such lesions are not impaired in basic processes of feeding, drinking, locomotion, or grooming and simple observation of lesioned rats' normal behaviour reveals no obvious gross impairment in function. However, the results of more subtle tests reveal a wide variety of deficits in various tasks. The outcome of these experiments are in many ways contradictory, but in the vast majority of cases, the changes can be described as involving disinhibition of behaviour, the release of inappropriate behaviour, and the production of perseverative behaviour. Anatomical and behavioural data support the conclusion that there are functional connections between frontal systems and the PPTg. This review also discusses what psychological processes might be served by such connections.

Reaction time responding in rats

The use of reaction time has a great tradition in the field of human information processing research. In animal research the use of reaction time test paradigms is mainly limited to two research fields: the role of the striatum in movement initiation; and aging. It was discussed that reaction time responding can be regarded as "single behavior", this term was used to indicate that only one behavioral category is measured, allowing a better analysis of brain-behavior relationships. Reaction time studies investigating the role of the striatum in motor functions revealed that the initiation of a behavioral response is dependent on the interaction of different neurotransmitters (viz. dopamine, glutamate, GABA). Studies in which lesions were made in different brain structures suggested that motor initiation is dependent on defined brain structures (e.g. medialldorsal striatum, prefrontal cortex). It was concluded that the use of reaction time measures can indeed be a powerful tool in studying brain-behavior relationships. However, there are some methodological constraints with respect to the assessment of reaction time in rats, as was tried to exemplify by the experiments described in the present paper. On the one hand one should try to control for behavioral characteristics of rats that may affect the validity of the parameter reaction time. On the other hand, the mean value of reaction time should be in the range of what has been reported in man. Although these criteria were not always met in several studies, it was concluded that reaction time can be validly assessed in rats. Finally, it was discussed that the use of reaction time may go beyond studies that investigate the role of the basal ganglia in motor output. Since response latency is a direct measure of information processing this parameter may provide insight into basic elements of cognition. Based on the significance of reaction times in human studies the use of this dependent variable in rats may provide a fruitful approach in studying brain-behavior relationships in cognitive functions.

Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials

Single neurons in the ventral striatum of primates carry signals that are related to reward and motivation. When monkeys performed a task requiring one to three bar release trials to be completed successfully before a reward was given, they seemed more motivated as the rewarded trials approached; they responded more quickly and accurately. When the monkeys were cued as to the progress of the schedule, 89 out of 150 ventral striatal neurons responded in at least one part of the task: (1) at the onset of the visual cue, (2) near the time of bar release, and/or (3) near the time of reward delivery. When the cue signaled progress through the schedule, the neuronal activity was related to the progress through the schedule. For example, one large group of these neurons responded in the first trial of every schedule, another large group responded in trials other than the first of a schedule, and a third large group responded in the first trial of schedules longer than one. Thus, these neurons coded the state of the cue, i.e., the neurons carried the information about how the monkey was progressing through the task. The differential activity disappeared on the first trial after randomizing the relation of the cue to the schedule. Considering the anatomical loop structure that includes ventral striatum and prefrontal cortex, we suggest that the ventral striatum might be part of a circuit that supports keeping track of progress through learned behavioral sequences that, when successfully completed, lead to reward.