Dopamine neuronal responses in monkeys performing visually cued reward schedules

Single caudate neurons encode temporally discounted value for formulating motivation for action

The term 'temporal discounting' describes both choice preferences and motivation for delayed rewards. Here we show that neuronal activity in the dorsal part of the primate caudate head (dCDh) signals the temporally discounted value needed to compute the motivation for delayed rewards. Macaque monkeys performed an instrumental task, in which visual cues indicated the forthcoming size and delay duration before reward. Single dCDh neurons represented the temporally discounted value without reflecting changes in the animal's physiological state. Bilateral pharmacological or chemogenetic inactivation of dCDh markedly distorted the normal task performance based on the integration of reward size and delay, but did not affect the task performance for different reward sizes without delay. These results suggest that dCDh is involved in encoding the integrated multi-dimensional information critical for motivation.

D1- and D2-like receptors differentially mediate the effects of dopaminergic transmission on cost-benefit evaluation and motivation in monkeys

It has been widely accepted that dopamine (DA) plays a major role in motivation, yet the specific contribution of DA signaling at D₁-like receptor (D₁R) and D₂-like receptor (D₂R) to cost–benefit trade-off remains unclear. Here, by combining pharmacological manipulation of DA receptors (DARs) and positron emission tomography (PET) imaging, we assessed the relationship between the degree of D₁R/D₂R blockade and changes in benefit- and cost-based motivation for goal-directed behavior of macaque monkeys. We found that the degree of blockade of either D₁R or D₂R was associated with a reduction of the positive impact of reward amount and increasing delay discounting. Workload discounting was selectively increased by D₂R antagonism. In addition, blocking both D₁R and D₂R had a synergistic effect on delay discounting but an antagonist effect on workload discounting. These results provide fundamental insight into the distinct mechanisms of DA action in the regulation of the benefit- and cost-based motivation, which have important implications for motivational alterations in both neurological and psychiatric disorders.

Using quantitatively controlled pharmacological manipulations, this study teases apart the role of D1- and D2-like dopamine receptors in motivation and goal-directed behavior in monkeys, revealing complementary roles of two dopamine receptor subtypes in the computation of the cost/benefit trade-off to guide action.

Noradrenergic But Not Dopaminergic Neurons Signal Task State Changes and Predict Reengagement After a Failure

The two catecholamines, noradrenaline and dopamine, have been shown to play comparable roles in behavior. Both noradrenergic and dopaminergic neurons respond to cues predicting reward availability and novelty. However, even though both are thought to be involved in motivating actions, their roles in motivation have seldom been directly compared. We therefore examined the activity of putative noradrenergic neurons in the locus coeruleus and putative midbrain dopaminergic neurons in monkeys cued to perform effortful actions for rewards. The activity in both regions correlated with engagement with a presented option. By contrast, only noradrenaline neurons were also (i) predictive of engagement in a subsequent trial following a failure to engage and (ii) more strongly activated in nonrepeated trials, when cues indicated a new task condition. This suggests that while both catecholaminergic neurons are involved in promoting action, noradrenergic neurons are sensitive to task state changes, and their influence on behavior extends beyond the immediately rewarded action.

What Is the Relationship between Dopamine and Effort?

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remains poorly understood, both at the behavioral and neurophysiological levels. This is important because unwillingness to overcome effort to gain reward is a common feature of many neuropsychiatric and neurological disorders. A recent surge in interest in the neurobiological basis of effort has led to seemingly conflicting results regarding the role of dopamine. We argue here that, upon closer examination, there is actually striking consensus across studies: dopamine primarily codes for future reward but is less sensitive to anticipated effort cost. This strong association between dopamine and the incentive effects of rewards places dopamine in a key position to promote reward-directed action.

Motivational neural circuits underlying reinforcement learning

Reinforcement learning (RL) is the behavioral process of learning the values of actions and objects. Most models of RL assume that the dopaminergic prediction error signal drives plasticity in frontal-striatal circuits. The striatum then encodes value representations that drive decision processes. However, the amygdala has also been shown to play an important role in forming Pavlovian stimulus-outcome associations. These Pavlovian associations can drive motivated behavior via the amygdala projections to the ventral striatum or the ventral tegmental area. The amygdala may, therefore, play a central role in RL. Here we compare the contributions of the amygdala and the striatum to RL and show that both the amygdala and striatum learn and represent expected values in RL tasks. Furthermore, value representations in the striatum may be inherited, to some extent, from the amygdala. The striatum may, therefore, play less of a primary role in learning stimulus-outcome associations in RL than previously suggested.

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

Dopamine-associated cached values are not sufficient as the basis for action selection

Phasic dopamine transmission is posited to act as a critical teaching signal that updates the stored (or "cached") values assigned to reward-predictive stimuli and actions. It is widely hypothesized that these cached values determine the selection among multiple courses of action, a premise that has provided a foundation for contemporary theories of decision making. In the current work we used fast-scan cyclic voltammetry to probe dopamine-associated cached values from cue-evoked dopamine release in the nucleus accumbens of rats performing cost-benefit decision-making paradigms to evaluate critically the relationship between dopamine-associated cached values and preferences. By manipulating the amount of effort required to obtain rewards of different sizes, we were able to bias rats toward preferring an option yielding a high-value reward in some sessions and toward instead preferring an option yielding a low-value reward in others. Therefore, this approach permitted the investigation of dopamine-associated cached values in a context in which reward magnitude and subjective preference were dissociated. We observed greater cue-evoked mesolimbic dopamine release to options yielding the high-value reward even when rats preferred the option yielding the low-value reward. This result identifies a clear mismatch between the ordinal utility of the available options and the rank ordering of their cached values, thereby providing robust evidence that dopamine-associated cached values cannot be the sole determinant of choices in simple economic decision making.

Injection parameters and virus dependent choice of promoters to improve neuron targeting in the nonhuman primate brain

We, like many others, wish to use modern molecular methods to alter neuronal functionality in primates. For us, this requires expression in a large proportion of the targeted cell population. Long generation times make germline modification of limited use. The size and intricate primate brain anatomy poses additional challenges. We surved methods using lentiviruses and serotypes of adeno-associated viruses (AAVs) to introduce active molecular material into cortical and subcortical regions of old-world monkey brains. Slow injections of AAV2 give well-defined expression of neurons in the cortex surrounding the injection site. Somewhat surprisingly we find that in the monkey the use of cytomegalovirus promoter in lentivirus primarily targets glial cells but few neurons. In contrast, with a synapsin promoter fragment the lentivirus expression is neuron specific at high transduction levels in all cortical layers. We also achieve specific targeting of tyrosine hydroxlase (TH)- rich neurons in the locus coeruleus and substantia nigra with a lentvirus carrying a fragment of the TH promoter. Lentiviruses carrying neuron specific promoters are suitable for both cortical and subcortical injections even when injected quickly.

The discovery of novel actions is affected by very brief reinforcement delays and reinforcement modality

ABSTRACT The authors investigated the ability of human participants to discover novel actions under conditions of delayed reinforcement. Participants used a joystick to search for a target indicated by visual or auditory reinforcement. Reinforcement delays of 75-150 ms were found to significantly impair action acquisition. They also found an effect of modality, with acquisition superior with auditory feedback. The duration at which delay was found to impede action discovery is, to the authors' knowledge, shorter than that previously reported from work with operant and causal learning paradigms. The sensitivity to delay reported, and the difference between modalities, is consistent with accounts of action discovery that emphasize the importance of a time stamp in the motor record for solving the credit assignment problem.

Neurons in monkey dorsal raphe nucleus code beginning and progress of step-by-step schedule, reward expectation, and amount of reward outcome in the reward schedule task

The dorsal raphe nucleus is the major source of serotonin in the brain. It is connected to brain regions related to reward processing, and the neurons show activity related to predicted reward outcome. Clinical observations also suggest that it is important in maintaining alertness and its apparent role in addiction seems to be related to reward processing. Here, we examined whether the neurons in dorsal raphe carry signals about reward outcome and task progress during multitrial schedules. We recorded from 98 single neurons in dorsal raphe of two monkeys. The monkeys perform one, two, or three visual discrimination trials (schedule), obtaining one, two, or three drops of liquid. In the valid cue condition, the length and brightness of a visual cue indicated schedule progress and reward amount, respectively. In the random cue condition, the visual cue was randomly presented with respect to schedule length and reward amount. We found information encoded about (1) schedule onset, (2) reward expectation, (3) reward outcome, and (4) reward amount in the mean firing rates. Information theoretic analysis showed that the temporal variation of the neuronal responses contained additional information related to the progress of the schedule toward the reward rather than only discriminating schedule onset or reward/no reward. When considered in light of all that is known about the raphe in anatomy, physiology, and behavior, the rich encoding about both task progress and predicted reward outcome makes the raphe a strong candidate for providing signals throughout the brain to coordinate persistent goal-seeking behavior.

Complementary neural correlates of motivation in dopaminergic and noradrenergic neurons of monkeys

Rewards have many influences on learning, decision-making, and performance. All seem to rely on complementary actions of two closely related catecholaminergic neuromodulators, dopamine (DA), and noradrenaline (NA). We compared single unit activity of dopaminergic neurons of the substantia nigra pars compacta (SNc) and noradrenergic neurons of the locus coeruleus (LC) in monkeys performing a reward schedule task. Their motivation, indexed using operant performance, increased as they progressed through schedules ending in reward delivery. The responses of dopaminergic and noradrenergic neurons around the time of major task events, visual cues predicting trial outcome and operant action to complete a trial were similar in that they occurred at the same time. They were also similar in that they both responded most strongly to the first cues in schedules, which are the most informative cues. The neuronal responses around the time of the monkeys' actions were different, in that the response intensity profiles changed in opposite directions. Dopaminergic responses were stronger around predictably rewarded correct actions whereas noradrenergic responses were greater around predictably unrewarded correct actions. The complementary response profiles related to the monkeys operant actions suggest that DA neurons might relate to the value of the current action whereas the noradrenergic neurons relate to the psychological cost of that action.

Encoding of reward expectation by monkey anterior insular neurons

The insula, a cortical brain region that is known to encode information about autonomic, visceral, and olfactory functions, has recently been shown to encode information during reward-seeking tasks in both single neuronal recording and functional magnetic resonance imaging studies. To examine the reward-related activation, we recorded from 170 single neurons in anterior insula of 2 monkeys during a multitrial reward schedule task, where the monkeys had to complete a schedule of 1, 2, 3, or 4 trials to earn a reward. In one block of trials a visual cue indicated whether a reward would or would not be delivered in the current trial after the monkey successfully detected that a red spot turned green, and in other blocks the visual cue was random with respect to reward delivery. Over one-quarter of 131 responsive neurons were activated when the current trial would (certain or uncertain) be rewarded if performed correctly. These same neurons failed to respond in trials that were certain, as indicated by the cue, to be unrewarded. Another group of neurons responded when the reward was delivered, similar to results reported previously. The dynamics of population activity in anterior insula also showed strong signals related to knowing when a reward is coming. The most parsimonious explanation is that this activity codes for a type of expected outcome, where the expectation encompasses both certain and uncertain rewards.

The prefrontal cortex and hybrid learning during iterative competitive games

Behavioral changes driven by reinforcement and punishment are referred to as simple or model-free reinforcement learning. Animals can also change their behaviors by observing events that are neither appetitive nor aversive when these events provide new information about payoffs available from alternative actions. This is an example of model-based reinforcement learning and can be accomplished by incorporating hypothetical reward signals into the value functions for specific actions. Recent neuroimaging and single-neuron recording studies showed that the prefrontal cortex and the striatum are involved not only in reinforcement and punishment, but also in model-based reinforcement learning. We found evidence for both types of learning, and hence hybrid learning, in monkeys during simulated competitive games. In addition, in both the dorsolateral prefrontal cortex and orbitofrontal cortex, individual neurons heterogeneously encoded signals related to actual and hypothetical outcomes from specific actions, suggesting that both areas might contribute to hybrid learning.

Dopamine neurons learn to encode the long-term value of multiple future rewards

Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multistep choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.

Dopamine in motivational control: rewarding, aversive, and alerting

Midbrain dopamine neurons are well known for their strong responses to rewards and their critical role in positive motivation. It has become increasingly clear, however, that dopamine neurons also transmit signals related to salient but nonrewarding experiences such as aversive and alerting events. Here we review recent advances in understanding the reward and nonreward functions of dopamine. Based on this data, we propose that dopamine neurons come in multiple types that are connected with distinct brain networks and have distinct roles in motivational control. Some dopamine neurons encode motivational value, supporting brain networks for seeking, evaluation, and value learning. Others encode motivational salience, supporting brain networks for orienting, cognition, and general motivation. Both types of dopamine neurons are augmented by an alerting signal involved in rapid detection of potentially important sensory cues. We hypothesize that these dopaminergic pathways for value, salience, and alerting cooperate to support adaptive behavior.

Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli

Midbrain dopamine neurons respond to reward-predictive stimuli. In the natural environment reward-predictive stimuli are often perceptually complicated. Thus, to discriminate one stimulus from another, elaborate sensory processing is necessary. Given that previous studies have used simpler types of reward-predictive stimuli, it has yet to be clear whether and, if so, how dopamine neurons obtain reward information from perceptually complicated stimuli. To investigate this, we recorded the activities of monkey dopamine neurons while they were performing discrimination between two coherent motion directions in random-dot motion stimuli. These coherent directions were paired with different magnitudes of reward. We found that dopamine neurons showed reward-predictive responses to random-dot motion stimuli. Moreover, dopamine neurons showed temporally extended activity correlated with changes in reward prediction (i.e., reward prediction error) from coarse to fine scales between initial motion detection and subsequent motion discrimination phases. Noticeably, dopamine reward-predictive responses became differential in a later phase than previously reported. This response pattern was consistent with the time course of processing required for the estimation of expected reward value that parallels the motion direction discrimination processing. The results demonstrate that dopamine neurons are able to reflect the reward value of perceptually complicated stimuli, and suggest that dopamine neurons use the moment-to-moment reward prediction associated with environmental stimuli to compute a reward prediction error.

Effort-based cost-benefit valuation and the human brain

In both the wild and the laboratory, animals' preferences for one course of action over another reflect not just reward expectations but also the cost in terms of effort that must be invested in pursuing the course of action. The ventral striatum and dorsal anterior cingulate cortex (ACCd) are implicated in the making of cost-benefit decisions in the rat, but there is little information about how effort costs are processed and influence calculations of expected net value in other mammals including humans. We performed a functional magnetic resonance imaging study to determine whether and where activity in the human brain was available to guide effort-based cost-benefit valuation. Subjects were scanned while they performed a series of effortful actions to obtain secondary reinforcers. At the beginning of each trial, subjects were presented with one of eight different visual cues that they had learned indicated how much effort the course of action would entail and how much reward could be expected at its completion. Cue-locked activity in the ventral striatum and midbrain reflected the net value of the course of action, signaling the expected amount of reward discounted by the amount of effort to be invested. Activity in ACCd also reflected the interaction of both expected reward and effort costs. Posterior orbitofrontal and insular activity, however, only reflected the expected reward magnitude. The ventral striatum and anterior cingulate cortex may be the substrate of effort-based cost-benefit valuation in primates as well as in rats.

Comparing the role of the anterior cingulate cortex and 6-hydroxydopamine nucleus accumbens lesions on operant effort-based decision making

Both the anterior cingulate cortex (ACC) and mesolimbic dopamine, particularly in the nucleus accumbens (NAc), have been implicated in allowing an animal to overcome effort constraints to obtain greater benefits. However, their exact contribution to such decisions has, to date, never been directly compared. To investigate this issue we tested rats on an operant effort-related cost-benefit decision-making task where animals selected between two response alternatives, one of which involved investing effort by lever pressing on a high fixed-ratio (FR) schedule to gain high reward [four food pellets (HR)], whereas the other led to a small amount of food on an FR schedule entailing less energetic cost [two food pellets, low reward (LR)]. All animals initially preferred to put in work to gain the HR. Systemic administration of a D2 antagonist caused a significant switch in choices towards the LR option. Similarly, post-operatively, excitotoxic ACC lesions caused a significant bias away from HR choices compared with sham-lesioned animals. There was no slowing in the speed of lever pressing and no correlation between time to complete the FR requirement and choice performance. Unexpectedly, no such alteration in choice allocation was observed in animals following 6-hydroxydopamine NAc lesions. However, these rats were consistently slower to initiate responding when cued to commence each trial and also showed a reduction in food hoarding on a species-typical foraging task. Taken together, this implies that only ACC lesions, and not 6-hydroxydopamine NAc lesions as performed here, cause a bias away from investing effort for greater reward when choosing between competing options

Prospective and retrospective temporal difference learning

A striking recent finding is that monkeys behave maladaptively in a class of tasks in which they know that reward is going to be systematically delayed. This may be explained by a malign Pavlovian influence arising from states with low predicted values. However, by very carefully analyzing behavioral data from such tasks, La Camera and Richmond (2008) observed the additional important characteristic that subjects perform differently on states in the task that are at equal distances from the future reward, depending on what has happened in the recent past. The authors pointed out that this violates the definition of state value in the standard reinforcement learning models that are ubiquitous as accounts of operant and classical conditioned behavior; they suggested and analyzed an alternative temporal difference (TD) model in which past and future are melded. Here, we show that, in fact, a standard TD model can actually exhibit the same behavior, and that this avoids deleterious consequences for choice. At the heart of the model is the average reward per step, which acts as a baseline for measuring immediate rewards. Relatively subtle changes to this baseline occasioned by the past can markedly influence predictions and thus behavior.

Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials

Midbrain dopaminergic neurons (DANs) typically increase their discharge rate in response to appetitive predictive cues and outcomes, whereas striatal cholinergic tonically active interneurons (TANs) decrease their rate. This may indicate that the activity of TANs and DANs is negatively correlated and that TANs can broaden the basal ganglia reinforcement teaching signal, for instance by encoding worse than predicted events. We studied the activity of 106 DANs and 180 TANs of two monkeys recorded during the performance of a classical conditioning task with cues predicting the probability of food, neutral, and air puff outcomes. DANs responded to all cues with elevations of discharge rate, whereas TANs depressed their discharge rate. Nevertheless, although dopaminergic responses to appetitive cues were larger than their responses to neutral or aversive cues, the TAN responses were more similar. Both TANs and DANs responded faster to an air puff than to a food outcome; however, DANs responded with a discharge elevation, whereas the TAN responses included major negative and positive deflections. Finally, food versus air puff omission was better encoded by TANs. In terms of the activity of single neurons with distinct responses to the different behavioral events, both DANs and TANs were more strongly modulated by reward than by aversive related events and better reflected the probability of reward than aversive outcome. Thus, TANs and DANs encode the task episodes differentially. The DANs encode mainly the cue and outcome delivery, whereas the TANs mainly encode outcome delivery and omission at termination of the behavioral trial episode.

Relation of locus coeruleus neurons in monkeys to Pavlovian and operant behaviors

Noradrenaline is released throughout the forebrain from locus coeruleus (LC) projections in close temporal proximity to emotional and goal-directed events. To examine interactive influences of these processes on LC neuronal activity, we used a task where Pavlovian and operant processes vary and can be easily identified. We recorded 69 single LC neurons from two monkeys performing a task where cues indicate the progression through schedules of one, two, or three operant trials. Pavlovian responses and phasic LC activations occur following the appearance of conditioned visual cues (54/69 neurons), especially those at the beginning of new schedules, whether the current trial will be rewarded (single trial schedule) or not (2 or 3 trial schedules), and after visual imperative signals eliciting the operant response (64/69 neurons), whether the current trial will be rewarded or not. The modulation of LC responses seems to be relatively independent of attention or motivation, because the responses do not covary with operant performance in the task. The magnitude of LC responses across the schedules varied in close relation to the intensity of Pavlovian behavior but these responses were also modulated by operant processes. Our conclusion is that LC activation occurs when task-relevant stimuli evoke a conditioned instinctive (Pavlovian) response, with the strength of the activation also being modulated by goal-directed processes. Thus locus coeruleus neurons broadcast information about stimulus-elicited primitive and goal-directed behaviors to forebrain structures important for executive functions and emotions.

Modeling the violation of reward maximization and invariance in reinforcement schedules

It is often assumed that animals and people adjust their behavior to maximize reward acquisition. In visually cued reinforcement schedules, monkeys make errors in trials that are not immediately rewarded, despite having to repeat error trials. Here we show that error rates are typically smaller in trials equally distant from reward but belonging to longer schedules (referred to as "schedule length effect"). This violates the principles of reward maximization and invariance and cannot be predicted by the standard methods of Reinforcement Learning, such as the method of temporal differences. We develop a heuristic model that accounts for all of the properties of the behavior in the reinforcement schedule task but whose predictions are not different from those of the standard temporal difference model in choice tasks. In the modification of temporal difference learning introduced here, the effect of schedule length emerges spontaneously from the sensitivity to the immediately preceding trial. We also introduce a policy for general Markov Decision Processes, where the decision made at each node is conditioned on the motivation to perform an instrumental action, and show that the application of our model to the reinforcement schedule task and the choice task are special cases of this general theoretical framework. Within this framework, Reinforcement Learning can approach contextual learning with the mixture of empirical findings and principled assumptions that seem to coexist in the best descriptions of animal behavior. As examples, we discuss two phenomena observed in humans that often derive from the violation of the principle of invariance: "framing," wherein equivalent options are treated differently depending on the context in which they are presented, and the "sunk cost" effect, the greater tendency to continue an endeavor once an investment in money, effort, or time has been made. The schedule length effect might be a manifestation of these phenomena in monkeys.

What is reinforced by phasic dopamine signals?

The basal ganglia have been associated with processes of reinforcement learning. A strong line of supporting evidence comes from the recording of dopamine (DA) neurones in behaving monkeys. Unpredicted, biologically salient events, including rewards cause a stereotypic short-latency (70-100 ms), short-duration (100-200 ms) burst of DA activity - the phasic response. This response is widely considered to represent reward prediction errors used as teaching signals in appetitive learning to promote actions that will maximise future reward acquisition. For DA signalling to perform this function, sensory processing afferent to DA neurones should discriminate unpredicted reward-related events. However, the comparative response latencies of DA neurones and orienting gaze-shifts indicate that phasic DA responses are triggered by pre-attentive sensory processing. Consequently, in circumstances where biologically salient events are both spatially and temporally unpredictable, it is unlikely their identity will be known at the time of DA signalling. The limited quality of afferent sensory processing and the precise timing of phasic DA signals, suggests that they may play a less direct role in 'Law of Effect' appetitive learning. Rather, the 'time-stamp' nature of the phasic response, in conjunction with the other signals likely to be present in the basal ganglia at the time of phasic DA input, suggests it may reinforce the discovery of unpredicted sensory events for which the organism is responsible. Furthermore, DA-promoted repetition of preceding actions/movements should enable the system to converge on those aspects of context and behavioural output that lead to the discovery of novel actions.

The influence of expected value on saccadic preparation

Basing higher-order decisions on expected value (reward probability x reward magnitude) maximizes an agent's accruement of reward over time. The goal of this study was to determine whether the advanced preparation of simple actions reflected the expected value of the potential outcomes. Human subjects were required to direct a saccadic eye movement to a visual target that was presented either to the left or right of a central fixation point on each trial. Expected value was manipulated by adjusting the probability of presenting each target and their associated magnitude of monetary reward across 15 blocks of trials. We found that saccadic reaction times (SRTs) were negatively correlated to the relative expected value of the targets. Occasionally, an irrelevant visual distractor was presented before the target to probe the spatial allocation of saccadic preparation. Distractor-directed errors (oculomotor captures) varied as a function of the relative expected value of, and the distance of distractors from, the potential valued targets. SRTs and oculomotor captures were better correlated to the relative expected value of actions than to reward probability, reward magnitude, or overall motivation. Together, our results suggest that the level and spatial distribution of competitive dynamic neural fields representing saccadic preparation reflect the relative expected value of the potential actions.

Dynamic Changes in Representations of Preceding and Upcoming Reward in Monkey Orbitofrontal Cortex

We investigated how orbitofrontal cortex (OFC) contributes to adaptability in the face of changing reward contingencies by examining how reward representations in monkey orbitofrontal neurons change during a visually cued, multi-trial reward schedule task. A large proportion of orbitofrontal neurons were sensitive to events in this task (69/80 neurons in the valid and 48/58 neurons in the random cue context). Neuronal activity depended upon preceding reward, upcoming reward, reward delivery, and schedule state. Preceding reward-dependent activity occurred in both the valid and random cue contexts, whereas upcoming reward-dependent activity was observed only in the valid context. A greater proportion of neurons encoded preceding reward in the random than the valid cue context. The proportion of neurons with preceding reward-dependent activity declined as each trial progressed, whereas the proportion encoding upcoming reward increased. Reward information was represented by ensembles of neurons, the composition of which changed with task context and time. Overall, neuronal activity in OFC adapted to reflect the importance of different types of reward information in different contexts and time periods. This contextual and temporal adaptability is one hallmark of neurons participating in executive functions.

Leading tonically active neurons of the striatum from reward detection to context recognition

Tonically active neurons (TANs) in the primate striatum, which are presumed to be cholinergic interneurons, carry signals that are traditionally considered to be important for reward-related learning. Recent studies investigating the functional properties of TANs in behaving monkeys have shown that other factors beyond motivation can affect their responsiveness. There is now evidence that TAN responses reflect stimulus detection, movement control and recognition of a specific context, suggesting that these local circuit neurons contribute to different computations used in learning and action functions of the striatum. This is consistent with the view that TAN responses could represent an important component of the processes that are responsible for the ability to select the appropriate behavioral response to environmental events.

Cue and reward signals carried by monkey entorhinal cortex neurons during reward schedules

Ablation of entorhinal/perirhinal cortices prevents learning associations between visual stimuli used as cues in reward schedules and the schedule state. Single neurons in perirhinal cortex are sensitive to associations between the cues and the reward schedules. To investigate whether neurons in the entorhinal cortex have similar sensitivities, we recorded single neuronal activity from two rhesus monkeys while the monkeys performed a visually cued reward schedule task. When the cue was related to the reward schedules, the monkeys made progressively fewer errors as the schedule state became closer to the reward state, showing that the monkeys were sensitive to the cue and the schedule state. Of 75 neurons recorded in the entorhinal cortex during task performance, about 30% responded. About half of these responded after cue presentation. When the relation of the cue to the reward schedules was random, the cue-related responses disappeared or lost their selectivity for schedule states. The responses of the entorhinal cortex neurons are similar to responses of perirhinal cortex neurons in that they are selective for the associative relationships between cues and reward schedules. However, they are particularly selective for the first trial of a new schedule, in contrast to perirhinal cortex where responsivity to all schedule states is seen. A different subpopulation of entorhinal neurons responded to the reward, unlike perirhinal neurons which respond solely to the cue. These results indicate that the entorhinal signals carry associative relationships between the visual cues and reward schedules, and between rewards and reward schedules that are not simply derived from perirhinal cortex by feed-forward serial processing.