Matching behavior and the representation of value in the parietal cortex

The Actor-Critic Learning Is Behind the Matching Law: Matching Versus Optimal Behaviors

The ability to make a correct choice of behavior from various options is crucial for animals' survival. The neural basis for the choice of behavior has been attracting growing attention in research on biological and artificial neural systems. Alternative choice tasks with variable ratio (VR) and variable interval (VI) schedules of reinforcement have often been employed in studying decision making by animals and humans. In the VR schedule task, alternative choices are reinforced with different probabilities, and subjects learn to select the behavioral response rewarded more frequently. In the VI schedule task, alternative choices are reinforced at different average intervals independent of the choice frequencies, and the choice behavior follows the so-called matching law. The two policies appear robustly in subjects' choice of behavior, but the underlying neural mechanisms remain unknown. Here, we show that these seemingly different policies can appear from a common computational algorithm known as actor-critic learning. We present experimentally testable variations of the VI schedule in which the matching behavior gives only a suboptimal solution to decision making and show that the actor-critic system exhibits the matching behavior in the steady state of the learning even when the matching behavior is suboptimal. However, it is found that the matching behavior can earn approximately the same reward as the optimal one in many practical situations.

Trial-by-trial updating of the gain in preparation for smooth pursuit eye movement based on past experience in humans

To understand how the CNS uses past experiences to generate movements that accommodate minute-by-minute environmental changes, we studied the trial-by-trial updating of the gain for initiating smooth pursuit eye movements and how this relates to the history of previous trials. Ocular responses in humans elicited by a small perturbing motion presented 300 ms after appearance of a target were used as a measure of the gain of visuomotor transmission. After the perturbation, the target was either moved horizontally (pursuit trial) or remained in a stationary position (fixation trial). The trial sequence randomly included pursuit and fixation. The amplitude of the response to the perturbation was modulated in a trial-by-trial manner based on the immediately preceding trial, with preceding fixation and pursuit trials decreasing and increasing the gain, respectively. The effect of the previous trial was larger with shorter intertrial intervals, but did not diminish for at least 2,000 ms. A time-series analysis showed that the response amplitude was significantly correlated with the past few trials, with dynamics that could be approximated by a first-order linear system. The results suggest that the CNS integrates recent experiences to set the gain in preparation for upcoming tracking movements in a changing environment.

The representation of economic value in the orbitofrontal cortex is invariant for changes of menu

Economic choice entails assigning values to the available options and is impaired by lesions to the orbitofrontal cortex (OFC). Recent results show that some neurons in the OFC encode the values that monkeys (Macaca mulatta) assign to different goods when they choose between them. A broad and fundamental question is how this neuronal representation of value depends on the behavioral context. Here we show that neuronal responses in the OFC are typically invariant for changes of menu. In other words, the activity of a neuron in response to one particular good usually does not depend on what other goods are available at the same time. Neurons in the OFC encode economic value, not relative preference. The fact that their responses are menu invariant suggests that transitivity, a fundamental trait of economic choice, may be rooted in the activity of individual neurons.

Positive and Negative Modulation of Motor Response in Primate Superior Colliculus by Reward Expectation

Expectation of reward is crucial for goal-directed behavior of animals. However, little is known about how reward information is used in the brain at the time of action. We investigated this question by recording from single neurons in the macaque superior colliculus (SC) while the animal was performing a memory-guided saccade task with an asymmetrical reward schedule. The SC is an ideal structure to ask this question because it receives inputs from many brain areas including the prefrontal cortex and the basal ganglia where reward information is thought to be encoded and sends motor commands to the brain stem saccade generators. We found two groups of SC neurons that encoded reward information in the presaccadic period: positive reward-coding neurons that showed higher activity when reward was expected and negative reward-coding neurons that showed higher activity when reward was not expected. The positive reward-coding usually started even before a cue for target position was presented, whereas the negative reward-coding was largely restricted to the presaccadic period. The two kinds of reward-coding may be useful for the animal to select an appropriate behavior in a complex environment.

Probing human and monkey anterior cingulate cortex in variable environments

Previous research has identified the anterior cingulate cortex (ACC) as an important node in the neural network underlying decision making in primates. Decision making can, however, be studied under a large variety of circumstances, ranging from the standard well-controlled lab situation to more natural, stochastic settings, in which multiple agents interact. Here, we illustrate how these different varieties of decision making studied can influence theories ofACC function in monkeys. Converging evidence from unit recordings and lesion studies now suggest that the ACC is important for interpreting outcome information according to the current task context to guide future action selection. We then apply this framework to the study of human ACC function and discuss its potential implications.

Do we expect natural selection to produce rational behaviour?

We expect that natural selection should result in behavioural rules which perform well; however, animals (including humans) sometimes make bad decisions. Researchers account for these with a variety of explanations; we concentrate on two of them. One explanation is that the outcome is a side effect; what matters is how a rule performs (in terms of reproductive success). Several rules may perform well in the environment in which they have evolved, but their performance may differ in a 'new' environment (e.g. the laboratory). Some rules may perform very badly in this environment. We use the debate about whether animals follow the matching law rather than maximizing their gains as an illustration. Another possibility is that we were wrong about what is optimal. Here, the general idea is that the setting in which optimal decisions are investigated is too simple and may not include elements that add extra degrees of freedom to the situation.

Cortical mechanisms of action selection: the affordance competition hypothesis

At every moment, the natural world presents animals with two fundamental pragmatic problems: selection between actions that are currently possible and specification of the parameters or metrics of those actions. It is commonly suggested that the brain addresses these by first constructing representations of the world on which to build knowledge and make a decision, and then by computing and executing an action plan. However, neurophysiological data argue against this serial viewpoint. In contrast, it is proposed here that the brain processes sensory information to specify, in parallel, several potential actions that are currently available. These potential actions compete against each other for further processing, while information is collected to bias this competition until a single response is selected. The hypothesis suggests that the dorsal visual system specifies actions which compete against each other within the fronto-parietal cortex, while a variety of biasing influences are provided by prefrontal regions and the basal ganglia. A computational model is described, which illustrates how this competition may take place in the cerebral cortex. Simulations of the model capture qualitative features of neurophysiological data and reproduce various behavioural phenomena.

Extending a biologically inspired model of choice: multi-alternatives, nonlinearity and value-based multidimensional choice

The leaky competing accumulator (LCA) is a biologically inspired model of choice. It describes the processes of leaky accumulation and competition observed in neuronal populations during choice tasks and it accounts for reaction time distributions observed in psychophysical experiments. This paper discusses recent analyses and extensions of the LCA model. First, it reviews the dynamics and examines the conditions that make the model achieve optimal performance. Second, it shows that nonlinearities of the type present in biological neurons improve performance when the number of choice alternatives increases. Third, the model is extended to value-based choice, where it is shown that nonlinearities in the value function explain risk aversion in risky choice and preference reversals in choice between alternatives characterized across multiple dimensions.

The neural correlates of subjective value during intertemporal choice

Neuroimaging studies of decision-making have generally related neural activity to objective measures (such as reward magnitude, probability or delay), despite choice preferences being subjective. However, economic theories posit that decision-makers behave as though different options have different subjective values. Here we use functional magnetic resonance imaging to show that neural activity in several brain regions--particularly the ventral striatum, medial prefrontal cortex and posterior cingulate cortex--tracks the revealed subjective value of delayed monetary rewards. This similarity provides unambiguous evidence that the subjective value of potential rewards is explicitly represented in the human brain.

Impaired strategic decision making in schizophrenia

Adaptive decision making in dynamic social settings requires frequent re-evaluation of choice outcomes and revision of strategies. This requires an array of multiple cognitive abilities, such as working memory and response inhibition. Thus, the disruption of such abilities in schizophrenia can have significant implications for social dysfunctions in affected patients. In the present study, 20 schizophrenia patients and 20 control subjects completed two computerized binary decision-making tasks. In the first task, the participants played a competitive zero-sum game against a computer in which the predictable choice behavior was penalized and the optimal strategy was to choose the two targets stochastically. In the second task, the expected payoffs of the two targets were fixed and unaffected by the subject's choices, so the optimal strategy was to choose the target with the higher expected payoff exclusively. The schizophrenia patients earned significantly less money during the first task, even though their overall choice probabilities were not significantly different from the control subjects. This was mostly because patients were impaired in integrating the outcomes of their previous choices appropriately in order to maintain the optimal strategy. During the second task, the choices of patients and control subjects displayed more similar patterns. This study elucidated the specific components in strategic decision making that are impaired in schizophrenia. The deficit, which can be characterized as strategic stiffness, may have implications for the poor social adjustment in schizophrenia patients.

The role of the superior intraparietal sulcus in supporting visual short-term memory for multifeature objects

Everyday objects can vary in a number of feature dimensions, such as color and shape. To identify and recognize a particular object, often times we need to encode and store multiple features of an object simultaneously. Previous studies have highlighted the role of the superior intraparietal sulcus (IPS) in storing single object features in visual short-term memory (VSTM), such as color, orientation, shape outline, and shape topology. The role of this brain area in storing multiple features of an object together in VSTM, however, remains mostly unknown. In this study, using an event-related functional magnetic resonance imaging design and an independent region-of-interest-based approach, how an object's color and shape may be retained together in the superior IPS during VSTM was investigated. Results from four experiments indicate that the superior IPS holds neither integrated whole objects nor the total number of objects (both whole and partial) stored in VSTM. Rather, it represents the total amount of feature information retained in VSTM. The ability to accumulate information acquired from different visual feature dimensions suggests that the superior IPS may be a flexible information storage device, consistent with the involvement of the parietal cortex in a variety of other cognitive tasks. These results also bring new understanding to the object benefit reported in behavioral VSTM studies and provide new insights into solving the binding problem in the brain.

Estimating the hidden learning representations

Successful adaptation relies on the ability to learn the consequence of our actions in different environments. However, understanding the neural bases of this ability still represents one of the great challenges of system neuroscience. In fact, the neuronal plasticity changes occurring during learning cannot be fully controlled experimentally and their evolution is hidden. Our approach is to provide hypotheses about the structure and dynamics of the hidden plasticity changes using behavioral learning theory. In fact, behavioral models of animal learning provide testable predictions about the hidden learning representations by formalizing their relation with the observables of the experiment (stimuli, actions and outcomes). Thus, we can understand whether and how the predicted learning processes are represented at the neural level by estimating their evolution and correlating them with neural data. Here, we present a bayesian model approach to estimate the evolution of the internal learning representations from the observations of the experiment (state estimation), and to identify the set of models' parameters (parameter estimation) and the class of behavioral model (model selection) that are most likely to have generated a given sequence of actions and outcomes. More precisely, we use Sequential Monte Carlo methods for state estimation and the maximum likelihood principle (MLP) for model selection and parameter estimation. We show that the method recovers simulated trajectories of learning sessions on a single-trial basis and provides predictions about the activity of different categories of neurons that should participate in the learning process. By correlating the estimated evolutions of the learning variables, we will be able to test the validity of different models of instrumental learning and possibly identify the neural bases of learning.

Expectation modulates neural responses to pleasant and aversive stimuli in primate amygdala

Animals and humans learn to approach and acquire pleasant stimuli and to avoid or defend against aversive ones. However, both pleasant and aversive stimuli can elicit arousal and attention, and their salience or intensity increases when they occur by surprise. Thus, adaptive behavior may require that neural circuits compute both stimulus valence--or value--and intensity. To explore how these computations may be implemented, we examined neural responses in the primate amygdala to unexpected reinforcement during learning. Many amygdala neurons responded differently to reinforcement depending upon whether or not it was expected. In some neurons, this modulation occurred only for rewards or aversive stimuli, but not both. In other neurons, expectation similarly modulated responses to both rewards and punishments. These different neuronal populations may subserve two sorts of processes mediated by the amygdala: those activated by surprising reinforcements of both valences-such as enhanced arousal and attention-and those that are valence-specific, such as fear or reward-seeking behavior.

Temporal filtering of reward signals in the dorsal anterior cingulate cortex during a mixed-strategy game

The process of decision making in humans and other animals is adaptive and can be tuned through experience so as to optimize the outcomes of their choices in a dynamic environment. Previous studies have demonstrated that the anterior cingulate cortex plays an important role in updating the animal's behavioral strategies when the action outcome contingencies change. Moreover, neurons in the anterior cingulate cortex often encode the signals related to expected or actual reward. We investigated whether reward-related activity in the anterior cingulate cortex is affected by the animal's previous reward history. This was tested in rhesus monkeys trained to make binary choices in a computer-simulated competitive zero-sum game. The animal's choice behavior was relatively close to the optimal strategy but also revealed small systematic biases that are consistent with the use of a reinforcement learning algorithm. In addition, the activity of neurons in the dorsal anterior cingulate cortex that was related to the reward received by the animal in a given trial often was modulated by the rewards in the previous trials. Some of these neurons encoded the rate of rewards in previous trials, whereas others displayed activity modulations more closely related to the reward prediction errors. In contrast, signals related to the animal's choices were represented only weakly in this cortical area. These results suggest that neurons in the dorsal anterior cingulate cortex might be involved in the subjective evaluation of choice outcomes based on the animal's reward history.

Understanding neural coding through the model-based analysis of decision making

The study of decision making poses new methodological challenges for systems neuroscience. Whereas our traditional approach linked neural activity to external variables that the experimenter directly observed and manipulated, many of the key elements that contribute to decisions are internal to the decider. Variables such as subjective value or subjective probability may be influenced by experimental conditions and manipulations but can neither be directly measured nor precisely controlled. Pioneering work on the neural basis of decision circumvented this difficulty by studying behavior in static conditions, in which knowledge of the average state of these quantities was sufficient. More recently, a new wave of studies has confronted the conundrum of internal decision variables more directly by leveraging quantitative behavioral models. When these behavioral models are successful in predicting a subject's choice, the model's internal variables may serve as proxies for the unobservable decision variables that actually drive behavior. This new methodology has allowed researchers to localize neural subsystems that encode hidden decision variables related to free choice and to study these variables under dynamic conditions.

Control of sensorimotor variability by consequences

Studies of reaction-time distributions provide a useful quantitative approach to understand decision processes at the neural level and at the behavioral level. A strong relationship between the spread of latencies and the median is generally accepted even though there has been no attempt to disentangle experimentally these two parameters. Here we test the ability to independently control the median and the variability in reaction times. Reaction times were measured in human subjects instructed to make a discrimination between a target and a distractor in a 2AFC task. In a first experiment, saccadic latencies were measured. In a second experiment, we used manual response reaction times. Subjects were trained to produce four different reaction-time distributions. A reinforcing feedback was given depending on both the variability and the median of the latency distributions. When low variability was reinforced, the standard deviation (SD) of reaction-time distributions were reduced by a factor of two and when high variability was reinforced, the SD returned to baseline level. Our procedure independently affected the spread and the median of the distribution patterns. By fitting the latency distributions using the Reddi and Carpenter LATER model, we found that these effects could be simulated by changing the distribution of the noise affecting the decision process. Our results demonstrate that learned contingencies can affect reaction time variability and support the view that the so-called noise level in decision processes can undergo long-term changes.

Learning the value of information in an uncertain world

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

The Neural Basis of Decision Making

The study of decision making spans such varied fields as neuroscience, psychology, economics, statistics, political science, and computer science. Despite this diversity of applications, most decisions share common elements including deliberation and commitment. Here we evaluate recent progress in understanding how these basic elements of decision formation are implemented in the brain. We focus on simple decisions that can be studied in the laboratory but emphasize general principles likely to extend to other settings.

Eye movements during rapid pointing under risk

We recorded saccadic eye movements during visually-guided rapid pointing movements under risk. We intended to determine whether saccadic end points are necessarily tied to the goals of rapid pointing movements or whether, when the visual features of a display and the goals of a pointing movement are different, saccades are driven by low-level features of the visual stimulus. Subjects pointed at a stimulus configuration consisting of a target region and a penalty region. Each target hit yielded a gain of points; each penalty hit incurred a loss of points. Late responses were penalized. The luminance of either target or penalty region was indicated by a disk which differed significantly from the background in luminance, while the other region was indicated by a thin circle. In subsequent experiments, we varied the visual salience of the stimulus configuration and found that manual responses followed near-optimal strategies maximizing expected gain, independent of the salience of the target region. We suggest that the final eye position is partially pre-programmed prior to hand movement initiation. While we found that manipulations of the visual salience of the display determined the end point of the initial saccade we also found that subsequent saccades are driven by the goal of the hand movement.

Optimal sensorimotor integration in recurrent cortical networks: a neural implementation of Kalman filters

Several behavioral experiments suggest that the nervous system uses an internal model of the dynamics of the body to implement a close approximation to a Kalman filter. This filter can be used to perform a variety of tasks nearly optimally, such as predicting the sensory consequence of motor action, integrating sensory and body posture signals, and computing motor commands. We propose that the neural implementation of this Kalman filter involves recurrent basis function networks with attractor dynamics, a kind of architecture that can be readily mapped onto cortical circuits. In such networks, the tuning curves to variables such as arm velocity are remarkably noninvariant in the sense that the amplitude and width of the tuning curves of a given neuron can vary greatly depending on other variables such as the position of the arm or the reliability of the sensory feedback. This property could explain some puzzling properties of tuning curves in the motor and premotor cortex, and it leads to several new predictions.

Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task

In probabilistic decision tasks, an expected value (EV) of a choice is calculated, and after the choice has been made, this can be updated based on a temporal difference (TD) prediction error between the EV and the reward magnitude (RM) obtained. The EV is measured as the probability of obtaining a reward x RM. To understand the contribution of different brain areas to these decision-making processes, functional magnetic resonance imaging activations related to EV versus RM (or outcome) were measured in a probabilistic decision task. Activations in the medial orbitofrontal cortex were correlated with both RM and with EV and confirmed in a conjunction analysis to extend toward the pregenual cingulate cortex. From these representations, TD reward prediction errors could be produced. Activations in areas that receive from the orbitofrontal cortex including the ventral striatum, midbrain, and inferior frontal gyrus were correlated with the TD error. Activations in the anterior insula were correlated negatively with EV, occurring when low reward outcomes were expected, and also with the uncertainty of the reward, implicating this region in basic and crucial decision-making parameters, low expected outcomes, and uncertainty.

Probabilistic reasoning by neurons

Our brains allow us to reason about alternatives and to make choices that are likely to pay off. Often there is no one correct answer, but instead one that is favoured simply because it is more likely to lead to reward. A variety of probabilistic classification tasks probe the covert strategies that humans use to decide among alternatives based on evidence that bears only probabilistically on outcome. Here we show that rhesus monkeys can also achieve such reasoning. We have trained two monkeys to choose between a pair of coloured targets after viewing four shapes, shown sequentially, that governed the probability that one of the targets would furnish reward. Monkeys learned to combine probabilistic information from the shape combinations. Moreover, neurons in the parietal cortex reveal the addition and subtraction of probabilistic quantities that underlie decision-making on this task.

A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales

Volitional behavior relies on the brain's ability to remap sensory flow to motor programs whenever demanded by a changed behavioral context. To investigate the circuit basis of such flexible behavior, we have developed a biophysically based decision-making network model of spiking neurons for arbitrary sensorimotor mapping. The model quantitatively reproduces behavioral and prefrontal single-cell data from an experiment in which monkeys learn visuomotor associations that are reversed unpredictably from time to time. We show that when synaptic modifications occur on multiple timescales, the model behavior becomes flexible only when needed: slow components of learning usually dominate the decision process. However, if behavioral contexts change frequently enough, fast components of plasticity take over, and the behavior exhibits a quick forget-and-learn pattern. This model prediction is confirmed by monkey data. Therefore, our work reveals a scenario for conditional associative learning that is distinct from instant switching between sets of well-established sensorimotor associations.

Modulation of visual processing by attention and emotion: windows on causal interactions between human brain regions

Visual processing is not determined solely by retinal inputs. Attentional modulation can arise when the internal attentional state (current task) of the observer alters visual processing of the same stimuli. This can influence visual cortex, boosting neural responses to an attended stimulus. Emotional modulation can also arise, when affective properties (emotional significance) of stimuli, rather than their strictly visual properties, influence processing. This too can boost responses in visual cortex, as for fear-associated stimuli. Both attentional and emotional modulation of visual processing may reflect distant influences upon visual cortex, exerted by brain structures outside the visual system per se. Hence, these modulations may provide windows onto causal interactions between distant but interconnected brain regions. We review recent evidence, noting both similarities and differences between attentional and emotional modulation. Both can affect visual cortex, but can reflect influences from different regions, such as fronto-parietal circuits versus the amygdala. Recent work on this has developed new approaches for studying causal influences between human brain regions that may be useful in other cognitive domains. The new methods include application of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) measures in brain-damaged patients to study distant functional impacts of their focal lesions, and use of transcranial magnetic stimulation concurrently with fMRI or EEG in the normal brain. Cognitive neuroscience is now moving beyond considering the putative functions of particular brain regions, as if each operated in isolation, to consider, instead, how distinct brain regions (such as visual cortex, parietal or frontal regions, or amygdala) may mutually influence each other in a causal manner.

Neural signature of fictive learning signals in a sequential investment task

Reinforcement learning models now provide principled guides for a wide range of reward learning experiments in animals and humans. One key learning (error) signal in these models is experiential and reports ongoing temporal differences between expected and experienced reward. However, these same abstract learning models also accommodate the existence of another class of learning signal that takes the form of a fictive error encoding ongoing differences between experienced returns and returns that "could-have-been-experienced" if decisions had been different. These observations suggest the hypothesis that, for all real-world learning tasks, one should expect the presence of both experiential and fictive learning signals. Motivated by this possibility, we used a sequential investment game and fMRI to probe ongoing brain responses to both experiential and fictive learning signals generated throughout the game. Using a large cohort of subjects (n = 54), we report that fictive learning signals strongly predict changes in subjects' investment behavior and correlate with fMRI signals measured in dopaminoceptive structures known to be involved in valuation and choice.

Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration

Many large and small decisions we make in our daily lives-which ice cream to choose, what research projects to pursue, which partner to marry-require an exploration of alternatives before committing to and exploiting the benefits of a particular choice. Furthermore, many decisions require re-evaluation, and further exploration of alternatives, in the face of changing needs or circumstances. That is, often our decisions depend on a higher level choice: whether to exploit well known but possibly suboptimal alternatives or to explore risky but potentially more profitable ones. How adaptive agents choose between exploitation and exploration remains an important and open question that has received relatively limited attention in the behavioural and brain sciences. The choice could depend on a number of factors, including the familiarity of the environment, how quickly the environment is likely to change and the relative value of exploiting known sources of reward versus the cost of reducing uncertainty through exploration. There is no known generally optimal solution to the exploration versus exploitation problem, and a solution to the general case may indeed not be possible. However, there have been formal analyses of the optimal policy under constrained circumstances. There have also been specific suggestions of how humans and animals may respond to this problem under particular experimental conditions as well as proposals about the brain mechanisms involved. Here, we provide a brief review of this work, discuss how exploration and exploitation may be mediated in the brain and highlight some promising future directions for research.

Responses of neurons in the lateral intraparietal area to central visual cues

Goal-directed behavior is characterized by flexible stimulus-action mappings. The lateral intraparietal area (area LIP) contains a representation of extra-personal space that is used to guide goal-directed behavior. To examine further how area LIP contributes to these flexible stimulus-action mappings, we recorded LIP activity while rhesus monkeys participated in two different cueing tasks. In the first task, the color of a central light indicated the location of a monkey's saccadic endpoint in the absence of any other visual stimuli. In the second task, the color of a central light indicated which of two visual targets was the saccadic goal. In both tasks, LIP activity was modulated by these non-spatial cues. These observations further suggest a role for area LIP in mediating endogenous associations that link stimuli with actions.

Dynamic circuitry for updating spatial representations. III. From neurons to behavior

Each time the eyes move, the visual system must adjust internal representations to account for the accompanying shift in the retinal image. In the lateral intraparietal cortex (LIP), neurons update the spatial representations of salient stimuli when the eyes move. In previous experiments, we found that split-brain monkeys were impaired on double-step saccade sequences that required updating across visual hemifields, as compared to within hemifield. Here we describe a subsequent experiment to characterize the relationship between behavioral performance and neural activity in LIP in the split-brain monkey. We recorded from single LIP neurons while split-brain and intact monkeys performed two conditions of the double-step saccade task: one required across-hemifield updating and the other required within-hemifield updating. We found that, despite extensive experience with the task, the split-brain monkeys were significantly more accurate for within-hemifield than for across-hemifield sequences. In parallel, we found that population activity in LIP of the split-brain monkeys was significantly stronger for the within-hemifield than for the across-hemifield condition of the double-step task. In contrast, in the normal monkey, both the average behavioral performance and population activity showed no bias toward the within-hemifield condition. Finally, we found that the difference between within-hemifield and across-hemifield performance in the split-brain monkeys was reflected at the level of single-neuron activity in LIP. These findings indicate that remapping activity in area LIP is present in the split-brain monkey for the double-step task and covaries with spatial behavior on within-hemifield compared to across-hemifield sequences.

Reinforcement learning: Computational theory and biological mechanisms

Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.

Reinforcement learning: Computational theory and biological mechanisms

Reinforcement learning is a computational framework for an active agent to learn behaviors on the basis of a scalar reward signal. The agent can be an animal, a human, or an artificial system such as a robot or a computer program. The reward can be food, water, money, or whatever measure of the performance of the agent. The theory of reinforcement learning, which was developed in an artificial intelligence community with intuitions from animal learning theory, is now giving a coherent account on the function of the basal ganglia. It now serves as the "common language" in which biologists, engineers, and social scientists can exchange their problems and findings. This article reviews the basic theoretical framework of reinforcement learning and discusses its recent and future contributions toward the understanding of animal behaviors and human decision making.

A scale-free systems theory of motivation and addiction

Scale-free organizations, characterized by uneven distributions of linkages between nodal elements, describe the structure and function of many life-based complex systems developing under evolutionary pressures. We explore motivated behavior as a scale-free map toward a comprehensive translational theory of addiction. Motivational and behavioral repertoires are reframed as link and nodal element sets, respectively, comprising a scale-free structure. These sets are generated by semi-independent information-processing streams within cortical-striatal circuits that cooperatively provide decision-making and sequential processing functions necessary for traversing maps of motivational links connecting behavioral nodes. Dopamine modulation of cortical-striatal plasticity serves a central-hierarchical mechanism for survival-adaptive sculpting and development of motivational-behavioral repertoires by guiding a scale-free design. Drug-induced dopamine activity promotes drug taking as a highly connected behavioral hub at the expense of natural-adaptive motivational links and behavioral nodes. Conceptualizing addiction as pathological alteration of scale-free motivational-behavioral repertoires unifies neurobiological, neurocomputational and behavioral research while addressing addiction vulnerability in adolescence and psychiatric illness. This model may inform integrative research in defining more effective prevention and treatment strategies for addiction.

Basal Ganglia Mechanisms of Reward-Oriented Eye Movement

Expectation of reward facilitates motor behaviors that enable the animal to approach a location in space where the reward is expected. It is now known that the same expectation of reward profoundly modifies sensory, motor, and cognitive information processing in the brain. However, it is still unclear which brain regions are responsible for causing the reward-approaching behavior. One candidate is the dorsal striatum where cortical and dopaminergic inputs converge. We tested this hypothesis by injecting dopamine antagonists into the caudate nucleus (CD) while the monkey was performing a saccade task with a position-dependent asymmetric reward schedule. We previously had shown that: (1) serial GABAergic connections from the CD to the superior colliculus (SC) via the substantia nigra pars reticulata (SNr) exert powerful control over the initiation of saccadic eye movement and (2) these GABAergic neurons encode target position and are strongly influenced by expected reward, while dopaminergic neurons in the substantia nigra pars compacta (SNc) encode only reward-related information. Before injections of dopamine antagonists the latencies of saccades to a given target were shorter when the saccades were followed by a large reward than when they were followed by a small reward. After injections of dopamine D1 receptor antagonist the reward-dependent latency bias became smaller. This was due to an increase in saccade latency on large-reward trials. After injections of D2 antagonist the latency bias became larger, largely due to an increase in saccade latency on small-reward trials. These results indicate that: (1) dopamine-dependent information processing in the CD is necessary for the reward-dependent modulation of saccadic eye movement and (2) D1 and D2 receptors play differential roles depending on the positive and negative reward outcomes.

Dynamic Changes in Representations of Preceding and Upcoming Reward in Monkey Orbitofrontal Cortex

We investigated how orbitofrontal cortex (OFC) contributes to adaptability in the face of changing reward contingencies by examining how reward representations in monkey orbitofrontal neurons change during a visually cued, multi-trial reward schedule task. A large proportion of orbitofrontal neurons were sensitive to events in this task (69/80 neurons in the valid and 48/58 neurons in the random cue context). Neuronal activity depended upon preceding reward, upcoming reward, reward delivery, and schedule state. Preceding reward-dependent activity occurred in both the valid and random cue contexts, whereas upcoming reward-dependent activity was observed only in the valid context. A greater proportion of neurons encoded preceding reward in the random than the valid cue context. The proportion of neurons with preceding reward-dependent activity declined as each trial progressed, whereas the proportion encoding upcoming reward increased. Reward information was represented by ensembles of neurons, the composition of which changed with task context and time. Overall, neuronal activity in OFC adapted to reflect the importance of different types of reward information in different contexts and time periods. This contextual and temporal adaptability is one hallmark of neurons participating in executive functions.

Receptor theory and biological constraints on value

Modern economic theories of value derive from expected utility theory. Behavioral evidence points strongly toward departures from linear value weighting, which has given rise to alternative formulations that include prospect theory and rank-dependent utility theory. Many of the nonlinear forms for value assumed by these theories can be derived from the assumption that value is signaled by neurotransmitters in the brain, which obey simple laws of molecular movement. From the laws of mass action and receptor occupancy, we show how behaviorally observed forms of nonlinear value functions can arise.

Multiple representations of belief states and action values in corticobasal ganglia loops

Reward-related neural activities have been found in a variety of cortical and subcortical areas by neurophysiological and neuroimaging experiments. Here we present a unified view on how three subloops of the corticobasal ganglia network are involved in reward prediction and action selection using different types of information. The motor/premotor-posterior striatum loop is specialized for action-based value representation and movement selection. The orbitofrontal-ventral striatum loop is specialized for object-based value representation and target selection. The lateral prefrontal-anterior striatum loop is specialized for context-based value representation and context estimation. Furthermore, the medial prefrontal cortex (MPFC) coordinates these multiple value representations and actions at different levels of hierarchy by monitoring the error in predictions.

Cost, benefit, tonic, phasic: what do response rates tell us about dopamine and motivation?

The role of dopamine in decision making has received much attention from both the experimental and computational communities. However, because reinforcement learning models concentrate on discrete action selection and on phasic dopamine signals, they are silent as to how animals decide upon the rate of their actions, and they fail to account for the prominent effects of dopamine on response rates. We suggest an extension to reinforcement learning models in which response rates are optimally determined by balancing the tradeoff between the cost of fast responding and the benefit of rapid reward acquisition. The resulting behavior conforms well with numerous characteristics of free-operant responding. More importantly, this framework highlights a role for a tonic signal corresponding to the net rate of rewards, in determining the optimal rate of responding. We hypothesize that this critical quantity is conveyed by tonic levels of dopamine, explaining why dopaminergic manipulations exert a global affect on response rates. We further suggest that the effects of motivation on instrumental rates of responding are mediated through its influence on the net reward rate, implying a tight coupling between motivational states and tonic dopamine. The relationships between phasic and tonic dopamine signaling, and between directing and energizing effects of motivation, as well as the implications for motivational control of habitual and goal-directed instrumental action selection, are discussed.

Model-based fMRI and its application to reward learning and decision making

In model-based functional magnetic resonance imaging (fMRI), signals derived from a computational model for a specific cognitive process are correlated against fMRI data from subjects performing a relevant task to determine brain regions showing a response profile consistent with that model. A key advantage of this technique over more conventional neuroimaging approaches is that model-based fMRI can provide insights into how a particular cognitive process is implemented in a specific brain area as opposed to merely identifying where a particular process is located. This review will briefly summarize the approach of model-based fMRI, with reference to the field of reward learning and decision making, where computational models have been used to probe the neural mechanisms underlying learning of reward associations, modifying action choice to obtain reward, as well as in encoding expected value signals that reflect the abstract structure of a decision problem. Finally, some of the limitations of this approach will be discussed.

Role of anticipated reward in cognitive behavioral control

The lateral prefrontal cortex (LPFC), which is important for higher cognitive activity, is also concerned with motivational operations; this is exemplified by its activity in relation to expectancy of rewards. In the LPFC, motivational information is integrated with cognitive information, as demonstrated by the enhancement of working-memory-related activity by reward expectancy. Such activity would be expected to induce changes in attention and, subsequently, to modify behavioral performance. Recently, the effects of motivation and emotion on neural activities have been examined in several areas of the brain in relation to cognitive-task performance. Of these areas, the LPFC seems to have the most important role in adaptive goal-directed behavior, by sending top-down attention-control signals to other areas of the brain.

Adaptive decision making and value in the anterior cingulate cortex

Choosing an appropriate response in an uncertain and varying world is central to adaptive behaviour. The frequent activation of the anterior cingulate cortex (ACC) in a diverse range of tasks has lead to intense interest in and debate over its role in the guidance and control of performance. Here, we consider how this issue can be informed by a series of studies considering the ACC's role in more naturalistic situations where there is no single certain correct response and the relationships between choices and their consequences vary. A neuroimaging study of response switching demonstrates that dorsal ACC is not simply concerned with self-generated responses or error monitoring in isolation, but is instead involved in evaluating the outcome of choices, positive or negative, that have been voluntarily chosen. By contrast, an interconnected part of the orbitofrontal cortex is shown to be more active when attending to consequences of actions instructed by the experimenter. This dissociation is explained with reference to the anatomy of these regions in humans as demonstrated by diffusion weighted imaging. Lesions to a corresponding ACC region in monkeys has no effect on animals' ability to detect or immediately correct errors when response contingencies reverse, but renders them unable to sustain appropriate behaviour due to an impairment in the ability to integrate over time their recent history of choices and outcomes. Taken together, this implies a prominent role for the ACC within a distributed network of regions that determine the dynamic value of actions and guide decision making appropriately.

Representing spatial relationships in posterior parietal cortex: single neurons code object-referenced position

The brain computes spatial relationships as necessary to achieve behavioral goals. Loss of this spatial cognitive ability after damage to posterior parietal cortex may contribute to constructional apraxia, a syndrome in which a patient's ability to reproduce spatial relationships between the parts of an object is disrupted. To explore neural correlates of object-relative spatial representation, we recorded neural activity in parietal area 7a of monkeys performing an object construction task. We found that neurons were activated as a function of the spatial relationship between a task-critical coordinate and a reference object. Individual neurons exhibited an object-relative spatial preference, such that different neural populations were activated when the spatial coordinate was located to the left or right of the reference object. In each case, the representation was robust to translation of the reference object, and neurons maintained their object-relative preference when the position of the object varied relative to the angle of gaze and viewer-centered frames of reference. This provides evidence that the activity of a subpopulation of parietal neurons active in the construction task represented relative position as referenced to an object and not absolute position with respect to the viewer.

Target selection signals for arm reaching in the posterior parietal cortex

The selection of visual stimuli as a target for a motor action may depend on external as well as internal variables. The parietal reach region (PRR) in the posterior parietal cortex plays an important role in the transformation of visual information into reach movement plans. We asked how neurons in PRR of macaque monkeys reflect the decision process of selecting one of two visual stimuli as a target for a reach movement. Spiking activity was recorded while the animal performed a free-choice task with one target presented in the preferred direction and the other in the off direction of the cell. Stimulus-onset asynchrony (SOA) was adjusted to ensure that both targets were selected equally often and the amount of reward was fixed. Neural activity in PRR was action specific for arm reaching and reflected the timing of the SOA as well as the selection of reach targets. In individual trials, activity was strongly linked to the choice of the animal, and, for the majority of cells, target selections could be predicted from activity in the stimulation or planning period, i.e., before the movement started. Many neurons were gain modulated by the fixation position, but gain modulation did not influence the target selection process directly. Finally, it was found that target selection for saccade movements was only weakly represented in PRR. These findings suggest that PRR is involved in decision making for reach movements and that separate cortical networks exist for target selection of different types of action.

Temporal discounting predicts risk sensitivity in rhesus macaques

Humans and animals tend both to avoid uncertainty and to prefer immediate over future rewards. The comorbidity of psychiatric disorders such as impulsivity, problem gambling, and addiction suggests that a common mechanism may underlie risk sensitivity and temporal discounting. Nonetheless, the precise relationship between these two traits remains largely unknown. To examine whether risk sensitivity and temporal discounting reflect a common process, we recorded choices made by two rhesus macaques in a visual gambling task while we varied the delay between trials. We found that preference for the risky option declined with increasing delay between sequential choices in the task, even when all other task parameters were held constant. These results were quantitatively predicted by a model that assumed that the subjective expected utility of the risky option is evaluated based on the expected time of the larger payoff. The importance of the larger payoff in this model suggests that the salience of larger payoffs played a critical role in determining the value of risky options. These data suggest that risk sensitivity may be a product of other cognitive processes, and specifically that myopia for the future and the salience of jackpots control the propensity to take a gamble.

Efficient reinforcement learning: computational theories, neuroscience and robotics

Reinforcement learning algorithms have provided some of the most influential computational theories for behavioral learning that depends on reward and penalty. After briefly reviewing supporting experimental data, this paper tackles three difficult theoretical issues that remain to be explored. First, plain reinforcement learning is much too slow to be considered a plausible brain model. Second, although the temporal-difference error has an important role both in theory and in experiments, how to compute it remains an enigma. Third, function of all brain areas, including the cerebral cortex, cerebellum, brainstem and basal ganglia, seems to necessitate a new computational framework. Computational studies that emphasize meta-parameters, hierarchy, modularity and supervised learning to resolve these issues are reviewed here, together with the related experimental data.

Calculating the cost of acting in frontal cortex

To make informed and successful decisions, it is vital to be able to evaluate whether the expected benefits of a course of action make it worth tolerating the costs incurred to obtain them. The frontal lobe has been implicated in several aspects of goal-directed action selection, social interaction, and optimal choice behavior. However, its exact contribution has remained elusive. Here, we discuss a series of studies in rats and primates examining the effect of discrete lesions on different aspects of cost-benefit decision making. Rats with excitotoxic lesions of the anterior cingulate cortex became less willing to invest effort for reward but showed no change when having to tolerate delays. Orbitofrontal cortex-lesioned rats, by contrast, became more impulsive, yet were just as prepared as normal animals to expend energy to obtain reward. The sulcal region of primate anterior cingulate cortex was also shown to be essential for dynamically integrating over time the recent history of choices and outcomes. Selecting a particular course of action may also come at the expense of gathering important information about other individuals. Evaluating social information when deciding whether to respond was demonstrated to be a function of the anterior cingulate gyrus. Taken together, this indicates that there may be dissociable pathways in the frontal lobe for managing different types of response cost and for gathering social information.

Mechanisms of reinforcement learning and decision making in the primate dorsolateral prefrontal cortex

To a first approximation, decision making is a process of optimization in which the decision maker tries to maximize the desirability of the outcomes resulting from chosen actions. Estimates of desirability are referred to as utilities or value functions, and they must be continually revised through experience according to the discrepancies between the predicted and obtained rewards. Reinforcement learning theory prescribes various algorithms for updating value functions and can parsimoniously account for the results of numerous behavioral, neurophysiological, and imaging studies in humans and other primates. In this article, we first discuss relative merits of various decision-making tasks used in neurophysiological studies of decision making in nonhuman primates. We then focus on how reinforcement learning theory can shed new light on the function of the primate dorsolateral prefrontal cortex. Similar to the findings from other brain areas, such as cingulate cortex and basal ganglia, activity in the dorsolateral prefrontal cortex often signals the value of expected reward and actual outcome. Thus, the dorsolateral prefrontal cortex is likely to be a part of the broader network involved in adaptive decision making. In addition, reward-related activity in the dorsolateral prefrontal cortex is influenced by the animal's choices and other contextual information, and therefore may provide a neural substrate by which the animals can flexibly modify their decision-making strategies according to the demands of specific tasks.

Contrasting roles for cingulate and orbitofrontal cortex in decisions and social behaviour

There is general acknowledgement that both the anterior cingulate and orbitofrontal cortex are implicated in reinforcement-guided decision making, and emotion and social behaviour. Despite the interest that these areas generate in both the cognitive neuroscience laboratory and the psychiatric clinic, ideas about the distinctive contributions made by each have only recently begun to emerge. This reflects an increasing understanding of the component processes that underlie reinforcement-guided decision making, such as the representation of reinforcement expectations, the exploration, updating and representation of action values, and the appreciation that choices are guided not just by the prospect of reward but also by the costs that action entails. Evidence is emerging to suggest that the anterior cingulate and orbitofrontal cortex make distinct contributions to each of these aspects of decision making.

Is matching innate?

Experimentally naive mice matched the proportions of their temporal investments (visit durations) in two feeding hoppers to the proportions of the food income (pellets per unit session time) derived from them in three experiments that varied the coupling between the behavioral investment and food income, from no coupling to strict coupling. Matching was observed from the outset; it did not improve with training. When the numbers of pellets received were proportional to time invested, investment was unstable, swinging abruptly from sustained, almost complete investment in one hopper, to sustained, almost complete investment in the other-in the absence of appropriate local fluctuations in returns (pellets obtained per time invested). The abruptness of the swings strongly constrains possible models. We suggest that matching reflects an innate (unconditioned) program that matches the ratio of expected visit durations to the ratio between the current estimates of expected incomes. A model that processes the income stream looking for changes in the income and generates discontinuous income estimates when a change is detected is shown to account for salient features of the data.