Matching behavior and the representation of value in the parietal cortex

Reward encoding in the monkey anterior cingulate cortex

The anterior cingulate cortex (ACC) is known to play a crucial role in the fast adaptations of behavior based on immediate reward values. What is less certain is whether the ACC is also involved in long-term adaptations to situations with uncertain outcomes. To study this issue, we placed macaque monkeys in a probabilistic context in which the appropriate strategy to maximize reward was to identify the stimulus with the highest reward value (optimal stimulus). Only knowledge of the theoretical average reward value associated with this stimulus--referred to as 'the task value'--was available. Remarkably, in each trial, ACC pre-reward activity correlated with the task value. Importantly, this neuronal activity was observed prior to the discovery of the optimal stimulus. We hypothesize that the received rewards and the task value, constructed a priori through learning, are used to guide behavior and identify the optimal stimulus. We tested this hypothesis by muscimol deactivation of the ACC. As predicted, this inactivation impaired the search for the optimal stimulus. We propose that ACC participates in long-term adaptation of voluntary reward-based behaviors by encoding general task values and received rewards.

Implicit short-term memory and event frequency effects in visual search

Numerous experiments have shown that animals and humans behave as if guided by an implicit knowledge of the relative frequency of occurrence of events in their environment. A well-known example of such trait is "Hick's Law" for reaction times: responses to more frequent stimuli are faster than to less frequent ones. In the present study, we demonstrate that an important source of the effects produced by Hick's law in the context of a visual search task is to be found in a form of implicit short-term memory, previously described as Priming of Pop-out. We report the results of experiments in which we have disrupted or enhanced the accumulation of implicit short-lived memory traces in the context of visual search tasks where stimulus frequency was varied. With target frequencies greater than 20%, these memory manipulations resulted in the elimination or enhancement of the effect of stimulus frequency on reaction times, thus indicating that an implicit, finite-memory accumulator is an important underlying mechanism for frequency effects in visual search paradigms. We characterize the properties of the accumulator and discuss the kinds of behaviors in which it may be implicated.

Dynamics of Parietal Neural Activity during Spatial Cognitive Processing

Dynamic neural processing unrelated to changes in sensory input or motor output is likely to be a hallmark of cognitive operations. Here we show that neural representations of space in parietal cortex are dynamic while monkeys perform a spatial cognitive operation on a static visual stimulus. We recorded neural activity in area 7a during a visual maze task in which monkeys mentally followed a path without moving their eyes. We found that the direction of the followed path could be recovered from neuronal population activity. When the monkeys covertly processed a path that turned, the population representation of path direction shifted in the direction of the turn. This neural population dynamic took place during a period of unchanging visual input and showed characteristics of both serial and parallel processing. The data suggest that the dynamic evolution of parietal neuronal activity is associated with the progression of spatial cognitive operations.

The urgency to look: prompt saccades to the benefit of perception

Researchers have shown that the promptness to initiate a saccade is modulated by countless factors pertaining to the visual context and the task. However, experiments on saccadic eye movements are usually designed in such a way that oculomotor performance is dissociated from the natural role of saccades, namely that of making crucial perceptual information rapidly available for high-resolution, foveal analysis. Here, we demonstrate that the requirement to perform a difficult perceptual judgment at the saccade landing location can reduce saccadic latency (by >15%) and increase saccadic peak velocity. Importantly, the effect cannot be explained in terms of arousal, as latency changes are specific to the location where the perceptual judgement is required. These results indicate that mechanisms for voluntary saccade initiation are under the powerful indirect control of perceptual goals.

Risk-sensitive neurons in macaque posterior cingulate cortex

People and animals often demonstrate strong attraction or aversion to options with uncertain or risky rewards, yet the neural substrate of subjective risk preferences has rarely been investigated. Here we show that monkeys systematically preferred the risky target in a visual gambling task in which they chose between two targets offering the same mean reward but differing in reward uncertainty. Neuronal activity in posterior cingulate cortex (CGp), a brain area linked to visual orienting and reward processing, increased when monkeys made risky choices and scaled with the degree of risk. CGp activation was better predicted by the subjective salience of a chosen target than by its actual value. These data suggest that CGp signals the subjective preferences that guide visual orienting.

Learning and decision making in monkeys during a rock-paper-scissors game

Game theory provides a solution to the problem of finding a set of optimal decision-making strategies in a group. However, people seldom play such optimal strategies and adjust their strategies based on their experience. Accordingly, many theories postulate a set of variables related to the probabilities of choosing various strategies and describe how such variables are dynamically updated. In reinforcement learning, these value functions are updated based on the outcome of the player's choice, whereas belief learning allows the value functions of all available choices to be updated according to the choices of other players. We investigated the nature of learning process in monkeys playing a competitive game with ternary choices, using a rock-paper-scissors game. During the baseline condition in which the computer selected its targets randomly, each animal displayed biases towards some targets. When the computer exploited the pattern of animal's choice sequence but not its reward history, the animal's choice was still systematically biased by the previous choice of the computer. This bias was reduced when the computer exploited both the choice and reward histories of the animal. Compared to simple models of reinforcement learning or belief learning, these adaptive processes were better described by a model that incorporated the features of both models. These results suggest that stochastic decision-making strategies in primates during social interactions might be adjusted according to both actual and hypothetical payoffs.

Neuronal Activity Dependent on Anticipated and Elapsed Delay in Macaque Prefrontal Cortex, Frontal and Supplementary Eye Fields, and Premotor Cortex

In macaque monkeys performing a memory-guided saccade task for a reward of variable size, neuronal activity in several areas of frontal cortex is stronger when the monkey anticipates a larger reward. This effect might depend on either the size or the value of the reward. To distinguish between these possibilities, we recorded from neurons in frontal cortex while controlling value through a manipulation of time rather than amount. A cue presented at the beginning of each trial, predicted the length of the delay during which the monkey would have to maintain fixation before performing a saccade and receiving a reward of fixed size. Predicting a short delay had effects closely similar to those of predicting a large reward: 1) monkeys were more motivated when working for a reward at short delay, 2) neurons tended to fire more strongly before a short delay, 3) individual neurons firing more strongly before a short delay tended also to fire more strongly before a large reward, and 4) the tendency to fire more strongly before a short delay was far more pronounced in premotor areas caudal to the arcuate sulcus than in association areas rostral to it. The association areas, in contrast, were marked by a tendency for neurons to fire more strongly at the end of the long delay. We conclude that predicting a short delay, like predicting a large reward, induces an enhancement of neuronal activity related to motivational modulation of the monkey's preparatory state.

Supplementary Motor Area Encodes Reward Expectancy in Eye-Movement Tasks

Neural activity signifying the expectation of reward has been found recently in many parts of the brain, including midbrain and cortical structures. These signals can facilitate goal-directed behavior or the learning of new skills based on reinforcements. Here we show that neurons in the supplementary motor area (SMA), an area concerned with movements of the body and limbs, also carry a reward expectancy signal in the postsaccadic period of oculomotor tasks. While the monkeys performed blocks of memory-guided and object-based saccades, the neurons discharged a burst after a ∼200-ms delay following the target-acquiring saccade in the memory task but often fired concurrently with the target-acquiring saccade in the object task. The hypothesis that this postsaccadic bursting activity reflects the expectation of a reward was tested with a series of manipulations to the memory-guided saccade task. It was found that although the timing of the bursting activity corresponds to a visual feedback stimulus, the visual feedback is not required for the neurons to discharge a burst. Second, blocks of no-reward trials reveal an extinction of the bursting activity as the monkeys come to understand that they would not be rewarded for properly generated saccades. Finally, the delivery of unexpected rewards confirmed that in many of the neurons, the activity is not related to a motor plan to acquire the reward (e.g., licking). Thus we conclude that reward expectancy is represented by the activity of SMA neurons, even in the context of an oculomotor task. These results suggest that the reward expectancy signal is broadcast over a large extent of motor cortex, and may facilitate the learning of new, coordinated behavior between different body parts.

Multi-stage mental process for economic choice in capuchins

We studied economic choice behavior in capuchin monkeys by offering them to choose between two different foods available in variable amounts. When monkeys selected between familiar foods, their choice patterns were well-described in terms of relative value of the two foods. A leading view in economics and biology is that such behavior results from stimulus-response associations acquired through experience. According to this view, values are not psychologically real; they can only be defined a posteriori. One prediction of this associative model is that animals faced for the first time with a new pair of foods learn to choose between them gradually. We tested this prediction. Surprisingly, we find that monkeys choose as effectively between new pairs of foods as they choose between familiar pairs of foods. We therefore, propose a cognitive model in which economic choice results from a two-stage mental process of value-assignment and decision-making. In a follow-up experiment, we find that the relative value assigned to three foods in sessions in which we tested them against each other combine according to transitivity.

Functional significance of delay-period activity of primate prefrontal neurons in relation to spatial working memory and reward/omission-of-reward expectancy

The lateral prefrontal cortex (LPFC) is important in cognitive control. During the delay period of a working memory (WM) task, primate LPFC neurons show sustained activity that is related to retaining task-relevant cognitive information in WM. However, it has not yet been determined whether LPFC delay neurons are concerned exclusively with the cognitive control of WM task performance. Recent studies have indicated that LPFC neurons also show reward and/or omission-of-reward expectancy-related delay activity, while the functional relationship between WM-related and reward/omission-of-reward expectancy-related delay activity remains unclear. To clarify the functional significance of LPFC delay-period activity for WM task performance, and particularly the functional relationship between these two types of activity, we examined individual delay neurons in the primate LPFC during spatial WM (delayed response) and non-WM (reward-no-reward delayed reaction) tasks. We found significant interactions between these two types of delay activity. The majority of the reward expectancy-related neurons and the minority of the omission-of-reward expectancy-related neurons were involved in spatial WM processes. Spatial WM-related neurons were more likely to be involved in reward expectancy than in omission-of-reward expectancy. In addition, LPFC delay neurons observed during the delayed response task were not concerned exclusively with the cognitive control of task performance; some were related to reward/omission-of-reward expectancy but not to WM, and many showed more memory-related activity for preferred rewards than for less-desirable rewards. Since employing a more preferred reward induced better task performance in the monkeys, as well as enhanced WM-related neuronal activity in the LPFC, the principal function of the LPFC appears to be the integration of cognitive and motivational operations in guiding the organism to obtain a reward more effectively.

Complementary Process to Response Bias in the Centromedian Nucleus of the Thalamus

Activity in several areas of the human brain and the monkey brain increases when a subject anticipates events associated with a reward, implicating a role for bias of decision and action. However, in real life, events do not always appear as expected, and we must choose an undesirable action. More than half of the neurons in the monkey centromedian (CM) thalamus were selectively activated when a small-reward action was required but a large-reward option was anticipated. Electrical stimulation of the CM after a large-reward action request substituted a brisk performance with a sluggish performance. These results suggest involvement of the CM in a mechanism complementary to decision and action bias.

Functional-anatomic correlates of memory retrieval that suggest nontraditional processing roles for multiple distinct regions within posterior parietal cortex

Current theories of posterior parietal cortex (PPC) function emphasize space-based attention and motor intention. Imaging studies of long-term memory have demonstrated PPC activation during successful memory retrieval. Here, we explored the relationship between memory processes and classical notions of PPC function. Study 1 investigated old-new recognition using picture and sound stimuli to test whether PPC memory effects were dependent on visuospatial attention. A region lateral to the intraparietal sulcus [inferior parietal lobule complex (IPLC)] and two regions in the medial PPC [precuneus complex (PCC) and posterior cingulate/retrosplenial cortex (pC/Rsp)] showed strong retrieval success effects for both picture and sound stimuli. Study 2 explored a recognition task with varied response contingencies to investigate whether these retrieval success effects are dependent on motor intentions. In one condition, subjects responded to both old and new items. In two other conditions, subjects responded only to old or only to new items. IPLC, PCC, and pC/Rsp continued to show retrieval success effects with similar magnitudes for all response contingencies, including a condition in which no responses were made to old items. In a third study, IPLC and PCC activity was modulated at retrieval based on levels of processing at study, suggesting sensitivity to memory demands. These studies demonstrate that retrieval success effects in lateral and medial PPC regions are not affected by manipulations predicted by classical theories of PPC function but can be modulated by memory-related manipulations. PPC regions thus have prominent response properties associated with memory, which may arise through interactions with medial temporal cortex.

Independent coding of movement direction and reward prediction by single pallidal neurons

Associating action with its reward value is a basic ability needed by adaptive organisms and requires the convergence of limbic, motor, and associative information. To chart the basal ganglia (BG) involvement in this association, we recorded the activity of 61 well isolated neurons in the external segment of the globus pallidus (GPe) of two monkeys performing a probabilistic visuomotor task. Our results indicate that most (96%) neurons responded to multiple phases of the task. The activity of many (34%) pallidal neurons was modulated solely by direction of movement, and the activity of only a few (3%) pallidal neurons was modulated exclusively by reward prediction. However, the activity of a large number (41%) of single pallidal neurons was comodulated by both expected trial outcome and direction of arm movement. The information carried by the neuronal activity of single pallidal neurons dynamically changed as the trial progressed. The activity was predominantly modulated by both outcome prediction and future movement direction at the beginning of trials and became modulated mainly by movement-direction toward the end of trials. GPe neurons can either increase or decrease their discharge rate in response to predicted future reward. The effects of movement-direction and reward probability on neural activity are linearly summed and thus reflect two independent modulations of pallidal activity. We propose that GPe neurons are uniquely suited for independent processing of a multitude of parameters. This is enabled by the funnel-structure characteristic of the BG architecture, as well as by the anatomical and physiological properties of GPe neurons.

Monkeys pay per view: adaptive valuation of social images by rhesus macaques

Individuals value information that improves decision making. When social interactions complicate the decision process, acquiring information about others should be particularly valuable. In primate societies, kinship, dominance, and reproductive status regulate social interactions and should therefore systematically influence the value of social information, but this has never been demonstrated. Here, we show that monkeys differentially value the opportunity to acquire visual information about particular classes of social images. Male rhesus macaques sacrificed fluid for the opportunity to view female perinea and the faces of high-status monkeys but required fluid overpayment to view the faces of low-status monkeys. Social value was highly consistent across subjects, independent of particular images displayed, and only partially predictive of how long subjects chose to view each image. These data demonstrate that visual orienting decisions reflect the specific social content of visual information and provide the first experimental evidence that monkeys spontaneously discriminate images of others based on social status.

Prospect theory on the brain? Toward a cognitive neuroscience of decision under risk

Most decisions must be made without advance knowledge of their consequences. Economists and psychologists have devoted much attention to modeling decisions made under conditions of risk in which options can be characterized by a known probability distribution over possible outcomes. The descriptive shortcomings of classical economic models motivated the development of prospect theory (D. Kahneman, A. Tversky, Prospect theory: An analysis of decision under risk. Econometrica, 4 (1979) 263-291; A. Tversky, D. Kahneman, Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5 (4) (1992) 297-323) the most successful behavioral model of decision under risk. In the prospect theory, subjective value is modeled by a value function that is concave for gains, convex for losses, and steeper for losses than for gains; the impact of probabilities are characterized by a weighting function that overweights low probabilities and underweights moderate to high probabilities. We outline the possible neural bases of the components of prospect theory, surveying evidence from human imaging, lesion, and neuropharmacology studies as well as animal neurophysiology studies. These results provide preliminary suggestions concerning the neural bases of prospect theory that include a broad set of brain regions and neuromodulatory systems. These data suggest that focused studies of decision making in the context of quantitative models may provide substantial leverage towards a fuller understanding of the cognitive neuroscience of decision making.

Choosing the greater of two goods: neural currencies for valuation and decision making

To make adaptive decisions, animals must evaluate the costs and benefits of available options. The nascent field of neuroeconomics has set itself the ambitious goal of understanding the brain mechanisms that are responsible for these evaluative processes. A series of recent neurophysiological studies in monkeys has begun to address this challenge using novel methods to manipulate and measure an animal's internal valuation of competing alternatives. By emphasizing the behavioural mechanisms and neural signals that mediate decision making under conditions of uncertainty, these studies might lay the foundation for an emerging neurobiology of choice behaviour.

Eye movements in natural behavior

The classic experiments of Yarbus over 50 years ago revealed that saccadic eye movements reflect cognitive processes. But it is only recently that three separate advances have greatly expanded our understanding of the intricate role of eye movements in cognitive function. The first is the demonstration of the pervasive role of the task in guiding where and when to fixate. The second has been the recognition of the role of internal reward in guiding eye and body movements, revealed especially in neurophysiological studies. The third important advance has been the theoretical developments in the fields of reinforcement learning and graphic simulation. All of these advances are proving crucial for understanding how behavioral programs control the selection of visual information.

Learned predictions of error likelihood in the anterior cingulate cortex

The anterior cingulate cortex (ACC) and the related medial wall play a critical role in recruiting cognitive control. Although ACC exhibits selective error and conflict responses, it has been unclear how these develop and become context-specific. With use of a modified stop-signal task, we show from integrated computational neural modeling and neuroimaging studies that ACC learns to predict error likelihood in a given context, even for trials in which there is no error or response conflict. These results support a more general error-likelihood theory of ACC function based on reinforcement learning, of which conflict and error detection are special cases.

Reinforcement learning and decision making in monkeys during a competitive game

Animals living in a dynamic environment must adjust their decision-making strategies through experience. To gain insights into the neural basis of such adaptive decision-making processes, we trained monkeys to play a competitive game against a computer in an oculomotor free-choice task. The animal selected one of two visual targets in each trial and was rewarded only when it selected the same target as the computer opponent. To determine how the animal's decision-making strategy can be affected by the opponent's strategy, the computer opponent was programmed with three different algorithms that exploited different aspects of the animal's choice and reward history. When the computer selected its targets randomly with equal probabilities, animals selected one of the targets more often, violating the prediction of probability matching, and their choices were systematically influenced by the choice history of the two players. When the computer exploited only the animal's choice history but not its reward history, animal's choice became more independent of its own choice history but was still related to the choice history of the opponent. This bias was substantially reduced, but not completely eliminated, when the computer used the choice history of both players in making its predictions. These biases were consistent with the predictions of reinforcement learning, suggesting that the animals sought optimal decision-making strategies using reinforcement learning algorithms.

Neural circuitry of judgment and decision mechanisms

Tracing the neural circuitry of decision formation is a critical step in the understanding of higher cognitive function. To make a decision, the primate brain coordinates dynamic interactions between several cortical and subcortical areas that process sensory, cognitive, and reward information. In selecting the optimal behavioral response, decision mechanisms integrate the accumulating evidence with reward expectation and knowledge from prior experience, and deliberate about the choice that matches the expected outcome. Linkages between sensory input and behavioral output responsible for response selection are shown in the neural activity of structures from the prefrontal-basal ganglia-thalamo-cortical loop. The deliberation process can be best described in terms of sensitivity, selection bias, and activation threshold. Here, we show a systems neuroscience approach of the visual saccade decision circuit and the interaction between its components during decision formation.

Neural correlates of spatial judgement during object construction in parietal cortex

We recorded the activity of parietal area 7a neurons in monkeys performing an object construction task. In each trial, a model object consisting of a variable arrangement of squares was presented, followed after a delay by a copy of the model object that was missing a single square. Monkeys replaced the missing square to reconstruct the model configuration. Activity of many 7a neurons varied systematically with the position of the missing square and predicted where monkeys were going to add parts to the object they were building. The location of the missing square was a computed spatial datum important to object construction which did not correlate with the retinal location of a visual stimulus or the direction of the required motor response. The population of cells coding this coordinate was generally inactive when the same spatial locations were made relevant by visual targets to which monkeys either planned saccades or directed attention in other behavioral contexts. The data suggest that some parietal neurons participate in neural representations of space that reflect spatial cognitive as opposed to sensorimotor processing, coding the results of spatial computations performed on visual stimuli to meet cognitive objectives.

Neuroeconomics

This paper introduces an emerging transdisciplinary field known as neuroeconomics. Neuroeconomics uses neuroscientific measurement techniques to investigate how decisions are made. First, I present a basic overview of neuroanatomy and explain how brain activity is measured. I then survey findings from the neuroeconomics literature on acquiring rewards and avoiding losses, learning, choice under risk and ambiguity, delay of gratification, the role of emotions in decision-making, strategic decisions and social decisions. I conclude by identifying new directions that neuroeconomics is taking, including applications to public policy and law.

Simultaneous representation of saccade targets and visual onsets in monkey lateral intraparietal area

The monkey's lateral intraparietal area (LIP) has been associated with attention and saccades. LIP neurons have visual on-responses to objects abruptly appearing in their receptive fields (RFs) and sustained activity preceding saccades to the RF. We studied the relationship between the on-responses and delay activity in LIP using a 'stable-array' task. Monkeys viewed eight distinct, continuously illuminated objects, arranged in a circle with at least one object in the RF. A cue flashed instructing the monkey to make a saccade, after a delay, to the stable object physically matching the cue. The location of the cue was fixed in trial blocks, either in or out of the RF. If the cue was outside the RF, neurons developed delay-period activity tuned for the direction of the saccade target at approximately 190 ms after cue onset. If the cue appeared in the RF, neurons initially responded to cue onset and developed tuning for saccade direction only toward the end of the delay period, 390 ms after cue onset. The cue- and saccade-target responses coexisted throughout a significant portion of the delay period. The results show that visual-on responses and delay-period activity in LIP are functionally separable, and that, although highly selective, the salience representation in LIP can contain more than one object at a time.

Neural computations of decision utility

How are decision alternatives represented in the primate brain? A recent study by Sugrue et al. sought to answer this question by integrating behavioral, computational and physiological methods in examining the choice patterns of monkeys placed in a dynamic foraging environment. They observed specific encoding of the relative value of alternatives by neurons in the parietal cortex, providing an important starting point for researchers interested in how value and probability are combined in the brain to arrive at decision outcomes.

Eye movements: viewing the window of opportunity

When searching with our eyes, parallel programming of successive eye movements ensures that visual information arriving too late to alter the direction of one eye movement can still influence the direction of the next. Paradoxically, we can use random noise to probe the time period over which visual information influences where next to direct our gaze.

Neuroeconomics: the consilience of brain and decision

Economics, psychology, and neuroscience are converging today into a single, unified discipline with the ultimate aim of providing a single, general theory of human behavior. This is the emerging field of neuroeconomics in which consilience, the accordance of two or more inductions drawn from different groups of phenomena, seems to be operating. Economists and psychologists are providing rich conceptual tools for understanding and modeling behavior, while neurobiologists provide tools for the study of mechanism. The goal of this discipline is thus to understand the processes that connect sensation and action by revealing the neurobiological mechanisms by which decisions are made. This review describes recent developments in neuroeconomics from both behavioral and biological perspectives.

A general mechanism for perceptual decision-making in the human brain

Findings from single-cell recording studies suggest that a comparison of the outputs of different pools of selectively tuned lower-level sensory neurons may be a general mechanism by which higher-level brain regions compute perceptual decisions. For example, when monkeys must decide whether a noisy field of dots is moving upward or downward, a decision can be formed by computing the difference in responses between lower-level neurons sensitive to upward motion and those sensitive to downward motion. Here we use functional magnetic resonance imaging and a categorization task in which subjects decide whether an image presented is a face or a house to test whether a similar mechanism is also at work for more complex decisions in the human brain and, if so, where in the brain this computation might be performed. Activity within the left dorsolateral prefrontal cortex is greater during easy decisions than during difficult decisions, covaries with the difference signal between face- and house-selective regions in the ventral temporal cortex, and predicts behavioural performance in the categorization task. These findings show that even for complex object categories, the comparison of the outputs of different pools of selectively tuned neurons could be a general mechanism by which the human brain computes perceptual decisions.

Activity in Posterior Parietal Cortex Is Correlated with the Relative Subjective Desirability of Action

Behavioral studies suggest that making a decision involves representing the overall desirability of all available actions and then selecting that action that is most desirable. Physiological studies have proposed that neurons in the parietal cortex play a role in selecting movements for execution. To test the hypothesis that these parietal neurons encode the subjective desirability of making particular movements, we exploited Nash's game theoretic equilibrium, during which the subjective desirability of multiple actions should be equal for human players. Behavior measured during a strategic game suggests that monkeys' choices, like those of humans, are guided by subjective desirability. Under these conditions, activity in the parietal cortex was correlated with the relative subjective desirability of actions irrespective of the specific combination of reward magnitude, reward probability, and response probability associated with each action. These observations may help place many recent findings regarding the posterior parietal cortex into a common conceptual framework.