Neuronal activity in primate dorsolateral and orbital prefrontal cortex during performance of a reward preference task

Category representation and generalization in the prefrontal cortex

Categorization is a function of the brain that serves to group together items and events in our environments. Here we review the following important issues related to category representation and generalization: namely, where categories are presented in the brain, and how the brain utilizes categorical membership to generate new information. Accumulated experimental evidence shows that the prefrontal cortex (PFC) plays a critical role in category formation and generalization. We propose that prefrontal neurons abstract the commonality beyond individual stimuli, and categorize these based on their common meaning by ignoring their physical properties and learning to represent the boundaries between behaviorally significant categories. We also claim that a subgroup of prefrontal neurons simultaneously receives the category-related information and specific property information (e.g. reward) associated with an exemplar, to form a category-based representation of that property, and propagates it among stimuli of the same category, possibly reflecting a neural basis for category generalization in the PFC. These results suggest that the PFC is involved in representing abstract rules, and generating new information on the basis of previously acquired knowledge.

Decision value computation in DLPFC and VMPFC adjusts to the available decision time

It is increasingly clear that simple decisions are made by computing decision values for the options under consideration, and then comparing these values to make a choice. Computational models of this process suggest that it involves the accumulation of information over time, but little is known about the temporal course of valuation in the brain. To examine this, we manipulated the available decision time and observed the consequences in the brain and behavioral correlates of choice. Participants were scanned with functional magnetic resonance imaging while they chose to eat or not eat basic food items, in two conditions differing in the amount of time provided for choice. After identifying valuation-related regions with unbiased whole-brain general linear models, we analyzed two regions of interest: ventromedial prefrontal cortex (VMPFC) and dorsolateral prefrontal cortex (DLPFC). Finite impulse response models of the upsampled estimated neural activity from those regions allowed us to examine the onset, duration and termination of decision value signals, and to compare across regions. We found evidence for the immediate onset of value computation in both regions, but an extended duration with longer decision time. However, this was not accompanied by behavioral changes in either the accuracy or determinants of choice. Finally, there was modest evidence that DLPFC computation correlated with, but lagged behind, VMPFC computation, suggesting the sharing of information across these regions. These findings have important implications for models of decision value computation and choice.

Neural correlates of spontaneous deception: A functional near-infrared spectroscopy (fNIRS)study

Deception is commonly seen in everyday social interactions. However, most of the knowledge about the underlying neural mechanism of deception comes from studies where participants were instructed when and how to lie. To study spontaneous deception, we designed a guessing game modeled after Greene and Paxton (2009) "Proceedings of the National Academy of Sciences, 106(30), 12506-12511", in which lying is the only way to achieve the performance level needed to end the game. We recorded neural responses during the game using near-infrared spectroscopy (NIRS). We found that when compared to truth-telling, spontaneous deception, like instructed deception, engenders greater involvement of such prefrontal regions as the left superior frontal gyrus. We also found that the correct-truth trials produced greater neural activities in the left middle frontal gyrus and right superior frontal gyrus than the incorrect-truth trials, suggesting the involvement of the reward system. Furthermore, the present study confirmed the feasibility of using NIRS to study spontaneous deception.

The decision to engage cognitive control is driven by expected reward-value: neural and behavioral evidence

Cognitive control is a fundamental skill reflecting the active use of task-rules to guide behavior and suppress inappropriate automatic responses. Prior work has traditionally used paradigms in which subjects are told when to engage cognitive control. Thus, surprisingly little is known about the factors that influence individuals' initial decision of whether or not to act in a reflective, rule-based manner. To examine this, we took three classic cognitive control tasks (Stroop, Wisconsin Card Sorting Task, Go/No-Go task) and created novel ‘free-choice’ versions in which human subjects were free to select an automatic, pre-potent action, or an action requiring rule-based cognitive control, and earned varying amounts of money based on their choices. Our findings demonstrated that subjects' decision to engage cognitive control was driven by an explicit representation of monetary rewards expected to be obtained from rule-use. Subjects rarely engaged cognitive control when the expected outcome was of equal or lesser value as compared to the value of the automatic response, but frequently engaged cognitive control when it was expected to yield a larger monetary outcome. Additionally, we exploited fMRI-adaptation to show that the lateral prefrontal cortex (LPFC) represents associations between rules and expected reward outcomes. Together, these findings suggest that individuals are more likely to act in a reflective, rule-based manner when they expect that it will result in a desired outcome. Thus, choosing to exert cognitive control is not simply a matter of reason and willpower, but rather, conforms to standard mechanisms of value-based decision making. Finally, in contrast to current models of LPFC function, our results suggest that the LPFC plays a direct role in representing motivational incentives.

Action selection in multi-effector decision making

Decision making and reinforcement learning over movements suffer from the curse of dimensionality: the space of possible movements is too vast to search or even represent in its entirety. When actions involve only a single effector, this problem can be ameliorated by considering that effector separately; accordingly, the brain's sensorimotor systems can subdivide choice by representing values and actions separately for each effector. However, for many actions, such as playing the piano, the value of an action by an effector (e.g., a hand) depends inseparably on the actions of other effectors. By definition, the values of such coordinated multi-effector actions cannot be represented by effector-specific action values, such as those that have been most extensively investigated in parietal and premotor regions. For such actions, one possible solution is to choose according to more abstract valuations over different goods or options, which can then be mapped onto the necessary motor actions. Such an approach separates the learning and decision problem, which will often be lower-dimensional than the space of possible movements, from the multi-effector movement planning problem. The ventromedial prefrontal cortex (vmPFC) is thought to contain goods-based value signals, so we hypothesized that this region might preferentially drive multi-effector action selection. To examine how the brain solves this problem, we used fMRI to compare patterns of BOLD activity in humans during reward learning tasks in which options were selected through either unimanual or bimanual actions, and in which the response requirements in the latter condition inseparably coupled valuation across both hands. We found value signals in the bilateral medial motor cortex and vmPFC, and consistent with previous studies, the medial motor value signals contained contra-lateral biases indicating effector-specificity, while the vmPFC value signals did not exhibit detectable effector specificity. Although neither region's value signaling differed significantly between bimanual and unimanual conditions, the vmPFC value region showed greater connectivity with the medial motor cortex during bimanual than during unimanual choices. The specific region implicated, the anterior mid-cingulate cortex, is thought to act as a hub that links subjective value signals to motor control centers. These results are consistent with the idea that while valuation for unilateral actions may be subserved by an effector-specific network, complex multi-effector actions preferentially implicate communication between motor regions and prefrontal regions, which may reflect increased top-down input into motor regions to guide action selection.

Sensorimotor learning biases choice behavior: a learning neural field model for decision making

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject's learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.

Decision making requires the selection between alternative actions. It has been suggested that action selection is not separate from motor preparation of the according actions, but rather that the selection emerges from the competition between different movement plans. We expand on this idea, and ask how action selection mechanisms interact with the learning of new action choices. We present a neurodynamic model that provides an integrated account of action selection and the learning of sensorimotor associations. The model explains recent electrophysiological findings from monkeys' sensorimotor cortex, and correctly predicted a newly described characteristic pattern of their choice errors. Based on the model, we present a theory of how geometrical sensorimotor mapping rules can be learned by association without the need for an explicit representation of the transformation rule, and how the learning history of these associations can have a direct influence on later decision making.

High and low sensation seeking adolescents show distinct patterns of brain activity during reward processing

Previous research has shown that personality characteristics, such as sensation seeking (SS), are strong predictors of risk-taking behavior during adolescence. However, the relationship between levels of SS and brain response has not been studied during this time period. Given the prevalence of risky behavior during adolescence, it is important to understand neurobiological differences in reward sensitivity between youth with high and low SS personalities. To this end, we used functional magnetic resonance imaging (fMRI) to examine differences in brain activity in an adolescent sample that included 27 high (HSS) and 27 low sensation seekers (LSS), defined by the Impulsive Sensation Seeking scale of the Zuckerman-Kuhlman Personality Questionnaire (Zuckerman et al., 1993). In the scanner, participants played a modified Wheel of Fortune decision-making task (Cservenka and Nagel, 2012) that resulted in trials with monetary Wins or No Wins. We compared age- and sex-matched adolescent HSS and LSS (mean age=13.94±1.05) on brain activity by contrasting Win vs. No Win trials. Our findings indicate that HSS show greater bilateral insular and prefrontal cortex (PFC) brain response on Win vs. No Win compared to LSS. Analysis of simple effects showed that while LSS showed comparable brain activity in these areas during Wins and No Wins, HSS showed significant differences in brain response to winning (activation) vs. not winning (deactivation), with between-group comparison suggesting significant differences in brain response, largely to reward absence. Group differences in insular activation between reward receipt and absence may suggest weak autonomic arousal to negative outcomes in HSS compared with LSS. Additionally, since the PFC is important for goal-directed behavior and attention, the current results may reflect that HSS allocate fewer attentional resources to negative outcomes than LSS. This insensitivity to reward absence in HSS may lead to a greater likelihood of maladaptive choices when negative consequences are not considered, and may be an early neural marker of decreased loss sensitivity that has been seen in addiction. This neurobiological information may ultimately be helpful in establishing prevention strategies aimed at reducing youth risk-taking and suggests value in further examination of neural associations with personality characteristics during adolescence.

Brain volumetric and microstructural correlates of executive and motor performance in aged rhesus monkeys

The aged rhesus macaque exhibits brain atrophy and behavioral deficits similar to normal aging in humans. Here we studied the association between cognitive and motor performance and anatomic and microstructural brain integrity measured with 3T magnetic resonance imaging in aged monkeys. About half of these animals were maintained on moderate calorie restriction (CR), the only intervention shown to delay the aging process in lower animals. T1-weighted anatomic and diffusion tensor images were used to obtain gray matter (GM) volume and fractional anisotropy (FA) and mean diffusivity (MD), respectively. We tested the extent to which brain health indexed by GM volume, FA, and MD were related to executive and motor function, and determined the effect of the dietary intervention on this relationship. We hypothesized that fewer errors on the executive function test and faster motor response times would be correlated with higher volume, higher FA, and lower MD in frontal areas that mediate executive function, and in motor, premotor, subcortical, and cerebellar areas underlying goal-directed motor behaviors. Higher error percentage on a cognitive conceptual shift task was significantly associated with lower GM volume in frontal and parietal cortices, and lower FA in major association fiber bundles. Similarly, slower performance time on the motor task was significantly correlated with lower volumetric measures in cortical, subcortical, and cerebellar areas and decreased FA in several major association fiber bundles. Notably, performance during the acquisition phase of the hardest level of the motor task was significantly associated with anterior mesial temporal lobe volume. Finally, these brain-behavior correlations for the motor task were attenuated in CR animals compared to controls, indicating a potential protective effect of the dietary intervention.

Social signals in primate orbitofrontal cortex

Primate evolution produced an increased capacity to respond flexibly to varying social contexts as well as expansion of the prefrontal cortex. Despite this association, how prefrontal neurons respond to social information remains virtually unknown. People with damage to their orbitofrontal cortex (OFC) struggle to recognize facial expressions, make poor social judgments, and frequently make social faux pas. Here we test explicitly whether neurons in primate OFC signal social information and, if so, how such signals compare with responses to primary fluid rewards. We find that OFC neurons distinguish images that belong to socially defined categories, such as female perinea and faces, as well as the social dominance of those faces. These modulations signaled both how much monkeys valued these pictures and their interest in continuing to view them. Far more neurons signaled social category than signaled fluid value, despite the stronger impact of fluid reward on monkeys' choices. These findings indicate that OFC represents both the motivational value and attentional priority of other individuals, thus contributing to both the acquisition of information about others and subsequent social decisions. Our results betray a fundamental disconnect between preferences expressed through overt choice, which were primarily driven by the desire for more fluid, and preferential neuronal processing, which favored social computations.

The road not taken: neural correlates of decision making in orbitofrontal cortex

Empirical research links human orbitofrontal cortex (OFC) to the evaluation of outcomes during decision making and the representation of alternative (better) outcomes after failures. When faced with a difficult decision, rats sometimes pause and turn back-and-forth toward goals, until finally orienting toward the chosen direction. Neural representations of reward in rodent OFC increased immediately following each reorientation, implying a transient representation of the expected outcome following self-initiated decisions. Upon reaching reward locations and finding no reward (having made an error), OFC representations of reward decreased locally indicating a disappointment signal that then switched to represent the unrewarded, non-local, would-have-been rewarded site. These results illustrate that following a decision to act, neural ensembles in OFC represent reward, and upon the realization of an error, represent the reward that could have been.

Asymmetric frontal brain activity and parental rejection predict altruistic behavior: moderation of oxytocin effects

Asymmetric frontal brain activity has been widely implicated in reactions to emotional stimuli and is thought to reflect individual differences in approach–withdrawal motivation. Here, we investigate whether asymmetric frontal activity, as a measure of approach–withdrawal motivation, also predicts charitable donations after a charity’s (emotion-eliciting) promotional video showing a child in need is viewed, in a sample of 47 young adult women. In addition, we explore possibilities for mediation and moderation, by asymmetric frontal activity, of the effects of intranasally administered oxytocin and parental love withdrawal on charitable donations. Greater relative left frontal activity was related to larger donations. In addition, we found evidence of moderation: Low levels of parental love withdrawal predicted larger donations in the oxytocin condition for participants showing greater relative right frontal activity. We suggest that when approach motivation is high (reflected in greater relative left frontal activity), individuals are generally inclined to take action upon seeing someone in need and, thus, to donate money to actively help out. Only when approach motivation is low (reflected in less relative left/greater relative right activity) do empathic concerns affected by oxytocin and experiences of love withdrawal play an important part in deciding about donations.

Efficient coding and the neural representation of value

To survive in a dynamic environment, an organism must be able to effectively learn, store, and recall the expected benefits and costs of potential actions. The nature of the valuation and decision processes is thus of fundamental interest to researchers at the intersection of psychology, neuroscience, and economics. Although normative theories of choice have outlined the theoretical structure of these valuations, recent experiments have begun to reveal how value is instantiated in the activity of neurons and neural circuits. Here, we review the various forms of value coding that have been observed in different brain systems and examine the implications of these value representations for both neural circuits and behavior. In particular, we focus on emerging evidence that value coding in a number of brain areas is context dependent, varying as a function of both the current choice set and previously experienced values. Similar contextual modulation occurs widely in the sensory system, and efficient coding principles derived in the sensory domain suggest a new framework for understanding the neural coding of value.

Prefrontal and striatal activity related to values of objects and locations

The value of an object acquired by a particular action often determines the motivation to produce that action. Previous studies found neural signals related to the values of different objects or goods in the orbitofrontal cortex, while the values of outcomes expected from different actions are broadly represented in multiple brain areas implicated in movement planning. However, how the brain combines the values associated with various objects and the information about their locations is not known. In this study, we tested whether the neurons in the dorsolateral prefrontal cortex (DLPFC) and striatum in rhesus monkeys might contribute to translating the value signals between multiple frames of reference. Monkeys were trained to perform an oculomotor intertemporal choice in which the color of a saccade target and the number of its surrounding dots signaled the magnitude of reward and its delay, respectively. In both DLPFC and striatum, temporally discounted values (DVs) associated with specific target colors and locations were encoded by partially overlapping populations of neurons. In the DLPFC, the information about reward delays and DVs of rewards available from specific target locations emerged earlier than the corresponding signals for target colors. Similar results were reproduced by a simple network model built to compute DVs of rewards in different locations. Therefore, DLPFC might play an important role in estimating the values of different actions by combining the previously learned values of objects and their present locations.

Proactive and reactive control by the medial frontal cortex

Adaptive behavior requires the ability to flexibly control actions. This can occur either proactively to anticipate task requirements, or reactively in response to sudden changes. Recent work in humans has identified a network of cortical and subcortical brain region that might have an important role in proactive and reactive control. However, due to technical limitations, such as the spatial and temporal resolution of the BOLD signal, human imaging experiments are not able to disambiguate the specific function(s) of these brain regions. These limitations can be overcome through single-unit recordings in non-human primates. In this article, we describe the behavioral and physiological evidence for dual mechanisms of control in response inhibition in the medial frontal cortex of monkeys performing the stop signal or countermanding task.

Changes in brain activity in response to problem solving during the abstinence from online game play

BACKGROUND AND AIMS

Several studies have suggested that addictive disorders including substance abuse and pathologic gambling might be associated with dysfunction on working memory and prefrontal activity. We hypothesized that excessive online game playing is associated with deficits in prefrontal cortex function and that recovery from excessive online game playing might improve prefrontal cortical activation in response to working memory stimulation.

METHODS

Thirteen adolescents with excessive online game playing (AEOP) and ten healthy adolescents (HC) agreed to participate in this study. The severity of online game play and playing time were evaluated for a baseline measurement and again following four weeks of treatment. Brain activation in response to working memory tasks (simple and complex calculations) at baseline and subsequent measurements was assessed using BOLD functional magnetic resonance imaging (fMRI).

RESULTS

Compared to the HC subjects, the AEOP participants exhibited significantly greater activity in the right middle occipital gyrus, left cerebellum posterior lobe, left premotor cortex and left middle temporal gyrus in response to working memory tasks during baseline measurements. After four weeks of treatment, the AEOP subjects showed increased activity within the right dorsolateral prefrontal cortex and left occipital fusiform gyrus. After four weeks of treatment, changes in the severity of online game playing were negatively correlated with changes in the mean β value of the right dorsolateral prefrontal cortex in response to complex stimulation.

CONCLUSIONS

We suggest that the effects of online game addiction on working memory may be similar to those observed in patients with substance dependence.

Developmental effects of incentives on response inhibition

Inhibitory control and incentive processes underlie decision making, yet few studies have explicitly examined their interaction across development. Here, the effects of potential rewards and losses on inhibitory control in 64 adolescents (13- to 17-year-olds) and 42 young adults (18- to 29-year-olds) were examined using an incentivized antisaccade task. Notably, measures were implemented to minimize age-related differences in reward valuation and potentially confounding motivation effects. Incentives affected antisaccade metrics differently across the age groups. Younger adolescents generated more errors than adults on reward trials, but all groups performed well on loss trials. Adolescent saccade latencies also differed from adults across the range of reward trials. Overall, results suggest persistent immaturities in the integration of reward and inhibitory control processes across adolescence.

Neural basis of reinforcement learning and decision making

Reinforcement learning is an adaptive process in which an animal utilizes its previous experience to improve the outcomes of future choices. Computational theories of reinforcement learning play a central role in the newly emerging areas of neuroeconomics and decision neuroscience. In this framework, actions are chosen according to their value functions, which describe how much future reward is expected from each action. Value functions can be adjusted not only through reward and penalty, but also by the animal's knowledge of its current environment. Studies have revealed that a large proportion of the brain is involved in representing and updating value functions and using them to choose an action. However, how the nature of a behavioral task affects the neural mechanisms of reinforcement learning remains incompletely understood. Future studies should uncover the principles by which different computational elements of reinforcement learning are dynamically coordinated across the entire brain.

Orbitofrontal cortical activity during repeated free choice

Neurons in the orbitofrontal cortex (OFC) have been shown to encode subjective values, suggesting a role in preference-based decision-making, although the precise relation to choice behavior is unclear. In a repeated two-choice task, subjective values of each choice can account for aggregate choice behavior, which is the overall likelihood of choosing one option over the other. Individual choices, however, are impossible to predict with knowledge of relative subjective values alone. In this study we investigated the role of internal factors in choice behavior with a simple but novel free-choice task and simultaneous recording from individual neurons in nonhuman primate OFC. We found that, first, the observed sequences of choice behavior included periods of exceptionally long runs of each of two available options and periods of frequent switching. Neither a satiety-based mechanism nor a random selection process could explain the observed choice behavior. Second, OFC neurons encode important features of the choice behavior. These features include activity selective for exceptionally long runs of a given choice (stay selectivity) as well as activity selective for switches between choices (switch selectivity). These results suggest that OFC neural activity, in addition to encoding subjective values on a long timescale that is sensitive to satiety, also encodes a signal that fluctuates on a shorter timescale and thereby reflects some of the statistically improbable aspects of free-choice behavior.

Empathic choice involves vmPFC value signals that are modulated by social processing implemented in IPL

Empathic decision-making involves making choices on behalf of others in order to maximize their well-being. Examples include the choices that parents make for their children, as well as the decisions of a politician trying to make good choices on behalf of his constituency. We investigated the neurobiological and computational basis of empathic choice using a human fMRI task in which subjects purchased DVDs for themselves with their own money, or DVDs for others with the other's money. We found that empathic choices engage the same regions of ventromedial prefrontal cortex that are known to compute stimulus values, and that these value signals were modulated by activity from a region of inferior parietal lobule (IPL) known to play a critical role in social processes such as empathy. We also found that the stimulus value signals used to make empathic choices were computed using a mixture of self-simulation and other-simulation processes, and that activity in IPL encoded a variable measuring the distance between the other's and self preferences, which provides a hint for how the mixture of self- and other-simulation might be implemented.

The neurobiology of intertemporal choice: insight from imaging and lesion studies

People are frequently faced with intertemporal choices, i.e., choices differing in the timing of their consequences, preferring smaller rewards available immediately over larger rewards delivered after a delay. The inability to forgo sooner gratification to favor delayed reward (e.g., impulsivity) has been related to several pathological conditions characterized by poor self-control, including drug addiction and obesity. Comparative and functional human studies have implicated a network of brain areas involved in intertemporal choice, including the medial portion of the orbitofrontal cortex (mOFC). Moreover, damage to this cortical area increases preference for immediate gratification in intertemporal decisions. Here, we review recent neuroscientific studies concerning intertemporal choice, suggesting that the mOFC contributes to preference for delayed rewards, either by computing the value of future outcomes (i.e., valuation), or by enabling people to imagine and represent future rewards and their consequences (e.g., prospection).

Contrasting reward signals in the orbitofrontal cortex and anterior cingulate cortex

Damage to the orbitofrontal cortex (OFC) and anterior cingulate cortex (ACC) impairs decision making, but the underlying value computations that cause such impairments remain unclear. Both the OFC and ACC encode a wide variety of signals correlated with decision making. The current challenge is to determine how these two different areas support decision-making processes. Here, we review a series of experiments that have helped define these roles. A special population of neurons in the ACC, but not the OFC, multiplex value information across decision parameters using a unified encoding scheme, and encode reward prediction errors. In contrast, neurons in the OFC, but not the ACC, encode the value of a choice relative to the recent history of choice values. Together, these results suggest complementary valuation processes: OFC neurons dynamically evaluate current choices relative to the value contexts recently experienced, while ACC neurons encode choice predictions and prediction errors using a common valuation currency reflecting the integration of multiple decision parameters.

The orbitofrontal cortex and response selection

Orbitofrontal cortex (OFC) function is often characterized in terms of stimulus-reward mapping; however, more recent evidence suggests that the OFC may play a role in selecting and representing extended actions. First, previously encoded reward associations in the OFC could be used to inform responding in novel but similar situations. Second, when evaluated in tasks requiring the animal to perform extended actions, response selective activity can be recorded in the OFC. Finally, the interaction between the OFC and hippocampus illustrates OFC's role in response selection. The OFC may facilitate reward-guided memory retrieval by selecting the memories most relevant to achieve a goal. This model for OFC function places it within the hierarchy of increasingly complex action representations that support decision making.

Giving credit where credit is due: orbitofrontal cortex and valuation in an uncertain world

The orbitofrontal cortex (OFC) has long been implicated in aspects of learning and adaptive decision making in changeable environments, but its precise role has remained elusive. One potential reason is that anatomical and functional distinctions within the OFC have often been overlooked. Here, we review findings centered largely on recent lesion studies in macaque monkeys from our laboratories that have investigated the causal role of the lateral and medial parts of the OFC (LOFC and MOFC) in choice behavior in uncertain, multioption environments. MOFC appears necessary for focusing attention on only the relevant decision variables to achieve a goal. By contrast, LOFC is required to allow rapid learning in changeable environments by enabling the credit for a particular outcome to be assigned to a specific choice.

Balkanizing the primate orbitofrontal cortex: distinct subregions for comparing and contrasting values

The primate orbitofrontal cortex (OFC) is often treated as a single entity, but architectonic and connectional neuroanatomy indicate that it has distinguishable parts. Nevertheless, few studies have attempted to dissociate the functions of its subregions. Here we review findings from recent neuropsychological and neurophysiological studies that do so. The lateral OFC seems to be important for learning, representing, and updating specific object-reward associations. The medial OFC seems to be important for value comparisons and choosing among objects on that basis. Rather than viewing this dissociation of function in terms of learning versus choosing, however, we suggest that it reflects the distinction between contrasts and comparisons: differences versus similarities. Making use of high-dimensional representations that arise from the convergence of several sensory modalities, the lateral OFC encodes contrasts among outcomes. The medial OFC reduces these contrasting representations of value to a single dimension, a common currency, in order to compare alternative choices.

Representations of appetitive and aversive information in the primate orbitofrontal cortex

Individuals weigh information about both rewarding and aversive stimuli to make adaptive decisions. Most studies of the orbitofrontal cortex (OFC), an area where appetitive and aversive neural subsystems might interact, have focused only on reward. Using a classical conditioning task where novel stimuli are paired with a reward or an aversive air puff, we discovered that two groups of orbitofrontal neurons respond preferentially to conditioned stimuli associated with rewarding and aversive outcomes; however, information about appetitive and aversive stimuli converges on individual neurons from both populations. Therefore, neurons in the OFC might participate in appetitive and aversive networks that track the motivational significance of stimuli even when they vary in valence and sensory modality. Further, we show that these networks, which also extend to the amygdala, exhibit different rates of change during reversal learning. Thus, although both networks represent appetitive and aversive associations, their distinct temporal dynamics might indicate different roles in learning processes.

The impact of orbitofrontal dysfunction on cocaine addiction

Cocaine addiction is characterized by poor judgment and maladaptive decision-making. Here we review evidence implicating the orbitofrontal cortex in such behavior. This evidence suggests that cocaine-induced changes in orbitofrontal cortex disrupt the representation of states and transition functions that form the basis of flexible and adaptive 'model-based' behavioral control. By impairing this function, cocaine exposure leads to an overemphasis on less flexible, maladaptive 'model-free' control systems. We propose that such an effect accounts for the complex pattern of maladaptive behaviors associated with cocaine addiction.

Differential encoding of factors influencing predicted reward value in monkey rostral anterior cingulate cortex

Background

The value of a predicted reward can be estimated based on the conjunction of both the intrinsic reward value and the length of time to obtain it. The question we addressed is how the two aspects, reward size and proximity to reward, influence the responses of neurons in rostral anterior cingulate cortex (rACC), a brain region thought to play an important role in reward processing.

Methods and Findings

We recorded from single neurons while two monkeys performed a multi-trial reward schedule task. The monkeys performed 1–4 sequential color discrimination trials to obtain a reward of 1–3 liquid drops. There were two task conditions, a valid cue condition, where the number of trials and reward amount were associated with visual cues, and a random cue condition, where the cue was picked from the cue set at random. In the valid cue condition, the neuronal firing is strongly modulated by the predicted reward proximity during the trials. Information about the predicted reward amount is almost absent at those times. In substantial subpopulations, the neuronal responses decreased or increased gradually through schedule progress to the predicted outcome. These two gradually modulating signals could be used to calculate the effect of time on the perception of reward value. In the random cue condition, little information about the reward proximity or reward amount is encoded during the course of the trial before reward delivery, but when the reward is actually delivered the responses reflect both the reward proximity and reward amount.

Conclusions

Our results suggest that the rACC neurons encode information about reward proximity and amount in a manner that is dependent on utility of reward information. The manner in which the information is represented could be used in the moment-to-moment calculation of the effect of time and amount on predicted outcome value.

Anatomical insights into the interaction of emotion and cognition in the prefrontal cortex

Psychological research increasingly indicates that emotional processes interact with other aspects of cognition. Studies have demonstrated both the ability of emotional stimuli to influence a broad range of cognitive operations, and the ability of humans to use top-down cognitive control mechanisms to regulate emotional responses. Portions of the prefrontal cortex appear to play a significant role in these interactions. However, the manner in which these interactions are implemented remains only partially elucidated. In the present review we describe the anatomical connections between ventral and dorsal prefrontal areas as well as their connections with limbic regions. Only a subset of prefrontal areas are likely to directly influence amygdalar processing, and as such models of prefrontal control of emotions and models of emotional regulation should be constrained to plausible pathways of influence. We also focus on how the specific pattern of feedforward and feedback connections between these regions may dictate the nature of information flow between ventral and dorsal prefrontal areas and the amygdala. These patterns of connections are inconsistent with several commonly expressed assumptions about the nature of communications between emotion and cognition.

Transient inactivation of orbitofrontal cortex blocks reinforcer devaluation in macaques

The orbitofrontal cortex (OFC) and its interactions with the basolateral amygdala (BLA) are critical for goal-directed behavior, especially for adapting to changes in reward value. Here we used a reinforcer devaluation paradigm to investigate the contribution of OFC to this behavior in four macaques. Subjects that had formed associations between objects and two different primary reinforcers (foods) were presented with choices of objects overlying the two different foods. When one of the two foods was devalued by selective satiation, the subjects shifted their choices toward the objects that represented the nonsated food reward (devaluation effect). Transient inactivation of OFC by infusions of the GABA(A) receptor agonist muscimol into area 13 blocked the devaluation effect: the monkeys did not reduce their selection of objects associated with the devalued food. This effect was observed when OFC was inactivated during both satiation and the choice test, and during the choice test only. This supports our hypothesis that OFC activity is required during the postsatiety object choice period to guide the selection of objects. This finding sharply contrasts with the role of BLA in the same devaluation process (Wellman et al., 2005). Whereas activity in BLA was required during the selective satiation procedure, it was not necessary for guiding the subsequent object choice. Our results are the first to demonstrate that transient inactivation of OFC is sufficient to disrupt the devaluation effect, and to document a role for OFC distinct from that of BLA for the conditioned reinforcer devaluation process in monkeys.

Sexual dimorphism of sulcal length asymmetry in the cerebrum of adult cynomolgus monkeys (Macaca fascicularis)

The present study aimed to quantitatively clarify the gross anatomical asymmetry and sexual dimorphism of the cerebral hemispheres of cynomolgus monkeys. While the fronto-occipital length of the right and left cerebral hemispheres was not different between sexes, a statistically significant rightward asymmetry was detected in the cerebral width at the perisylvian region in females, but not in males (narrower width of the left side in the females). An asymmetry quotient of the sulcal lengths revealed a rightward asymmetry in the inferior occipital sulcus and a leftward asymmetry in the central and intraparietal sulci in both sexes. However, the laterality of the lengths of other sulci was different for males and females. The arcuate sulcus was directed rightward in males but there was no rightward bias in females. Interestingly, the principle sulcus and lateral fissure were left-lateralized in the males, but right-lateralized in the females. The results suggest that lateralization patterns are regionally and sexually different in the cerebrum of cynomolgus monkeys. The present results provide a reference for quantitatively evaluating the normality of the cerebral cortical morphology in cynomolgus monkeys.

Challenges of Interpreting Frontal Neurons during Value-Based Decision-Making

The frontal cortex is crucial to sound decision-making, and the activity of frontal neurons correlates with many aspects of a choice, including the reward value of options and outcomes. However, rewards are of high motivational significance and have widespread effects on neural activity. As such, many neural signals not directly involved in the decision process can correlate with reward value. With correlative techniques such as electrophysiological recording or functional neuroimaging, it can be challenging to distinguish neural signals underlying value-based decision-making from other perceptual, cognitive, and motor processes. In the first part of the paper, we examine how different value-related computations can potentially be confused. In particular, error-related signals in the anterior cingulate cortex, generated when one discovers the consequences of an action, might actually represent violations of outcome expectation, rather than errors per se. Also, signals generated at the time of choice are typically interpreted as reflecting predictions regarding the outcomes associated with the different choice alternatives. However, these signals could instead reflect comparisons between the presented choice options and previously presented choice alternatives. In the second part of the paper, we examine how value signals have been successfully dissociated from saliency-related signals, such as attention, arousal, and motor preparation in studies employing outcomes with both positive and negative valence. We hope that highlighting these issues will prove useful for future studies aimed at disambiguating the contribution of different neuronal populations to choice behavior.

Cross-species studies of orbitofrontal cortex and value-based decision-making

Recent work has emphasized the role that orbitofrontal cortex (OFC) has in value-based decision-making. However, it is also clear that a number of discrepancies have arisen when comparing the findings from animal models to those from humans. Here, we examine several possibilities that might explain these discrepancies, including anatomical difference between species, the behavioral tasks used to probe decision-making and the methodologies used to assess neural function. Understanding how these differences affect the interpretation of experimental results will help us to better integrate future results from animal models. This will enable us to fully realize the benefits of using multiple approaches to understand OFC function.

Double dissociation of value computations in orbitofrontal and anterior cingulate neurons

Damage to prefrontal cortex (PFC) impairs decision-making, but the underlying value computations that might cause such impairments remain unclear. Here we report that value computations are doubly dissociable among PFC neurons. Although many PFC neurons encoded chosen value, they used opponent encoding schemes such that averaging the neuronal population extinguished value coding. However, a special population of neurons in anterior cingulate cortex (ACC), but not in orbitofrontal cortex (OFC), multiplexed chosen value across decision parameters using a unified encoding scheme and encoded reward prediction errors. In contrast, neurons in OFC, but not ACC, encoded chosen value relative to the recent history of choice values. Together, these results suggest complementary valuation processes across PFC areas: OFC neurons dynamically evaluate current choices relative to recent choice values, whereas ACC neurons encode choice predictions and prediction errors using a common valuation currency reflecting the integration of multiple decision parameters.

Interaction between dysfunctional connectivity at rest and heroin cues-induced brain responses in male abstinent heroin-dependent individuals

Background

The majority of previous heroin cue-reactivity functional magnetic resonance imaging (fMRI) studies focused on local function impairments, such as inhibitory control, decision-making and stress regulation. Our previous studies have demonstrated that these brain circuits also presented dysfunctional connectivity during the resting state. Yet few studies considered the relevance of resting state dysfunctional connectivity to task-related neural activity in the same chronic heroin user (CHU).

Methodology/Principal Findings

We employed the method of graph theory analysis, which detected the abnormality of brain regions and dysregulation of brain connections at rest between 16 male abstinent chronic heroin users (CHUs) and 16 non-drug users (NDUs). Using a cue-reactivity task, we assessed the relationship between drug-related cue-induced craving activity and the abnormal topological properties of the CHUs' resting networks. Comparing NDUs' brain activity to that of CHUs, the intensity of functional connectivity of the medial frontal gyrus (meFG) in patients' resting state networks was prominently greater and positively correlated with the same region's neural activity in the heroin-related task; decreased functional connectivity intensity of the anterior cingulate cortex (ACC) in CHUs at rest was associated with more drug-related cue-induced craving activities.

Conclusions

These results may indicate that there exist two brain systems interacting simultaneously in the heroin-addicted brain with regards to a cue-reactivity task. The current study may shed further light on the neural architecture that supports craving responses in heroin dependence.

Transformation of stimulus value signals into motor commands during simple choice

Decision-making can be broken down into several component processes: assigning values to stimuli under consideration, selecting an option by comparing those values, and initiating motor responses to obtain the reward. Although much is known about the neural encoding of stimulus values and motor commands, little is known about the mechanisms through which stimulus values are compared, and the resulting decision is transmitted to motor systems. We investigated this process using human fMRI in a task where choices were indicated using the left or right hand. We found evidence consistent with the hypothesis that value signals are computed in the ventral medial prefrontal cortex, they are passed to regions of dorsomedial prefrontal cortex and intraparietal sulcus, implementing a comparison process, and the output of the comparator regions modulates activity in motor cortex to implement the choice. These results describe the network through which stimulus values are transformed into actions during a simple choice task.

The neural basis of drug stimulus processing and craving: an activation likelihood estimation meta-analysis

BACKGROUND

The capacity of drug cues to elicit drug-seeking behavior is believed to play a fundamental role in drug dependence; yet the neurofunctional basis of human drug cue-reactivity is not fully understood. We performed a meta-analysis to identify brain regions that are consistently activated by presentation of drug cues. Studies involving treatment-seeking and nontreatment-seeking substance users were contrasted to determine whether there were consistent differences in the neural response to drug cues between these populations. Finally, to assess the neural basis of craving, consistency across studies in brain regions that show correlated activation with craving was assessed.

METHODS

Appropriate studies, assessing the effect of drug-related cues or manipulations of drug craving in drug-user populations across the whole brain, were obtained via the PubMed database and literature search. Activation likelihood estimation, a method of quantitative meta-analysis that estimates convergence across experiments by modeling the spatial uncertainty of neuroimaging data, was used to identify consistent regions of activation.

RESULTS

Cue-related activation was observed in the ventral striatum (across both subgroups), amygdala (in the treatment-seeking subgroup and overall), and orbitofrontal cortex (in the nontreatment-seeking subgroup and overall) but not insula cortex. Although a different pattern of frontal and temporal lobe activation between the subgroups was observed, these differences were not significant. Finally, right amygdala and left middle frontal gyrus activity were positively associated with craving.

CONCLUSIONS

These results substantiate the key neural substrates underlying reactivity to drug cues and drug craving.

Exploring prefrontal cortical memory mechanisms with eyeblink conditioning

Several studies in nonhuman primates have shown that neurons in the dorsolateral prefrontal cortex have activity that persists throughout the delay period in delayed matching to sample tasks, and age-related changes in the microcolumnar organization of the prefrontal cortex are significantly correlated with age-related declines in cognition. Activity that persists beyond the presentation of a stimulus could mediate working memory processes, and disruption of those processes could account for memory deficits that often accompany the aging process. These potential memory and aging mechanisms are being systematically examined with eyeblink conditioning paradigms in nonprimate mammalian animal models including the rabbit. The trace version of the conditioning paradigm is a particularly good system to explore declarative memory since humans do not acquire trace conditioning if they are unable to become cognitively aware of the association between a conditioning tone and an airpuff to the eye. This conditioning paradigm has been used to show that the hippocampus and cerebellum interact functionally since both conditioned responses and conditioned hippocampal pyramidal neuron activity are abolished following lesions of the cerebellar nuclei and since hippocampal lesions prevent or abolish trace conditioned blinks. However, because there are no direct connections between the hippocampal formation and the cerebellum, and because the hippocampus is not necessary for trace conditioning after a period of consolidation has elapsed, we and others have been examining the prefrontal cortex for its role in forebrain-dependent trace eyeblink conditioning. This review examines some of the literature which suggests that the prefrontal cortex serves to orchestrate a neuronal network that interacts with the cerebellum to mediate adaptively timed conditioned responses.

Decision making and reward in frontal cortex: complementary evidence from neurophysiological and neuropsychological studies

Patients with damage to the prefrontal cortex (PFC)—especially the ventral and medial parts of PFC—often show a marked inability to make choices that meet their needs and goals. These decision-making impairments often reflect both a deficit in learning concerning the consequences of a choice, as well as deficits in the ability to adapt future choices based on experienced value of the current choice. Thus, areas of PFC must support some value computations that are necessary for optimal choice. However, recent frameworks of decision making have highlighted that optimal and adaptive decision making does not simply rest on a single computation, but a number of different value computations may be necessary. Using this framework as a guide, we summarize evidence from both lesion studies and single-neuron physiology for the representation of different value computations across PFC areas.

Dissociable effects of subtotal lesions within the macaque orbital prefrontal cortex on reward-guided behavior

The macaque orbital prefrontal cortex (PFo) has been implicated in a wide range of reward-guided behaviors essential for efficient foraging. The PFo, however, is not a homogeneous structure. Two major subregions, distinct by their cytoarchitecture and connections to other brain structures, compose the PFo. One subregion encompasses Walker's areas 11 and 13 and the other centers on Walker's area 14. Although it has been suggested that these subregions play dissociable roles in reward-guided behavior, direct neuropsychological evidence for this hypothesis is limited. To explore the independent contributions of PFo subregions to behavior, we studied rhesus monkeys (Macaca mulatta) with restricted excitotoxic lesions targeting either Walker's areas 11/13 or area 14. The performance of these two groups was compared to that of a group of unoperated controls on a series of reward-based tasks that has been shown to be sensitive to lesions of the PFo as a whole (Walker's areas 11, 13, and 14). Lesions of areas 11/13, but not area 14, disrupted the rapid updating of object value during selective satiation. In contrast, lesions targeting area 14, but not areas 11/13, impaired the ability of monkeys to learn to stop responding to a previously rewarded object. Somewhat surprisingly, neither lesion disrupted performance on a serial object reversal learning task, although aspiration lesions of the entire PFo produce severe deficits on this task. Our data indicate that anatomically defined subregions within macaque PFo make dissociable contributions to reward-guided behavior.

Dissociable effects of natural image structure and color on LFP and spiking activity in the lateral prefrontal cortex and extrastriate visual area V4

Visual perception is mediated by unique contributions of the numerous brain regions that constitute the visual system. We performed simultaneous recordings of local field potentials (LFPs) and single unit activity (SUA) in areas V4 and lateral prefrontal cortex to characterize their contribution to visual processing. Here, we trained monkeys to identify natural images at different degradation levels in a visual recognition task. We parametrically varied color and structural information of natural images while the animals were performing the task. We show that the visual-evoked potential (VEP) of the LFP in V4 is highly sensitive to color, whereas the VEP in prefrontal cortex predominantly depends on image structure. When examining the relationship between VEP and SUA, we found that stimulus sensitivity for SUA was well predicted by the VEP in PF cortex but not in V4. Our results first reveal a functional specialization in both areas at the level of the LFP and further suggest that the degree to which mesoscopic signals, such as the VEP, are representative of the underlying SUA neural processing may be brain region specific within the context of visual recognition.

Dorsolateral prefrontal cortex drives mesolimbic dopaminergic regions to initiate motivated behavior

How does the brain translate information signaling potential rewards into motivation to get them? Motivation to obtain reward is thought to depend on the midbrain [particularly the ventral tegmental area (VTA)], the nucleus accumbens (NAcc), and the dorsolateral prefrontal cortex (dlPFC), but it is not clear how the interactions among these regions relate to reward-motivated behavior. To study the influence of motivation on these reward-responsive regions and on their interactions, we used dynamic causal modeling to analyze functional magnetic resonance imaging (fMRI) data from humans performing a simple task designed to isolate reward anticipation. The use of fMRI permitted the simultaneous measurement of multiple brain regions while human participants anticipated and prepared for opportunities to obtain reward, thus allowing characterization of how information about reward changes physiology underlying motivational drive. Furthermore, we modeled the impact of external reward cues on causal relationships within this network, thus elaborating a link between physiology, connectivity, and motivation. Specifically, our results indicated that dlPFC was the exclusive entry point of information about reward in this network, and that anticipated reward availability caused VTA activation only via its effect on the dlPFC. Anticipated reward thus increased dlPFC activation directly, whereas it influenced VTA and NAcc only indirectly, by enhancing intrinsically weak or inactive pathways from the dlPFC. Our findings of a directional prefrontal influence on dopaminergic regions during reward anticipation suggest a model in which the dlPFC integrates and transmits representations of reward to the mesolimbic and mesocortical dopamine systems, thereby initiating motivated behavior.

Distributed coding of actual and hypothetical outcomes in the orbital and dorsolateral prefrontal cortex

Knowledge about hypothetical outcomes from unchosen actions is beneficial only when such outcomes can be correctly attributed to specific actions. Here we show that during a simulated rock-paper-scissors game, rhesus monkeys can adjust their choice behaviors according to both actual and hypothetical outcomes from their chosen and unchosen actions, respectively. In addition, neurons in both dorsolateral prefrontal cortex and orbitofrontal cortex encoded the signals related to actual and hypothetical outcomes immediately after they were revealed to the animal. Moreover, compared to the neurons in the orbitofrontal cortex, those in the dorsolateral prefrontal cortex were more likely to change their activity according to the hypothetical outcomes from specific actions. Conjunctive and parallel coding of multiple actions and their outcomes in the prefrontal cortex might enhance the efficiency of reinforcement learning and also contribute to their context-dependent memory.

Domain expertise insulates against judgment bias by monetary favors through a modulation of ventromedial prefrontal cortex

Recent work using an art-viewing paradigm shows that monetary sponsorship of the experiment by a company (a favor) increases the valuation of paintings placed next to the sponsoring corporate logo, an effect that correlates with modulation of the ventromedial prefrontal cortex (VMPFC). We used the same art-viewing paradigm to test a prevailing idea in the domain of conflict-of-interest: that expertise in a domain insulates against judgment bias even in the presence of a monetary favor. Using a cohort of art experts, we show that monetary favors do not bias the experts' valuation of art, an effect that correlates with a lack of modulation of the VMPFC across sponsorship conditions. The lack of sponsorship effect in the VMPFC suggests the hypothesis that their brains remove the behavioral sponsorship effect by censoring sponsorship-dependent modulation of VMPFC activity. We tested the hypothesis that prefrontal regions play a regulatory role in mediating the sponsorship effect. We show that the dorsolateral prefrontal cortex (DLPFC) is recruited in the expert group. Furthermore, we tested the hypothesis in nonexpert controls by contrasting brain responses in controls who did not show a sponsorship effect to controls who did. Changes in effective connectivity between the DLPFC and VMPFC were greater in nonexpert controls, with an absence of the sponsorship effect relative to those with a presence of the sponsorship effect. The role of the DLPFC in cognitive control and emotion regulation suggests that it removes the influence of a monetary favor by controlling responses in known valuation regions of the brain including the the VMPFC.

Comparison of strategy signals in the dorsolateral and orbital prefrontal cortex

Abstract behavior-guiding rules and strategies allow monkeys to avoid errors in rarely encountered situations. In the present study, we contrasted strategy-related neuronal activity in the dorsolateral prefrontal cortex (PFdl) and the orbital prefrontal cortex (PFo) of rhesus monkeys. On each trial of their behavioral task, the monkeys responded to a foveal visual cue by making a saccade to one of two spatial targets. One response required a leftward saccade, the other required a saccade of equal magnitude to the right. The cues instructed the monkeys to follow one of two response strategies: to stay with their most recent successful response or to shift to the alternative response. Neurons in both areas encoded the stay and shift strategies after the cue appeared, but there were three major differences between the PFo and the PFdl: (1) many strategy-encoding cells in PFdl also encoded the response (left or right), but few, if any, PFo cells did so; (2) strategy selectivity appeared earlier in PFo than in PFdl; and (3) on error trials, PFo neurons encoded the correct strategy-the one that had been cued but not implemented-whereas in PFdl the strategy signals were weak or absent on error trials. These findings indicate that PFo and PFdl both contribute to behaviors guided by abstract response strategies, but do so differently, with PFo encoding a strategy and PFdl encoding a response based on a strategy.

Dynamic coding of goal-directed paths by orbital prefrontal cortex

Adapting successfully to new situations relies on integrating memory of similar circumstances with the outcomes of past actions. Here, we tested how reward history and recent memory influenced coding by orbital prefrontal cortex (OFC) neurons. Rats were trained to find food in plus maze tasks that required both the OFC and the hippocampus, and unit activity was recorded during stable performance, reversal learning, and strategy switching. OFC firing distinguished different rewarded paths, journeys from a start arm to a goal arm. Activity of individual cells and the population correlated with performance as rats learned newly rewarded outcomes. Activity was similar during reversal, an OFC-dependent task, and strategy switching, an OFC-independent task, suggesting that OFC associates information about paths and outcomes both when it is required for performance and when it is not. Path-selective OFC cells fired differently during overlapping journeys that led to different goals or from different starts, resembling journey-dependent coding by hippocampal neurons. Local field potentials (LFPs) recorded simultaneously in the OFC and the hippocampus oscillated coherently in the theta band (5-12 Hz) during stable performance. LFP coherence diminished when rats adapted to altered reward contingencies and followed different paths. Thus, OFC neurons appear to participate in a distributed network including the hippocampus that associates spatial paths, recent memory, and integrated reward history.

Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning

In many cases, learning is thought to be driven by differences between the value of rewards we expect and rewards we actually receive. Yet learning can also occur when the identity of the reward we receive is not as expected, even if its value remains unchanged. Learning from changes in reward identity implies access to an internal model of the environment, from which information about the identity of the expected reward can be derived. As a result, such learning is not easily accounted for by model-free reinforcement learning theories such as temporal difference reinforcement learning (TDRL), which predicate learning on changes in reward value, but not identity. Here, we used unblocking procedures to assess learning driven by value- versus identity-based prediction errors. Rats were trained to associate distinct visual cues with different food quantities and identities. These cues were subsequently presented in compound with novel auditory cues and the reward quantity or identity was selectively changed. Unblocking was assessed by presenting the auditory cues alone in a probe test. Consistent with neural implementations of TDRL models, we found that the ventral striatum was necessary for learning in response to changes in reward value. However, this area, along with orbitofrontal cortex, was also required for learning driven by changes in reward identity. This observation requires that existing models of TDRL in the ventral striatum be modified to include information about the specific features of expected outcomes derived from model-based representations, and that the role of orbitofrontal cortex in these models be clearly delineated.

Social and monetary reward learning engage overlapping neural substrates

Learning to make choices that yield rewarding outcomes requires the computation of three distinct signals: stimulus values that are used to guide choices at the time of decision making, experienced utility signals that are used to evaluate the outcomes of those decisions and prediction errors that are used to update the values assigned to stimuli during reward learning. Here we investigated whether monetary and social rewards involve overlapping neural substrates during these computations. Subjects engaged in two probabilistic reward learning tasks that were identical except that rewards were either social (pictures of smiling or angry people) or monetary (gaining or losing money). We found substantial overlap between the two types of rewards for all components of the learning process: a common area of ventromedial prefrontal cortex (vmPFC) correlated with stimulus value at the time of choice and another common area of vmPFC correlated with reward magnitude and common areas in the striatum correlated with prediction errors. Taken together, the findings support the hypothesis that shared anatomical substrates are involved in the computation of both monetary and social rewards.