Differential encoding of losses and gains in the human striatum

Sleep deprivation biases the neural mechanisms underlying economic preferences

A single night of sleep deprivation (SD) evoked a strategy shift during risky decision making such that healthy human volunteers moved from defending against losses to seeking increased gains. This change in economic preferences was correlated with the magnitude of an SD-driven increase in ventromedial prefrontal activation as well as by an SD-driven decrease in anterior insula activation during decision making. Analogous changes were observed during receipt of reward outcomes: elevated activation to gains in ventromedial prefrontal cortex and ventral striatum, but attenuated anterior insula activation following losses. Finally, the observed shift in economic preferences was not correlated with change in psychomotor vigilance. These results suggest that a night of total sleep deprivation affects the neural mechanisms underlying economic preferences independent of its effects on vigilant attention.

A role for the striatum in regret-related choice repetition

"Regret aversion" is proposed to explain a tendency to avoid future choices that have induced past regret. However, regret might also motivate us to repeat previous regret-related choices to make up for their previous selection, a behavior resembling "chasing" in the context of gambling. In the current experiment, we acquired fMRI brain data while participants placed monetary bets on repeated gambles. Behaviorally, participants showed a tendency to repeat previously regret-related choices (operationalized as those leading to an outcome worse than what might have been), an effect restricted to early runs of the task. At gamble outcome, we show a reduction in ventral striatal activity for regret-related relative to relief-related outcomes. Critically, this modulation was only seen when subjects were responsible for the bet choice. Activity in dorsal striatum was associated with an influence of previous regret on participants' subsequent choices, which is evident in increased activity when regret-related choices were repeated, relative to avoided, on the next trial. Our findings indicate that regret can lead to choice repetition as if seeking to make up for our mistakes and in so doing may lead to subsequent chasing behavior.

Effect of reinforcement history on hand choice in an unconstrained reaching task

Choosing which hand to use for an action is one of the most frequent decisions people make in everyday behavior. We developed a simple reaching task in which we vary the lateral position of a target and the participant is free to reach to it with either the right or left hand. While people exhibit a strong preference to use the hand ipsilateral to the target, there is a region of uncertainty within which hand choice varies across trials. We manipulated the reinforcement rates for the two hands, either by increasing the likelihood that a reach with the non-dominant hand would successfully intersect the target or decreasing the likelihood that a reach with the dominant hand would be successful. While participants had minimal awareness of these manipulations, we observed an increase in the use of the non-dominant hand for targets presented in the region of uncertainty. We modeled the shift in hand use using a Q-learning model of reinforcement learning. The results provided a good fit of the data and indicate that the effects of increasing and decreasing the rate of positive reinforcement are additive. These experiments emphasize the role of decision processes for effector selection, and may point to a novel approach for physical rehabilitation based on intrinsic reinforcement.

Social and monetary reward learning engage overlapping neural substrates

Learning to make choices that yield rewarding outcomes requires the computation of three distinct signals: stimulus values that are used to guide choices at the time of decision making, experienced utility signals that are used to evaluate the outcomes of those decisions and prediction errors that are used to update the values assigned to stimuli during reward learning. Here we investigated whether monetary and social rewards involve overlapping neural substrates during these computations. Subjects engaged in two probabilistic reward learning tasks that were identical except that rewards were either social (pictures of smiling or angry people) or monetary (gaining or losing money). We found substantial overlap between the two types of rewards for all components of the learning process: a common area of ventromedial prefrontal cortex (vmPFC) correlated with stimulus value at the time of choice and another common area of vmPFC correlated with reward magnitude and common areas in the striatum correlated with prediction errors. Taken together, the findings support the hypothesis that shared anatomical substrates are involved in the computation of both monetary and social rewards.

Category learning in the brain

The ability to group items and events into functional categories is a fundamental characteristic of sophisticated thought. It is subserved by plasticity in many neural systems, including neocortical regions (sensory, prefrontal, parietal, and motor cortex), the medial temporal lobe, the basal ganglia, and midbrain dopaminergic systems. These systems interact during category learning. Corticostriatal loops may mediate recursive, bootstrapping interactions between fast reward-gated plasticity in the basal ganglia and slow reward-shaded plasticity in the cortex. This can provide a balance between acquisition of details of experiences and generalization across them. Interactions between the corticostriatal loops can integrate perceptual, response, and feedback-related aspects of the task and mediate the shift from novice to skilled performance. The basal ganglia and medial temporal lobe interact competitively or cooperatively, depending on the demands of the learning task.

Aberrant Effective Connectivity in Schizophrenia Patients during Appetitive Conditioning

It has recently been suggested that schizophrenia involves dysfunction in brain connectivity at a neural level, and a dysfunction in reward processing at a behavioral level. The purpose of the present study was to link these two levels of analyses by examining effective connectivity patterns between brain regions mediating reward learning in patients with schizophrenia and healthy, age-matched controls. To this aim, we used functional magnetic resonance imaging and galvanic skin recordings (GSR) while patients and controls performed an appetitive conditioning experiment with visual cues as the conditioned (CS) stimuli, and monetary reward as the appetitive unconditioned stimulus (US). Based on explicit stimulus contingency ratings, conditioning occurred in both groups; however, based on implicit, physiological GSR measures, patients failed to show differences between CS+ and CS− conditions. Healthy controls exhibited increased blood-oxygen-level dependent (BOLD) activity across striatal, hippocampal, and prefrontal regions and increased effective connectivity from the ventral striatum to the orbitofrontal cortex (OFC BA 11) in the CS+ compared to the CS− condition. Compared to controls, patients showed increased BOLD activity across a similar network of brain regions, and increased effective connectivity from the striatum to hippocampus and prefrontal regions in the CS− compared to the CS+ condition. The findings of increased BOLD activity and effective connectivity in response to the CS− in patients with schizophrenia offer insight into the aberrant assignment of motivational salience to non-reinforced stimuli during conditioning that is thought to accompany schizophrenia.

Striatal responses to negative monetary outcomes differ between temperamentally inhibited and non-inhibited adolescents

The present study compared blood oxygen level dependent (BOLD) response in behaviorally inhibited and behaviorally non-inhibited adolescents to positive and negative feedback following their choice in a reward task. Previous data in these same subjects showed enhanced activation in striatal areas in behaviorally inhibited subjects to cues predicting gain or a loss. However, no analyses had examined responses following actual gains or losses. Relative to non-inhibited subjects, behaviorally inhibited subjects in the current study showed enhanced caudate response to negative but not positive feedback, indicating that striatal sensitivity to feedback may be specific to aversive information. In addition, compared to non-inhibited subjects, behaviorally inhibited subjects exhibited reduced differentiation between positive and negative feedback in ventromedial prefrontal cortex (vmPFC). This suggests a perturbed ability to encode reward value.

From bad to worse: striatal coding of the relative value of painful decisions

The majority of decision-related research has focused on how the brain computes decisions over outcomes that are positive in expectation. However, much less is known about how the brain integrates information when all possible outcomes in a decision are negative. To study decision-making over negative outcomes, we used fMRI along with a task in which participants had to accept or reject 50/50 lotteries that could result in more or fewer electric shocks compared to a reference amount. We hypothesized that behaviorally, participants would treat fewer shocks from the reference amount as a gain, and more shocks from the reference amount as a loss. Furthermore, we hypothesized that this would be reflected by a greater BOLD response to the prospect of fewer shocks in regions typically associated with gain, including the ventral striatum and orbitofrontal cortex. The behavioral data suggest that participants in our study viewed all outcomes as losses, despite our attempt to induce a status quo. We find that the ventral striatum showed an increase in BOLD response to better potential gambles (i.e., fewer expected shocks). This lends evidence to the idea that the ventral striatum is not solely responsible for reward processing but that it might also signal the relative value of an expected outcome or action, regardless of whether the outcome is entirely appetitive or aversive. We also find a greater response to worse gambles in regions previously associated with aversive valuation, suggesting an opposing but simultaneous valuation signal to that conveyed by the striatum.

Internal-time temporal difference model for neural value-based decision making

The temporal difference (TD) learning framework is a major paradigm for understanding value-based decision making and related neural activities (e.g., dopamine activity). The representation of time in neural processes modeled by a TD framework, however, is poorly understood. To address this issue, we propose a TD formulation that separates the time of the operator (neural valuation processes), which we refer to as internal time, from the time of the observer (experiment), which we refer to as conventional time. We provide the formulation and theoretical characteristics of this TD model based on internal time, called internal-time TD, and explore the possible consequences of the use of this model in neural value-based decision making. Due to the separation of the two times, internal-time TD computations, such as TD error, are expressed differently, depending on both the time frame and time unit. We examine this operator-observer problem in relation to the time representation used in previous TD models. An internal time TD value function exhibits the co-appearance of exponential and hyperbolic discounting at different delays in intertemporal choice tasks. We further examine the effects of internal time noise on TD error, the dynamic construction of internal time, and the modulation of internal time with the internal time hypothesis of serotonin function. We also relate the internal TD formulation to research on interval timing and subjective time.

Subjective and model-estimated reward prediction: association with the feedback-related negativity (FRN) and reward prediction error in a reinforcement learning task

In this study, we examined whether the feedback-related negativity (FRN) is associated with both subjective and objective (model-estimated) reward prediction errors (RPE) per trial in a reinforcement learning task in healthy adults (n=25). The level of RPE was assessed by 1) subjective ratings per trial and by 2) a computational model of reinforcement learning. As results, model-estimated RPE was highly correlated with subjective RPE (r=.82), and the grand-averaged ERP waves based on the trials with high and low model-estimated RPE showed the significant difference only in the time period of the FRN component (p<.05). Regardless of the time course of learning, FRN was associated with both subjective and model-estimated RPEs within subject (r=.47, p<.001; r=.40, p<.05) and between subjects (r=.33, p<.05; r=.41, p<.005) only in the Learnable condition where the internal reward prediction varied enough with a behavior-reward contingency.

Appetitive and aversive goal values are encoded in the medial orbitofrontal cortex at the time of decision making

An essential feature of choice is the assignment of goal values (GVs) to the different options under consideration at the time of decision making. This computation is done when choosing among appetitive and aversive items. Several groups have studied the location of GV computations for appetitive stimuli, but the problem of valuation in aversive contexts at the time of decision making has been ignored. Thus, although dissociations between appetitive and aversive components of value signals have been shown in other domains such as anticipatory and outcome values, it is not known whether appetitive and aversive GVs are computed in similar brain regions or in separate ones. We investigated this question using two different functional magnetic resonance imaging studies while human subjects placed real bids in an economic auction for the right to eat/avoid eating liked/disliked foods. We found that activity in a common area of the medial orbitofrontal cortex and the dorsolateral prefrontal cortex correlated with both appetitive and aversive GVs. These findings suggest that these regions might form part of a common network.

Experience and choice shape expected aversive outcomes

The value assigned to aversive events is susceptible to contextual influences. Here, we asked whether a change in the valuation of negative events is reflected in an altered neuronal representation of their expected aversive outcome. We show that experiencing an aversive event in the past, and choosing to experience it in the future, reduces its aversive value. This psychological change is mirrored in an altered neural representation of aversive value in the caudate nucleus and anterior cingulate cortex. Our findings indicate that subcortical regions known to track expected value such as the caudate nucleus, together with anterior cingulate cortical regions implicated in emotional modulation, mediate a revaluation in expectancies of aversive states. The results provide a striking example of a contextual sensitivity in how the brain ascribes value to events, in a manner that may foster resilience in the face of adversity.

Diffusion tensor imaging and decision making in cocaine dependence

BACKGROUND

Chronic stimulant abuse is associated with both impairment in decision making and structural abnormalities in brain gray and white matter. Recent data suggest these structural abnormalities may be related to functional impairment in important behavioral processes.

METHODOLOGY/PRINCIPAL FINDINGS

In 15 cocaine-dependent and 18 control subjects, we examined relationships between decision-making performance on the Iowa Gambling Task (IGT) and white matter integrity as measured by diffusion tensor imaging (DTI). Whole brain voxelwise analyses showed that, relative to controls, the cocaine group had lower fractional anisotropy (FA) and higher mean of the second and third eigenvalues (lambda perpendicular) in frontal and parietal white matter regions and the corpus callosum. Cocaine subjects showed worse performance on the IGT, notably over the last 40 trials. Importantly, FA and lambda perpendicular values in these regions showed a significant relationship with IGT performance on the last 40 trials.

CONCLUSIONS

Compromised white matter integrity in cocaine dependence may be related to functional impairments in decision making.

Dissociable responses to punishment in distinct striatal regions during reversal learning

Adaptive behavior depends on the ability to flexibly alter our choices in response to changes in reward and punishment contingencies. One brain region frequently implicated in such behavior is the striatum. However, this region is functionally diverse and there are a number of apparent inconsistencies across previous studies. For instance, how can significant BOLD responses in the ventral striatum during punishment-based reversal learning be reconciled with the frequently demonstrated role of the ventral striatum in reward processing? Here we attempt to address this question by separately examining BOLD responses during reversal learning driven by reward and during reversal learning driven by punishment. We demonstrate simultaneous valence-specific and valence-nonspecific signals in the striatum, with the posterior dorsal striatum responding only to unexpected reward, and the anterior ventral striatum responding to both unexpected punishment as well as unexpected reward. These data help to reconcile conflicting findings from previous studies by showing that distinct regions of the striatum exhibit dissociable responses to punishment during reversal learning.

Reputation for reciprocity engages the brain reward center

Brain reward circuitry, including ventral striatum and orbitofrontal cortex, has been independently implicated in preferences for fair and cooperative outcomes as well as learning of reputations. Using functional MRI (fMRI) and a "trust game" task involving iterative exchanges with fictive partners who acquire different reputations for reciprocity, we measured brain responses in 36 healthy adults when positive actions (entrust investment to partners) yield positive returns (reciprocity) and how these brain responses are modulated by partner reputation for repayment. Here we show that positive reciprocity robustly engages the ventral striatum and orbitofrontal cortex. Moreover, this signal of reciprocity in the ventral striatum appears selectively in response to partners who have consistently returned the investment (e.g., a reputation for reciprocity) and is absent for partners who lack a reputation for reciprocity. These findings elucidate a fundamental brain mechanism, via reward-related neural substrates, by which human cooperative relationships are initiated and sustained.

Overlapping neural systems mediating extinction, reversal and regulation of fear

Learned fear is a process allowing quick detection of associations between cues in the environment and prediction of imminent threat. Adaptive function in a changing environment, however, requires organisms to quickly update this learning and have the ability to hinder fear responses when predictions are no longer correct. Here we focus on three strategies that can modify conditioned fear, namely extinction, reversal and regulation of fear, and review their underlying neural mechanisms. By directly comparing neuroimaging data from three separate studies that employ each strategy, we highlight overlapping brain structures that comprise a general circuitry in the human brain. This circuitry potentially enables the flexible control of fear, regardless of the particular task demands.

Decision neuroscience: neuroeconomics

Few aspects of human cognition are more personal than the choices we make. Our decisions-from the mundane to the impossibly complex-continually shape the courses of our lives. In recent years, researchers have applied the tools of neuroscience to understand the mechanisms that underlie decision making, as part of the new discipline of decision neuroscience. A primary goal of this emerging field has been to identify the processes that underlie specific decision variables, including the value of rewards, the uncertainty associated with particular outcomes, and the consequences of social interactions. Recent work suggests potential neural substrates that integrate these variables, potentially reflecting a common neural currency for value, to facilitate value comparisons. Despite the successes of decision neuroscience research for elucidating brain mechanisms, significant challenges remain. These include building new conceptual frameworks for decision making, integrating research findings across disparate techniques and species, and extending results from neuroscience to shape economic theory. To overcome these challenges, future research will likely focus on interpersonal variability in decision making, with the eventual goal of creating biologically plausible models for individual choice. WIREs Cogn Sci 2010 1 854-871 This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition.

Choosing to make an effort: the role of striatum in signaling physical effort of a chosen action

The possibility that we will have to invest effort influences our future choice behavior. Indeed deciding whether an action is actually worth taking is a key element in the expression of human apathy or inertia. There is a well developed literature on brain activity related to the anticipation of effort, but how effort affects actual choice is less well understood. Furthermore, prior work is largely restricted to mental as opposed to physical effort or has confounded temporal with effortful costs. Here we investigated choice behavior and brain activity, using functional magnetic resonance imaging, in a study where healthy participants are required to make decisions between effortful gripping, where the factors of force (high and low) and reward (high and low) were varied, and a choice of merely holding a grip device for minimal monetary reward. Behaviorally, we show that force level influences the likelihood of choosing an effortful grip. We observed greater activity in the putamen when participants opt to grip an option with low effort compared with when they opt to grip an option with high effort. The results suggest that, over and above a nonspecific role in movement anticipation and salience, the putamen plays a crucial role in computations for choice that involves effort costs.

Stop looking angry and smile, please: start and stop of the very same facial expression differentially activate threat- and reward-related brain networks

Static pictures of emotional facial expressions have been found to activate brain structures involved in the processing of emotional stimuli. However, in everyday live, emotional expressions are changing rapidly, and the processing of the onset vs the offset of the very same emotional expression might rely on different brain networks, presumably leading to different behavioral and physiological reactions (e.g. approach or avoidance). Using functional magnetic resonance imaging, this was examined by presenting video clips depicting onsets and offsets of happy and angry facial expressions. Subjective valence and threat ratings clearly depended on the direction of change. Blood oxygen level dependent responses indicate both reward- and threat-related activations for the offset of angry expressions. Comparing onsets and offsets, angry offsets were associated with stronger ventral striatum activation than angry onsets. Additionally, the offset of happy and the onset of angry expressions showed strong common activity in the lateral orbitofrontal cortex bilaterally, the left amygdala and the left insula, whereas the onset of happy and the offset of angry expressions induced significant activation in the left dorsal striatum. In sum, the results confirm different activity in motivation-related brain areas in response to the onset and offset of the same emotional expression and highlight the importance of temporal characteristics of facial expressions for social communication.

Dopaminergic modulation of appetitive and aversive predictive learning

Temporal contiguity between two stimuli is insufficient for the establishment of a predictive relation between those stimuli. Rather, learning about predictive relations is influenced by a prediction error mechanism: the discrepancy between actual and expected outcomes. Although the neural substrates of contiguous stimuli presentation have been the focus of research for decades, relatively little empirical evidence exists with regard to the neural mechanisms of prediction error. Recent work has implicated the neurotransmitter dopamine in regulation of predictive learning. If dopamine modulates prediction error then it should do so despite the nature (appetitive or aversive) of the biological stimuli that serve to drive learning. The exact role of dopamine in appetitive and aversive predictive learning, however, remains the focus of continuous debate. This review focuses on the behavioural, neuropharmacological and electrophysiological evidence implicating dopamine in prediction error in appetitive and aversive predictive learning. In addition, recent work in the area of fear conditioning implicating other neurochemical substrates, namely opioids, in the process of prediction error is discussed. Finally, some predictions are made with regard to the neurochemical circuitry involved in modulating learning and behaviour based on prediction error.

Executive function and error detection: The effect of motivation on cingulate and ventral striatum activity

Reacting appropriately to errors during task performance is fundamental to successful negotiation of our environment. This is especially true when errors will result in a significant penalty for the person performing a given task, be they financial or otherwise. Error responses and monitoring states were manipulated in a GO/NOGO task by introducing a financial punishment for errors. This study employed a mixed block design alternating between punishment and no punishment (neutral) conditions, enabling an assessment of tonic changes associated with cognitive control as well as trial-specific effects. Behavioural results revealed slower responses and fewer commission errors in the punishment condition. The dorsal anterior cingulate cortex (ACC) had equal trial-specific activity for errors in the neutral and punishment conditions but had greater tonic activity throughout the punishment condition. A region of interest analysis revealed different activation patterns between the dorsal and the rostral parts of the ACC with the rostral ACC having only trial-specific activity for errors in the punishment condition, an activity profile similar to one observed in the nucleus accumbens. This study suggests that there is a motivational influence on cognitive processes in the ACC and nucleus accumbens and hints at a dissociation between tonic proactive activity and phasic reactive error-related activity.

Dissociating valuation and saliency signals during decision-making

There is a growing consensus that the brain computes value and saliency-like signals at the time of decision-making. Value signals are essential for making choices. Saliency signals are related to motivation, attention, and arousal. Unfortunately, an unequivocal characterization of the areas involved in these 2 distinct sets of processes is made difficult by the fact that, in most experiments, both types of signals are highly correlated. We dissociated value and saliency signals using a novel human functional magnetic resonance imaging decision-making task. Activity in the medial orbitofrontal, rostral anterior cingulate, and posterior cingulate cortices was modulated by value but not saliency. The opposite was true for dorsal anterior cingulate, supplementary motor area, insula, and the precentral and fusiform gyri. Only the ventral striatum and the cuneus were modulated by both value and saliency.

Astrocyte-mediated hepatocyte growth factor/scatter factor supplementation restores GABAergic interneurons and corrects reversal learning deficits in mice

Many psychiatric and neurological disorders present persistent neuroanatomical abnormalities in multiple brain regions that may reflect a common origin for a developmental disturbance. In mammals, many of the local GABAergic inhibitory interneurons arise from a single subcortical source. Perturbations in the ontogeny of the GABAergic interneurons may be reflected in the adult by interneuron deficits in both frontal cerebral cortical and striatal regions. Disrupted GABAergic circuitry has been reported in patients with schizophrenia and frontal lobe epilepsy and may contribute to their associated impairments in behavioral flexibility. The present study demonstrates that one type of behavioral flexibility, reversal learning, is dependent upon proper numbers of GABAergic interneurons. Mice with abnormal interneuron ontogeny have reduced numbers of parvalbumin-expressing GABAergic local interneurons in the orbitofrontal cortical and striatal regions and impaired reversal leaning. Using a genetic approach, both the anatomical and functional deficiencies are restored with exogenous postnatal growth factor supplementation. These results show that GABAergic local circuitry is critical for modulating behavioral flexibility and that birth defects can be corrected by replenishing crucial growth factors.

Cerebral maturation in adolescence: behavioral vulnerability

Recent neurodevelopmental research has been focusing on the transition period of adolescence into adulthood. This growing interest was spurred by the long-standing realization of the high cost of this transition period in terms of morbidity and mortality, and the emergence of research tools that permit direct examination of brain function in humans. The cost of reaching adulthood is understood as resulting from the typical behavioral and environmental changes that accompany adolescence [4]. The present review describes how the current research helps formulate neurobiological models that can be used to guide future work. One example of such a model, the triadic neural systems model [8], will be examined in more detail. This review will proceed in three stages. First, we will show how neural development results from the confluence of maturational changes that are quantitatively and qualitatively heterogeneous across brain regions, neurochemical and molecular systems. This normative developmental heterogeneity is translated into typical adolescent behavioral patterns, including risk-taking, novelty-seeking, emotional intensity and lability, and peer-group social primacy [5, 9]. Second, based on the notion that motivated behavior can be operationally decomposed into the three core modules of approach, avoidance and control, any alteration in the balance of these three core entities can affect behavior in unique ways. This formulation will serve as the foundation of the neural systems model framework proposed in this review. Third, functional neuroimaging is being used to examine how neural systems underlie this balance within the neural systems model. We will provide a summary of the state of research in this area. Finally, we will show how this research is important for understanding not only normal development, but also the psychopathology. The role of genetic or environmental factors will not be addressed here, despite their critical roles in determining and modulating the balance among the nodes of the triadic model.

The current status of research on the structure of evaluative space

The structure of evaluative space shapes emotional life and must be capable of an astonishing range of emotional experience and expression. In this article, we outline the current status of research on the Evaluative Space Model (ESM; Cacioppo et al., 1997, 1999), which proposes that behavioral predispositions (e.g., approach, withdrawal) are the ultimate output of the affect system, which is defined by operating characteristics that differ for positivity and negativity, and across levels of the nervous system. First, we summarize the tenets of the model, as well as counterarguments raised by other theorists. To address these counterarguments, we discuss the postulates of affective oscillation and calibration, two features of the affect system proposed to underlie the durability and adaptability of affect. Finally, we consider the implications of disorder in the structure of evaluative space for the comprehension and treatment of depression and anxiety.

Action selection: a race model for selected and non-selected actions distinguishes the contribution of premotor and prefrontal areas

Race models have been used to explain perceptual, motor and oculomotor decisions. Here we developed a race model to explain how human subjects select actions when there are no overt rewards and no external cues to specify which action to make. Critically, we were able to estimate the cumulative activity of neuronal decision-units for selected and non-selected actions. We used functional magnetic resonance imaging (fMRI) to test for regional brain activity that correlated with the predictions of this race model. Activity in the pre-SMA, cingulate motor and premotor areas correlated with prospective selection between responses according to the race model. Activity in the lateral prefrontal cortex did not correlate with the race model, even though this area was active during action selection. This activity related to the degree to which individuals switched between alternative actions. Crucially, a follow-up experiment showed that it was not present on the first trial. Taken together, these results suggest that the lateral prefrontal cortex is not the source for the generation of action. It is more likely that it is involved in switching to alternatives or monitoring previous actions. Thus, our experiment shows the power of the race model in distinguishing the contribution of different areas in the selection of action.

Probing compulsive and impulsive behaviors, from animal models to endophenotypes: a narrative review

Failures in cortical control of fronto-striatal neural circuits may underpin impulsive and compulsive acts. In this narrative review, we explore these behaviors from the perspective of neural processes and consider how these behaviors and neural processes contribute to mental disorders such as obsessive-compulsive disorder (OCD), obsessive-compulsive personality disorder, and impulse-control disorders such as trichotillomania and pathological gambling. We present findings from a broad range of data, comprising translational and human endophenotypes research and clinical treatment trials, focussing on the parallel, functionally segregated, cortico-striatal neural projections, from orbitofrontal cortex (OFC) to medial striatum (caudate nucleus), proposed to drive compulsive activity, and from the anterior cingulate/ventromedial prefrontal cortex to the ventral striatum (nucleus accumbens shell), proposed to drive impulsive activity, and the interaction between them. We suggest that impulsivity and compulsivity each seem to be multidimensional. Impulsive or compulsive behaviors are mediated by overlapping as well as distinct neural substrates. Trichotillomania may stand apart as a disorder of motor-impulse control, whereas pathological gambling involves abnormal ventral reward circuitry that identifies it more closely with substance addiction. OCD shows motor impulsivity and compulsivity, probably mediated through disruption of OFC-caudate circuitry, as well as other frontal, cingulate, and parietal connections. Serotonin and dopamine interact across these circuits to modulate aspects of both impulsive and compulsive responding and as yet unidentified brain-based systems may also have important functions. Targeted application of neurocognitive tasks, receptor-specific neurochemical probes, and brain systems neuroimaging techniques have potential for future research in this field.

Dopamine precursor depletion improves punishment prediction during reversal learning in healthy females but not males

INTRODUCTION

The neurotransmitter dopamine has frequently been implicated in reward processing but is also, increasingly, implicated in punishment processing. We have previously shown that both patients with Parkinson's disease and healthy individuals with low dopamine (DA) synthesis are better at reversal learning based on punishment than reward. Here, we extend these prior findings by examining the effects of artificially reducing DA synthesis in healthy individuals performing this previously employed task.

METHODS

In a double-blind, placebo-controlled crossover design, we applied the acute tyrosine and phenylalanine depletion (ATPD) procedure to reduce global DA synthesis in 15 female and 14 male subjects. Each subject performed the reward- and punishment-based reversal-learning paradigm.

RESULTS

There was a significant three-way interaction between ATPD, the valence of the outcome signalling reversal and the gender of the participants. Examination of punishment and reward-based reversals separately revealed that this was driven by a significant improvement in punishment processing in female but not male subjects following DA depletion.

CONCLUSIONS

Reducing DA synthesis in healthy individuals shifted sensitivity of performance from reward to punishment processing. Gender differences in DA synthesis might underlie the selectivity of this effect to female subjects. Such gender biases may go some way towards explaining the gender biases in certain psychiatric disorders such as depression and Parkinson's disease.

Dopamine-induced changes in neural network patterns supporting aversive conditioning

The aim of the present paper is to assess the effects of altered dopamine (DA) transmission on the functional connectivity among brain regions mediating aversive conditioning in humans. To this aim, we analyzed a previous published data set from a double-blind design combined with functional magnetic resonance imaging (fMRI) recordings in which healthy volunteers were randomly assigned to one of three drug groups: amphetamine (an indirect DA agonist), haloperidol (DA D2 receptor antagonist), and placebo. Participants were exposed to an aversive classical conditioning paradigm using cutaneous electrical stimulation as the unconditioned stimulus (US), and visual cues as the conditioned stimuli (CS) where one colour (CS+) was followed by the US in 33% of the trials and another colour (CS-) had no consequences. All participants reported awareness of stimulus contingencies. Group analysis of fMRI data revealed that the left ventral striatum (VS) and amygdala activated in response to the CS+ in all the three groups. Because of their activation patterns and documented involvement in aversive conditioning, both regions were used as seeds in the functional connectivity analysis. To constrain the functional networks obtained to relate to the conditioned response, we also correlated seed activity with the Galvanic Skin Response (GSR). In the placebo group, the right ventral tegmental area/substantia nigra (VTA/SN), bilateral caudate, right parahippocampal gyrus, left inferior parietal lobule (IPL), bilateral postcentral gyrus, bilateral middle frontal (BA 46), orbitofrontal, and ventromedial prefrontal cortices (PFC, BA 10/11) correlated with the VS and amygdala seeds in response to the CS+ compared to the CS-. Enhancing dopamine transmission via amphetamine was associated with reduced task differences and significant functional connectivity for both CS+ and CS- conditions between the left VS seed and regions modulated by DA, such as the left VTA/SN, right caudate, left amygdala, left middle frontal gyrus (BA 46), and bilateral ventromedial PFC (BA 10). Blocking dopamine transmission via haloperidol was associated with significant functional connectivity across an alternate network of regions including the left amygdala seed and the right insula, the left ACC (BA 24/32), bilateral IPL (BA 40), precuneus (BA 7), post-central gyrus, middle frontal gyrus (BA 46), and supplementary motor area (SMA, BA 6) to the CS+ versus the CS-. These data provide insight into the distinct effects of DA agents on the functional connectivity between striatal, limbic, and prefrontal areas.

Available alternative incentives modulate anticipatory nucleus accumbens activation

A reward or punishment can seem better or worse depending on what else might have happened. Little is known, however, about how neural representations of an anticipated incentive might be influenced by the available alternatives. We used event-related FMRI to investigate the activation in the nucleus accumbens (NAcc), while we varied the available alternative incentives in a monetary incentive delay task. Some task blocks included only uncertain gains and losses; others included the same uncertain gains and losses intermixed with certain gains and losses. The availability of certain gains and losses increased NAcc activation for uncertain losses and decreased the difference between uncertain gains and losses. We suggest that this pattern of activation can result from reference point changes across blocks, and that the worst available loss may serve as an important anchor for NAcc activation. These findings imply that NAcc activation represents anticipated incentive value relative to the current context of available alternative gains and losses.

Avoiding negative outcomes: tracking the mechanisms of avoidance learning in humans during fear conditioning

Previous research across species has shown that the amygdala is critical for learning about aversive outcomes, while the striatum is involved in reward-related processing. Less is known, however, about the role of the amygdala and the striatum in learning how to exert control over emotions and avoid negative outcomes. One potential mechanism for active avoidance of stressful situations is postulated to involve amygdala–striatal interactions. The goal of this study was to investigate the physiological and neural correlates underlying avoidance learning in humans. Specifically, we used a classical conditioning paradigm where three different conditioned stimuli (CS) were presented. One stimulus predicted the delivery of a shock upon stimulus offset (CS+), while another predicted no negative consequences (CS−). A third conditioned cue also predicted delivery of a shock, but participants were instructed that upon seeing this stimulus, they could avoid the shock if they chose the correct action (AV+). After successful learning, participants could then easily terminate the shock during subsequent stimulus presentations (AV−). Physiological responses (as measured by skin conductance responses) confirmed a main effect of conditioning, particularly showing higher arousal responses during pre (AV+) compared to post (AV−) learning of an avoidance response. Consistent with animal models, amygdala–striatal interactions were observed to underlie the acquisition of an avoidance response. These results support a mechanism of active coping with conditioned fear that allows for the control over emotional responses such as fears that can become maladaptive and influence our decision-making.

The subjective value of delayed and probabilistic outcomes: Outcome size matters for gains but not for losses

The subjective value of a reward (gain) is related to factors such as its size, the delay to its receipt and the probability of its receipt. We examined whether the subjective value of losses was similarly affected by these factors in 128 adults. Participants chose between immediate/certain gains or losses and larger delayed/probabilistic gains or losses. Rewards of $100 were devalued as a function of their delay ("discounted") relatively less than $10 gains while probabilistic $100 rewards were discounted relatively more than $10 rewards. However, there was no effect of outcome size on discounting of delayed or probabilistic losses. For delayed outcomes of each size, the degree to which gains were discounted was positively correlated with the degree to which losses were discounted, whereas for probabilistic outcomes, no such correlation was observed. These results suggest that the processes underlying the subjective valuation of losses are different from those underlying the subjective valuation of gains.

A neurochemical approach to valuation sensitivity over gains and losses

Prospect theory proposes the hypothesis that people have diminishing sensitivity in valuing increases in the size of monetary outcomes, for both gains and losses. For decision-making under risk, this implies a tendency to be risk-tolerant over losses while being generally risk averse over gains. We offer a neurochemistry-based model of the diminishing valuation sensitivity hypothesis. Specifically, we propose that dopamine tone modulates the sensitivity towards valuation of gains while serotonin tone modulates the sensitivity towards valuation of losses. Consequently, higher dopamine tone would yield a more concave valuation function over gains while higher serotonin tone would yield a more convex valuation function over losses. Using a neurogenetics strategy to test our neurochemical model, we find that subjects with the 9-repeat allele of DAT1 (lower DA tone) are more risk-tolerant over gains than subjects with the 10-repeat allele, and that subjects with the 10-repeat allele of STin2 (higher 5HT tone) are more risk-tolerant over losses than subjects with the 12-repeat allele. Overall, our results support the implications of our model and provide the first neurogenetics evidence that risk attitudes are partially hard-wired in differentiating between gain- and loss-oriented risks.

Perceived social isolation and cognition

Social species, from Drosophila melanogaster to Homo sapiens, fare poorly when isolated. Homo sapiens, an irrepressibly meaning-making species, are, in normal circumstances, dramatically affected by perceived social isolation. Research indicates that perceived social isolation (i.e. loneliness) is a risk factor for, and may contribute to, poorer overall cognitive performance, faster cognitive decline, poorer executive functioning, increased negativity and depressive cognition, heightened sensitivity to social threats, a confirmatory bias in social cognition that is self-protective and paradoxically self-defeating, heightened anthropomorphism and contagion that threatens social cohesion. These differences in attention and cognition impact on emotions, decisions, behaviors and interpersonal interactions that can contribute to the association between loneliness and cognitive decline and between loneliness and morbidity more generally.

Activation in the VTA and nucleus accumbens increases in anticipation of both gains and losses

To represent value for learning and decision making, the brain must encode information about both the motivational relevance and affective valence of anticipated outcomes. The nucleus accumbens (NAcc) and ventral tegmental area (VTA) are thought to play key roles in representing these and other aspects of valuation. Here, we manipulated the valence (i.e., monetary gain or loss) and personal relevance (i.e., self-directed or charity-directed) of anticipated outcomes within a variant of the monetary incentive delay task. We scanned young-adult participants using functional magnetic resonance imaging (fMRI), utilizing imaging parameters targeted for the NAcc and VTA. For both self-directed and charity-directed trials, activation in the NAcc and VTA increased to anticipated gains, as predicted by prior work, but also increased to anticipated losses. Moreover, the magnitude of responses in both regions was positively correlated for gains and losses, across participants, while an independent reward-sensitivity covariate predicted the relative difference between and gain- and loss-related activation on self-directed trials. These results are inconsistent with the interpretation that these regions reflect anticipation of only positive-valence events. Instead, they indicate that anticipatory activation in reward-related regions largely reflects the motivational relevance of an upcoming event.

Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study

Animal studies have found that the phasic activity of dopamine neurons during reward-related learning resembles a "prediction error" (PE) signal derived from a class of computational models called reinforcement learning (RL). An apparently similar signal can be measured using fMRI in the human striatum, a primary dopaminergic target. However, the fMRI signal does not measure dopamine per se, and therefore further evidence is needed to determine if these signals are related to each other. Parkinson's disease (PD) involves the neurodegeneration of the dopamine system and is accompanied by deficits in reward-related decision-making tasks. In the current study we used a computational RL model to assess striatal error signals in PD patients performing an RL task during fMRI scanning. Results show that error signals were preserved in ventral striatum of PD patients, but impaired in dorsolateral striatum, relative to healthy controls, a pattern reflecting the known selective anatomical degeneration of dopamine nuclei in PD. These findings support the notion that PE signals measured in the human striatum by the BOLD signal may reflect phasic DA activity. These results also provide evidence for a deficiency in PE signaling in the dorsolateral striatum of PD patients that may offer an explanation for their deficits observed in other reward learning tasks.

The Wick in the Candle of Learning

Curiosity has been described as a desire for learning and knowledge, but its underlying mechanisms are not well understood. We scanned subjects with functional magnetic resonance imaging while they read trivia questions. The level of curiosity when reading questions was correlated with activity in caudate regions previously suggested to be involved in anticipated reward. This finding led to a behavioral study, which showed that subjects spent more scarce resources (either limited tokens or waiting time) to find out answers when they were more curious. The functional imaging also showed that curiosity increased activity in memory areas when subjects guessed incorrectly, which suggests that curiosity may enhance memory for surprising new information. This prediction about memory enhancement was confirmed in a behavioral study: Higher curiosity in an initial session was correlated with better recall of surprising answers 1 to 2 weeks later.

The influence of context valence in the neural coding of monetary outcomes

The emotional significance of objects and events depends on the context in which they occur. Using functional magnetic resonance imaging, we examined the modulation of neural responses to monetary outcomes while subjects performed a decision-making task in a positive and a negative economic context. Neural responses indicated a relative regional specialization in the neural coding of outcome valence and followed three distinct patterns. The nucleus accumbens (NAc) and orbital frontal cortex (OFC) appeared to code the most extreme outcome in each context, with a potentiated response for favorable outcomes by a positive context. The amygdala and insula appeared to also code highly salient outcomes, but showed a potentiated response to unfavorable outcomes occurring in a negative context. The medial prefrontal cortex (medPFC), on the other hand, only coded favorable responses occurring in a positive context. Moreover, the medPFC showed large inter-individual variability when responding to outcomes in a negative context, suggesting that its role in a negative context may depend on a number of individual factors. The results of this work provide evidence of complex valence-based regional dissociations that are influenced by contextual factors.

Separate neural mechanisms underlie choices and strategic preferences in risky decision making

Adaptive decision making in real-world contexts often relies on strategic simplifications of decision problems. Yet, the neural mechanisms that shape these strategies and their implementation remain largely unknown. Using an economic decision-making task, we dissociate brain regions that predict specific choices from those predicting an individual's preferred strategy. Choices that maximized gains or minimized losses were predicted by functional magnetic resonance imaging activation in ventromedial prefrontal cortex or anterior insula, respectively. However, choices that followed a simplifying strategy (i.e., attending to overall probability of winning) were associated with activation in parietal and lateral prefrontal cortices. Dorsomedial prefrontal cortex, through differential functional connectivity with parietal and insular cortex, predicted individual variability in strategic preferences. Finally, we demonstrate that robust decision strategies follow from neural sensitivity to rewards. We conclude that decision making reflects more than compensatory interaction of choice-related regions; in addition, specific brain systems potentiate choices depending on strategies, traits, and context.

The cerebellum is involved in reward-based reversal learning

The cerebellum has recently been discussed in terms of a possible involvement in reward-based associative learning. To clarify the cerebellar contribution, eight patients with focal vascular lesions of the cerebellum and a group of 24 healthy subjects matched for age and IQ were compared on a range of different probabilistic outcome-based associative learning tasks, assessing acquisition, reversal, cognitive transfer, and generalization as well as the effect of reward magnitude. Cerebellar patients showed intact acquisition of stimulus contingencies, while reward-based reversal learning was significantly impaired. In addition, the patients showed slower acquisition of new stimulus contingencies in a second reward-based learning task, which might reflect reduced carry-over effects. Reward magnitude affected learning only during initial acquisition, with better learning on trials with high rewards in patients and control subjects. Overall, the findings suggest that the cerebellum is implicated in reversal learning as a dissociable component of reward-based learning.

The neurobiology of reference-dependent value computation

A key focus of current research in neuroeconomics concerns how the human brain computes value. Although, value has generally been viewed as an absolute measure (e.g., expected value, reward magnitude), much evidence suggests that value is more often computed with respect to a changing reference point, rather than in isolation. Here, we present the results of a study aimed to dissociate brain regions involved in reference-independent (i.e., "absolute") value computations, from those involved in value computations relative to a reference point. During functional magnetic resonance imaging, subjects acted as buyers and sellers during a market exchange of lottery tickets. At a behavioral level, we demonstrate that subjects systematically accorded a higher value to objects they owned relative to those they did not, an effect that results from a shift in reference point (i.e., status quo bias or endowment effect). Our results show that activity in orbitofrontal cortex and dorsal striatum track parameters such as the expected value of lottery tickets indicating the computation of reference-independent value. In contrast, activity in ventral striatum indexed the degree to which stated prices, at a within-subjects and between-subjects level, were distorted with respect to a reference point. The findings speak to the neurobiological underpinnings of reference dependency during real market value computations.

How choice reveals and shapes expected hedonic outcome

Humans tend to modify their attitudes to align with past action. For example, after choosing between similarly valued alternatives, people rate the selected option as better than they originally did, and the rejected option as worse. However, it is unknown whether these modifications in evaluation reflect an underlying change in the physiological representation of a stimulus' expected hedonic value and our emotional response to it. Here, we addressed this question by combining participants' estimations of the pleasure they will derive from future events, with brain imaging data recorded while they imagined those events, both before, and after, choosing between them. Participants rated the selected alternatives as better after the decision stage relative to before, whereas discarded alternatives were valued less. Our functional magnetic resonance imaging findings reveal that postchoice changes in preference are tracked in caudate nucleus activity. Specifically, the difference in blood oxygenation level-dependent (BOLD) signal associated with the selected and rejected stimuli was enhanced after a decision was taken, reflecting the choice that had just been made. This finding suggests that the physiological representation of a stimulus' expected hedonic value is altered by a commitment to it. Furthermore, before any revaluation induced by the decision process, our data show that BOLD signal in this same region reflects the choices we are likely to make at a later time.

Behavioral and neural changes after gains and losses of conditioned reinforcers

Human behaviors can be more powerfully influenced by conditioned reinforcers, such as money, than by primary reinforcers. Moreover, people often change their behaviors to avoid monetary losses. However, the effect of removing conditioned reinforcers on choices has not been explored in animals, and the neural mechanisms mediating the behavioral effects of gains and losses are not well understood. To investigate the behavioral and neural effects of gaining and losing a conditioned reinforcer, we trained rhesus monkeys for a matching pennies task in which the positive and negative values of its payoff matrix were realized by the delivery and removal of a conditioned reinforcer. Consistent with the findings previously obtained with non-negative payoffs and primary rewards, the animal's choice behavior during this task was nearly optimal. Nevertheless, the gain and loss of a conditioned reinforcer significantly increased and decreased, respectively, the tendency for the animal to choose the same target in subsequent trials. We also found that the neurons in the dorsomedial frontal cortex, dorsal anterior cingulate cortex, and dorsolateral prefrontal cortex often changed their activity according to whether the animal earned or lost a conditioned reinforcer in the current or previous trial. Moreover, many neurons in the dorsomedial frontal cortex also signaled the gain or loss occurring as a result of choosing a particular action as well as changes in the animal's behaviors resulting from such gains or losses. Thus, primate medial frontal cortex might mediate the behavioral effects of conditioned reinforcers and their losses.

Segregated and integrated coding of reward and punishment in the cingulate cortex

Reward and punishment have opposite affective value but are both processed by the cingulate cortex. However, it is unclear whether the positive and negative affective values of monetary reward and punishment are processed by separate or common subregions of the cingulate cortex. We performed a functional magnetic resonance imaging study using a free-choice task and compared cingulate activations for different levels of monetary gain and loss. Gain-specific activation (increasing activation for increasing gain, but no activation change in relation to loss) occurred mainly in the anterior part of the anterior cingulate and in the posterior cingulate cortex. Conversely, loss-specific activation (increasing activation for increasing loss, but no activation change in relation to gain) occurred between these areas, in the middle and posterior part of the anterior cingulate. Integrated coding of gain and loss (increasing activation throughout the full range, from biggest loss to biggest gain) occurred in the dorsal part of the anterior cingulate, at the border with the medial prefrontal cortex. Finally, unspecific activation increases to both gains and losses (increasing activation to increasing gains and increasing losses, possibly reflecting attention) occurred in dorsal and middle regions of the cingulate cortex. Together, these results suggest separate and common coding of monetary reward and punishment in distinct subregions of the cingulate cortex. Further meta-analysis suggested that the presently found reward- and punishment-specific areas overlapped with those processing positive and negative emotions, respectively.

In the eye of the beholder: individual differences in perceived social isolation predict regional brain activation to social stimuli

Prior research has shown that perceived social isolation (loneliness) motivates people to attend to and connect with others but to do so in a self-protective and paradoxically self-defeating fashion. Although recent research has shed light on the neural correlates of social perception, cooperation, empathy, rejection, and love, little is known about how individual differences in loneliness relate to neural responses to social and emotional stimuli. Using functional magnetic resonance imaging, we show that there are at least two neural mechanisms differentiating social perception in lonely and nonlonely young adults. For pleasant depictions, lonely individuals appear to be less rewarded by social stimuli, as evidenced by weaker activation of the ventral striatum to pictures of people than of objects, whereas nonlonely individuals showed stronger activation of the ventral striatum to pictures of people than of objects. For unpleasant depictions, lonely individuals were characterized by greater activation of the visual cortex to pictures of people than of objects, suggesting that their attention is drawn more to the distress of others, whereas nonlonely individuals showed greater activation of the right and left temporo-parietal junction to pictures of people than of objects, consistent with the notion that they are more likely to reflect spontaneously on the perspective of distressed others.