Differential encoding of losses and gains in the human striatum

Do patients with schizophrenia exhibit aberrant salience?

BACKGROUND

It has been suggested that some psychotic symptoms reflect 'aberrant salience', related to dysfunctional reward learning. To test this hypothesis we investigated whether patients with schizophrenia showed impaired learning of task-relevant stimulus-reinforcement associations in the presence of distracting task-irrelevant cues.

METHOD

We tested 20 medicated patients with schizophrenia and 17 controls on a reaction time game, the Salience Attribution Test. In this game, participants made a speeded response to earn money in the presence of conditioned stimuli (CSs). Each CS comprised two visual dimensions, colour and form. Probability of reinforcement varied over one of these dimensions (task-relevant), but not the other (task-irrelevant). Measures of adaptive and aberrant motivational salience were calculated on the basis of latency and subjective reinforcement probability rating differences over the task-relevant and task-irrelevant dimensions respectively.

RESULTS

Participants rated reinforcement significantly more likely and responded significantly faster on high-probability-reinforced relative to low-probability-reinforced trials, representing adaptive motivational salience. Patients exhibited reduced adaptive salience relative to controls, but the two groups did not differ in terms of aberrant salience. Patients with delusions exhibited significantly greater aberrant salience than those without delusions, and aberrant salience also correlated with negative symptoms. In the controls, aberrant salience correlated significantly with 'introvertive anhedonia' schizotypy.

CONCLUSIONS

These data support the hypothesis that aberrant salience is related to the presence of delusions in medicated patients with schizophrenia, but are also suggestive of a link with negative symptoms. The relationship between aberrant salience and psychotic symptoms warrants further investigation in unmedicated patients.

Neural mechanism of intertemporal choice: from discounting future gains to future losses

Intertemporal choice, the tradeoff among outcomes occurring at different points in time, involves not only benefit options but also those associated with cost. Previous neuroimaging studies have primarily focused on discounting future gains; thus the neural mechanism underlying discounting future losses remains unidentified. Using event-related functional magnetic resonance imaging, we comprehensively investigated the neural mechanism of temporal discounting using two decision-making tasks with a symmetric pattern of gains and losses. Our results revealed that the lateral prefrontal and posterior parietal areas were activated in discounting both future gains and future losses, but their activations were stronger when discounting losses. Moreover, we found that the insula, thalamus and dorsal striatum were more activated during intertemporal choices involving losses, suggesting that the enhanced sensitivity to losses may be driven by negative emotions. In addition, whereas the posterior cingulate cortex and medial prefrontal cortex were activated when the choices included immediate options, extra regions including the anterior cingulate cortex, insula and superior frontal gyrus were preferentially activated when the choices involved immediate losses. Taken together, our findings suggest that a fronto-parietal network supports the common discounting process, and more importantly, discounting future losses and gains occurs asymmetrically in the brain. We speculate that this may provide a neural basis for the phenomenon that future losses are discounted less steeply than future gains.

The role of the striatum in aversive learning and aversive prediction errors

Neuroeconomic studies of decision making have emphasized reward learning as critical in the representation of value-driven choice behaviour. However, it is readily apparent that punishment and aversive learning are also significant factors in motivating decisions and actions. In this paper, we review the role of the striatum and amygdala in affective learning and the coding of aversive prediction errors (PEs). We present neuroimaging results showing aversive PE-related signals in the striatum in fear conditioning paradigms with both primary (shock) and secondary (monetary loss) reinforcers. These results and others point to the general role for the striatum in coding PEs across a broad range of learning paradigms and reinforcer types.

Lesions of the medial striatum in monkeys produce perseverative impairments during reversal learning similar to those produced by lesions of the orbitofrontal cortex

The ability to switch responding between two visual stimuli based on their changing relationship with reward is dependent on the orbitofrontal cortex (OFC). OFC lesions in humans, monkeys, and rats disrupt performance on a common test of this ability, the visual serial discrimination reversal task. This finding is of particular significance to our understanding of psychiatric disorders such as obsessive-compulsive disorder (OCD) and schizophrenia, in which behavioral inflexibility is a prominent symptom. Although OFC dysfunction can occur in these disorders, there is considerable evidence for more widespread dysfunction within frontostriatal and frontoamygdalar circuitry. Because the contribution of these subcortical structures to behavioral flexibility is poorly understood, the present study compared the effects of excitotoxic lesions of the medial striatum (MS), amygdala, and OFC in the marmoset monkey on performance of the serial reversal task. All monkeys were able to learn a novel stimulus-reward association but, compared with both control and amygdala-lesioned monkeys, those with MS or OFC lesions showed a perseverative impairment in their ability to reverse this association. However, whereas both MS and OFC groups showed insensitivity to negative feedback, only OFC-lesioned monkeys showed insensitivity to positive feedback. These findings suggest that, for different reasons, both the MS and OFC support behavioral flexibility after changes in reward contingencies, and are consistent with the hypothesis that striatal and OFC dysfunction can contribute to pathological perseveration.

Foundations of neuroeconomics: from philosophy to practice

How can we use neuroscience to better understand economic behavior? By quelling concerns about the nascent field of neuroeconomics, the authors defend future integrations of the biological and social sciences.

A developmental neurobiological model of motivated behavior: anatomy, connectivity and ontogeny of the triadic nodes

Adolescence is the transition period that prepares individuals for fulfilling their role as adults. Most conspicuous in this transition period is the peak level of risk-taking behaviors that characterize adolescent motivated behavior. Significant neural remodeling contributes to this change. This review focuses on the functional neuroanatomy underlying motivated behavior, and how ontogenic changes can explain the typical behavioral patterns in adolescence. To help model these changes and provide testable hypotheses, a neural systems-based theory is presented. In short, the Triadic Model proposes that motivated behavior is governed by a carefully orchestrated articulation among three systems, approach, avoidance and regulatory. These three systems map to distinct, but overlapping, neural circuits, whose representatives are the striatum, the amygdala and the medial prefrontal cortex. Each of these system-representatives will be described from a functional anatomy perspective that includes a review of their connectivity and what is known of their ontogenic changes.

Striatal outcome processing in healthy aging

Functional MRI of young adults has implicated the striatum in the processing of rewarding and punishing events. To date, only two published experiments (Samanez-Larkin et al., 2007; Schott et al., 2007) have explored similar phenomena in older adults, with both studies emphasizing the anticipation of monetary outcomes. To better understand older participants' striatal responses to delivered outcomes, we engaged 20 older adults and 13 younger adults in a card-guessing task that rewarded correct guesses with monetary gain and punished incorrect guesses with monetary loss. Overall, the older adults retained most of the typical features of the striatal response, so that activity in the caudate head showed reliable differentiation between rewards and punishments during the 6- to 9-sec postoutcome window. Comparison of the older and younger adults also pointed to some potential aging effects on outcome activity, including reductions in the magnitude and extent of striatal activation, and a trend for the older adults to show a decreased early punishment response. In sum, our data suggest that the signaling of outcome valence remains relatively stable into late adulthood, although more research is needed to understand some subtle changes that might occur across the life span.

The hidden island of addiction: the insula

Most prior research on the neurobiology of addiction has focused on the role of subcortical systems, such as the amygdala, the ventral striatum and mesolimbic dopamine system, in promoting the motivation to seek drugs. Recent evidence indicates that a largely overlooked structure, the insula, plays a crucial part in conscious urges to take drugs. The insula has been highlighted as a region that integrates interoceptive (i.e. bodily) states into conscious feelings and into decision-making processes that involve uncertain risk and reward. Here, we propose a model in which the processing of the interoceptive effects of drug use by the insula contributes to conscious drug urges and to decision-making processes that precipitate relapse.

Neurocomputational models of basal ganglia function in learning, memory and choice

The basal ganglia (BG) are critical for the coordination of several motor, cognitive, and emotional functions and become dysfunctional in several pathological states ranging from Parkinson's disease to Schizophrenia. Here we review principles developed within a neurocomputational framework of BG and related circuitry which provide insights into their functional roles in behavior. We focus on two classes of models: those that incorporate aspects of biological realism and constrained by functional principles, and more abstract mathematical models focusing on the higher level computational goals of the BG. While the former are arguably more "realistic", the latter have a complementary advantage in being able to describe functional principles of how the system works in a relatively simple set of equations, but are less suited to making specific hypotheses about the roles of specific nuclei and neurophysiological processes. We review the basic architecture and assumptions of these models, their relevance to our understanding of the neurobiological and cognitive functions of the BG, and provide an update on the potential roles of biological details not explicitly incorporated in existing models. Empirical studies ranging from those in transgenic mice to dopaminergic manipulation, deep brain stimulation, and genetics in humans largely support model predictions and provide the basis for further refinement. Finally, we discuss possible future directions and possible ways to integrate different types of models.

Neurocomputational mechanisms of reinforcement-guided learning in humans: a review

Adapting decision making according to dynamic and probabilistic changes in action-reward contingencies is critical for survival in a competitive and resource-limited world. Much research has focused on elucidating the neural systems and computations that underlie how the brain identifies whether the consequences of actions are relatively good or bad. In contrast, less empirical research has focused on the mechanisms by which reinforcements might be used to guide decision making. Here, I review recent studies in which an attempt to bridge this gap has been made by characterizing how humans use reward information to guide and optimize decision making. Regions that have been implicated in reinforcement processing, including the striatum, orbitofrontal cortex, and anterior cingulate, also seem to mediate how reinforcements are used to adjust subsequent decision making. This research provides insights into why the brain devotes resources to evaluating reinforcements and suggests a direction for future research, from studying the mechanisms of reinforcement processing to studying the mechanisms of reinforcement learning.

Brain correlates of aesthetic expertise: a parametric fMRI study

Several studies have demonstrated that acquired expertise influences aesthetic judgments. In this paradigm we used functional magnetic resonance imaging (fMRI) to study aesthetic judgments of visually presented architectural stimuli and control-stimuli (faces) for a group of architects and a group of non-architects. This design allowed us to test whether level of expertise modulates neural activity in brain areas associated with either perceptual processing, memory, or reward processing. We show that experts and non-experts recruit bilateral medial orbitofrontal cortex (OFC) and subcallosal cingulate gyrus differentially during aesthetic judgment, even in the absence of behavioural aesthetic rating differences between experts and non-experts. By contrast, activity in nucleus accumbens (NAcc) exhibits a differential response profile compared to OFC and subcallosal cingulate gyrus, suggesting a dissociable role between these regions in the reward processing of expertise. Finally, categorical responses (irrespective of aesthetic ratings) resulted in expertise effects in memory-related areas such as hippocampus and precuneus. These results highlight the fact that expertise not only modulates cognitive processing, but also modulates the response in reward related brain areas.

Reinforcement learning: the good, the bad and the ugly

Reinforcement learning provides both qualitative and quantitative frameworks for understanding and modeling adaptive decision-making in the face of rewards and punishments. Here we review the latest dispatches from the forefront of this field, and map out some of the territories where lie monsters.

Anchors, scales and the relative coding of value in the brain

People are alarmingly susceptible to manipulations that change both their expectations and experience of the value of goods. Recent studies in behavioral economics suggest such variability reflects more than mere caprice. People commonly judge options and prices in relative terms, rather than absolutely, and display strong sensitivity to exemplar and price anchors. We propose that these findings elucidate important principles about reward processing in the brain. In particular, relative valuation may be a natural consequence of adaptive coding of neuronal firing to optimise sensitivity across large ranges of value. Furthermore, the initial apparent arbitrariness of value may reflect the brains' attempts to optimally integrate diverse sources of value-relevant information in the face of perceived uncertainty. Recent findings in neuroscience support both accounts, and implicate regions in the orbitofrontal cortex, striatum, and ventromedial prefrontal cortex in the construction of value.

Individual differences in reinforcement learning: behavioral, electrophysiological, and neuroimaging correlates

During reinforcement learning, phasic modulations of activity in midbrain dopamine neurons are conveyed to the dorsal anterior cingulate cortex (dACC) and basal ganglia (BG) and serve to guide adaptive responding. While the animal literature supports a role for the dACC in integrating reward history over time, most human electrophysiological studies of dACC function have focused on responses to single positive and negative outcomes. The present electrophysiological study investigated the role of the dACC in probabilistic reward learning in healthy subjects using a task that required integration of reinforcement history over time. We recorded the feedback-related negativity (FRN) to reward feedback in subjects who developed a response bias toward a more frequently rewarded ("rich") stimulus ("learners") versus subjects who did not ("non-learners"). Compared to non-learners, learners showed more positive (i.e., smaller) FRNs and greater dACC activation upon receiving reward for correct identification of the rich stimulus. In addition, dACC activation and a bias to select the rich stimulus were positively correlated. The same participants also completed a monetary incentive delay (MID) task administered during functional magnetic resonance imaging. Compared to non-learners, learners displayed stronger BG responses to reward in the MID task. These findings raise the possibility that learners in the probabilistic reinforcement task were characterized by stronger dACC and BG responses to rewarding outcomes. Furthermore, these results highlight the importance of the dACC to probabilistic reward learning in humans.

The neural mechanisms underlying the influence of pavlovian cues on human decision making

In outcome-specific transfer, pavlovian cues that are predictive of specific outcomes bias action choice toward actions associated with those outcomes. This transfer occurs despite no explicit training of the instrumental actions in the presence of pavlovian cues. The neural substrates of this effect in humans are unknown. To address this, we scanned 23 human subjects with functional magnetic resonance imaging while they made choices between different liquid food rewards in the presence of pavlovian cues previously associated with one of these outcomes. We found behavioral evidence of outcome-specific transfer effects in our subjects, as well as differential blood oxygenation level-dependent activity in a region of ventrolateral putamen when subjects chose, respectively, actions consistent and inconsistent with the pavlovian-predicted outcome. Our results suggest that choosing an action incompatible with a pavlovian-predicted outcome might require the inhibition of feasible but nonselected action-outcome associations. The results of this study are relevant for understanding how marketing actions can affect consumer choice behavior as well as for how environmental cues can influence drug-seeking behavior in addiction.

Episodic simulation of future events: concepts, data, and applications

This article focuses on the neural and cognitive processes that support imagining or simulating future events, a topic that has recently emerged in the forefront of cognitive neuroscience. We begin by considering concepts of simulation from a number of areas of psychology and cognitive neuroscience in order to place our use of the term in a broader context. We then review neuroimaging, neuropsychological, and cognitive studies that have examined future-event simulation and its relation to episodic memory. This research supports the idea that simulating possible future events depends on much of the same neural machinery, referred to here as a core network, as does remembering past events. After discussing several theoretical accounts of the data, we consider applications of work on episodic simulation for research concerning clinical populations suffering from anxiety or depression. Finally, we consider other aspects of future-oriented thinking that we think are related to episodic simulation, including planning, prediction, and remembering intentions. These processes together comprise what we have termed "the prospective brain," whose primary function is to use past experiences to anticipate future events.

A framework for studying the neurobiology of value-based decision making

Neuroeconomics is the study of the neurobiological and computational basis of value-based decision making. Its goal is to provide a biologically based account of human behaviour that can be applied in both the natural and the social sciences. This Review proposes a framework to investigate different aspects of the neurobiology of decision making. The framework allows us to bring together recent findings in the field, highlight some of the most important outstanding problems, define a common lexicon that bridges the different disciplines that inform neuroeconomics, and point the way to future applications.

Modulators of decision making

Human and animal decisions are modulated by a variety of environmental and intrinsic contexts. Here I consider computational factors that can affect decision making and review anatomical structures and neurochemical systems that are related to contextual modulation of decision making. Expectation of a high reward can motivate a subject to go for an action despite a large cost, a decision that is influenced by dopamine in the anterior cingulate cortex. Uncertainty of action outcomes can promote risk taking and exploratory choices, in which norepinephrine and the orbitofrontal cortex appear to be involved. Predictable environments should facilitate consideration of longer-delayed rewards, which depends on serotonin in the dorsal striatum and dorsal prefrontal cortex. This article aims to sort out factors that affect the process of decision making from the viewpoint of reinforcement learning theory and to bridge between such computational needs and their neurophysiological substrates.

A common neurobiology for pain and pleasure

Pain and pleasure are powerful motivators of behaviour and have historically been considered opposites. Emerging evidence from the pain and reward research fields points to extensive similarities in the anatomical substrates of painful and pleasant sensations. Recent molecular-imaging and animal studies have demonstrated the important role of the opioid and dopamine systems in modulating both pain and pleasure. Understanding the mutually inhibitory effects that pain and reward processing have on each other, and the neural mechanisms that underpin such modulation, is important for alleviating unnecessary suffering and improving well-being.

High-frequency oscillations in distributed neural networks reveal the dynamics of human decision making

We examine the relative timing of numerous brain regions involved in human decisions that are based on external criteria, learned information, personal preferences, or unconstrained internal considerations. Using magnetoencephalography (MEG) and advanced signal analysis techniques, we were able to non-invasively reconstruct oscillations of distributed neural networks in the high-gamma frequency band (60–150 Hz). The time course of the observed neural activity suggested that two-alternative forced choice tasks are processed in four overlapping stages: processing of sensory input, option evaluation, intention formation, and action execution. Visual areas are activated first, and show recurring activations throughout the entire decision process. The temporo-occipital junction and the intraparietal sulcus are active during evaluation of external values of the options, 250–500 ms after stimulus presentation. Simultaneously, personal preference is mediated by cortical midline structures. Subsequently, the posterior parietal and superior occipital cortices appear to encode intention, with different subregions being responsible for different types of choice. The cerebellum and inferior parietal cortex are recruited for internal generation of decisions and actions, when all options have the same value. Action execution was accompanied by activation peaks in the contralateral motor cortex. These results suggest that high-gamma oscillations as recorded by MEG allow a reliable reconstruction of decision processes with excellent spatiotemporal resolution.

Risky business: the neuroeconomics of decision making under uncertainty

Many decisions involve uncertainty, or imperfect knowledge about how choices lead to outcomes. Colloquial notions of uncertainty, particularly when describing a decision as 'risky', often carry connotations of potential danger as well. Gambling on a long shot, whether a horse at the racetrack or a foreign oil company in a hedge fund, can have negative consequences, but the impact of uncertainty on decision making extends beyond gambling. Indeed, uncertainty in some form pervades nearly all our choices in daily life. Stepping into traffic to hail a cab, braving an ice storm to be the first at work, or dating the boss's son or daughter also offer potentially great windfalls, at the expense of surety. We continually face trade-offs between options that promise safety and others that offer an uncertain potential for jackpot or bust. When mechanisms for dealing with uncertain outcomes fail, as in mental disorders such as problem gambling or addiction, the results can be disastrous. Thus, understanding decision making-indeed, understanding behavior itself-requires knowing how the brain responds to and uses information about uncertainty.

Prefrontal Cortex, Emotion, and Approach/Withdrawal Motivation

This article provides a selective review of the literature and current theories regarding the role of prefrontal cortex, along with some other critical brain regions, in emotion and motivation. Seemingly contradictory findings have often appeared in this literature. Research attempting to resolve these contradictions has been the basis of new areas of growth and has led to more sophisticated understandings of emotional and motivational processes as well as neural networks associated with these processes. Progress has, in part, depended on methodological advances that allow for increased resolution in brain imaging. A number of issues are currently in play, among them the role of prefrontal cortex in emotional or motivational processes. This debate fosters research that will likely lead to further refinement of conceptualizations of emotion, motivation, and the neural processes associated with them.

Attention and emotion influence the relationship between extraversion and neural response

Extraversion has been shown to positively correlate with activation within the ventral striatum, amygdala and other dopaminergically innervated, reward-sensitive regions. These regions are implicated in emotional responding, in a manner sensitive to attentional focus. However, no study has investigated the interaction among extraversion, emotion and attention. We used fMRI and dynamic, evocative film clips to elicit amusement and sadness in a sample of 28 women. Participants were instructed either to respond naturally (n = 14) or to attend to and continuously rate their emotions (n = 14) while watching the films. Contrary to expectations, striatal response was negatively associated with extraversion during amusement, regardless of attention. A negative association was also observed during sad films, but only when attending to emotion. These findings suggest that attentional focus does not influence the relationship between extraversion and neural response to positive (amusing) stimuli but does impact the response to negative (sad) stimuli.

Social neuroeconomics: the neural circuitry of social preferences

Combining the methods of neuroscience and economics generates powerful tools for studying the brain processes behind human social interaction. We argue that hedonic interpretations of theories of social preferences provide a useful framework that generates interesting predictions and helps interpret brain activations involved in altruistic, fair and trusting behaviors. These behaviors are consistently associated with activation in reward-related brain areas, such as the striatum, and with prefrontal activity implicated in cognitive control, the processing of emotions, and integration of benefits and costs, consistent with resolution of a conflict between self-interest and other-regarding motives.

Prospection: experiencing the future

All animals can predict the hedonic consequences of events they've experienced before. But humans can predict the hedonic consequences of events they've never experienced by simulating those events in their minds. Scientists are beginning to understand how the brain simulates future events, how it uses those simulations to predict an event's hedonic consequences, and why these predictions so often go awry.

The cerebral signature for pain perception and its modulation

Our understanding of the neural correlates of pain perception in humans has increased significantly since the advent of neuroimaging. Relating neural activity changes to the varied pain experiences has led to an increased awareness of how factors (e.g., cognition, emotion, context, injury) can separately influence pain perception. Tying this body of knowledge in humans to work in animal models of pain provides an opportunity to determine common features that reliably contribute to pain perception and its modulation. One key system that underpins the ability to change pain intensity is the brainstem's descending modulatory network with its pro- and antinociceptive components. We discuss not only the latest data describing the cerebral signature of pain and its modulation in humans, but also suggest that the brainstem plays a pivotal role in gating the degree of nociceptive transmission so that the resultant pain experienced is appropriate for the particular situation of the individual.

Dissociation of neural regions associated with anticipatory versus consummatory phases of incentive processing

Incentive delay tasks implicate the striatum and medial frontal cortex in reward processing. However, prior studies delivered more rewards than penalties, possibly leading to unwanted differences in signal-to-noise ratio. Also, whether particular brain regions are specifically involved in anticipation or consumption is unclear. We used a task featuring balanced incentive delivery and an analytic strategy designed to identify activity specific to anticipation or consumption. Reaction time data in two independent samples (n=13 and n=8) confirmed motivated responding. Functional magnetic resonance imaging revealed regions activated by anticipation (anterior cingulate) versus consumption (orbital and medial frontal cortex). Ventral striatum was active during reward anticipation but not significantly more so than during consumption. Although the study features several methodological improvements and helps clarify the neural basis of incentive processing, replications in larger samples are needed.