Testing the reward prediction error hypothesis with an axiomatic model

Understanding psychiatric disorder by capturing ecologically relevant features of learning and decision-making

Recent research in cognitive neuroscience has begun to uncover the processes underlying increasingly complex voluntary behaviours, including learning and decision-making. Partly this success has been possible by progressing from simple experimental tasks to paradigms that incorporate more ecological features. More specifically, the premise is that to understand cognitions and brain functions relevant for real life, we need to introduce some of the ecological challenges that we have evolved to solve. This often entails an increase in task complexity, which can be managed by using computational models to help parse complex behaviours into specific component mechanisms. Here we propose that using computational models with tasks that capture ecologically relevant learning and decision-making processes may provide a critical advantage for capturing the mechanisms underlying symptoms of disorders in psychiatry. As a result, it may help develop mechanistic approaches towards diagnosis and treatment. We begin this review by mapping out the basic concepts and models of learning and decision-making. We then move on to consider specific challenges that emerge in realistic environments and describe how they can be captured by tasks. These include changes of context, uncertainty, reflexive/emotional biases, cost-benefit decision-making, and balancing exploration and exploitation. Where appropriate we highlight future or current links to psychiatry. We particularly draw examples from research on clinical depression, a disorder that greatly compromises motivated behaviours in real-life, but where simpler paradigms have yielded mixed results. Finally, we highlight several paradigms that could be used to help provide new insights into the mechanisms of psychiatric disorders.

Computing Value from Quality and Quantity in Human Decision-Making

How organisms learn the value of single stimuli through experience is well described. In many decisions, however, value estimates are computed “on the fly” by combining multiple stimulus attributes. The neural basis of this computation is poorly understood. Here we explore a common scenario in which decision-makers must combine information about quality and quantity to determine the best option. Using fMRI, we examined the neural representation of quality, quantity, and their integration into an integrated subjective value signal in humans of both genders. We found that activity within inferior frontal gyrus (IFG) correlated with offer quality, while activity in the intraparietal sulcus (IPS) specifically correlated with offer quantity. Several brain regions, including the anterior cingulate cortex (ACC), were sensitive to an interaction of quality and quantity. However, the ACC was uniquely activated by quality, quantity, and their interaction, suggesting that this region provides a substrate for flexible computation of value from both quality and quantity. Furthermore, ACC signals across subjects correlated with the strength of quality and quantity signals in IFG and IPS, respectively. ACC tracking of subjective value also correlated with choice predictability. Finally, activity in the ACC was elevated for choice trials, suggesting that ACC provides a nexus for the computation of subjective value in multiattribute decision-making.

SIGNIFICANCE STATEMENT Would you prefer three apples or two oranges? Many choices we make each day require us to weigh up the quality and quantity of different outcomes. Using fMRI, we show that option quality is selectively represented in the inferior frontal gyrus, while option quantity correlates with areas of the intraparietal sulcus that have previously been associated with numerical processing. We show that information about the two is integrated into a value signal in the anterior cingulate cortex, and the fidelity of this integration predicts choice predictability. Our results demonstrate how on-the-fly value estimates are computed from multiple attributes in human value-based decision-making.

Neural mechanisms for learning self and other ownership

Sense of ownership is a ubiquitous and fundamental aspect of human cognition. Here we used model-based functional magnetic resonance imaging and a novel minimal ownership paradigm to probe the behavioural and neural mechanisms underpinning ownership acquisition for ourselves, friends and strangers. We find a self-ownership bias at multiple levels of behaviour from initial preferences to reaction times and computational learning rates. Ventromedial prefrontal cortex (vmPFC) and anterior cingulate sulcus (ACCs) responded more to self vs. stranger associations, but despite a pervasive neural bias to track self-ownership, no brain area tracked self-ownership exclusively. However, ACC gyrus (ACCg) specifically coded ownership prediction errors for strangers and ownership associative strength for friends and strangers but not for self. Core neural mechanisms for associative learning are biased to learn in reference to self but also engaged when learning in reference to others. In contrast, ACC gyrus exhibits specialization for learning about others.

The neuroanatomical basis of the Gambler's fallacy: A univariate and multivariate morphometric study

Human decision-making can be irrational, as in the case of the gambler's fallacy (GF). Converging evidence from behavioral and functional neuroimaging studies has suggested that a hyperactive cognitive system and a hypo-active affective system contribute to the false world model that generates the GF. However, the neuroanatomical basis underlying the GF remains unclear. The current study addressed this issue by collecting high-resolution magnetic resonance anatomical images from a large sample of 350 healthy Chinese adults. Univariate voxel-based morphometry (VBM) analysis suggested that the gray matter volume (GMV) in the anterior cingulate cortex (ACC) and medial temporal lobe (MTL) (two regions of the cognitive system) showed negative correlations with the degree of GF, while the GMV in the striatum and orbitofrontal cortex (OFC; two regions of the affective system) showed positive correlations. Further multivariate VBM analysis showed that the GMV in these regions could potentially predict the degree of GF. Moreover, a mediation analysis suggested that the GMV in MTL, ACC, and OFC mediated the relationships between the cognitive abilities or affective decision-making performance and the GF. Results of our study help us to understand the potential neural bases of the cognitive system's constructive role and the affective system's destructive role in decision making.

What is mood? A computational perspective

The neurobiological understanding of mood, and by extension mood disorders, remains elusive despite decades of research implicating several neuromodulator systems. This review considers a new approach based on existing theories of functional brain organisation. The free energy principle (a.k.a. active inference), and its instantiation in the Bayesian brain, offers a complete and simple formulation of mood. It has been proposed that emotions reflect the precision of - or certainty about - the predicted sensorimotor/interoceptive consequences of action. By extending this reasoning, in a hierarchical setting, we suggest mood states act as (hyper) priors over uncertainty (i.e. emotions). Here, we consider the same computational pathology in the proprioceptive and interoceptive (behavioural and autonomic) domain in order to furnish an explanation for mood disorders. This formulation reconciles several strands of research at multiple levels of enquiry.

Neural activity and fundamental learning, motivated by monetary loss and reward, are intact in mild to moderate major depressive disorder

INTRODUCTION

Reduced motivation is an important symptom of major depression, thought to impair recovery by reducing opportunities for rewarding experiences. We characterized motivation for monetary outcomes in depressed outpatients (N = 39, 22 female) and controls (N = 22, 11 female) in terms of their effectiveness in seeking rewards and avoiding losses. We assessed motivational function during learning of associations between stimuli and actions, as well as when learning was complete. We compared the activity within neural circuits underpinning these behaviors between depressed patients and controls.

METHODS

We used a Go/No-Go task that assessed subjects' abilities in learning to emit or withhold actions to obtain monetary rewards or avoid losses. We derived motivation-relevant parameters of behavior (learning rate, Pavlovian bias, and motivational influence of gains and losses). After learning, participants performed the task during functional magnetic resonance imaging (fMRI). We compared neural activation during anticipation of action emission vs. action inhibition, and for actions performed to obtain rewards compared to actions that avoid losses.

RESULTS

Depressed patients showed a similar Pavlovian bias to controls and were equivalent in terms of withholding action to gain rewards and emitting action to avoid losses, behaviors that conflict with well-described Pavlovian tendencies to approach rewards and avoid losses. Patients were not impaired in overall performance or learning and showed no abnormal neural responses, for example in bilateral midbrain or striatum. We conclude that basic mechanisms subserving motivated learning are thus intact in moderate depression.

IMPLICATIONS

Therapeutically, the intact mechanisms identified here suggest that learning-based interventions may be particularly effective in encouraging recovery. Etiologically, our results suggest that the severe motivational deficits clinically observed in depression are likely to have complex origins, possibly related to an impairment in the representation of future states necessary for long-term planning.

What's in a word? How instructions, suggestions, and social information change pain and emotion

Instructions, suggestions, and other types of social information can have powerful effects on pain and emotion. Prominent examples include observational learning, social influence, placebo, and hypnosis. These different phenomena and their underlying brain mechanisms have been studied in partially separate literatures, which we discuss, compare, and integrate in this review. Converging findings from these literatures suggest that (1) instructions and social information affect brain systems associated with the generation of pain and emotion, and with reinforcement learning, and that (2) these changes are mediated by alterations in prefrontal systems responsible for top-down control and the generation of affective meaning. We argue that changes in expectation and appraisal, a process of assessing personal meaning and implications for wellbeing, are two potential key mediators of the effects of instructions and social information on affective experience. Finally, we propose a tentative model of how prefrontal regions, especially dorsolateral and ventromedial prefrontal cortex may regulate affective processing based on instructions and socially transmitted expectations more broadly.

Valuation of knowledge and ignorance in mesolimbic reward circuitry

Humans desire to know what the future holds. Yet, at times they decide to remain ignorant (e.g., reject medical screenings). These decisions have important societal implications in domains ranging from health to finance. We show how the opportunity to gain information is valued and explain why knowledge is not always preferred. Specifically, the mesolimbic reward circuitry selectively treats the opportunity to gain knowledge about favorable, but not unfavorable, outcomes as a reward to be approached. This coding predicts biased information seeking: Participants choose knowledge about future desirable outcomes more than about undesirable ones, vice versa for ignorance, and are willing to pay for both. This work demonstrates a role for valence in how the human brain values knowledge.

The pursuit of knowledge is a basic feature of human nature. However, in domains ranging from health to finance people sometimes choose to remain ignorant. Here, we show that valence is central to the process by which the human brain evaluates the opportunity to gain information, explaining why knowledge may not always be preferred. We reveal that the mesolimbic reward circuitry selectively treats the opportunity to gain knowledge about future favorable outcomes, but not unfavorable outcomes, as if it has positive utility. This neural coding predicts participants’ tendency to choose knowledge about future desirable outcomes more often than undesirable ones, and to choose ignorance about future undesirable outcomes more often than desirable ones. Strikingly, participants are willing to pay both for knowledge and ignorance as a function of the expected valence of knowledge. The orbitofrontal cortex (OFC), however, responds to the opportunity to receive knowledge over ignorance regardless of the valence of the information. Connectivity between the OFC and mesolimbic circuitry could contribute to a general preference for knowledge that is also modulated by valence. Our findings characterize the importance of valence in information seeking and its underlying neural computation. This mechanism could lead to suboptimal behavior, such as when people reject medical screenings or monitor investments more during bull than bear markets.

Knowledge acquisition is governed by striatal prediction errors

Discrepancies between expectations and outcomes, or prediction errors, are central to trial-and-error learning based on reward and punishment, and their neurobiological basis is well characterized. It is not known, however, whether the same principles apply to declarative memory systems, such as those supporting semantic learning. Here, we demonstrate with fMRI that the brain parametrically encodes the degree to which new factual information violates expectations based on prior knowledge and beliefs-most prominently in the ventral striatum, and cortical regions supporting declarative memory encoding. These semantic prediction errors determine the extent to which information is incorporated into long-term memory, such that learning is superior when incoming information counters strong incorrect recollections, thereby eliciting large prediction errors. Paradoxically, by the same account, strong accurate recollections are more amenable to being supplanted by misinformation, engendering false memories. These findings highlight a commonality in brain mechanisms and computational rules that govern declarative and nondeclarative learning, traditionally deemed dissociable.

Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory

Learning from rewards and punishments is essential to survival and facilitates flexible human behavior. It is widely appreciated that multiple cognitive and reinforcement learning systems contribute to decision-making, but the nature of their interactions is elusive. Here, we leverage methods for extracting trial-by-trial indices of reinforcement learning (RL) and working memory (WM) in human electro-encephalography to reveal single-trial computations beyond that afforded by behavior alone. Neural dynamics confirmed that increases in neural expectation were predictive of reduced neural surprise in the following feedback period, supporting central tenets of RL models. Within- and cross-trial dynamics revealed a cooperative interplay between systems for learning, in which WM contributes expectations to guide RL, despite competition between systems during choice. Together, these results provide a deeper understanding of how multiple neural systems interact for learning and decision-making and facilitate analysis of their disruption in clinical populations.

Identification of Neurotensin Receptor Expressing Cells in the Ventral Tegmental Area across the Lifespan

Neurotensin (Nts) promotes activation of dopamine (DA) neurons in the ventral tegmental area (VTA) via incompletely understood mechanisms. Nts can signal via the G protein-coupled Nts receptors 1 and 2 (NtsR1 and NtsR2), but the lack of methods to detect NtsR1- and NtsR2-expressing cells has limited mechanistic understanding of Nts action. To overcome this challenge, we generated dual recombinase mice that express FlpO-dependent Cre recombinase in NtsR1 or NtsR2 cells. This strategy permitted temporal control over recombination, such that we could identify NtsR1- or NtsR2-expressing cells and determine whether their distributions differed between the developing and adult brain. Using this system, we found that NtsR1 is transiently expressed in nearly all DA neurons and in many non-DA neurons in the VTA during development. However, NtsR1 expression is more restricted within the adult brain, where only two thirds of VTA DA neurons expressed NtsR1. By contrast, NtsR2 expression remains constant throughout lifespan, but it is predominantly expressed within glia. Anterograde tract tracing revealed that NtsR1 is expressed by mesolimbic, not mesocortical DA neurons, suggesting that VTA NtsR1 neurons may represent a functionally unique subset of VTA DA neurons. Collectively, this work reveals a cellular mechanism by which Nts can directly engage NtsR1-expressing DA neurons to modify DA signaling. Going forward, the dual recombinase strategy developed here will be useful to selectively modulate NtsR1- and NtsR2-expressing cells and to parse their contributions to Nts-mediated behaviors.

Altered ventral striatal-medial prefrontal cortex resting-state connectivity mediates adolescent social problems after early institutional care

Early caregiving adversity is associated with increased risk for social difficulties. The ventral striatum and associated corticostriatal circuitry, which have demonstrated vulnerability to early exposures to adversity, are implicated in many aspects of social behavior, including social play, aggression, and valuation of social stimuli across development. Here, we used resting-state functional magnetic resonance imaging to assess the degree to which early caregiving adversity was associated with altered coritocostriatal resting connectivity in previously institutionalized youth (n = 41) relative to youth who were raised with their biological families from birth (n = 47), and the degree to which this connectivity was associated with parent-reported social problems. Using a seed-based approach, we observed increased positive coupling between the ventral striatum and anterior regions of medial prefrontal cortex (mPFC) in previously institutionalized youth. Stronger ventral striatum-mPFC coupling was associated with parent reports of social problems. A moderated-mediation analysis showed that ventral striatal-mPFC connectivity mediated group differences in social problems, and more so with increasing age. These findings show that early institutional care is associated with differences in resting-state connectivity between the ventral striatum and the mPFC, and this connectivity seems to play an increasingly important role in social behaviors as youth enter adolescence.

Alterations in functional brain networks associated with loss-chasing in gambling disorder and cocaine-use disorder

BACKGROUND

Continued, persistent gambling to recover accumulating losses, or 'loss-chasing', is a behavioral pattern linked particularly closely to gambling disorder (GD) but may reflect impaired decision-making processes relevant to drug addictions like cocaine-use disorder (CUD). However, little is known regarding the neurocognitive mechanisms of this complex, maladaptive behavior, particularly in individuals with addictive disorders.

METHODS

Seventy participants (25 GD, 18 CUD, and 27 healthy comparison (HC)) completed a loss-chase task during fMRI. Engagement of functional brain networks in response to losing outcomes and during decision-making periods preceding choices to loss-chase or to quit chasing losses were investigated using independent component analysis (ICA). An exploratory factor analysis was performed to examine patterns of coordinated engagement across identified networks.

RESULTS

In GD relative to HC and CUD participants, choices to quit chasing were associated with greater engagement of a medial frontal executive-processing network. By comparison, CUD participants exhibited altered engagement of a striato-amygdala motivational network in response to losing outcomes as compared to HC, and during decision-making as compared to GD. Several other networks were differentially engaged during loss-chase relative to quit-chasing choices, but did not differ across participant groups. Exploratory factor analysis identified a system of coordinated activity across prefrontal executive-control networks that was greater in GD and CUD relative to HC participants and was associated with increased chasing persistence across all participants.

CONCLUSIONS

Results provide evidence of shared and distinct neurobiological mechanisms in substance and behavioral addictions, and lend insight into potential cognitive interventions targeting loss-chasing behavior in GD.

Anatomical and functional organization of the human substantia nigra and its connections

We investigated the anatomical and functional organization of the human substantia nigra (SN) using diffusion and functional MRI data from the Human Connectome Project. We identified a tripartite connectivity-based parcellation of SN with a limbic, cognitive, motor arrangement. The medial SN connects with limbic striatal and cortical regions and encodes value (greater response to monetary wins than losses during fMRI), while the ventral SN connects with associative regions of cortex and striatum and encodes salience (equal response to wins and losses). The lateral SN connects with somatomotor regions of striatum and cortex and also encodes salience. Behavioral measures from delay discounting and flanker tasks supported a role for the value-coding medial SN network in decisional impulsivity, while the salience-coding ventral SN network was associated with motor impulsivity. In sum, there is anatomical and functional heterogeneity of human SN, which underpins value versus salience coding, and impulsive choice versus impulsive action.

Temporal dissociation of salience and prediction error responses to appetitive and aversive taste

The feedback-related negativity (FRN), a frontocentral ERP occurring 200-350 ms after emotionally valued outcomes, has been posited as the neural correlate of reward prediction error, a key component of associative learning. Recent evidence challenged this interpretation and has led to the suggestion that this ERP expresses salience instead. Here, we distinguish between utility prediction error and salience by delivering or withholding hedonistically matched appetitive and aversive tastes, and measure ERPs to cues signaling each taste. We observed a typical FRN (computed as the loss-minus-gain difference wave) to appetitive taste, but a reverse FRN to aversive taste. When tested axiomatically, frontocentral ERPs showed a salience response across tastes, with a particularly early response to outcome delivery, supporting recent propositions of a fast, unsigned, and unspecific response to salient stimuli. ERPs also expressed aversive prediction error peaking at 285 ms, which conformed to the logic of an axiomatic model of prediction error. With stimuli that most resemble those used in animal models, we did not detect any frontocentral ERP signal for utility prediction error, in contrast with dominant views of the functional role of the FRN ERP. We link the animal and human literature and present a challenge for current perspectives on associative learning research using ERPs.

Separate mesocortical and mesolimbic pathways encode effort and reward learning signals

Optimal decision making mandates organisms learn the relevant features of choice options. Likewise, knowing how much effort we should expend can assume paramount importance. A mesolimbic network supports reward learning, but it is unclear whether other choice features, such as effort learning, rely on this same network. Using computational fMRI, we show parallel encoding of effort and reward prediction errors (PEs) within distinct brain regions, with effort PEs expressed in dorsomedial prefrontal cortex and reward PEs in ventral striatum. We show a common mesencephalic origin for these signals evident in overlapping, but spatially dissociable, dopaminergic midbrain regions expressing both types of PE. During action anticipation, reward and effort expectations were integrated in ventral striatum, consistent with a computation of an overall net benefit of a stimulus. Thus, we show that motivationally relevant stimulus features are learned in parallel dopaminergic pathways, with formation of an integrated utility signal at choice.

Ventral striatum lesions do not affect reinforcement learning with deterministic outcomes on slow time scales

A large body of work has implicated the ventral striatum (VS) in aspects of reinforcement learning (RL). However, less work has directly examined the effects of lesions in the VS, or other forms of inactivation, on 2-armed bandit RL tasks. We have recently found that lesions in the VS in macaque monkeys affect learning with stochastic schedules but have minimal effects with deterministic schedules. The reasons for this are not currently clear. Because our previous work used short intertrial intervals, one possibility is that the animals were using working memory to bridge stimulus-reward associations from 1 trial to the next. In the present study, we examined learning of 60 pairs of objects, in which the animals received only 1 trial per day with each pair. The large number of object pairs and the long interval (approximately 24 hr) between trials with a given pair minimized the chances that the animals could use working memory to bridge trials. We found that monkeys with VS lesions were unimpaired relative to controls, which suggests that animals with VS lesions can still learn to select rewarded objects even when they cannot make use of working memory. (PsycINFO Database Record

Motivational Deficits in Schizophrenia Are Associated With Reduced Differentiation Between Gain and Loss-Avoidance Feedback in the Striatum

BACKGROUND

The current study was designed to test the hypothesis that motivational deficits in schizophrenia (SZ) are tied to a reduced ability to differentially signal gains and instances of loss-avoidance in the brain, leading to reduced ability to form adaptive representations of expected value.

METHODS

We administered a reinforcement learning paradigm to 27 medicated SZ patients and 27 control subjects in which participants learned three probabilistic discriminations. In regions of interest in reward networks identified a priori, we examined contrasts between trial types with different expected values (e.g., expected gain-nonmonetary) and between outcomes with the same prediction error valence but different experienced values (e.g., gain-loss-avoidance outcome, miss-loss outcome).

RESULTS

Both whole-brain and region of interest analyses revealed that SZ patients showed reduced differentiation between gain and loss-avoidance outcomes in the dorsal anterior cingulate cortex and bilateral anterior insula. That is, SZ patients showed reduced contrasts between positive prediction errors of different objective values in these areas. In addition, we observed significant correlations between gain-loss-avoidance outcome contrasts in the ventral striatum and ratings for avolition/anhedonia and between expected gain-nonmonetary contrasts in the ventral striatum and ventromedial prefrontal cortex.

CONCLUSIONS

These results provide further evidence for intact prediction error signaling in medicated SZ patients, especially with regard to loss-avoidance. By contrast, components of frontostriatal circuits appear to show reduced sensitivity to the absolute valence of expected and experienced outcomes, suggesting a mechanism by which motivational deficits may emerge.

Association of Neural and Emotional Impacts of Reward Prediction Errors With Major Depression

IMPORTANCE

Major depressive disorder (MDD) is associated with deficits in representing reward prediction errors (RPEs), which are the difference between experienced and predicted reward. Reward prediction errors underlie learning of values in reinforcement learning models, are represented by phasic dopamine release, and are known to affect momentary mood.

OBJECTIVE

To combine functional neuroimaging, computational modeling, and smartphone-based large-scale data collection to test, in the absence of learning-related concerns, the hypothesis that depression attenuates the impact of RPEs.

DESIGN, SETTING, AND PARTICIPANTS

Functional magnetic resonance imaging (fMRI) data were collected on 32 individuals with moderate MDD and 20 control participants who performed a probabilistic reward task. A risky decision task with repeated happiness ratings as a measure of momentary mood was also tested in the laboratory in 74 participants and with a smartphone-based platform in 1833 participants. The study was conducted from November 20, 2012, to February 17, 2015.

MAIN OUTCOMES AND MEASURES

Blood oxygen level-dependent activity was measured in ventral striatum, a dopamine target area known to represent RPEs. Momentary mood was measured during risky decision making.

RESULTS

Of the 52 fMRI participants (mean [SD] age, 34.0 [9.1] years), 30 (58%) were women and 32 had MDD. Of the 74 participants in the laboratory risky decision task (mean age, 34.2 [10.3] years), 44 (59%) were women and 54 had MDD. Of the smartphone group, 543 (30%) had a depression history and 1290 (70%) had no depression history; 918 (50%) were women, and 593 (32%) were younger than 30 years. Contrary to previous results in reinforcement learning tasks, individuals with moderate depression showed intact RPE signals in ventral striatum (z = 3.16; P = .002) that did not differ significantly from controls (z = 0.91; P = .36). Symptom severity correlated with baseline mood parameters in laboratory (ρ = -0.54; P < 1 × 10-6) and smartphone (ρ = -0.30; P < 1 × 10-39) data. However, participants with depression showed an intact association between RPEs and happiness in a computational model of momentary mood dynamics (z = 4.55; P < .001) that was not attenuated compared with controls (z = -0.42; P = .67).

CONCLUSIONS AND RELEVANCE

The neural and emotional impact of RPEs is intact in major depression. These results suggest that depression does not affect the expression of dopaminergic RPEs and that attenuated RPEs in previous reports may reflect downstream effects more closely related to aberrant behavior. The correlation between symptom severity and baseline mood parameters supports an association between depression and momentary mood fluctuations during cognitive tasks. These results demonstrate a potential for smartphones in large-scale computational phenotyping, which is a goal for computational psychiatry.

Spatiotemporal neural characterization of prediction error valence and surprise during reward learning in humans

Reward learning depends on accurate reward associations with potential choices. These associations can be attained with reinforcement learning mechanisms using a reward prediction error (RPE) signal (the difference between actual and expected rewards) for updating future reward expectations. Despite an extensive body of literature on the influence of RPE on learning, little has been done to investigate the potentially separate contributions of RPE valence (positive or negative) and surprise (absolute degree of deviation from expectations). Here, we coupled single-trial electroencephalography with simultaneously acquired fMRI, during a probabilistic reversal-learning task, to offer evidence of temporally overlapping but largely distinct spatial representations of RPE valence and surprise. Electrophysiological variability in RPE valence correlated with activity in regions of the human reward network promoting approach or avoidance learning. Electrophysiological variability in RPE surprise correlated primarily with activity in regions of the human attentional network controlling the speed of learning. Crucially, despite the largely separate spatial extend of these representations our EEG-informed fMRI approach uniquely revealed a linear superposition of the two RPE components in a smaller network encompassing visuo-mnemonic and reward areas. Activity in this network was further predictive of stimulus value updating indicating a comparable contribution of both signals to reward learning.

A Selective Role for Dopamine in Learning to Maximize Reward But Not to Minimize Effort: Evidence from Patients with Parkinson's Disease

Instrumental learning is a fundamental process through which agents optimize their choices, taking into account various dimensions of available options such as the possible reward or punishment outcomes and the costs associated with potential actions. Although the implication of dopamine in learning from choice outcomes is well established, less is known about its role in learning the action costs such as effort. Here, we tested the ability of patients with Parkinson's disease (PD) to maximize monetary rewards and minimize physical efforts in a probabilistic instrumental learning task. The implication of dopamine was assessed by comparing performance ON and OFF prodopaminergic medication. In a first sample of PD patients (n = 15), we observed that reward learning, but not effort learning, was selectively impaired in the absence of treatment, with a significant interaction between learning condition (reward vs effort) and medication status (OFF vs ON). These results were replicated in a second, independent sample of PD patients (n = 20) using a simplified version of the task. According to Bayesian model selection, the best account for medication effects in both studies was a specific amplification of reward magnitude in a Q-learning algorithm. These results suggest that learning to avoid physical effort is independent from dopaminergic circuits and strengthen the general idea that dopaminergic signaling amplifies the effects of reward expectation or obtainment on instrumental behavior.SIGNIFICANCE STATEMENT Theoretically, maximizing reward and minimizing effort could involve the same computations and therefore rely on the same brain circuits. Here, we tested whether dopamine, a key component of reward-related circuitry, is also implicated in effort learning. We found that patients suffering from dopamine depletion due to Parkinson's disease were selectively impaired in reward learning, but not effort learning. Moreover, anti-parkinsonian medication restored the ability to maximize reward, but had no effect on effort minimization. This dissociation suggests that the brain has evolved separate, domain-specific systems for instrumental learning. These results help to disambiguate the motivational role of prodopaminergic medications: they amplify the impact of reward without affecting the integration of effort cost.

Reward Prediction Errors in Drug Addiction and Parkinson's Disease: from Neurophysiology to Neuroimaging

PURPOSE OF REVIEW

Surprises are important sources of learning. Cognitive scientists often refer to surprises as "reward prediction errors," a parameter that captures discrepancies between expectations and actual outcomes. Here, we integrate neurophysiological and functional magnetic resonance imaging (fMRI) results addressing the processing of reward prediction errors and how they might be altered in drug addiction and Parkinson's disease.

RECENT FINDINGS

By increasing phasic dopamine responses, drugs might accentuate prediction error signals, causing increases in fMRI activity in mesolimbic areas in response to drugs. Chronic substance dependence, by contrast, has been linked with compromised dopaminergic function, which might be associated with blunted fMRI responses to pleasant non-drug stimuli in mesocorticolimbic areas. In Parkinson's disease, dopamine replacement therapies seem to induce impairments in learning from negative outcomes. The present review provides a holistic overview of reward prediction errors across different pathologies and might inform future clinical strategies targeting impulsive/compulsive disorders.

Reward Processing, Neuroeconomics, and Psychopathology

Abnormal reward processing is a prominent transdiagnostic feature of psychopathology. The present review provides a framework for considering the different aspects of reward processing and their assessment, and highlights recent insights from the field of neuroeconomics that may aid in understanding these processes. Although altered reward processing in psychopathology has often been treated as a general hypo- or hyperresponsivity to reward, increasing data indicate that a comprehensive understanding of reward dysfunction requires characterization within more specific reward-processing domains, including subjective valuation, discounting, hedonics, reward anticipation and facilitation, and reinforcement learning. As such, more nuanced models of the nature of these abnormalities are needed. We describe several processing abnormalities capable of producing the types of selective alterations in reward-related behavior observed in different forms of psychopathology, including (mal)adaptive scaling and anchoring, dysfunctional weighting of reward and cost variables, competition between valuation systems, and reward prediction error signaling.

Increased fronto-striatal reward prediction errors moderate decision making in obsessive-compulsive disorder

BACKGROUND

Obsessive-compulsive disorder (OCD) has been linked to functional abnormalities in fronto-striatal networks as well as impairments in decision making and learning. Little is known about the neurocognitive mechanisms causing these decision-making and learning deficits in OCD, and how they relate to dysfunction in fronto-striatal networks.

METHOD

We investigated neural mechanisms of decision making in OCD patients, including early and late onset of disorder, in terms of reward prediction errors (RPEs) using functional magnetic resonance imaging. RPEs index a mismatch between expected and received outcomes, encoded by the dopaminergic system, and are known to drive learning and decision making in humans and animals. We used reinforcement learning models and RPE signals to infer the learning mechanisms and to compare behavioural parameters and neural RPE responses of the OCD patients with those of healthy matched controls.

RESULTS

Patients with OCD showed significantly increased RPE responses in the anterior cingulate cortex (ACC) and the putamen compared with controls. OCD patients also had a significantly lower perseveration parameter than controls.

CONCLUSIONS

Enhanced RPE signals in the ACC and putamen extend previous findings of fronto-striatal deficits in OCD. These abnormally strong RPEs suggest a hyper-responsive learning network in patients with OCD, which might explain their indecisiveness and intolerance of uncertainty.

Brain networks for confidence weighting and hierarchical inference during probabilistic learning

Learning is difficult when the world fluctuates randomly and ceaselessly. Classical learning algorithms, such as the delta rule with constant learning rate, are not optimal. Mathematically, the optimal learning rule requires weighting prior knowledge and incoming evidence according to their respective reliabilities. This "confidence weighting" implies the maintenance of an accurate estimate of the reliability of what has been learned. Here, using fMRI and an ideal-observer analysis, we demonstrate that the brain's learning algorithm relies on confidence weighting. While in the fMRI scanner, human adults attempted to learn the transition probabilities underlying an auditory or visual sequence, and reported their confidence in those estimates. They knew that these transition probabilities could change simultaneously at unpredicted moments, and therefore that the learning problem was inherently hierarchical. Subjective confidence reports tightly followed the predictions derived from the ideal observer. In particular, subjects managed to attach distinct levels of confidence to each learned transition probability, as required by Bayes-optimal inference. Distinct brain areas tracked the likelihood of new observations given current predictions, and the confidence in those predictions. Both signals were combined in the right inferior frontal gyrus, where they operated in agreement with the confidence-weighting model. This brain region also presented signatures of a hierarchical process that disentangles distinct sources of uncertainty. Together, our results provide evidence that the sense of confidence is an essential ingredient of probabilistic learning in the human brain, and that the right inferior frontal gyrus hosts a confidence-based statistical learning algorithm for auditory and visual sequences.

Beyond negative valence: 2-week administration of a serotonergic antidepressant enhances both reward and effort learning signals

To make good decisions, humans need to learn about and integrate different sources of appetitive and aversive information. While serotonin has been linked to value-based decision-making, its role in learning is less clear, with acute manipulations often producing inconsistent results. Here, we show that when the effects of a selective serotonin reuptake inhibitor (SSRI, citalopram) are studied over longer timescales, learning is robustly improved. We measured brain activity with functional magnetic resonance imaging (fMRI) in volunteers as they performed a concurrent appetitive (money) and aversive (effort) learning task. We found that 2 weeks of citalopram enhanced reward and effort learning signals in a widespread network of brain regions, including ventromedial prefrontal and anterior cingulate cortex. At a behavioral level, this was accompanied by more robust reward learning. This suggests that serotonin can modulate the ability to learn via a mechanism that is independent of stimulus valence. Such effects may partly underlie SSRIs’ impact in treating psychological illnesses. Our results highlight both a specific function in learning for serotonin and the importance of studying its role across longer timescales.

Drugs acting on the neurotransmitter serotonin in the brain are commonly prescribed to treat depression, but we still lack a complete understanding of their effects on the brain and behavior. We do, however, know that patients who suffer from depression learn about the links between their choices and pleasant and unpleasant outcomes in a different manner than healthy controls. Neural markers of learning are also weakened in depressed people. Here, we looked at the effects of a short-term course (2 weeks) of a serotonergic antidepressant on brain and behavior in healthy volunteers while they learnt to predict what consequences their choices had in a simple computer task. We found that the antidepressant increased how strongly brain areas concerned with predictions of pleasant and unpleasant consequences became active during learning of the task. At the same time, participants who had taken the antidepressant also performed better on the task. Our results suggest that serotonergic drugs might exert their beneficial clinical effects by changing how the brain learns.

Inflammation Effects on Motivation and Motor Activity: Role of Dopamine

Motivational and motor deficits are common in patients with depression and other psychiatric disorders, and are related to symptoms of anhedonia and motor retardation. These deficits in motivation and motor function are associated with alterations in corticostriatal neurocircuitry, which may reflect abnormalities in mesolimbic and mesostriatal dopamine (DA). One pathophysiologic pathway that may drive changes in DAergic corticostriatal circuitry is inflammation. Biomarkers of inflammation such as inflammatory cytokines and acute-phase proteins are reliably elevated in a significant proportion of psychiatric patients. A variety of inflammatory stimuli have been found to preferentially target basal ganglia function to lead to impaired motivation and motor activity. Findings have included inflammation-associated reductions in ventral striatal neural responses to reward anticipation, decreased DA and DA metabolites in cerebrospinal fluid, and decreased availability, and release of striatal DA, all of which correlated with symptoms of reduced motivation and/or motor retardation. Importantly, inflammation-associated symptoms are often difficult to treat, and evidence suggests that inflammation may decrease DA synthesis and availability, thus circumventing the efficacy of standard pharmacotherapies. This review will highlight the impact of administration of inflammatory stimuli on the brain in relation to motivation and motor function. Recent data demonstrating similar relationships between increased inflammation and altered DAergic corticostriatal circuitry and behavior in patients with major depressive disorder will also be presented. Finally, we will discuss the mechanisms by which inflammation affects DA neurotransmission and relevance to novel therapeutic strategies to treat reduced motivation and motor symptoms in patients with high inflammation.

Motivational Context Modulates Prediction Error Response in Schizophrenia

BACKGROUND

Recent findings demonstrate that patients with schizophrenia are worse at learning to predict rewards than losses, suggesting that motivational context modulates learning in this disease. However, these findings derive from studies in patients treated with antipsychotic medications, D2 receptor antagonists that may interfere with the neural systems that underlie motivation and learning. Thus, it remains unknown how motivational context affects learning in schizophrenia, separate from the effects of medication.

METHODS

To examine the impact of motivational context on learning in schizophrenia, we tested 16 unmedicated patients with schizophrenia and 23 matched controls on a probabilistic learning task while they underwent functional magnetic resonance imaging (fMRI) under 2 conditions: one in which they pursued rewards, and one in which they avoided losses. Computational models were used to derive trial-by-trial prediction error responses to feedback.

RESULTS

Patients performed worse than controls on the learning task overall, but there were no behavioral effects of condition. FMRI revealed an attenuated prediction error response in patients in the medial prefrontal cortex, striatum, and medial temporal lobe when learning to predict rewards, but not when learning to avoid losses.

CONCLUSIONS

Patients with schizophrenia showed differences in learning-related brain activity when learning to predict rewards, but not when learning to avoid losses. Together with prior work, these results suggest that motivational deficits related to learning in schizophrenia are characteristic of the disease and not solely a result of antipsychotic treatment.

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

Dopamine (DA) has been suggested to have two reward-related roles: (1) representing reward-prediction-error (RPE), and (2) providing motivational drive. Role(1) is based on the physiological results that DA responds to unpredicted but not predicted reward, whereas role(2) is supported by the pharmacological results that blockade of DA signaling causes motivational impairments such as slowdown of self-paced behavior. So far, these two roles are considered to be played by two different temporal patterns of DA signals: role(1) by phasic signals and role(2) by tonic/sustained signals. However, recent studies have found sustained DA signals with features indicative of both roles (1) and (2), complicating this picture. Meanwhile, whereas synaptic/circuit mechanisms for role(1), i.e., how RPE is calculated in the upstream of DA neurons and how RPE-dependent update of learned-values occurs through DA-dependent synaptic plasticity, have now become clarified, mechanisms for role(2) remain unclear. In this work, we modeled self-paced behavior by a series of ‘Go’ or ‘No-Go’ selections in the framework of reinforcement-learning assuming DA's role(1), and demonstrated that incorporation of decay/forgetting of learned-values, which is presumably implemented as decay of synaptic strengths storing learned-values, provides a potential unified mechanistic account for the DA's two roles, together with its various temporal patterns.

Amygdala and Ventral Striatum Make Distinct Contributions to Reinforcement Learning

Reinforcement learning (RL) theories posit that dopaminergic signals are integrated within the striatum to associate choices with outcomes. Often overlooked is that the amygdala also receives dopaminergic input and is involved in Pavlovian processes that influence choice behavior. To determine the relative contributions of the ventral striatum (VS) and amygdala to appetitive RL, we tested rhesus macaques with VS or amygdala lesions on deterministic and stochastic versions of a two-arm bandit reversal learning task. When learning was characterized with an RL model relative to controls, amygdala lesions caused general decreases in learning from positive feedback and choice consistency. By comparison, VS lesions only affected learning in the stochastic task. Moreover, the VS lesions hastened the monkeys' choice reaction times, which emphasized a speed-accuracy trade-off that accounted for errors in deterministic learning. These results update standard accounts of RL by emphasizing distinct contributions of the amygdala and VS to RL.

Taming the beast: extracting generalizable knowledge from computational models of cognition

Generalizing knowledge from experimental data requires constructing theories capable of explaining observations and extending beyond them. Computational modeling offers formal quantitative methods for generating and testing theories of cognition and neural processing. These techniques can be used to extract general principles from specific experimental measurements, but introduce dangers inherent to theory: model-based analyses are conditioned on a set of fixed assumptions that impact the interpretations of experimental data. When these conditions are not met, model-based results can be misleading or biased. Recent work in computational modeling has highlighted the implications of this problem and developed new methods for minimizing its negative impact. Here we discuss the issues that arise when data is interpreted through models and strategies for avoiding misinterpretation of data through model fitting.

Shared Neural Mechanisms for the Evaluation of Intense Sensory Stimulation and Economic Reward, Dependent on Stimulation-Seeking Behavior

Why are some people strongly motivated by intense sensory experiences? Here we investigated how people encode the value of an intense sensory experience compared with economic reward, and how this varies according to stimulation-seeking preference. Specifically, we used a novel behavioral task in combination with computational modeling to derive the value individuals assigned to the opportunity to experience an intense tactile stimulus (mild electric shock). We then examined functional imaging data recorded during task performance to see how the opportunity to experience the sensory stimulus was encoded in stimulation-seekers versus stimulation-avoiders. We found that for individuals who positively sought out this kind of sensory stimulation, there was common encoding of anticipated economic and sensory rewards in the ventromedial prefrontal cortex. Conversely, there was robust encoding of the modeled probability of receiving such stimulation in the insula only in stimulation-avoidant individuals. Finally, we found preliminary evidence that sensory prediction error signals may be positively signed for stimulation-seekers, but negatively signed for stimulation-avoiders, in the posterior cingulate cortex. These findings may help explain why high intensity sensory experiences are appetitive for some individuals, but not for others, and may have relevance for the increased vulnerability for some psychopathologies, but perhaps increased resilience for others, in high sensation-seeking individuals.

SIGNIFICANCE STATEMENT

People vary in their preference for intense sensory experiences. Here, we investigated how different individuals evaluate the prospect of an unusual sensory experience (electric shock), compared with the opportunity to gain a more traditional reward (money). We found that in a subset of individuals who sought out such unusual sensory stimulation, anticipation of the sensory outcome was encoded in the same way as that of monetary gain, in the ventromedial prefrontal cortex. Further understanding of stimulation-seeking behavior may shed light on the etiology of psychopathologies such as addiction, for which high or low sensation-seeking personality has been identified as a risk factor.

Auditory Hallucinations and the Brain's Resting-State Networks: Findings and Methodological Observations

In recent years, there has been increasing interest in the potential for alterations to the brain's resting-state networks (RSNs) to explain various kinds of psychopathology. RSNs provide an intriguing new explanatory framework for hallucinations, which can occur in different modalities and population groups, but which remain poorly understood. This collaboration from the International Consortium on Hallucination Research (ICHR) reports on the evidence linking resting-state alterations to auditory hallucinations (AH) and provides a critical appraisal of the methodological approaches used in this area. In the report, we describe findings from resting connectivity fMRI in AH (in schizophrenia and nonclinical individuals) and compare them with findings from neurophysiological research, structural MRI, and research on visual hallucinations (VH). In AH, various studies show resting connectivity differences in left-hemisphere auditory and language regions, as well as atypical interaction of the default mode network and RSNs linked to cognitive control and salience. As the latter are also evident in studies of VH, this points to a domain-general mechanism for hallucinations alongside modality-specific changes to RSNs in different sensory regions. However, we also observed high methodological heterogeneity in the current literature, affecting the ability to make clear comparisons between studies. To address this, we provide some methodological recommendations and options for future research on the resting state and hallucinations.

Perimovement decrease of alpha/beta oscillations in the human nucleus accumbens

The present work clarifies how the nucleus accumbens contributes to action. This region is often assumed to influence behavior “off-line” by evaluating outcomes. Studying rare recordings of local field potentials from the human nucleus accumbens, we observe a perimovement decrease of alpha and beta oscillations in seven of eight individuals, a signal that, in the motor system, is directly related to action preparation. Our results support the idea of an online role of this region for imminent action.

The human nucleus accumbens is thought to play an important role in guiding future action selection via an evaluation of current action outcomes. Here we provide electrophysiological evidence for a more direct, i.e., online, role during action preparation. We recorded local field potentials from the nucleus accumbens in patients with epilepsy undergoing surgery for deep brain stimulation. We found a consistent decrease in the power of alpha/beta oscillations (10–30 Hz) before and around the time of movements. This perimovement alpha/beta desynchronization was observed in seven of eight patients and was present both before instructed movements in a serial reaction time task as well as before self-paced, deliberate choices in a decision making task. A similar beta decrease over sensorimotor cortex and in the subthalamic nucleus has been directly related to movement preparation and execution. Our results support the idea of a direct role of the human nucleus accumbens in action preparation and execution.

Effects of early life stress on amygdala and striatal development

Species-expected caregiving early in life is critical for the normative development and regulation of emotional behavior, the ability to effectively evaluate affective stimuli in the environment, and the ability to sustain social relationships. Severe psychosocial stressors early in life (early life stress; ELS) in the form of the absence of species expected caregiving (i.e., caregiver deprivation), can drastically impact one's social and emotional success, leading to the onset of internalizing illness later in life. Development of the amygdala and striatum, two key regions supporting affective valuation and learning, is significantly affected by ELS, and their altered developmental trajectories have important implications for cognitive, behavioral and socioemotional development. However, an understanding of the impact of ELS on the development of functional interactions between these regions and subsequent behavioral effects is lacking. In this review, we highlight the roles of the amygdala and striatum in affective valuation and learning in maturity and across development. We discuss their function separately as well as their interaction. We highlight evidence across species characterizing how ELS induced changes in the development of the amygdala and striatum mediate subsequent behavioral changes associated with internalizing illness, positing a particular import of the effect of ELS on their interaction.

Over a Decade of Neuroeconomics: What Have We Learned?

At its inception, neuroeconomics promised to revolutionize economics. That promise has not yet been realized, and neuroeconomics has seen limited penetration into mainstream economics. Nevertheless, it would be a mistake to declare that neuroeconomics has failed. Quite to the contrary, the yearly rate of neuroeconomics papers has roughly doubled since 2005. While the number of direct applications to economics remains limited, due to the infancy of the field, we have learned an amazing amount about how the brain makes decisions. In this article, we review some of the major topics that have emerged in neuroeconomics and highlight findings that we believe will form the basis for future applications to economics. When possible, we focus on existing applications to economics and future directions for that research.

Single subject analyses reveal consistent recruitment of frontal operculum in performance monitoring

There are continuing uncertainties regarding whether performance monitoring recruits the anterior insula (aI) and/or the frontal operculum (fO). The proximity and morphological complexity of these two regions make proper identification and isolation of the loci of activation extremely difficult. The use of group averaging methods in human neuroimaging might contribute to this problem. The result has been heterogeneous labeling of this region as aI, fO, or aI/fO, and a discussion of results oriented towards either cognitive or interoceptive functions depending on labeling. In the present article, we adapted the spatial preprocessing of functional magnetic resonance imaging data to account for group averaging artifacts and performed a subject-by-subject analysis in three performance monitoring tasks. Results show that functional activity related to feedback or action monitoring consistently follows local morphology in this region and demonstrate that the activity is located predominantly in the fO rather than in the aI. From these results, we propose that a full understanding of the respective role of aI and fO would benefit from increased spatial resolution and subject-by-subject analysis.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Computational Psychiatry of ADHD: Neural Gain Impairments across Marrian Levels of Analysis

Attention-deficit hyperactivity disorder (ADHD), one of the most common psychiatric disorders, is characterised by unstable response patterns across multiple cognitive domains. However, the neural mechanisms that explain these characteristic features remain unclear. Using a computational multilevel approach, we propose that ADHD is caused by impaired gain modulation in systems that generate this phenotypic increased behavioural variability. Using Marr's three levels of analysis as a heuristic framework, we focus on this variable behaviour, detail how it can be explained algorithmically, and how it might be implemented at a neural level through catecholamine influences on corticostriatal loops. This computational, multilevel, approach to ADHD provides a framework for bridging gaps between descriptions of neuronal activity and behaviour, and provides testable predictions about impaired mechanisms.

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

The good, the bad and the brain: Neural correlates of appetitive and aversive values underlying decision making

Approaching rewards and avoiding punishments could be considered as core principles governing behavior. Experiments from behavioral economics have shown that choices involving gains and losses follow different policy rules, suggesting that appetitive and aversive processes might rely on different brain systems. Here we contrast this hypothesis with recent neuroscience studies exploring the human brain from brainstem nuclei to cortical areas. Although some circuits show rigid specialization, many others appear to process both appetitive and aversive stimuli, in a flexible manner that depends on a context-wise subjective reference point. Moreover, appetitive and aversive aspects are often integrated into net values that are signaled with enhanced activity in 'positive regions', and suppressed activity in 'negative regions'. This dichotomy might explain why drugs or lesions can produce valence-specific effects, biasing decisions towards approaching a reward or avoiding a punishment.

Using Intermediate Cognitive Endpoints to Facilitate Translational Research in Psychosis

Recent advances in the understanding of psychosis have uncovered potential for a paradigm shift in related drug discovery efforts. The study of psychosis is evolving from its origins in serendipity and empiricism to more formal, hypothesis driven accounts of the cognitive substrates underlying hallucinations and delusions. Recent evidence suggests that misattribution of salience and abnormal prediction error might underlie some forms of psychosis. If substantiated, such intermediate constructs could significantly facilitate translational research for drug discovery. Aberrant salience and prediction error can be assayed with simple tests of associative learning in both species, and a convincing back translation of effects, when combined with measures of neurotransmitter release and brain activity could for the first time allow robust, causal connections to be made between molecular mechanisms in rodents and symptoms in patients.

No unified reward prediction error in local field potentials from the human nucleus accumbens: evidence from epilepsy patients

Functional magnetic resonance imaging (fMRI), cyclic voltammetry, and single-unit electrophysiology studies suggest that signals measured in the nucleus accumbens (Nacc) during value-based decision making represent reward prediction errors (RPEs), the difference between actual and predicted rewards. Here, we studied the precise temporal and spectral pattern of reward-related signals in the human Nacc. We recorded local field potentials (LFPs) from the Nacc of six epilepsy patients during an economic decision-making task. On each trial, patients decided whether to accept or reject a gamble with equal probabilities of a monetary gain or loss. The behavior of four patients was consistent with choices being guided by value expectations. Expected value signals before outcome onset were observed in three of those patients, at varying latencies and with nonoverlapping spectral patterns. Signals after outcome onset were correlated with RPE regressors in all subjects. However, further analysis revealed that these signals were better explained as outcome valence rather than RPE signals, with gamble gains and losses differing in the power of beta oscillations and in evoked response amplitudes. Taken together, our results do not support the idea that postsynaptic potentials in the Nacc represent a RPE that unifies outcome magnitude and prior value expectation. We discuss the generalizability of our findings to healthy individuals and the relation of our results to measurements of RPE signals obtained from the Nacc with other methods.

Expectation modulates neural representations of valence throughout the human brain

The brain's sensitivity to unexpected gains or losses plays an important role in our ability to learn new behaviors (Rescorla and Wagner, 1972; Sutton and Barto, 1990). Recent work suggests that gains and losses are ubiquitously encoded throughout the human brain (Vickery et al., 2011), however, the extent to which reward expectation modulates these valence representations is not known. To address this question, we analyzed recordings from 4306 intracranially implanted electrodes in 39 neurosurgical patients as they performed a two-alternative probability learning task. Using high-frequency activity (HFA, 70-200 Hz) as an indicator of local firing rates, we found that expectation modulated reward-related neural activity in widespread brain regions, including regions that receive sparse inputs from midbrain dopaminergic neurons. The strength of unexpected gain signals predicted subjects' abilities to encode stimulus-reward associations. Thus, neural signals that are functionally related to learning are widely distributed throughout the human brain.

Anticipatory reward processing in addicted populations: a focus on the monetary incentive delay task

Advances in brain imaging techniques have allowed neurobiological research to temporally analyze signals coding for the anticipation of reward. In addicted populations, both hyporesponsiveness and hyperresponsiveness of brain regions (e.g., ventral striatum) implicated in drug effects and reward system processing have been reported during anticipation of generalized reward. We discuss the current state of knowledge of reward processing in addictive disorders from a widely used and validated task: the monetary incentive delay task. Only studies applying the monetary incentive delay task in addicted and at-risk adult populations are reviewed, with a focus on anticipatory processing and striatal regions activated during task performance as well as the relationship of these regions with individual difference (e.g., impulsivity) and treatment outcome variables. We further review drug influences in challenge studies as a means to examine acute influences on reward processing in abstinent, recreationally using, and addicted populations. Generalized reward processing in addicted and at-risk populations is often characterized by divergent anticipatory signaling in the ventral striatum. Although methodologic and task variations may underlie some discrepant findings, anticipatory signaling in the ventral striatum may also be influenced by smoking status, drug metabolites, and treatment status in addicted populations. Divergent results across abstinent, recreationally using, and addicted populations demonstrate complexities in interpreting findings. Future studies would benefit from focusing on characterizing how impulsivity and other addiction-related features relate to anticipatory striatal signaling over time. Additionally, identifying how anticipatory signals recover or adjust after protracted abstinence will be important in understanding recovery processes.