Dorsal striatal-midbrain connectivity in humans predicts how reinforcements are used to guide decisions

Test-retest reliability of reinforcement learning parameters

It has recently been suggested that parameter estimates of computational models can be used to understand individual differences at the process level. One area of research in which this approach, called computational phenotyping, has taken hold is computational psychiatry. One requirement for successful computational phenotyping is that behavior and parameters are stable over time. Surprisingly, the test-retest reliability of behavior and model parameters remains unknown for most experimental tasks and models. The present study seeks to close this gap by investigating the test-retest reliability of canonical reinforcement learning models in the context of two often-used learning paradigms: a two-armed bandit and a reversal learning task. We tested independent cohorts for the two tasks (N = 69 and N = 47) via an online testing platform with a between-test interval of five weeks. Whereas reliability was high for personality and cognitive measures (with ICCs ranging from .67 to .93), it was generally poor for the parameter estimates of the reinforcement learning models (with ICCs ranging from .02 to .52 for the bandit task and from .01 to .71 for the reversal learning task). Given that simulations indicated that our procedures could detect high test-retest reliability, this suggests that a significant proportion of the variability must be ascribed to the participants themselves. In support of that hypothesis, we show that mood (stress and happiness) can partly explain within-participant variability. Taken together, these results are critical for current practices in computational phenotyping and suggest that individual variability should be taken into account in the future development of the field.

Observational reinforcement learning in children and young adults

Observational learning is essential for the acquisition of new behavior in educational practices and daily life and serves as an important mechanism for human cognitive and social-emotional development. However, we know little about its underlying neurocomputational mechanisms from a developmental perspective. In this study we used model-based fMRI to investigate differences in observational learning and individual learning between children and younger adults. Prediction errors (PE), the difference between experienced and predicted outcomes, related positively to striatal and ventral medial prefrontal cortex activation during individual learning and showed no age-related differences. PE-related activation during observational learning was more pronounced when outcomes were worse than predicted. Particularly, negative PE-coding in the dorsal medial prefrontal cortex was stronger in adults compared to children and was associated with improved observational learning in children and adults. The current findings pave the way to better understand observational learning challenges across development and educational settings.

Dynamics Learning Rate Bias in Pigeons: Insights from Reinforcement Learning and Neural Correlates

Research in reinforcement learning indicates that animals respond differently to positive and negative reward prediction errors, which can be calculated by assuming learning rate bias. Many studies have shown that humans and other animals have learning rate bias during learning, but it is unclear whether and how the bias changes throughout the entire learning process. Here, we recorded the behavior data and the local field potentials (LFPs) in the striatum of five pigeons performing a probabilistic learning task. Reinforcement learning models with and without learning rate biases were used to dynamically fit the pigeons' choice behavior and estimate the option values. Furthemore, the correlation between the striatal LFPs power and the model-estimated option values was explored. We found that the pigeons' learning rate bias shifted from negative to positive during the learning process, and the striatal Gamma (31 to 80 Hz) power correlated with the option values modulated by dynamic learning rate bias. In conclusion, our results support the hypothesis that pigeons employ a dynamic learning strategy in the learning process from both behavioral and neural aspects, providing valuable insights into reinforcement learning mechanisms of non-human animals.

Translational models of addiction phenotypes to advance addiction pharmacotherapy

Alcohol and substance use disorders are heterogeneous conditions with limited effective treatment options. While there have been prior attempts to classify addiction subtypes, they have not been translated into clinical practice. In an effort to better understand heterogeneity in psychiatric disorders, the National Institute for Mental Health Research Domain Criteria (RDoC) has challenged scientists to think beyond diagnostic symptoms and to consider the underlying features of psychopathology from a neuroscience-based framework. The field of addiction has grappled with this approach by considering several key constructs with the potential to capture RDoC domains. This critical review will focus on the efforts to apply translational models of addiction phenomenology in human clinical samples, including their relative strengths and weaknesses. Opportunities for forward and reverse translation are also discussed. Deep behavioral phenotyping using neuroscience-informed batteries shows promise for a better understanding of the clinical neuroscience of addiction and advancing precision medicine for alcohol and substance use disorders.

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

Learning under social versus nonsocial uncertainty: A meta-analytic approach

Much of the uncertainty that clouds our understanding of the world springs from the covert values and intentions held by other people. Thus, it is plausible that specialized mechanisms that compute learning signals under uncertainty of exclusively social origin operate in the brain. To test this hypothesis, we scoured academic databases for neuroimaging studies involving learning under uncertainty, and performed a meta‐analysis of brain activation maps that compared learning in the face of social versus nonsocial uncertainty. Although most of the brain activations associated with learning error signals were shared between social and nonsocial conditions, we found some evidence for functional segregation of error signals of exclusively social origin during learning in limited regions of ventrolateral prefrontal cortex and insula. This suggests that most behavioral adaptations to navigate social environments are reused from frontal and subcortical areas processing generic value representation and learning, but that a specialized circuitry might have evolved in prefrontal regions to deal with social context representation and strategic action.

The computational roots of positivity and confirmation biases in reinforcement learning

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Neurocomputational mechanisms of adaptive learning in social exchanges

Prior work on prosocial and self-serving behavior in human economic exchanges has shown that counterparts' high social reputations bias striatal reward signals and elicit cooperation, even when such cooperation is disadvantageous. This phenomenon suggests that the human striatum is modulated by the other's social value, which is insensitive to the individual's own choices to cooperate or defect. We tested an alternative hypothesis that, when people learn from their interactions with others, they encode prediction error updates with respect to their own policy. Under this policy update account striatal signals would reflect positive prediction errors when the individual's choices correctly anticipated not only the counterpart's cooperation but also defection. We examined behavior in three samples using reinforcement learning and model-free analyses and performed an fMRI study of striatal learning signals. In order to uncover the dynamics of goal-directed learning, we introduced reversals in the counterpart's behavior and provided counterfactual (would-be) feedback when the individual chose not to engage with the counterpart. Behavioral data and model-derived prediction error maps (in both whole-brain and a priori striatal region of interest analyses) supported the policy update model. Thus, as people continually adjust their rate of cooperation based on experience, their behavior and striatal learning signals reveal a self-centered instrumental process corresponding to reciprocal altruism.

Cross-Task Contributions of Frontobasal Ganglia Circuitry in Response Inhibition and Conflict-Induced Slowing

Why are we so slow in choosing the lesser of 2 evils? We considered whether such slowing relates to uncertainty about the value of these options, which arises from the tendency to avoid them during learning, and whether such slowing relates to frontosubthalamic inhibitory control mechanisms. In total, 49 participants performed a reinforcement-learning task and a stop-signal task while fMRI was recorded. A reinforcement-learning model was used to quantify learning strategies. Individual differences in lose-lose slowing related to information uncertainty due to sampling, and independently, to less efficient response inhibition in the stop-signal task. Neuroimaging analysis revealed an analogous dissociation: subthalamic nucleus (STN) BOLD activity related to variability in stopping latencies, whereas weaker frontosubthalamic connectivity related to slowing and information sampling. Across tasks, fast inhibitors increased STN activity for successfully canceled responses in the stop task, but decreased activity for lose-lose choices. These data support the notion that fronto-STN communication implements a rapid but transient brake on response execution, and that slowing due to decision uncertainty could result from an inefficient release of this "hold your horses" mechanism.

Learning in Visual Regions as Support for the Bias in Future Value-Driven Choice

Reinforcement learning can bias decision-making toward the option with the highest expected outcome. Cognitive learning theories associate this bias with the constant tracking of stimulus values and the evaluation of choice outcomes in the striatum and prefrontal cortex. Decisions however first require processing of sensory input, and to date, we know far less about the interplay between learning and perception. This functional magnetic resonance imaging study (N = 43) relates visual blood oxygen level-dependent (BOLD) responses to value beliefs during choice and signed prediction errors after outcomes. To understand these relationships, which co-occurred in the striatum, we sought relevance by evaluating the prediction of future value-based decisions in a separate transfer phase where learning was already established. We decoded choice outcomes with a 70% accuracy with a supervised machine learning algorithm that was given trial-by-trial BOLD from visual regions alongside more traditional motor, prefrontal, and striatal regions. Importantly, this decoding of future value-driven choice outcomes again highlighted an important role for visual activity. These results raise the intriguing possibility that the tracking of value in visual cortex is supportive for the striatal bias toward the more valued option in future choice.

White matter integrity alternations associated with cocaine dependence and long-term abstinence: Preliminary findings

Cocaine dependence has been associated with deficits in white matter (WM) integrity. Nevertheless, what happens to WM integrity after long-term abstinence is not fully understood. To bridge this gap, changes in WM integrity were examined with diffusion tensor imaging (DTI) applied to 39 participants: 12 participants who used cocaine in the last year (CURRENT USERS), 20 who were at different stages of cocaine abstinence (ABSTINENCE) [five with 1-5 years of abstinence (ABS1), five with 6-10 years of abstinence (ABS2), and 10 with over 10 years of abstinence (ABS3)], and 7 healthy controls (CONTROLS). The CONTROL group had higher fractional anisotropy (FA) compared to CURRENT USERS in frontal cortex tracts, including the bilateral corpus callosum, bilateral superior longitudinal fasciculus, bilateral inferior fronto-occipital fasciculus, left internal capsule, left middle cingulum, and left ventral and dorsal medial frontal regions. The ABSTINENCE group also had higher FA compared to CURRENT USERS in frontal cortex tracts, such as the bilateral corpus callosum, bilateral superior longitudinal fasciculus, left inferior longitudinal fasciculus, left uncinate fasciculus, left inferior fronto-occipital fasciculus, and the left ventral and dorsal medial frontal regions. Tractography analysis showed (1) deficits in terms of number of fibers and fiber length in these regions, and that (2) while there was some recovery of white matter in dorsolateral regions during abstinence, duration of abstinence was not associated with such recovery. The results identified WM differences among cocaine users, cocaine abstinent participants, and controls. These preliminary findings point to WM tracts that recover, and some that do not, after long-term abstinence from cocaine.

Spontaneous eye blink rate predicts individual differences in exploration and exploitation during reinforcement learning

Spontaneous eye blink rate (sEBR) has been linked to striatal dopamine function and to how individuals make value-based choices after a period of reinforcement learning (RL). While sEBR is thought to reflect how individuals learn from the negative outcomes of their choices, this idea has not been tested explicitly. This study assessed how individual differences in sEBR relate to learning by focusing on the cognitive processes that drive RL. Using Bayesian latent mixture modelling to quantify the mapping between RL behaviour and its underlying cognitive processes, we were able to differentiate low and high sEBR individuals at the level of these cognitive processes. Further inspection of these cognitive processes indicated that sEBR uniquely indexed explore-exploit tendencies during RL: lower sEBR predicted exploitative choices for high valued options, whereas higher sEBR predicted exploration of lower value options. This relationship was additionally supported by a network analysis where, notably, no link was observed between sEBR and how individuals learned from negative outcomes. Our findings challenge the notion that sEBR predicts learning from negative outcomes during RL, and suggest that sEBR predicts individual explore-exploit tendencies. These then influence value sensitivity during choices to support successful performance when facing uncertain reward.

Dopaminergic medication reduces striatal sensitivity to negative outcomes in Parkinson's disease

Reduced levels of dopamine in Parkinson's disease contribute to changes in learning, resulting from the loss of midbrain neurons that transmit a dopaminergic teaching signal to the striatum. Dopamine medication used by patients with Parkinson's disease has previously been linked to behavioural changes during learning as well as to adjustments in value-based decision-making after learning. To date, however, little is known about the specific relationship between dopaminergic medication-driven differences during learning and subsequent changes in approach/avoidance tendencies in individual patients. Twenty-four Parkinson's disease patients ON and OFF dopaminergic medication and 24 healthy controls subjects underwent functional MRI while performing a probabilistic reinforcement learning experiment. During learning, dopaminergic medication reduced an overemphasis on negative outcomes. Medication reduced negative (but not positive) outcome learning rates, while concurrent striatal blood oxygen level-dependent responses showed reduced prediction error sensitivity. Medication-induced shifts in negative learning rates were predictive of changes in approach/avoidance choice patterns after learning, and these changes were accompanied by systematic striatal blood oxygen level-dependent response alterations. These findings elucidate the role of dopamine-driven learning differences in Parkinson's disease, and show how these changes during learning impact subsequent value-based decision-making.

Altered monetary loss processing and reinforcement-based learning in individuals with obesity

Individuals with obesity are often characterized by alterations in reward processing. This may affect how new information is used to update stimulus values during reinforcement-based learning. Here, we investigated obesity-related changes in non-food reinforcement processing, their impact on learning performance as well as the neural underpinnings of reinforcement-based learning in obesity. Nineteen individuals with obesity (BMI > = 30 kg/m², 10 female) and 23 lean control participants (BMI 18.5–24.9 kg/m², 11 female) performed a probabilistic learning task during functional magnetic resonance imaging (fMRI), in which they learned to choose between advantageous and disadvantageous choice options in separate monetary gain, loss, and neutral conditions. During learning individuals with obesity made a significantly lower number of correct choices and accumulated a significantly lower overall monetary outcome than lean control participants. FMRI analyses revealed aberrant medial prefrontal cortex responses to monetary losses in individuals with obesity. There were no significant group differences in the regional representation of prediction errors. However, we found evidence for increased functional connectivity between the ventral striatum and insula in individuals with obesity. The present results suggest that obesity is associated with aberrant value representations for monetary losses, alterations in functional connectivity during the processing of learning outcomes, as well as a decresased reinforcement-based learning performance. This may affect how new information is incorporated to adjust dysfunctional behavior and could be a factor contributing to the maintenance of dysfunctional eating behavior in obesity.

Narcissism and risky decisions: a neurophysiological approach

Narcissists are prone to risky decision-making, but why? This study tested—via behavioral and event-related potential (ERP) measures—two accounts: deficiencies in error monitoring and deficiencies in action updating. High and low narcissists were engaged in a monetary gambling task by choosing between a high-risk and a low-risk option while the electroencephalogram (EEG) was being recorded. Two ERP components relevant to outcome evaluation—feedback-related negativity (FRN) and P3—were analyzed, with the FRN serving as an index of error monitoring and the P3 as an index of action updating. Generally, high and low narcissists differed in the high-risk condition but not in the low-risk condition. At the behavioral level, high (vs low) narcissists made riskier decisions following high-risk decision outcomes, which was in line with past findings; at the neurophysiological level, while no FRN difference emerged between high and low narcissists, the outcome valence effect (positive vs negative) on the P3 was stronger among low narcissists than high narcissists following high-risk decision outcomes. One possible interpretation of the results is that narcissism is associated with reduced action updating. The findings contribute to the understanding of narcissistic decision-making and self-regulation.

The computational basis of following advice in adolescents

Advice taking helps one to quickly acquire knowledge and make decisions. This age-comparative study (in children [8- to 10-year-olds], adolescents [13- to 15-year-olds], and adults [18- to 22-year-olds]) investigated developmental differences in how advice, experience, and exploration influence learning. The results showed that adolescents were initially easily swayed to follow peer advice but also switched more rapidly to exploring alternatives like children. Whereas adults stayed with the advice over the task, adolescents put more weight on their own experience compared with adults. A social learning model showed that although social influence most strongly affects adolescents' initial expectations (i.e., their priors), adolescents showed higher exploration and discovered the other good option in the current task. Thus, our model resolved the apparently conflicting findings of adolescents being more and less sensitive to peer influence and provides novel insights into the dynamic interaction between social and individual learning.

How pupil responses track value-based decision-making during and after reinforcement learning

Cognition can reveal itself in the pupil, as latent cognitive processes map onto specific pupil responses. For instance, the pupil dilates when we make decisions and these pupil size fluctuations reflect decision-making computations during and after a choice. Surprisingly little is known, however, about how pupil responses relate to decisions driven by the learned value of stimuli. This understanding is important, as most real-life decisions are guided by the outcomes of earlier choices. The goal of this study was to investigate which cognitive processes the pupil reflects during value-based decision-making. We used a reinforcement learning task to study pupil responses during value-based decisions and subsequent decision evaluations, employing computational modeling to quantitatively describe the underlying cognitive processes. We found that the pupil closely tracks reinforcement learning processes independently across participants and across trials. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs about the to-be chosen option and predicted an individual's tendency to exploit high value options. After feedback a biphasic pupil response was observed, the amplitude of which correlated with participants' learning rates. Furthermore, across trials, early feedback-related dilation scaled with value uncertainty, whereas later constriction scaled with signed reward prediction errors. These findings show that pupil size fluctuations can provide detailed information about the computations underlying value-based decisions and the subsequent updating of value beliefs. As these processes are affected in a host of psychiatric disorders, our results indicate that pupillometry can be used as an accessible tool to non-invasively study the processes underlying ongoing reinforcement learning in the clinic.

Reinforcement magnitudes modulate subthalamic beta band activity in patients with Parkinson's disease

We set out to investigate whether beta oscillations in the human basal ganglia are modulated during reinforcement learning. Based on previous research, we assumed that beta activity might either reflect the magnitudes of individuals' received reinforcements (reinforcement hypothesis), their reinforcement prediction errors (dopamine hypothesis) or their tendencies to repeat versus adapt responses based upon reinforcements (status-quo hypothesis). We tested these hypotheses by recording local field potentials (LFPs) from the subthalamic nuclei of 19 Parkinson's disease patients engaged in a reinforcement-learning paradigm. We then correlated patients' reinforcement magnitudes, reinforcement prediction errors and response repetition tendencies with task-related power changes in their LFP oscillations. During feedback presentation, activity in the frequency range of 14 to 27 Hz (beta spectrum) correlated positively with reinforcement magnitudes. During responding, alpha and low beta activity (6 to 18 Hz) was negatively correlated with previous reinforcement magnitudes. Reinforcement prediction errors and response repetition tendencies did not correlate significantly with LFP oscillations. These results suggest that alpha and beta oscillations during reinforcement learning reflect patients' observed reinforcement magnitudes, rather than their reinforcement prediction errors or their tendencies to repeat versus adapt their responses, arguing both against an involvement of phasic dopamine and against applicability of the status-quo theory.

Impaired reward prediction error encoding and striatal-midbrain connectivity in depression

Anhedonia (hyposensitivity to rewards) and negative bias (hypersensitivity to punishments) are core features of major depressive disorder (MDD), which could stem from abnormal reinforcement learning. Emerging evidence highlights blunted reward learning and reward prediction error (RPE) signaling in the striatum in MDD, although inconsistencies exist. Preclinical studies have clarified that ventral tegmental area (VTA) neurons encode RPE and habenular neurons encode punishment prediction error (PPE), which are then transmitted to the striatum and cortex to guide goal-directed behavior. However, few studies have probed striatal activation, and functional connectivity between VTA-striatum and VTA-habenula during reward and punishment learning respectively, in unmedicated MDD. To fill this gap, we acquired fMRI data from 25 unmedicated MDD and 26 healthy individuals during a monetary instrumental learning task and utilized a computational modeling approach to characterize underlying neural correlates of RPE and PPE. Relative to controls, MDD individuals showed impaired reward learning, blunted RPE signal in the striatum and overall reduced VTA-striatal connectivity to feedback. Critically, striatal RPE signal was increasingly blunted with more major depressive episodes (MDEs). No group differences emerged in PPE signals in the habenula and VTA or in connectivity between these regions. However, PPE signals in the habenula correlated positively with number of MDEs. These results highlight impaired reward learning, disrupted RPE signaling in the striatum (particularly among individuals with more lifetime MDEs) as well as reduced VTA-striatal connectivity in MDD. Collectively, these findings highlight reward-related learning deficits in MDD and their underlying pathophysiology.

Sensitivity to value-driven attention is predicted by how we learn from value

Reward learning is known to influence the automatic capture of attention. This study examined how the rate of learning, after high- or low-value reward outcomes, can influence future transfers into value-driven attentional capture. Participants performed an instrumental learning task that was directly followed by an attentional capture task. A hierarchical Bayesian reinforcement model was used to infer individual differences in learning from high or low reward. Results showed a strong relationship between high-reward learning rates (or the weight that is put on learning after a high reward) and the magnitude of attentional capture with high-reward colors. Individual differences in learning from high or low rewards were further related to performance differences when high- or low-value distractors were present. These findings provide novel insight into the development of value-driven attentional capture by showing how information updating after desired or undesired outcomes can influence future deployments of automatic attention.

White matter disruptions in male cocaine polysubstance users: Associations with severity of drug use and duration of abstinence

BACKGROUND

Cocaine dependence has been associated with alterations in the brain's white matter integrity, yet relevant questions remain about what alterations are linked to cocaine use and/or polysubstance use, and whether they are amenable to abstinence.

METHODS

This study applied a single measurement session of diffusion tensor imaging (DTI) to examine white matter structure in male cocaine polysubstance users (n=37) versus male healthy controls (n=38), along with correlations between DTI measures and patterns of polysubstance use and duration of abstinence. Specifically, we conducted voxel-wise analyses of fractional anisotropy (FA) in the corpus callosum, frontolimbic, striatal and cingulate tracts relevant to drug sequelae.

RESULTS

Cocaine polysubstance users, compared to controls, showed lower FA in the body of the corpus callosum, anterior cingulate, uncinate fasciculus and retrolenticular part of the internal capsule. Duration of cocaine use had a marginal negative association with FA in the corpus callosum, and duration of alcohol use was negatively associated with FA in the internal capsule and the uncinate fasciculus. Duration of cocaine abstinence was positively correlated with FA in the uncinate fasciculus, posterior cingulate and fornix-striatum. In the context of cocaine polysubstance use, chronicity of cocaine use is therefore likely to be associated with lower FA in the corpus callosum, and chronicity of alcohol use with lower FA in the frontal-striatal and frontal-limbic tracts. Longer abstinence was correlated to greater FA in frontal-striatal and frontal-limbic tracts, though the direction of causality remains unclear.

CONCLUSION

Since the results did not survive multiple comparison-corrected thresholds, more studies are needed to confirm these indications.

Neural Correlates of Learning from Induced Insight: A Case for Reward-Based Episodic Encoding

Experiencing insight when solving problems can improve memory formation for both the problem and its solution. The underlying neural processes involved in this kind of learning are, however, thus far insufficiently understood. Here, we conceptualized insight as the sudden understanding of a novel relationship between known stimuli that fits into existing knowledge and is accompanied by a positive emotional response. Hence, insight is thought to comprise associative novelty, schema congruency, and intrinsic reward, all of which are separately known to enhance memory performance. We examined the neural correlates of learning from induced insight with functional magnetic resonance imaging (fMRI) using our own version of the compound-remote-associates-task (CRAT) in which each item consists of three clue words and a solution word. (Pseudo-)Solution words were presented after a brief period of problem-solving attempts to induce either sudden comprehension (CRA items) or continued incomprehension (control items) at a specific time point. By comparing processing of the solution words of CRA with control items, we found induced insight to elicit activation of the rostral anterior cingulate cortex/medial prefrontal cortex (rACC/mPFC) and left hippocampus. This pattern of results lends support to the role of schema congruency (rACC/mPFC) and associative novelty (hippocampus) in the processing of induced insight. We propose that (1) the mPFC not only responds to schema-congruent information, but also to the detection of novel schemata, and (2) that the hippocampus responds to a form of associative novelty that is not just a novel constellation of familiar items, but rather comprises a novel meaningful relationship between the items—which was the only difference between our insight and no insight conditions. To investigate episodic long-term memory encoding, we compared CRA items whose solution word was recognized 24 h after encoding to those with forgotten solutions. We found activation in the left striatum and parts of the left amygdala, pointing to a potential role of brain reward circuitry in the encoding of the solution words. We propose that learning from induced insight mainly relies on the amygdala evaluating the internal value (as an affective evaluation) of the suddenly comprehended information, and striatum-dependent reward-based learning.

Memory-reliant Post-error Slowing Is Associated with Successful Learning and Fronto-occipital Activity

Negative feedback after an action in a cognitive task can lead to devaluing that action on future trials as well as to more cautious responding when encountering that same choice again. These phenomena have been explored in the past by reinforcement learning theories and cognitive control accounts, respectively. Yet, how cognitive control interacts with value updating to give rise to adequate adaptations under uncertainty is less clear. In this fMRI study, we investigated cognitive control-based behavioral adjustments during a probabilistic reinforcement learning task and studied their influence on performance in a later test phase in which the learned value of items is tested. We provide support for the idea that functionally relevant and memory-reliant behavioral adjustments in the form of post-error slowing during reinforcement learning are associated with test performance. Adjusting response speed after negative feedback was correlated with BOLD activity in right inferior frontal gyrus and bilateral middle occipital cortex during the event of receiving the feedback. Bilateral middle occipital cortex activity overlapped partly with activity reflecting feedback deviance from expectations as measured by unsigned prediction error. These results suggest that cognitive control and feature processing cortical regions interact to implement feedback-congruent adaptations beneficial to learning.

Neural and neurochemical basis of reinforcement-guided decision making

Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcement-guided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decision-related information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Using fMRI to study reward processing in humans: past, present, and future

Functional magnetic resonance imaging (fMRI) is a noninvasive tool used to probe cognitive and affective processes. Although fMRI provides indirect measures of neural activity, the advent of fMRI has allowed for1) the corroboration of significant animal findings in the human brain, and2) the expansion of models to include more common human attributes that inform behavior. In this review, we briefly consider the neural basis of the blood oxygenation level dependent signal to set up a discussion of how fMRI studies have applied it in examining cognitive models in humans and the promise of using fMRI to advance such models. Specifically, we illustrate the contribution that fMRI has made to the study of reward processing, focusing on the role of the striatum in encoding reward-related learning signals that drive anticipatory and consummatory behaviors. For instance, we discuss how fMRI can be used to link neural signals (e.g., striatal responses to rewards) to individual differences in behavior and traits. While this functional segregation approach has been constructive to our understanding of reward-related functions, many fMRI studies have also benefitted from a functional integration approach that takes into account how interconnected regions (e.g., corticostriatal circuits) contribute to reward processing. We contend that future work using fMRI will profit from using a multimodal approach, such as combining fMRI with noninvasive brain stimulation tools (e.g., transcranial electrical stimulation), that can identify causal mechanisms underlying reward processing. Consequently, advancements in implementing fMRI will promise new translational opportunities to inform our understanding of psychopathologies.

Contextual modulation of value signals in reward and punishment learning

Compared with reward seeking, punishment avoidance learning is less clearly understood at both the computational and neurobiological levels. Here we demonstrate, using computational modelling and fMRI in humans, that learning option values in a relative—context-dependent—scale offers a simple computational solution for avoidance learning. The context (or state) value sets the reference point to which an outcome should be compared before updating the option value. Consequently, in contexts with an overall negative expected value, successful punishment avoidance acquires a positive value, thus reinforcing the response. As revealed by post-learning assessment of options values, contextual influences are enhanced when subjects are informed about the result of the forgone alternative (counterfactual information). This is mirrored at the neural level by a shift in negative outcome encoding from the anterior insula to the ventral striatum, suggesting that value contextualization also limits the need to mobilize an opponent punishment learning system.

In contrast to predictions from learning theory, humans learn to seek rewards and avoid punishments equally well. Here the authors offer an elegant solution to this problem by demonstrating that humans learn option values relative to a reference point subserved by a common neural substrate.

Differential contributions of the globus pallidus and ventral thalamus to stimulus-response learning in humans

The ability to learn associations between stimuli, responses and rewards is a prerequisite for survival. Models of reinforcement learning suggest that the striatum, a basal ganglia input nucleus, vitally contributes to these learning processes. Our recently presented computational model predicts, first, that not only the striatum, but also the globus pallidus contributes to the learning (i.e., exploration) of stimulus-response associations based on rewards. Secondly, it predicts that the stable execution (i.e., exploitation) of well-learned associations involves further learning in the thalamus. To test these predictions, we postoperatively recorded local field potentials (LFPs) from patients that had undergone surgery for deep brain stimulation to treat severe movement disorders. Macroelectrodes were placed either in the globus pallidus or in the ventral thalamus. During recordings, patients performed a reward-based stimulus-response learning task that comprised periods of exploration and exploitation. We analyzed correlations between patients' LFP amplitudes and model-based estimates of their reward expectations and reward prediction errors. In line with our first prediction, pallidal LFP amplitudes during the presentation of rewards and reward omissions correlated with patients' reward prediction errors, suggesting pallidal access to reward-based teaching signals. Unexpectedly, the same was true for the thalamus. In further support of this prediction, pallidal LFP amplitudes during stimulus presentation correlated with patients' reward expectations during phases of low reward certainty - suggesting pallidal participation in the learning of stimulus-response associations. In line with our second prediction, correlations between thalamic stimulus-related LFP amplitudes and patients' reward expectations were significant within phases of already high reward certainty, suggesting thalamic participation in exploitation.

An evolutionary computational theory of prefrontal executive function in decision-making

The prefrontal cortex subserves executive control and decision-making, that is, the coordination and selection of thoughts and actions in the service of adaptive behaviour. We present here a computational theory describing the evolution of the prefrontal cortex from rodents to humans as gradually adding new inferential Bayesian capabilities for dealing with a computationally intractable decision problem: exploring and learning new behavioural strategies versus exploiting and adjusting previously learned ones through reinforcement learning (RL). We provide a principled account identifying three inferential steps optimizing this arbitration through the emergence of (i) factual reactive inferences in paralimbic prefrontal regions in rodents; (ii) factual proactive inferences in lateral prefrontal regions in primates and (iii) counterfactual reactive and proactive inferences in human frontopolar regions. The theory clarifies the integration of model-free and model-based RL through the notion of strategy creation. The theory also shows that counterfactual inferences in humans yield to the notion of hypothesis testing, a critical reasoning ability for approximating optimal adaptive processes and presumably endowing humans with a qualitative evolutionary advantage in adaptive behaviour.

Dopamine D2-receptor blockade enhances decoding of prefrontal signals in humans

The prefrontal cortex houses representations critical for ongoing and future behavior expressed in the form of patterns of neural activity. Dopamine has long been suggested to play a key role in the integrity of such representations, with D2-receptor activation rendering them flexible but weak. However, it is currently unknown whether and how D2-receptor activation affects prefrontal representations in humans. In the current study, we use dopamine receptor-specific pharmacology and multivoxel pattern-based functional magnetic resonance imaging to test the hypothesis that blocking D2-receptor activation enhances prefrontal representations. Human subjects performed a simple reward prediction task after double-blind and placebo controlled administration of the D2-receptor antagonist amisulpride. Using a whole-brain searchlight decoding approach we show that D2-receptor blockade enhances decoding of reward signals in the medial orbitofrontal cortex. Examination of activity patterns suggests that amisulpride increases the separation of activity patterns related to reward versus no reward. Moreover, consistent with the cortical distribution of D2 receptors, post hoc analyses showed enhanced decoding of motor signals in motor cortex, but not of visual signals in visual cortex. These results suggest that D2-receptor blockade enhances content-specific representations in frontal cortex, presumably by a dopamine-mediated increase in pattern separation. These findings are in line with a dual-state model of prefrontal dopamine, and provide new insights into the potential mechanism of action of dopaminergic drugs.

Adolescent-specific patterns of behavior and neural activity during social reinforcement learning

Humans are sophisticated social beings. Social cues from others are exceptionally salient, particularly during adolescence. Understanding how adolescents interpret and learn from variable social signals can provide insight into the observed shift in social sensitivity during this period. The present study tested 120 participants between the ages of 8 and 25 years on a social reinforcement learning task where the probability of receiving positive social feedback was parametrically manipulated. Seventy-eight of these participants completed the task during fMRI scanning. Modeling trial-by-trial learning, children and adults showed higher positive learning rates than did adolescents, suggesting that adolescents demonstrated less differentiation in their reaction times for peers who provided more positive feedback. Forming expectations about receiving positive social reinforcement correlated with neural activity within the medial prefrontal cortex and ventral striatum across age. Adolescents, unlike children and adults, showed greater insular activity during positive prediction error learning and increased activity in the supplementary motor cortex and the putamen when receiving positive social feedback regardless of the expected outcome, suggesting that peer approval may motivate adolescents toward action. While different amounts of positive social reinforcement enhanced learning in children and adults, all positive social reinforcement equally motivated adolescents. Together, these findings indicate that sensitivity to peer approval during adolescence goes beyond simple reinforcement theory accounts and suggest possible explanations for how peers may motivate adolescent behavior.

The effects of life stress and neural learning signals on fluid intelligence

Fluid intelligence (fluid IQ), defined as the capacity for rapid problem solving and behavioral adaptation, is known to be modulated by learning and experience. Both stressful life events (SLES) and neural correlates of learning [specifically, a key mediator of adaptive learning in the brain, namely the ventral striatal representation of prediction errors (PE)] have been shown to be associated with individual differences in fluid IQ. Here, we examine the interaction between adaptive learning signals (using a well-characterized probabilistic reversal learning task in combination with fMRI) and SLES on fluid IQ measures. We find that the correlation between ventral striatal BOLD PE and fluid IQ, which we have previously reported, is quantitatively modulated by the amount of reported SLES. Thus, after experiencing adversity, basic neuronal learning signatures appear to align more closely with a general measure of flexible learning (fluid IQ), a finding complementing studies on the effects of acute stress on learning. The results suggest that an understanding of the neurobiological correlates of trait variables like fluid IQ needs to take socioemotional influences such as chronic stress into account.

Chronic alcohol intake abolishes the relationship between dopamine synthesis capacity and learning signals in the ventral striatum

Drugs of abuse elicit dopamine release in the ventral striatum, possibly biasing dopamine-driven reinforcement learning towards drug-related reward at the expense of non-drug-related reward. Indeed, in alcohol-dependent patients, reactivity in dopaminergic target areas is shifted from non-drug-related stimuli towards drug-related stimuli. Such 'hijacked' dopamine signals may impair flexible learning from non-drug-related rewards, and thus promote craving for the drug of abuse. Here, we used functional magnetic resonance imaging to measure ventral striatal activation by reward prediction errors (RPEs) during a probabilistic reversal learning task in recently detoxified alcohol-dependent patients and healthy controls (N = 27). All participants also underwent 6-[(18) F]fluoro-DOPA positron emission tomography to assess ventral striatal dopamine synthesis capacity. Neither ventral striatal activation by RPEs nor striatal dopamine synthesis capacity differed between groups. However, ventral striatal coding of RPEs correlated inversely with craving in patients. Furthermore, we found a negative correlation between ventral striatal coding of RPEs and dopamine synthesis capacity in healthy controls, but not in alcohol-dependent patients. Moderator analyses showed that the magnitude of the association between dopamine synthesis capacity and RPE coding depended on the amount of chronic, habitual alcohol intake. Despite the relatively small sample size, a power analysis supports the reported results. Using a multimodal imaging approach, this study suggests that dopaminergic modulation of neural learning signals is disrupted in alcohol dependence in proportion to long-term alcohol intake of patients. Alcohol intake may perpetuate itself by interfering with dopaminergic modulation of neural learning signals in the ventral striatum, thus increasing craving for habitual drug intake.

The brain correlates of the effects of monetary and verbal rewards on intrinsic motivation

Apart from everyday duties, such as doing the laundry or cleaning the house, there are tasks we do for pleasure and enjoyment. We do such tasks, like solving crossword puzzles or reading novels, without any external pressure or force; instead, we are intrinsically motivated: we do the tasks because we enjoy doing them. Previous studies suggest that external rewards, i.e., rewards from the outside, affect the intrinsic motivation to engage in a task: while performance-based monetary rewards are perceived as controlling and induce a business-contract framing, verbal rewards praising one's competence can enhance the perceived self-determination. Accordingly, the former have been shown to decrease intrinsic motivation, whereas the latter have been shown to increase intrinsic motivation. The present study investigated the neural processes underlying the effects of monetary and verbal rewards on intrinsic motivation in a group of 64 subjects applying functional magnetic resonance imaging (fMRI). We found that, when participants received positive performance feedback, activation in the anterior striatum and midbrain was affected by the nature of the reward; compared to a non-rewarded control group, activation was higher while monetary rewards were administered. However, we did not find a decrease in activation after reward withdrawal. In contrast, we found an increase in activation for verbal rewards: after verbal rewards had been withdrawn, participants showed a higher activation in the aforementioned brain areas when they received success compared to failure feedback. We further found that, while participants worked on the task, activation in the lateral prefrontal cortex was enhanced after the verbal rewards were administered and withdrawn.

Functional connectivity and grey matter volume of the striatum in schizophrenia

BACKGROUND

Alterations in the dopaminergic reward system, predominantly the striatum, constitute core characteristics of schizophrenia.

AIMS

Functional connectivity of the dorsal striatum during reward-related trial-and-error learning was investigated in 17 people with schizophrenia and 18 healthy volunteers and related to striatal grey matter volume and psychopathology.

METHOD

We used voxel-based morphometry and psychophysiological interaction to examine striatal volume and connectivity.

RESULTS

A reduced functional connectivity between left striatum and temporo-occipital areas, precuneus and insula could be detected in the schizophrenia group. The positive correlation between grey matter volume and functional connectivity of the left striatum yielded significant results in a very similar network. Connectivity of the left striatum was negatively correlated with negative symptoms.

CONCLUSIONS

Present results suggest a disruption in striatal functional connectivity that is closely linked to grey matter morphometry of the striatum. Decreased connectivity between the striatum and psychopathologically relevant networks may explain the emergence of negative symptoms.

Different aspects of performance feedback engage different brain areas: disentangling valence and expectancy in feedback processing

Evaluating the positive and negative outcomes of our behaviour is important for action selection and learning. Such reinforcement learning has been shown to engage a specific neural circuitry including the mesencephalic dopamine system and its target areas, the striatum and medial frontal cortex, especially the anterior cingulate cortex (ACC). An intensively pursued debate regards the prevailing influence of feedback expectancy and feedback valence on the engagement of these two brain regions in reinforcement learning and their respective roles are far from being understood. To this end, we used a time estimation task with three different types of feedback that allows disentangling the effect of feedback valence and expectancy using functional magnetic resonance imaging (fMRI). Our results show greater ACC activation after unexpected positive and unexpected negative feedback than after expected feedback and by this sensitivity to unexpected events in general irrespective of their valence.

The role of dopamine in risk taking: a specific look at Parkinson's disease and gambling

An influential model suggests that dopamine signals the difference between predicted and experienced reward. In this way, dopamine can act as a learning signal that can shape behaviors to maximize rewards and avoid punishments. Dopamine is also thought to invigorate reward seeking behavior. Loss of dopamine signaling is the major abnormality in Parkinson’s disease. Dopamine agonists have been implicated in the occurrence of impulse control disorders in Parkinson’s disease patients, the most common being pathological gambling, compulsive sexual behavior, and compulsive buying. Recently, a number of functional imaging studies investigating impulse control disorders in Parkinson’s disease have been published. Here we review this literature, and attempt to place it within a decision-making framework in which potential gains and losses are evaluated to arrive at optimum choices. We also provide a hypothetical but still incomplete model on the effect of dopamine agonist treatment on these value and risk assessments. Two of the main brain structures thought to be involved in computing aspects of reward and loss are the ventral striatum (VStr) and the insula, both dopamine projection sites. Both structures are consistently implicated in functional brain imaging studies of pathological gambling in Parkinson’s disease.

Linking brain electrical signals elicited by current outcomes with future risk decision-making

The experience of current outcomes influences future decisions in various ways. The neural mechanism of this phenomenon may help to clarify the determinants of decision-making. In this study, thirty-nine young adults finished a risky gambling task by choosing between a high- and a low-risk option in each trial during electroencephalographic data collection. We found that risk-taking strategies significantly modulated mean amplitudes of the event-related potential (ERP) component P3, particularly at the central scalp. The event-related spectral perturbation and the inter-trial coherence measurements of the independent component analysis (ICA) data indicated that the “stay” vs. “switch” electrophysiological difference associated with subsequent decision-making was mainly due to fronto-central theta and left/right mu independent components. Event-related cross-coherence results suggested that the neural information of action monitoring and updating emerged in the fronto-central cortex and propagated to sensorimotor area for further behavior adjustment. Based on these findings of ERP and event-related oscillation (ERO) measures, we propose a neural model of the influence of current outcomes on future decisions.

The neurobiology of alcohol craving and relapse

A major block to recovery from alcoholism is substantial alcohol craving and the chronic relapsing nature of the illness. This chapter reviews relevant structural and functional neuroimaging studies and discusses neural mechanisms underlying alcohol craving and relapse in the context of influential risk factors (i.e., alcohol, alcohol cue, and stress). Review of neuroimaging studies suggests that neuroadaptations in the cortico-striatal-limbic circuit encompassing the medial prefrontal cortex, orbitofrontal cortex, anterior cingulate cortex, striatum, and amygdala significantly contribute to overwhelming alcohol craving and early relapse after a period of abstinence. The cortico-striatal-limbic circuit plays an important role in the modulation of emotion, reward, and decision making. As functional and structural chronic alcohol-related neuroadaptations are consistently reported in this circuit, it is likely that sensitization of this circuit from continued alcohol abuse may contribute to high alcohol craving and early relapse via impairments in the prefrontal executive function related to emotion regulation and decision making. This vulnerable neurobiologic state may be manifested as compulsive craving and intense urge to resume alcohol drinking in the face of environmental risk factors, including alcohol, alcohol cue, or stressful live events.

Striatal dysfunction during reversal learning in unmedicated schizophrenia patients

Subjects with schizophrenia are impaired at reinforcement-driven reversal learning from as early as their first episode. The neurobiological basis of this deficit is unknown. We obtained behavioral and fMRI data in 24 unmedicated, primarily first episode, schizophrenia patients and 24 age-, IQ- and gender-matched healthy controls during a reversal learning task. We supplemented our fMRI analysis, focusing on learning from prediction errors, with detailed computational modeling to probe task solving strategy including an ability to deploy an internal goal directed model of the task. Patients displayed reduced functional activation in the ventral striatum (VS) elicited by prediction errors. However, modeling task performance revealed that a subgroup did not adjust their behavior according to an accurate internal model of the task structure, and these were also the more severely psychotic patients. In patients who could adapt their behavior, as well as in controls, task solving was best described by cognitive strategies according to a Hidden Markov Model. When we compared patients and controls who acted according to this strategy, patients still displayed a significant reduction in VS activation elicited by informative errors that precede salient changes of behavior (reversals). Thus, our study shows that VS dysfunction in schizophrenia patients during reward-related reversal learning remains a core deficit even when controlling for task solving strategies. This result highlights VS dysfunction is tightly linked to a reward-related reversal learning deficit in early, unmedicated schizophrenia patients.

•

Probabilistic reversal learning was examined in unmedicated schizophrenia patients.

•

Computational modeling assessed subjects ability to use the latent task structure.

•

SZ patients showed lower reinforcement sensitivity and higher switch tendency.

•

Blunted striatal prediction error signal in unmedicated schizophrenia patients.

•

PFC activation during reversal errors intact in SZ patients able to do the task.

The expected value of control: an integrative theory of anterior cingulate cortex function

The dorsal anterior cingulate cortex (dACC) has a near-ubiquitous presence in the neuroscience of cognitive control. It has been implicated in a diversity of functions, from reward processing and performance monitoring to the execution of control and action selection. Here, we propose that this diversity can be understood in terms of a single underlying function: allocation of control based on an evaluation of the expected value of control (EVC). We present a normative model of EVC that integrates three critical factors: the expected payoff from a controlled process, the amount of control that must be invested to achieve that payoff, and the cost in terms of cognitive effort. We propose that dACC integrates this information, using it to determine whether, where and how much control to allocate. We then consider how the EVC model can explain the diverse array of findings concerning dACC function.

Categorization = decision making + generalization

We rarely, if ever, repeatedly encounter exactly the same situation. This makes generalization crucial for real world decision making. We argue that categorization, the study of generalizable representations, is a type of decision making, and that categorization learning research would benefit from approaches developed to study the neuroscience of decision making. Similarly, methods developed to examine generalization and learning within the field of categorization may enhance decision making research. We first discuss perceptual information processing and integration, with an emphasis on accumulator models. We then examine learning the value of different decision making choices via experience, emphasizing reinforcement learning modeling approaches. Next we discuss how value is combined with other factors in decision making, emphasizing the effects of uncertainty. Finally, we describe how a final decision is selected via thresholding processes implemented by the basal ganglia and related regions. We also consider how memory related functions in the hippocampus may be integrated with decision making mechanisms and contribute to categorization.

Value of freedom to choose encoded by the human brain

Humans and animals value the opportunity to choose by preferring alternatives that offer more rather than fewer choices. This preference for choice may arise not only from an increased probability of obtaining preferred outcomes but also from the freedom it provides. We used human neuroimaging to investigate the neural basis of the preference for choice as well as for the items that could be chosen. In each trial, participants chose between two options, a monetary amount option and a "choice option." The latter consisted of a number that corresponded to the number of everyday items participants would subsequently be able to choose from. We found that the opportunity to choose from a larger number of items was equivalent to greater amounts of money, indicating that participants valued having more choice; moreover, participants varied in the degree to which they valued having the opportunity to choose, with some valuing it more than the increased probability of obtaining preferred items. Neural activations in the mid striatum increased with the value of the opportunity to choose. The same region also coded the value of the items. Conversely, activation in the dorsolateral striatum was not related to the value of the items but was elevated when participants were offered more choices, particularly in those participants who overvalued the opportunity to choose. These data suggest a functional dissociation of value representations within the striatum, with general representations in mid striatum and specific representations of the value of freedom provided by the opportunity to choose in dorsolateral striatum.

Direct-pathway striatal neurons regulate the retention of decision-making strategies

The dorsal striatum has been implicated in reward-based decision making, but the role played by specific striatal circuits in these processes is essentially unknown. Using cell phenotype-specific viral vectors to express engineered G-protein-coupled DREADD (designer receptors exclusively activated by designer drugs) receptors, we enhanced Gi/o- or Gs-protein-mediated signaling selectively in direct-pathway (striatonigral) neurons of the dorsomedial striatum in Long-Evans rats during discrete periods of training of a high versus low reward-discrimination task. Surprisingly, these perturbations had no impact on reward preference, task performance, or improvement of performance during training. However, we found that transiently increasing Gi/o signaling during training significantly impaired the retention of task strategies used to maximize reward obtainment during subsequent preference testing, whereas increasing Gs signaling produced the opposite effect and significantly enhanced the encoding of a high-reward preference in this decision-making task. Thus, the fact that the endurance of this improved performance was significantly altered over time-long after these neurons were manipulated-indicates that it is under bidirectional control of canonical G-protein-mediated signaling in striatonigral neurons during training. These data demonstrate that cAMP-dependent signaling in direct-pathway neurons play a well-defined role in reward-related behavior; that is, they modulate the plasticity required for the retention of task-specific information that is used to improve performance on future renditions of the task.

Neural correlates of reinforcement learning and social preferences in competitive bidding

In competitive social environments, people often deviate from what rational choice theory prescribes, resulting in losses or suboptimal monetary gains. We investigate how competition affects learning and decision-making in a common value auction task. During the experiment, groups of five human participants were simultaneously scanned using MRI while playing the auction task. We first demonstrate that bidding is well characterized by reinforcement learning with biased reward representations dependent on social preferences. Indicative of reinforcement learning, we found that estimated trial-by-trial prediction errors correlated with activity in the striatum and ventromedial prefrontal cortex. Additionally, we found that individual differences in social preferences were related to activity in the temporal-parietal junction and anterior insula. Connectivity analyses suggest that monetary and social value signals are integrated in the ventromedial prefrontal cortex and striatum. Based on these results, we argue for a novel mechanistic account for the integration of reinforcement history and social preferences in competitive decision-making.

Salience signals in the right temporoparietal junction facilitate value-based decisions

Value-based decisions optimize the relation of costs and benefits. Costs and benefits confer not only value but also salience, which may influence decision making through attentional mechanisms. However, the computational and neurobiological role of salience in value-based decisions remains elusive. Here we develop and contrast two formal concepts of salience for value-based choices involving costs and benefits. Specifically, global salience (GS) first integrates costs and benefits and then determines salience based on this overall sum, whereas elemental salience (ES) first determines the salience of costs and benefits before integrating them. We dissociate the behavioral and neural effects of GS and ES from those of value using a value-based decision-making task and fMRI in humans. Specifically, we show that value guides choices and correlates with neural signals in the striatum. In contrast, only ES but not GS impacts decision making by speeding up reaction times. Moreover, activity in the right temporoparietal junction (RTPJ) reflects only ES and correlates with its response-accelerating behavioral effects. Finally, we report an ES-dependent change in functional connectivity between the RTPJ and the locus ceruleus, suggesting noradrenergic processes underlying the response-facilitating effects of ES on decision making. Together, these results support a novel concept of salience in value-based decision making and suggest a computational, anatomical, and neurochemical dissociation of value- and salience-based factors supporting value-based choices.

To you I am listening: perceived competence of advisors influences judgment and decision-making via recruitment of the amygdala

Considering advice from others is a pervasive element of human social life. We used the judge-advisor paradigm to investigate the neural correlates of advice evaluation and advice integration by means of functional magnetic resonance imaging. Our results demonstrate that evaluating advice recruits the "mentalizing network," brain regions activated when people think about others' mental states. Important activation differences exist, however, depending upon the perceived competence of the advisor. Consistently, additional analyses demonstrate that integrating others' advice, i.e., how much participants actually adjust their initial estimate, correlates with neural activity in the centromedial amygdala in the case of a competent and with activity in visual cortex in the case of an incompetent advisor. Taken together, our findings, therefore, demonstrate that advice evaluation and integration rely on dissociable neural mechanisms and that significant differences exist depending upon the advisor's reputation, which suggests different modes of processing advice depending upon the perceived competence of the advisor.

Salience network-midbrain dysconnectivity and blunted reward signals in schizophrenia

Theories of schizophrenia propose that abnormal functioning of the neural reward system is linked to negative and psychotic symptoms, by disruption of reward processing and promotion of context-independent false associations. Recently, it has been argued that an insula-anterior cingulate cortex (ACC) salience network system enables switching of brain states from the default mode to a task-related activity mode. Abnormal interaction between the insula-ACC system and reward processing regions may help explain abnormal reinforcer processing and symptoms. Here we use functional magnetic resonance imaging to assess the neural correlates of reward processing in schizophrenia. Furthermore, we investigated functional connectivity between the dopaminergic midbrain, a key region for the processing of reinforcers, and other brain regions. In response to rewards, controls activated task related regions (striatum, amygdala/hippocampus and midbrain) and the insula-ACC salience network. Patients similarly activated the insula-ACC salience network system but failed to activate task related regions. Reduced functional connectivity between the midbrain and the insula was found in schizophrenia, with the extent of this abnormality correlating with increased psychotic symptoms. The findings support the notion that reward processing is abnormal in schizophrenia and highlight the potential role of abnormal interactions between the insula-ACC salience network and reward regions.