Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study

A novel technique for delineating the effect of variation in the learning rate on the neural correlates of reward prediction errors in model-based fMRI

Introduction

Computational models play an increasingly important role in describing variation in neural activation in human neuroimaging experiments, including evaluating individual differences in the context of psychiatric neuroimaging. In particular, reinforcement learning (RL) techniques have been widely adopted to examine neural responses to reward prediction errors and stimulus or action values, and how these might vary as a function of clinical status. However, there is a lack of consensus around the importance of the precision of free parameter estimation for these methods, particularly with regard to the learning rate. In the present study, I introduce a novel technique which may be used within a general linear model (GLM) to model the effect of mis-estimation of the learning rate on reward prediction error (RPE)-related neural responses.

Methods

Simulations employed a simple RL algorithm, which was used to generate hypothetical neural activations that would be expected to be observed in functional magnetic resonance imaging (fMRI) studies of RL. Similar RL models were incorporated within a GLM-based analysis method including derivatives, with individual differences in the resulting GLM-derived beta parameters being evaluated with respect to the free parameters of the RL model or being submitted to other validation analyses.

Results

Initial simulations demonstrated that the conventional approach to fitting RL models to RPE responses is more likely to reflect individual differences in a reinforcement efficacy construct (lambda) rather than learning rate (alpha). The proposed method, adding a derivative regressor to the GLM, provides a second regressor which reflects the learning rate. Validation analyses were performed including examining another comparable method which yielded highly similar results, and a demonstration of sensitivity of the method in presence of fMRI-like noise.

Conclusion

Overall, the findings underscore the importance of the lambda parameter for interpreting individual differences in RPE-coupled neural activity, and validate a novel neural metric of the modulation of such activity by individual differences in the learning rate. The method is expected to find application in understanding aberrant reinforcement learning across different psychiatric patient groups including major depression and substance use disorder.

Impulse control disorder in Parkinson's disease is associated with abnormal frontal value signalling

Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson's disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson's disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression. One hundred and ninety-nine patients with Parkinson's disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered. These data substantiate the hypothesis that dopamine's effects on reinforcement learning in Parkinson's disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum. See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.

Hippocampal Contribution to Probabilistic Feedback Learning: Modeling Observation- and Reinforcement-based Processes

Simple probabilistic reinforcement learning is recognized as a striatum-based learning system, but in recent years, has also been associated with hippocampal involvement. This study examined whether such involvement may be attributed to observation-based learning (OL) processes, running in parallel to striatum-based reinforcement learning. A computational model of OL, mirroring classic models of reinforcement-based learning (RL), was constructed and applied to the neuroimaging data set of Palombo, Hayes, Reid, and Verfaellie (2019). Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia. Cognitive, Affective & Behavioral Neuroscience, 19(3), 523-536. Results suggested that OL processes may indeed take place concomitantly to reinforcement learning and involve activation of the hippocampus and central orbitofrontal cortex. However, rather than independent mechanisms running in parallel, the brain correlates of the OL and RL prediction errors indicated collaboration between systems, with direct implication of the hippocampus in computations of the discrepancy between the expected and actual reinforcing values of actions. These findings are consistent with previous accounts of a role for the hippocampus in encoding the strength of observed stimulus-outcome associations, with updating of such associations through striatal reinforcement-based computations. In addition, enhanced negative RL prediction error signaling was found in the anterior insula with greater use of OL over RL processes. This result may suggest an additional mode of collaboration between the OL and RL systems, implicating the error monitoring network.

Role of dopamine and clinical heterogeneity in cognitive dysfunction in Parkinson's disease

Parkinson's disease (PD) is commonly treated with dopaminergic medication, which enhances some, while impairing other cognitive functions. It can even contribute to impulse control disorder and addiction. We describe the history of research supporting the dopamine overdose hypothesis, which accounts for the large within-patient variability in dopaminergic medication effects across different tasks by referring to the spatially non-uniform pattern of dopamine depletion in dorsal versus ventral striatum. However, there is tremendous variability in dopaminergic medication effects not just within patients across distinct tasks, but also across different patients. In the second part of this chapter we review recent studies addressing the large individual variability in the negative side effects of dopaminergic medication on functions that implicate dopamine, such as value-based learning and choice. These studies begin to unravel the mechanisms of dopamine overdosing, thus revising the strict version of the overdose hypothesis. For example, the work shows that the canonical boosting of reward-versus punishment-based choice by medication is greater in patients with depression and a non-tremor phenotype, which both implicate, among other pathology, more rather than less severe dysregulation of the mesolimbic dopamine system. Future longitudinal cohort studies are needed to identify how to optimally combine different clinical, personality, cognitive, neural, genetic and molecular predictors of detrimental medication effects in order to account for as much of the relevant variability as possible. This will provide a useful tool for precision neurology, allowing individual and contextual tailoring of (the dose of) dopaminergic medication in order to maximize its cognitive benefits, yet minimize its side effects.

Value estimation and latent-state update-related neural activity during fear conditioning predict posttraumatic stress disorder symptom severity

Learning theories of posttraumatic stress disorder (PTSD) purport that fear-learning processes, such as those that support fear acquisition and extinction, are impaired. Computational models designed to capture specific processes involved in fear learning have primarily assessed model-free, or trial-and-error, reinforcement learning (RL). Although previous studies indicated that aspects of model-free RL are disrupted among individuals with PTSD, research has yet to identify whether model-based RL, which is inferential and contextually driven, is impaired. Given empirical evidence of aberrant contextual modulation of fear in PTSD, the present study sought to identify whether model-based RL processes are altered during fear conditioning among women with interpersonal violence (IPV)-related PTSD (n = 85) using computational modeling. Model-free, hybrid, and model-based RL models were applied to skin conductance responses (SCR) collected during fear acquisition and extinction, and the model-based RL model was found to provide the best fit to the SCR data. Parameters from the model-based RL model were carried forward to neuroimaging analyses (voxel-wise and independent component analysis). Results revealed that reduced activity within visual processing regions during model-based updating uniquely predicted higher PTSD symptoms. Additionally, after controlling for model-based updating, greater value estimation encoding within the left frontoparietal network during fear acquisition and reduced value estimation encoding within the dorsomedial prefrontal cortex during fear extinction predicted greater PTSD symptoms. Results provide evidence of disrupted RL processes in women with assault-related PTSD, which may contribute to impaired fear and safety learning, and, furthermore, may relate to treatment response (e.g., poorer response to exposure therapy).

Task Learnability Modulates Surprise but Not Valence Processing for Reinforcement Learning in Probabilistic Choice Tasks

The goal of temporal difference (TD) reinforcement learning is to maximize outcomes and improve future decision-making. It does so by utilizing a prediction error (PE), which quantifies the difference between the expected and the obtained outcome. In gambling tasks, however, decision-making cannot be improved because of the lack of learnability. On the basis of the idea that TD utilizes two independent bits of information from the PE (valence and surprise), we asked which of these aspects is affected when a task is not learnable. We contrasted behavioral data and ERPs in a learning variant and a gambling variant of a simple two-armed bandit task, in which outcome sequences were matched across tasks. Participants were explicitly informed that feedback could be used to improve performance in the learning task but not in the gambling task, and we predicted a corresponding modulation of the aspects of the PE. We used a model-based analysis of ERP data to extract the neural footprints of the valence and surprise information in the two tasks. Our results revealed that task learnability modulates reinforcement learning via the suppression of surprise processing but leaves the processing of valence unaffected. On the basis of our model and the data, we propose that task learnability can selectively suppress TD learning as well as alter behavioral adaptation based on a flexible cost-benefit arbitration.

Probabilistic value learning in medial temporal lobe amnesia

A prevailing view in cognitive neuroscience suggests that different forms of learning are mediated by dissociable memory systems, with a mesolimbic (i.e., midbrain and basal ganglia) system supporting incremental trial-and-error reinforcement learning and a hippocampal-based system supporting episodic memory. Yet, growing evidence suggests that the hippocampus may also be important for trial-and-error learning, particularly value or reward-based learning. In the present report, we use a lesion-based neuropsychological approach to clarify hippocampal contributions to such learning. Six amnesic patients with medial temporal lobe damage and a group of healthy controls were administered a simple value-based learning task involving probabilistic trial-and-error acquisition of stimulus-response-outcome (reward or none) contingencies modeled after Li et al. (Proceedings of the National Academy of Sciences , 2011, 108 (1), 55-60). As predicted, patients were significantly impaired on the task, demonstrating reduced learning of the contingencies. Our results provide further supportive evidence that the hippocampus' role in cognition extends beyond episodic memory tasks and call for further refinement of theoretical models of hippocampal functioning.

Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia

Recent evidence suggests that the human hippocampus-known primarily for its involvement in episodic memory-plays a role in a host of motivationally relevant behaviors, including some forms of value-based decision-making. However, less is known about the role of the hippocampus in value-based learning. Such learning is typically associated with a striatal system, yet a small number of studies, both in human and nonhuman species, suggest hippocampal engagement. It is not clear, however, whether this engagement is necessary for such learning. In the present study, we used both functional MRI (fMRI) and lesion-based neuropsychological methods to clarify hippocampal contributions to value-based learning. In Experiment 1, healthy participants were scanned while learning value-based contingencies (whether players in a "game" win money) in the context of a probabilistic learning task. Here, we observed recruitment of the hippocampus, in addition to the expected ventral striatal (nucleus accumbens) activation that typically accompanies such learning. In Experiment 2, we administered this task to amnesic patients with medial temporal lobe damage and to healthy controls. Amnesic patients, including those with damage circumscribed to the hippocampus, failed to acquire value-based contingencies, thus confirming that hippocampal engagement is necessary for task performance. Control experiments established that this impairment was not due to perceptual demands or memory load. Future research is needed to clarify the mechanisms by which the hippocampus contributes to value-based learning, but these findings point to a broader role for the hippocampus in goal-directed behaviors than previously appreciated.

Shared striatal activity in decisions to satisfy curiosity and hunger at the risk of electric shocks

Curiosity is often portrayed as a desirable feature of human faculty. However, curiosity may come at a cost that sometimes puts people in harmful situations. Here, using a set of behavioural and neuroimaging experiments with stimuli that strongly trigger curiosity (for example, magic tricks), we examine the psychological and neural mechanisms underlying the motivational effect of curiosity. We consistently demonstrate that across different samples, people are indeed willing to gamble, subjecting themselves to electric shocks to satisfy their curiosity for trivial knowledge that carries no apparent instrumental value. Also, this influence of curiosity shares common neural mechanisms with that of hunger for food. In particular, we show that acceptance (compared to rejection) of curiosity-driven or incentive-driven gambles is accompanied by enhanced activity in the ventral striatum when curiosity or hunger was elicited, which extends into the dorsal striatum when participants made a decision.

Cerebellar Contribution to Motor and Non-motor Functions in Parkinson's Disease: A Meta-Analysis of fMRI Findings

Background: Parkinson's disease (PD) results in both motor and non-motor symptoms. Traditionally, the underlying mechanism of PD has been linked to neurodegeneration of the basal ganglia. Yet it does not adequately account for the non-motor symptoms of the disease, suggesting that other brain regions may be involved. One such region is the cerebellum, which is known to be involved, together with the basal ganglia, in both motor and non-motor functions. Many studies have found the cerebellum to be hyperactive in PD patients, a finding that is seldom discussed in detail, and warrants further examination. The current study thus aims to examine quantitively the current literature on the cerebellar involvement in both motor and non-motor functioning in PD. Methods: A meta-analysis of functional neuroimaging literature was conducted with Seed-based D mapping. Only the studies testing functional activation in response to motor and non-motor paradigms in PD and healthy controls (HC) were included in the meta-analysis. Separate analyses were conducted by including only studies with non-motor paradigms, as well as meta-regressions with UPDRS III scores and disease duration. Results: A total of 57 studies with both motor and non-motor paradigms fulfilled our inclusion criteria and were included in the meta-analysis, which revealed hyperactivity in Crus I-II and vermal III in PD patients compared to HC. An analysis including only studies with cognitive paradigms revealed a cluster of increased activity in PD patients encompassing lobule VIIB and VIII. Another meta-analysis including the only 20 studies that employed motor paradigms did not reveal any significant group differences. However, a descriptive analysis of these studies revealed that 60% of them reported cerebellar hyperactivations in PD and included motor paradigm with significant cognitive task demands, as opposed to 40% presenting the opposite pattern and using mainly force grip tasks. The meta-regression with UPDRS III scores found a negative association between motor scores and activation in lobule VI and vermal VII-VIII. No correlation was found with disease duration. Discussion: The present findings suggest that one of the main cerebellar implications in PD is linked to cognitive functioning. The negative association between UPDRS scores and activation in regions implicated in motor functioning indicate that there is less involvement of these areas as the disease severity increases. In contrast, the lack of correlation with disease duration seems to indicate that the cerebellar activity may be a compensatory mechanism to the dysfunctional basal ganglia, where certain sub-regions of the cerebellum are employed to cope with motor demands. Yet future longitudinal studies are needed to fully address this possibility.

Modulating the Use of Multiple Memory Systems in Value-based Decisions with Contextual Novelty

With multiple learning and memory systems at its disposal, the human brain can represent the past in many ways, from extracting regularities across similar experiences (incremental learning) to storing rich, idiosyncratic details of individual events (episodic memory). The unique information carried by these neurologically distinct forms of memory can bias our behavior in different directions, raising crucial questions about how these memory systems interact to guide choice and the factors that cause one to dominate. Here, we devised a new approach to estimate how decisions are independently influenced by episodic memories and incremental learning. Furthermore, we identified a biologically motivated factor that biases the use of different memory types-the detection of novelty versus familiarity. Consistent with computational models of cholinergic memory modulation, we find that choices are more influenced by episodic memories following the recognition of an unrelated familiar image but more influenced by incrementally learned values after the detection of a novel image. Together this work provides a new behavioral tool enabling the disambiguation of key memory behaviors thought to be supported by distinct neural systems while also identifying a theoretically important and broadly applicable manipulation to bias the arbitration between these two sources of memories.

Paradoxical Decision-Making: A Framework for Understanding Cognition in Parkinson's Disease

People with Parkinson's disease (PD) show impaired decision-making when sensory and memory information must be combined. This recently identified impairment results from an inability to accumulate the proper amount of information needed to make a decision and appears to be independent of dopamine tone and reinforcement learning mechanisms. Although considerable work focuses on PD and decisions involving risk and reward, in this Opinion article we propose that the emerging findings in perceptual decision-making highlight the multisystem nature of PD, and that unraveling the neuronal circuits underlying perceptual decision-making impairment may help in understanding other cognitive impairments in people with PD. We also discuss how a decision-making framework may be extended to gain insights into mechanisms of motor impairments in PD.

Separate neural representations of prediction error valence and surprise: Evidence from an fMRI meta-analysis

Learning occurs when an outcome differs from expectations, generating a reward prediction error signal (RPE). The RPE signal has been hypothesized to simultaneously embody the valence of an outcome (better or worse than expected) and its surprise (how far from expectations). Nonetheless, growing evidence suggests that separate representations of the two RPE components exist in the human brain. Meta-analyses provide an opportunity to test this hypothesis and directly probe the extent to which the valence and surprise of the error signal are encoded in separate or overlapping networks. We carried out several meta-analyses on a large set of fMRI studies investigating the neural basis of RPE, locked at decision outcome. We identified two valence learning systems by pooling studies searching for differential neural activity in response to categorical positive-versus-negative outcomes. The first valence network (negative > positive) involved areas regulating alertness and switching behaviours such as the midcingulate cortex, the thalamus and the dorsolateral prefrontal cortex whereas the second valence network (positive > negative) encompassed regions of the human reward circuitry such as the ventral striatum and the ventromedial prefrontal cortex. We also found evidence of a largely distinct surprise-encoding network including the anterior cingulate cortex, anterior insula and dorsal striatum. Together with recent animal and electrophysiological evidence this meta-analysis points to a sequential and distributed encoding of different components of the RPE signal, with potentially distinct functional roles.

The control of tonic pain by active relief learning

Tonic pain after injury characterises a behavioural state that prioritises recovery. Although generally suppressing cognition and attention, tonic pain needs to allow effective relief learning to reduce the cause of the pain. Here, we describe a central learning circuit that supports learning of relief and concurrently suppresses the level of ongoing pain. We used computational modelling of behavioural, physiological and neuroimaging data in two experiments in which subjects learned to terminate tonic pain in static and dynamic escape-learning paradigms. In both studies, we show that active relief-seeking involves a reinforcement learning process manifest by error signals observed in the dorsal putamen. Critically, this system uses an uncertainty ('associability') signal detected in pregenual anterior cingulate cortex that both controls the relief learning rate, and endogenously and parametrically modulates the level of tonic pain. The results define a self-organising learning circuit that reduces ongoing pain when learning about potential relief.

Association of grey matter changes with stability and flexibility of prediction in akinetic-rigid Parkinson's disease

Parkinson's disease (PD), which is caused by degeneration of dopaminergic neurons in the midbrain, results in a heterogeneous clinical picture including cognitive decline. Since the phasic signal of dopamine neurons is proposed to guide learning by signifying mismatches between subjects' expectations and external events, we here investigated whether akinetic-rigid PD patients without mild cognitive impairment exhibit difficulties in dealing with either relevant (requiring flexibility) or irrelevant (requiring stability) prediction errors. Following our previous study on flexibility and stability in prediction (Trempler et al. J Cogn Neurosci 29(2):298-309, 2017), we then assessed whether deficits would correspond with specific structural alterations in dopaminergic regions as well as in inferior frontal cortex, medial prefrontal cortex, and the hippocampus. Twenty-one healthy controls and twenty-one akinetic-rigid PD patients on and off medication performed a task which required to serially predict upcoming items. Switches between predictable sequences had to be indicated via button press, whereas sequence omissions had to be ignored. Independent of the disease, midbrain volume was related to a general response bias to unexpected events, whereas right putamen volume correlated with the ability to discriminate between relevant and irrelevant prediction errors. However, patients compared with healthy participants showed deficits in stabilisation against irrelevant prediction errors, associated with thickness of right inferior frontal gyrus and left medial prefrontal cortex. Flexible updating due to relevant prediction errors was also affected in patients compared with controls and associated with right hippocampus volume. Dopaminergic medication influenced behavioural performance across, but not within the patients. Our exploratory study warrants further research on deficient prediction error processing and its structural correlates as a core of cognitive symptoms occurring already in early stages of the disease.

Learning about Expectation Violation from Prediction Error Paradigms - A Meta-Analysis on Brain Processes Following a Prediction Error

Modifying patients' expectations by exposing them to expectation violation situations (thus maximizing the difference between the expected and the actual situational outcome) is proposed to be a crucial mechanism for therapeutic success for a variety of different mental disorders. However, clinical observations suggest that patients often maintain their expectations regardless of experiences contradicting their expectations. It remains unclear which information processing mechanisms lead to modification or persistence of patients' expectations. Insight in the processing could be provided by Neuroimaging studies investigating prediction error (PE, i.e., neuronal reactions to non-expected stimuli). Two methods are often used to investigate the PE: (1) paradigms, in which participants passively observe PEs ("passive" paradigms) and (2) paradigms, which encourage a behavioral adaptation following a PE ("active" paradigms). These paradigms are similar to the methods used to induce expectation violations in clinical settings: (1) the confrontation with an expectation violation situation and (2) an enhanced confrontation in which the patient actively challenges his expectation. We used this similarity to gain insight in the different neuronal processing of the two PE paradigms. We performed a meta-analysis contrasting neuronal activity of PE paradigms encouraging a behavioral adaptation following a PE and paradigms enforcing passiveness following a PE. We found more neuronal activity in the striatum, the insula and the fusiform gyrus in studies encouraging behavioral adaptation following a PE. Due to the involvement of reward assessment and avoidance learning associated with the striatum and the insula we propose that the deliberate execution of action alternatives following a PE is associated with the integration of new information into previously existing expectations, therefore leading to an expectation change. While further research is needed to directly assess expectations of participants, this study provides new insights into the information processing mechanisms following an expectation violation.

Reinforcement Learning and Episodic Memory in Humans and Animals: An Integrative Framework

We review the psychology and neuroscience of reinforcement learning (RL), which has experienced significant progress in the past two decades, enabled by the comprehensive experimental study of simple learning and decision-making tasks. However, one challenge in the study of RL is computational: The simplicity of these tasks ignores important aspects of reinforcement learning in the real world: (a) State spaces are high-dimensional, continuous, and partially observable; this implies that (b) data are relatively sparse and, indeed, precisely the same situation may never be encountered twice; furthermore, (c) rewards depend on the long-term consequences of actions in ways that violate the classical assumptions that make RL tractable. A seemingly distinct challenge is that, cognitively, theories of RL have largely involved procedural and semantic memory, the way in which knowledge about action values or world models extracted gradually from many experiences can drive choice. This focus on semantic memory leaves out many aspects of memory, such as episodic memory, related to the traces of individual events. We suggest that these two challenges are related. The computational challenge can be dealt with, in part, by endowing RL systems with episodic memory, allowing them to (a) efficiently approximate value functions over complex state spaces, (b) learn with very little data, and (c) bridge long-term dependencies between actions and rewards. We review the computational theory underlying this proposal and the empirical evidence to support it. Our proposal suggests that the ubiquitous and diverse roles of memory in RL may function as part of an integrated learning system.

The effects of methylphenidate on cerebral responses to conflict anticipation and unsigned prediction error in a stop-signal task

To adapt flexibly to a rapidly changing environment, humans must anticipate conflict and respond to surprising, unexpected events. To this end, the brain estimates upcoming conflict on the basis of prior experience and computes unsigned prediction error (UPE). Although much work implicates catecholamines in cognitive control, little is known about how pharmacological manipulation of catecholamines affects the neural processes underlying conflict anticipation and UPE computation. We addressed this issue by imaging 24 healthy young adults who received a 45 mg oral dose of methylphenidate (MPH) and 62 matched controls who did not receive MPH prior to performing the stop-signal task. We used a Bayesian Dynamic Belief Model to make trial-by-trial estimates of conflict and UPE during task performance. Replicating previous research, the control group showed anticipation-related activation in the presupplementary motor area and deactivation in the ventromedial prefrontal cortex and parahippocampal gyrus, as well as UPE-related activations in the dorsal anterior cingulate, insula, and inferior parietal lobule. In group comparison, MPH increased anticipation activity in the bilateral caudate head and decreased UPE activity in each of the aforementioned regions. These findings highlight distinct effects of catecholamines on the neural mechanisms underlying conflict anticipation and UPE, signals critical to learning and adaptive behavior.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Dopamine selectively remediates 'model-based' reward learning: a computational approach

Patients with loss of dopamine due to Parkinson's disease are impaired at learning from reward. However, it remains unknown precisely which aspect of learning is impaired. In particular, learning from reward, or reinforcement learning, can be driven by two distinct computational processes. One involves habitual stamping-in of stimulus-response associations, hypothesized to arise computationally from 'model-free' learning. The other, 'model-based' learning, involves learning a model of the world that is believed to support goal-directed behaviour. Much work has pointed to a role for dopamine in model-free learning. But recent work suggests model-based learning may also involve dopamine modulation, raising the possibility that model-based learning may contribute to the learning impairment in Parkinson's disease. To directly test this, we used a two-step reward-learning task which dissociates model-free versus model-based learning. We evaluated learning in patients with Parkinson's disease tested ON versus OFF their dopamine replacement medication and in healthy controls. Surprisingly, we found no effect of disease or medication on model-free learning. Instead, we found that patients tested OFF medication showed a marked impairment in model-based learning, and that this impairment was remediated by dopaminergic medication. Moreover, model-based learning was positively correlated with a separate measure of working memory performance, raising the possibility of common neural substrates. Our results suggest that some learning deficits in Parkinson's disease may be related to an inability to pursue reward based on complete representations of the environment.

From Affective Experience to Motivated Action: Tracking Reward-Seeking and Punishment-Avoidant Behaviour in Real-Life

Many of the decisions and actions in everyday life result from implicit learning processes. Important to psychopathology are, for example, implicit reward-seeking and punishment-avoidant learning processes. It is known that when specific actions get associated with a rewarding experience, such as positive emotions, that this will increase the likelihood that an organism will engage in similar actions in the future. Similarly, when actions get associated with punishing experiences, such as negative emotions, this may reduce the likelihood that the organism will engage in similar actions in the future. This study examines whether we can observe these implicit processes prospectively in the flow of daily life. If such processes take place then we expect that current behaviour can be predicted by how similar behaviour was experienced (in terms of positive and negative affect) at previous measurement moments. This was examined in a sample of 621 female individuals that had participated in an Experience Sampling data collection. Measures of affect and behaviour were collected at 10 semi-random moments of the day for 5 consecutive days. It was examined whether affective experience that was paired with certain behaviours (physical activity and social context) at previous measurements modified the likelihood to show similar behaviours at next measurement moments. Analyses were performed both at the level of observations (a time scale with units of ± 90 min) and at day level (a time scale with units of 24 h). As expected, we found that affect indeed moderated the extent to which previous behaviour predicted similar behaviour later in time, at both beep- and day-level. This study showed that it is feasible to track reward-seeking and punishment-avoidant behaviour prospectively in humans in the flow of daily life. This opens up a new toolbox to examine processes determining goal-oriented behaviour in relation to psychopathology in humans.

Decision-making impairments in Parkinson's disease as a by-product of defective cost-benefit analysis and feedback processing

Studies examining decision-making in people with Parkinson's disease (PD) show impaired performance on a variety of tasks. However, there are also demonstrations that patients with PD can make optimal decisions just like healthy age-matched controls. We propose that the reason for these mixed findings is that PD does not produce a generalized impairment of decision-making, but rather affects sub-components of this process. In this review we evaluate this hypothesis by considering the empirical evidence examining decision-making in PD. We suggest that of the various stages of the decision-making process, the most affected in PD are (1) the cost-benefit analysis stage and (2) the outcome evaluation stage. We consider the implications of this proposal for research in this area.

The effects of life stress and neural learning signals on fluid intelligence

Fluid intelligence (fluid IQ), defined as the capacity for rapid problem solving and behavioral adaptation, is known to be modulated by learning and experience. Both stressful life events (SLES) and neural correlates of learning [specifically, a key mediator of adaptive learning in the brain, namely the ventral striatal representation of prediction errors (PE)] have been shown to be associated with individual differences in fluid IQ. Here, we examine the interaction between adaptive learning signals (using a well-characterized probabilistic reversal learning task in combination with fMRI) and SLES on fluid IQ measures. We find that the correlation between ventral striatal BOLD PE and fluid IQ, which we have previously reported, is quantitatively modulated by the amount of reported SLES. Thus, after experiencing adversity, basic neuronal learning signatures appear to align more closely with a general measure of flexible learning (fluid IQ), a finding complementing studies on the effects of acute stress on learning. The results suggest that an understanding of the neurobiological correlates of trait variables like fluid IQ needs to take socioemotional influences such as chronic stress into account.

Mind matters: placebo enhances reward learning in Parkinson's disease

Expectations have a powerful influence on how we experience the world. Neurobiological and computational models of learning suggest that dopamine is crucial for shaping expectations of reward and that expectations alone may influence dopamine levels. However, because expectations and reinforcers are typically manipulated together, the role of expectations per se has remained unclear. Here, we separated these two factors using a placebo dopaminergic manipulation in Parkinson’s patients. We combined a reward learning task with fMRI to test how expectations of dopamine release modulate learning-related activity in the brain. We found that the mere expectation of dopamine release enhances reward learning and modulates learning-related signals in the striatum and the ventromedial prefrontal cortex. These effects were selective to learning from reward: neither medication nor placebo had an effect on learning to avoid monetary loss. These findings suggest a neurobiological mechanism by which expectations shape learning and affect.

Methamphetamine-induced neurotoxicity disrupts pharmacologically evoked dopamine transients in the dorsomedial and dorsolateral striatum

Phasic dopamine (DA) signaling, during which burst firing by DA neurons generates short-lived elevations in extracellular DA in terminal fields called DA transients, is implicated in reinforcement learning. Disrupted phasic DA signaling is proposed to link DA depletions and cognitive-behavioral impairment in methamphetamine (METH)-induced neurotoxicity. Here, we further investigated this disruption by assessing effects of METH pretreatment on DA transients elicited by a drug cocktail of raclopride, a D2 DA receptor antagonist, and nomifensine, an inhibitor of the dopamine transporter (DAT). One advantage of this approach is that pharmacological activation provides a large, high-quality data set of transients elicited by endogenous burst firing of DA neurons for analysis of regional differences and neurotoxicity. These pharmacologically evoked DA transients were measured in the dorsomedial (DM) and dorsolateral (DL) striatum of urethane-anesthetized rats by fast-scan cyclic voltammetry. Electrically evoked DA levels were also recorded to quantify DA release and uptake, and DAT binding was determined by means of autoradiography to index DA denervation. Pharmacologically evoked DA transients in intact animals exhibited a greater amplitude and frequency and shorter duration in the DM compared to the DL striatum, despite similar pre- and post-drug assessments of DA release and uptake in both sub-regions as determined from the electrically evoked DA signals. METH pretreatment reduced transient activity. The most prominent effect of METH pretreatment on transients across striatal sub-region was decreased amplitude, which mirrored decreased DAT binding and was accompanied by decreased DA release. Overall, these results identify marked intrastriatal differences in the activity of DA transients that appear independent of presynaptic mechanisms for DA release and uptake and further support disrupted phasic DA signaling mediated by decreased DA release in rats with METH-induced neurotoxicity.

SK channel blockade reverses cognitive and motor deficits induced by nigrostriatal dopamine lesions in rats

Parkinson's disease has traditionally been viewed as a motor disorder caused by the loss of dopamine (DA) neurons. However, emotional and cognitive syndromes can precede the onset of the motor deficits and provide an opportunity for therapeutic intervention. Potassium channels have recently emerged as potential new targets in the treatment of Parkinson's disease. The selective blockade of small conductance calcium-activated K+ channels (SK channels) by apamin is known to increase burst firing in midbrain DA neurons and therefore DA release. We thus investigated the effects of systemic administration of apamin on the motor, cognitive deficits and anxiety present after bilateral nigrostriatal 6-hydroxydopamine (6-OHDA) lesions in rats. Apamin administration (0.1 or 0.3 mg/kg i.p.) counteracted the depression, anxiety-like behaviors evaluated on sucrose consumption and in the elevated plus maze, social recognition and spatial memory deficits produced by partial 6-OHDA lesions. Apamin also reduced asymmetric motor deficits on circling behavior and postural adjustments in the unilateral extensive 6-OHDA model. The partial 6-OHDA lesions (56% striatal DA depletion) produced 20% decrease of iodinated apamin binding sites in the substantia nigra pars compacta in correlation with the loss of tyrosine hydroxylase positive cells, without modifying apamin binding in brain regions receiving DAergic innervation. Striatal extracellular levels of DA, not detectable after 6-OHDA lesions, were enhanced by apamin treatment as measured by in vivo microdialysis. These results indicate that blocking SK channels may reinstate minimal DA activity in the striatum to alleviate the non-motor symptoms induced by partial striatal DA lesions.

Role of the primate ventral tegmental area in reinforcement and motivation

Monkey electrophysiology suggests that the activity of the ventral tegmental area (VTA) helps regulate reinforcement learning and motivated behavior, in part by broadcasting prediction error signals throughout the reward system. However, electrophysiological studies do not allow causal inferences regarding the activity of VTA neurons with respect to these processes because they require artificial manipulation of neuronal firing. Rodent studies fulfilled this requirement by demonstrating that electrical and optogenetic VTA stimulation can induce learning and modulate downstream structures. Still, the primate dopamine system has diverged significantly from that of rodents, exhibiting greatly expanded and uniquely distributed cortical and subcortical innervation patterns. Here, we bridge the gap between rodent perturbation studies and monkey electrophysiology using chronic electrical microstimulation of macaque VTA (VTA-EM). VTA-EM was found to reinforce cue selection in an operant task and to motivate future cue selection using a Pavlovian paradigm. Moreover, by combining VTA-EM with concurrent fMRI, we demonstrated that VTA-EM increased fMRI activity throughout most of the dopaminergic reward system. These results establish a causative role for primate VTA in regulating stimulus-specific reinforcement and motivation as well as in modulating activity throughout the reward system.

A universal role of the ventral striatum in reward-based learning: evidence from human studies

Reinforcement learning enables organisms to adjust their behavior in order to maximize rewards. Electrophysiological recordings of dopaminergic midbrain neurons have shown that they code the difference between actual and predicted rewards, i.e., the reward prediction error, in many species. This error signal is conveyed to both the striatum and cortical areas and is thought to play a central role in learning to optimize behavior. However, in human daily life rewards are diverse and often only indirect feedback is available. Here we explore the range of rewards that are processed by the dopaminergic system in human participants, and examine whether it is also involved in learning in the absence of explicit rewards. While results from electrophysiological recordings in humans are sparse, evidence linking dopaminergic activity to the metabolic signal recorded from the midbrain and striatum with functional magnetic resonance imaging (fMRI) is available. Results from fMRI studies suggest that the human ventral striatum (VS) receives valuation information for a diverse set of rewarding stimuli. These range from simple primary reinforcers such as juice rewards over abstract social rewards to internally generated signals on perceived correctness, suggesting that the VS is involved in learning from trial-and-error irrespective of the specific nature of provided rewards. In addition, we summarize evidence that the VS can also be implicated when learning from observing others, and in tasks that go beyond simple stimulus-action-outcome learning, indicating that the reward system is also recruited in more complex learning tasks.

Hierarchical prediction errors in midbrain and basal forebrain during sensory learning

In Bayesian brain theories, hierarchically related prediction errors (PEs) play a central role for predicting sensory inputs and inferring their underlying causes, e.g., the probabilistic structure of the environment and its volatility. Notably, PEs at different hierarchical levels may be encoded by different neuromodulatory transmitters. Here, we tested this possibility in computational fMRI studies of audio-visual learning. Using a hierarchical Bayesian model, we found that low-level PEs about visual stimulus outcome were reflected by widespread activity in visual and supramodal areas but also in the midbrain. In contrast, high-level PEs about stimulus probabilities were encoded by the basal forebrain. These findings were replicated in two groups of healthy volunteers. While our fMRI measures do not reveal the exact neuron types activated in midbrain and basal forebrain, they suggest a dichotomy between neuromodulatory systems, linking dopamine to low-level PEs about stimulus outcome and acetylcholine to more abstract PEs about stimulus probabilities.

Neural systems for choice and valuation with counterfactual learning signals

The purpose of this experiment was to test a computational model of reinforcement learning with and without fictive prediction error (FPE) signals to investigate how counterfactual consequences contribute to acquired representations of action-specific expected value, and to determine the functional neuroanatomy and neuromodulator systems that are involved. 80 male participants underwent dietary depletion of either tryptophan or tyrosine/phenylalanine to manipulate serotonin (5HT) and dopamine (DA), respectively. They completed 80 rounds (240 trials) of a strategic sequential investment task that required accepting interim losses in order to access a lucrative state and maximize long-term gains, while being scanned. We extended the standard Q-learning model by incorporating both counterfactual gains and losses into separate error signals. The FPE model explained the participants' data significantly better than a model that did not include counterfactual learning signals. Expected value from the FPE model was significantly correlated with BOLD signal change in the ventromedial prefrontal cortex (vmPFC) and posterior orbitofrontal cortex (OFC), whereas expected value from the standard model did not predict changes in neural activity. The depletion procedure revealed significantly different neural responses to expected value in the vmPFC, caudate, and dopaminergic midbrain in the vicinity of the substantia nigra (SN). Differences in neural activity were not evident in the standard Q-learning computational model. These findings demonstrate that FPE signals are an important component of valuation for decision making, and that the neural representation of expected value incorporates cortical and subcortical structures via interactions among serotonergic and dopaminergic modulator systems.

Cortical and hippocampal correlates of deliberation during model-based decisions for rewards in humans

How do we use our memories of the past to guide decisions we've never had to make before? Although extensive work describes how the brain learns to repeat rewarded actions, decisions can also be influenced by associations between stimuli or events not directly involving reward — such as when planning routes using a cognitive map or chess moves using predicted countermoves — and these sorts of associations are critical when deciding among novel options. This process is known as model-based decision making. While the learning of environmental relations that might support model-based decisions is well studied, and separately this sort of information has been inferred to impact decisions, there is little evidence concerning the full cycle by which such associations are acquired and drive choices. Of particular interest is whether decisions are directly supported by the same mnemonic systems characterized for relational learning more generally, or instead rely on other, specialized representations. Here, building on our previous work, which isolated dual representations underlying sequential predictive learning, we directly demonstrate that one such representation, encoded by the hippocampal memory system and adjacent cortical structures, supports goal-directed decisions. Using interleaved learning and decision tasks, we monitor predictive learning directly and also trace its influence on decisions for reward. We quantitatively compare the learning processes underlying multiple behavioral and fMRI observables using computational model fits. Across both tasks, a quantitatively consistent learning process explains reaction times, choices, and both expectation- and surprise-related neural activity. The same hippocampal and ventral stream regions engaged in anticipating stimuli during learning are also engaged in proportion to the difficulty of decisions. These results support a role for predictive associations learned by the hippocampal memory system to be recalled during choice formation.

We are always learning regularities in the world around us: where things are, and in what order we might find them. Our knowledge of these contingencies can be relied upon if we later want to use them to make decisions. However, there is little agreement about the neurobiological mechanism by which learned contingencies are deployed for decision making. These are different kinds of decisions than simple habits, in which we take actions that have in the past given us reward. Neural mechanisms of habitual decisions are well-described by computational reinforcement learning approaches, but have not often been applied to ‘model-based’ decisions that depend on learned contingencies. In this article, we apply reinforcement learning to investigate model-based decisions. We tested participants on a serial reaction time task with changing sequential contingencies, and choice probes that depend on these contingencies. Fitting computational models to reaction times, we show that two sets of predictions drive simple response behavior, only one of which is used to make choices. Using fMRI, we observed learning and decision-related activity in hippocampal and ventral cortical areas that is computationally linked to the learned contingencies used to make choices. These results suggest a critical role for a hippocampal-cortical network in model-based decisions for reward.

Migraine Mistakes: Error Awareness

Error awareness or detection is the conscious and subconscious processing to evaluate physiological signals that are different from a baseline or homeostatic level. Migraine is a unique neurological disorder in which there are repeated attacks interspersed by attack-free periods. These attacks are dynamic and multidimensional in the sense that sensory, affective, autonomic, and cognitive functions are altered and these changes evolve differently before (pre-ictal), during (ictal), and immediately after (post-ictal) an attack. Thus migraine serves as a model disease to understand how the brain monitors and react to the presence of errors.

Extraversion differentiates between model-based and model-free strategies in a reinforcement learning task

Prominent computational models describe a neural mechanism for learning from reward prediction errors, and it has been suggested that variations in this mechanism are reflected in personality factors such as trait extraversion. However, although trait extraversion has been linked to improved reward learning, it is not yet known whether this relationship is selective for the particular computational strategy associated with error-driven learning, known as model-free reinforcement learning, vs. another strategy, model-based learning, which the brain is also known to employ. In the present study we test this relationship by examining whether humans' scores on an extraversion scale predict individual differences in the balance between model-based and model-free learning strategies in a sequentially structured decision task designed to distinguish between them. In previous studies with this task, participants have shown a combination of both types of learning, but with substantial individual variation in the balance between them. In the current study, extraversion predicted worse behavior across both sorts of learning. However, the hypothesis that extraverts would be selectively better at model-free reinforcement learning held up among a subset of the more engaged participants, and overall, higher task engagement was associated with a more selective pattern by which extraversion predicted better model-free learning. The findings indicate a relationship between a broad personality orientation and detailed computational learning mechanisms. Results like those in the present study suggest an intriguing and rich relationship between core neuro-computational mechanisms and broader life orientations and outcomes.

Decision making: from neuroscience to psychiatry

Adaptive behaviors increase the likelihood of survival and reproduction and improve the quality of life. However, it is often difficult to identify optimal behaviors in real life due to the complexity of the decision maker's environment and social dynamics. As a result, although many different brain areas and circuits are involved in decision making, evolutionary and learning solutions adopted by individual decision makers sometimes produce suboptimal outcomes. Although these problems are exacerbated in numerous neurological and psychiatric disorders, their underlying neurobiological causes remain incompletely understood. In this review, theoretical frameworks in economics and machine learning and their applications in recent behavioral and neurobiological studies are summarized. Examples of such applications in clinical domains are also discussed for substance abuse, Parkinson's disease, attention-deficit/hyperactivity disorder, schizophrenia, mood disorders, and autism. Findings from these studies have begun to lay the foundations necessary to improve diagnostics and treatment for various neurological and psychiatric disorders.

Many hats: intratrial and reward level-dependent BOLD activity in the striatum and premotor cortex

Human functional magnetic resonance imaging (fMRI) studies, as well as lesion, drug, and single-cell recording studies in animals, suggest that the striatum plays a key role in associating sensory events with rewarding actions, both by facilitating reward processing and prediction (i.e., reinforcement learning) and by biasing and later updating action selection. Previous human neuroimaging research has failed to dissociate striatal activity associated with reward, stimulus, and response processing, and previous electrophysiological research in nonhuman animals has typically only examined single striatal subregions. Overcoming both these limitations, we isolated blood oxygen level-dependent (BOLD) signal associated with four intratrial processes (stimulus, preparation of response, response, and feedback) in a visuomotor learning task and examined activity associated with each within four striatal subregions (ventral striatum, putamen, head of the caudate nucleus, and body of the caudate) and the lateral premotor cortex. Overall, the striatum and lateral premotor cortex were recruited during all trial components, confirming their importance in all aspects of visuomotor learning. However, the caudate was most active at stimulus and feedback, whereas the putamen peaked in activity at response. Activation in the lateral premotor cortex was, surprisingly, strongest during stimulus and following response as feedback approached. Activity was additionally examined at three reward magnitudes. Reward magnitude affected neural activity only during stimulus in the caudate, putamen, and premotor cortex, whereas the ventral striatum showed reward sensitivity during both stimulus and feedback. Collectively, these results indicate that each striatal region makes a unique contribution to visuomotor learning through functions performed at different points within single trials.

Methamphetamine-induced neurotoxicity disrupts naturally occurring phasic dopamine signaling

Methamphetamine (METH) is a highly addictive drug that is also neurotoxic to central dopamine (DA) systems. Although striatal DA depletions induced by METH are associated with behavioral and cognitive impairments, the link between these phenomena remains poorly understood. Previous work in both METH-pretreated animals and the 6-hydroxydopamine model of Parkinson's disease suggests that a disruption of phasic DA signaling, which is important for learning and goal-directed behavior, may be such a link. However, previous studies used electrical stimulation to elicit phasic-like DA responses and were also performed under anesthesia, which alters DA neuron activity and presynaptic function. Here we investigated the consequences of METH-induced DA terminal loss on both electrically evoked phasic-like DA signals and so-called 'spontaneous' phasic DA transients measured by voltammetry in awake rats. Not ostensibly attributable to discrete stimuli, these subsecond DA changes may play a role in enhancing reward-cue associations. METH pretreatment reduced tissue DA content in the dorsomedial striatum and nucleus accumbens by ~55%. Analysis of phasic-like DA responses elicited by reinforcing stimulation revealed that METH pretreatment decreased their amplitude and underlying mechanisms for release and uptake to a similar degree as DA content in both striatal subregions. Most importantly, characteristics of DA transients were altered by METH-induced DA terminal loss, with amplitude and frequency decreased and duration increased. These results demonstrate for the first time that denervation of DA neurons alters naturally occurring DA transients and are consistent with diminished phasic DA signaling as a plausible mechanism linking METH-induced striatal DA depletions and cognitive deficits.

Attenuated neural response to gamble outcomes in drug-naive patients with Parkinson's disease

Parkinson's disease results from the degeneration of dopaminergic neurons in the substantia nigra, manifesting as a spectrum of motor, cognitive and affective deficits. Parkinson's disease also affects reward processing, but disease-related deficits in reinforcement learning are thought to emerge at a slower pace than motor symptoms as the degeneration progresses from dorsal to ventral striatum. Dysfunctions in reward processing are difficult to study in Parkinson's disease as most patients have been treated with dopaminergic drugs, which sensitize reward responses in the ventral striatum, commonly resulting in impulse control disorders. To circumvent this treatment confound, we assayed the neural basis of reward processing in a group of newly diagnosed patients with Parkinson's disease that had never been treated with dopaminergic drugs. Thirteen drug-naive patients with Parkinson's disease and 12 healthy age-matched control subjects underwent whole-brain functional magnetic resonance imaging while they performed a simple two-choice gambling task resulting in stochastic and parametrically variable monetary gains and losses. In patients with Parkinson's disease, the neural response to reward outcome (as reflected by the blood oxygen level-dependent signal) was attenuated in a large group of mesolimbic and mesocortical regions, comprising the ventral putamen, ventral tegmental area, thalamus and hippocampus. Although these regions showed a linear response to reward outcome in healthy individuals, this response was either markedly reduced or undetectable in drug-naive patients with Parkinson's disease. The results show that the core regions of the meso-cortico-limbic dopaminergic system, including the ventral tegmental area, ventral striatum, and medial orbitofrontal cortex, are already significantly compromised in the early stages of the disease and that these deficits cannot be attributed to the contaminating effect of dopaminergic treatment.

Dissociating hippocampal and striatal contributions to sequential prediction learning

Behavior may be generated on the basis of many different kinds of learned contingencies. For instance, responses could be guided by the direct association between a stimulus and response, or by sequential stimulus-stimulus relationships (as in model-based reinforcement learning or goal-directed actions). However, the neural architecture underlying sequential predictive learning is not well understood, in part because it is difficult to isolate its effect on choice behavior. To track such learning more directly, we examined reaction times (RTs) in a probabilistic sequential picture identification task in healthy individuals. We used computational learning models to isolate trial-by-trial effects of two distinct learning processes in behavior, and used these as signatures to analyse the separate neural substrates of each process. RTs were best explained via the combination of two delta rule learning processes with different learning rates. To examine neural manifestations of these learning processes, we used functional magnetic resonance imaging to seek correlates of time-series related to expectancy or surprise. We observed such correlates in two regions, hippocampus and striatum. By estimating the learning rates best explaining each signal, we verified that they were uniquely associated with one of the two distinct processes identified behaviorally. These differential correlates suggest that complementary anticipatory functions drive each region's effect on behavior. Our results provide novel insights as to the quantitative computational distinctions between medial temporal and basal ganglia learning networks and enable experiments that exploit trial-by-trial measurement of the unique contributions of both hippocampus and striatum to response behavior.

Generalization of value in reinforcement learning by humans

Research in decision-making has focused on the role of dopamine and its striatal targets in guiding choices via learned stimulus-reward or stimulus-response associations, behavior that is well described by reinforcement learning theories. However, basic reinforcement learning is relatively limited in scope and does not explain how learning about stimulus regularities or relations may guide decision-making. A candidate mechanism for this type of learning comes from the domain of memory, which has highlighted a role for the hippocampus in learning of stimulus-stimulus relations, typically dissociated from the role of the striatum in stimulus-response learning. Here, we used functional magnetic resonance imaging and computational model-based analyses to examine the joint contributions of these mechanisms to reinforcement learning. Humans performed a reinforcement learning task with added relational structure, modeled after tasks used to isolate hippocampal contributions to memory. On each trial participants chose one of four options, but the reward probabilities for pairs of options were correlated across trials. This (uninstructed) relationship between pairs of options potentially enabled an observer to learn about option values based on experience with the other options and to generalize across them. We observed blood oxygen level-dependent (BOLD) activity related to learning in the striatum and also in the hippocampus. By comparing a basic reinforcement learning model to one augmented to allow feedback to generalize between correlated options, we tested whether choice behavior and BOLD activity were influenced by the opportunity to generalize across correlated options. Although such generalization goes beyond standard computational accounts of reinforcement learning and striatal BOLD, both choices and striatal BOLD activity were better explained by the augmented model. Consistent with the hypothesized role for the hippocampus in this generalization, functional connectivity between the ventral striatum and hippocampus was modulated, across participants, by the ability of the augmented model to capture participants' choice. Our results thus point toward an interactive model in which striatal reinforcement learning systems may employ relational representations typically associated with the hippocampus.

Reward-related learning via multiple memory systems

The application of a neuroeconomic approach to the study of reward-related processes has provided significant insights in our understanding of human learning and decision making. Much of this research has focused primarily on the contributions of the corticostriatal circuitry, involved in trial-and-error reward learning. As a result, less consideration has been allotted to the potential influence of different neural mechanisms such as the hippocampus or to more common ways in human society in which information is acquired and utilized to reach a decision, such as through explicit instruction rather than trial-and-error learning. This review examines the individual contributions of multiple learning and memory neural systems and their interactions during human decision making in both normal and neuropsychiatric populations. Specifically, the anatomical and functional connectivity across multiple memory systems are highlighted to suggest that probing the role of the hippocampus and its interactions with the corticostriatal circuitry via the application of model-based neuroeconomic approaches may provide novel insights into neuropsychiatric populations that suffer from damage to one of these structures and as a consequence have deficits in learning, memory, or decision making.

Serotonin selectively modulates reward value in human decision-making

Establishing a function for the neuromodulator serotonin in human decision-making has proved remarkably difficult because if its complex role in reward and punishment processing. In a novel choice task where actions led concurrently and independently to the stochastic delivery of both money and pain, we studied the impact of decreased brain serotonin induced by acute dietary tryptophan depletion. Depletion selectively impaired both behavioral and neural representations of reward outcome value, and hence the effective exchange rate by which rewards and punishments were compared. This effect was computationally and anatomically distinct from a separate effect on increasing outcome-independent choice perseveration. Our results provide evidence for a surprising role for serotonin in reward processing, while illustrating its complex and multifarious effects.

Dopamine and performance in a reinforcement learning task: evidence from Parkinson's disease

The role dopamine plays in decision-making has important theoretical, empirical and clinical implications. Here, we examined its precise contribution by exploiting the lesion deficit model afforded by Parkinson’s disease. We studied patients in a two-stage reinforcement learning task, while they were ON and OFF dopamine replacement medication. Contrary to expectation, we found that dopaminergic drug state (ON or OFF) did not impact learning. Instead, the critical factor was drug state during the performance phase, with patients ON medication choosing correctly significantly more frequently than those OFF medication. This effect was independent of drug state during initial learning and appears to reflect a facilitation of generalization for learnt information. This inference is bolstered by our observation that neural activity in nucleus accumbens and ventromedial prefrontal cortex, measured during simultaneously acquired functional magnetic resonance imaging, represented learnt stimulus values during performance. This effect was expressed solely during the ON state with activity in these regions correlating with better performance. Our data indicate that dopamine modulation of nucleus accumbens and ventromedial prefrontal cortex exerts a specific effect on choice behaviour distinct from pure learning. The findings are in keeping with the substantial other evidence that certain aspects of learning are unaffected by dopamine lesions or depletion, and that dopamine plays a key role in performance that may be distinct from its role in learning.

Mechanisms of hierarchical reinforcement learning in corticostriatal circuits 1: computational analysis

Growing evidence suggests that the prefrontal cortex (PFC) is organized hierarchically, with more anterior regions having increasingly abstract representations. How does this organization support hierarchical cognitive control and the rapid discovery of abstract action rules? We present computational models at different levels of description. A neural circuit model simulates interacting corticostriatal circuits organized hierarchically. In each circuit, the basal ganglia gate frontal actions, with some striatal units gating the inputs to PFC and others gating the outputs to influence response selection. Learning at all of these levels is accomplished via dopaminergic reward prediction error signals in each corticostriatal circuit. This functionality allows the system to exhibit conditional if-then hypothesis testing and to learn rapidly in environments with hierarchical structure. We also develop a hybrid Bayesian-reinforcement learning mixture of experts (MoE) model, which can estimate the most likely hypothesis state of individual participants based on their observed sequence of choices and rewards. This model yields accurate probabilistic estimates about which hypotheses are attended by manipulating attentional states in the generative neural model and recovering them with the MoE model. This 2-pronged modeling approach leads to multiple quantitative predictions that are tested with functional magnetic resonance imaging in the companion paper.

Ventral striatal prediction error signaling is associated with dopamine synthesis capacity and fluid intelligence

Fluid intelligence represents the capacity for flexible problem solving and rapid behavioral adaptation. Rewards drive flexible behavioral adaptation, in part via a teaching signal expressed as reward prediction errors in the ventral striatum, which has been associated with phasic dopamine release in animal studies. We examined a sample of 28 healthy male adults using multimodal imaging and biological parametric mapping with (1) functional magnetic resonance imaging during a reversal learning task and (2) in a subsample of 17 subjects also with positron emission tomography using 6-[(18) F]fluoro-L-DOPA to assess dopamine synthesis capacity. Fluid intelligence was measured using a battery of nine standard neuropsychological tests. Ventral striatal BOLD correlates of reward prediction errors were positively correlated with fluid intelligence and, in the right ventral striatum, also inversely correlated with dopamine synthesis capacity (FDOPA K inapp). When exploring aspects of fluid intelligence, we observed that prediction error signaling correlates with complex attention and reasoning. These findings indicate that individual differences in the capacity for flexible problem solving relate to ventral striatal activation during reward-related learning, which in turn proved to be inversely associated with ventral striatal dopamine synthesis capacity.