Tonic dopamine modulates exploitation of reward learning

Ready, set, explore! Event-related potentials reveal the time-course of exploratory decisions

The decision trade-off between exploiting the known and exploring the unknown has been studied using a variety of approaches and techniques. Surprisingly, electroencephalography (EEG) has been underused in this area of study, even though its high temporal resolution has the potential to reveal the time-course of exploratory decisions. We addressed this issue by recording EEG data while participants tried to win as many points as possible in a two-choice gambling task called a two-armed bandit. After using a computational model to classify responses as either exploitations or explorations, we examined event-related potentials locked to two events preceding decisions to exploit/explore: the arrival of feedback, and the subsequent appearance of the next trial's choice stimuli. In particular, we examined the feedback-locked P300 component, thought to index a phasic release of norepinephrine (a neural interrupt signal), and the reward positivity, thought to index a phasic release of dopamine (a neural prediction error signal). We observed an exploration-dependent enhancement of the P300 only, suggesting a critical role of norepinephrine (but not dopamine) in triggering decisions to explore. Similarly, we examined the N200/P300 components evoked by the appearance of the choice stimuli. In this case, exploration was characterized by an enhancement of the N200, but not P300, a result we attribute to increased response conflict. These results demonstrate the usefulness of combining computational and EEG methodologies, and suggest that exploratory decisions are preceded by two characterizing events: a feedback-locked neural interrupt signal (enhanced P300), and a choice-locked increase in response conflict (enhanced N200).

Dopamine blockade impairs the exploration-exploitation trade-off in rats

In a volatile environment where rewards are uncertain, successful performance requires a delicate balance between exploitation of the best option and exploration of alternative choices. It has theoretically been proposed that dopamine contributes to the control of this exploration-exploitation trade-off, specifically that the higher the level of tonic dopamine, the more exploitation is favored. We demonstrate here that there is a formal relationship between the rescaling of dopamine positive reward prediction errors and the exploration-exploitation trade-off in simple non-stationary multi-armed bandit tasks. We further show in rats performing such a task that systemically antagonizing dopamine receptors greatly increases the number of random choices without affecting learning capacities. Simulations and comparison of a set of different computational models (an extended Q-learning model, a directed exploration model, and a meta-learning model) fitted on each individual confirm that, independently of the model, decreasing dopaminergic activity does not affect learning rate but is equivalent to an increase in random exploration rate. This study shows that dopamine could adapt the exploration-exploitation trade-off in decision-making when facing changing environmental contingencies.

Anhedonia modulates the effects of positive mood induction on reward-related brain activation

Blunted activation in the reward circuitry has been associated with anhedonia, the inability to experience pleasure in previously rewarding activities. In healthy individuals, reward-related activation has been found to be modulated by acute contextual factors such as induced positive mood. Accordingly, blunted reward response in anhedonia might involve a failure to appropriately modulate reward-related activation as a function of context. To test this hypothesis, 29 participants (19 females, mean age of 24.14 ± 4.61, age range 18-34), with a wide range of anhedonic symptoms, underwent functional MRI while anticipating and receiving monetary rewards, before and after a positive mood induction. Change in neural activation from before to after mood induction was quantified, and effects of anhedonia were investigated through whole-brain, ROI, and functional connectivity analyses. Contrary to hypotheses, results indicated that during reward anticipation (but not receipt), nucleus accumbens activation decreased while its connectivity with the dorsolateral prefrontal cortex increased, following positive mood induction. Critically, anhedonia modulated both effects. The unexpected finding of decreased activation to reward cues following positive mood induction is compelling as it aligns with a prominent behavioral model of the effect of positive mood on exploration of rewarding and neutral stimuli. Furthermore, the modulation of this effect by anhedonia suggests that it may be a key process altered in anhedonia.

The Role of Frontostriatal Systems in Instructed Reinforcement Learning: Evidence From Genetic and Experimentally-Induced Variation

Instructions have a powerful effect on learning and decision-making, biasing choice even in the face of disconfirming feedback. Detrimental biasing effects have been reported in a number of studies in which instruction was given prior to trial-and-error learning. Previous work has attributed individual differences in instructional bias to variations in prefrontal and striatal dopaminergic genes, suggesting a role for prefrontally-mediated cognitive control processes in biasing learning. The current study replicates and extends these findings. Human subjects performed a probabilistic reinforcement learning task after receiving inaccurate instructions about the quality of one of the options. In order to establish a causal relationship between prefrontal cortical mechanisms and instructional bias, we applied transcranial direct current stimulation over dorsolateral prefrontal cortex (anodal, cathodal, or sham) while subjects performed the task. We additionally genotyped subjects for the COMT Val158Met genetic polymorphism, which influences the breakdown of prefrontal dopamine, and for the DAT1/SLC6A3 variable number tandem repeat, which affects expression of striatal dopamine transporter. We replicated the finding that the COMT Met allele is associated with increased instructional bias and further demonstrated that variation in DAT1 has similar effects to variation in COMT, with 9-repeat carriers demonstrating increased bias relative to 10-repeat homozygotes. Consistent with increased top-down regulation of reinforcement learning, anodal subjects demonstrated greater bias relative to sham, though this effect was present only early in training. In contrast, there was no effect of cathodal stimulation. Finally, we fit computational models to subjects' data to better characterize the mechanisms underlying instruction bias. A novel choice bias model, in which instructions influence decision-making rather than learning, was found to best account for subjects' behavior. Overall, these data provide further evidence for the role of frontostriatal interactions in biasing instructed reinforcement learning, which adds to the growing literature documenting both costs and benefits of cognitive control.

Daily-life stress differentially impacts ventral striatal dopaminergic modulation of reward processing in first-degree relatives of individuals with psychosis

Emerging evidence shows that stress can impair the ability to learn from and pursue rewards, which in turn has been linked to motivational impairments characteristic of the psychotic disorder. Ventral striatal dopaminergic neurotransmission has been found to modulate reward processing, and appears to be disrupted by exposure to stress. We investigated the hypothesis that stress experienced in the everyday life has a blunting effect on reward-induced dopamine release in the ventral striatum of 16 individuals at a familial risk for psychosis compared to 16 matched control subjects. Six days of ecological momentary assessments quantified the amount of daily-life stress prior to [¹⁸F]fallypride PET imaging while performing a probabilistic reinforcement learning task. Relative to the controls, individuals at a familial risk for psychosis who encountered more daily-life stress showed significantly diminished extent of reward-induced dopamine release in the right ventral striatum, as well as poorer performance on the reward task. These findings provide the first neuromolecular evidence for stress-related deregulation of reward processing in familial predisposition to psychosis. The implication of daily-life stress in compromised modulation of reward function may facilitate the design of targeted neuropharmacological and ecological interventions.

L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action

Dopamine is a key neurotransmitter in action control. However, influential theories of dopamine function make conflicting predictions about the effect of boosting dopamine neurotransmission. Here, we tested if increases in dopamine tone by administration of L-DOPA upregulate reward learning as predicted by reinforcement learning theories, and if increases are specific for deliberative "model-based" control or reflexive "model-free" control. Alternatively, L-DOPA may impair learning as suggested by "value" or "thrift" theories of dopamine. To this end, we employed a two-stage Markov decision-task to investigate the effect of L-DOPA (randomized cross-over) on behavioral control while brain activation was measured using fMRI. L-DOPA led to attenuated model-free control of behavior as indicated by the reduced impact of reward on choice. Increased model-based control was only observed in participants with high working memory capacity. Furthermore, L-DOPA facilitated exploratory behavior, particularly after a stream of wins in the task. Correspondingly, in the brain, L-DOPA decreased the effect of reward at the outcome stage and when the next decision had to be made. Critically, reward-learning rates and prediction error signals were unaffected by L-DOPA, indicating that differences in behavior and brain response to reward were not driven by differences in learning. Taken together, our results suggest that L-DOPA reduces model-free control of behavior by attenuating the transfer of value to action. These findings provide support for the value and thrift accounts of dopamine and call for a refined integration of valuation and action signals in reinforcement learning models.

Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats

Risky decisions are inherently characterized by the potential to receive gains or incur losses, and these outcomes have distinct effects on subsequent decision-making. One important factor is that individuals engage in loss-chasing, in which the reception of a loss is followed by relatively increased risk-taking. Unfortunately, the mechanisms of loss-chasing are poorly understood, despite the potential importance for understanding pathological choice behavior. The goal of the present experiment was to illuminate the mechanisms governing individual differences in loss-chasing and risky-choice behaviors. Rats chose between a low-uncertainty outcome that always delivered a variable amount of reward and a high-uncertainty outcome that probabilistically delivered reward. Loss-processing and loss-chasing were assessed in the context of losses disguised as wins (LDWs), which are loss outcomes that are presented along with gain-related stimuli. LDWs have been suggested to interfere with adaptive decision-making in humans and thus potentially increase loss-making. Here, the rats presented with LDWs were riskier, in that they made more choices for the high-uncertainty outcome. A series of nonlinear models were fit to individual rats' data to elucidate the possible psychological mechanisms that best account for individual differences in high-uncertainty choices and loss-chasing behaviors. The models suggested that the rats presented with LDWs were more prone to showing a stay bias following high-uncertainty outcomes compared to rats not presented with LDWs. These results collectively suggest that LDWs acquire conditioned reinforcement properties that encourage continued risk-taking and increase loss-chasing following previous high-risk decisions. (PsycINFO Database Record

Phasic Stimulation of Midbrain Dopamine Neuron Activity Reduces Salt Consumption

Salt intake is an essential dietary requirement, but excessive consumption is implicated in hypertension and associated conditions. Little is known about the neural circuit mechanisms that control motivation to consume salt, although the midbrain dopamine system, which plays a key role in other reward-related behaviors, has been implicated. We, therefore, examined the effects on salt consumption of either optogenetic excitation or chemogenetic inhibition of ventral tegmental area (VTA) dopamine neurons in male mice. Strikingly, optogenetic excitation of dopamine neurons decreased salt intake in a rapid and reversible manner, despite a strong salt appetite. Importantly, optogenetic excitation was not aversive, did not induce hyperactivity, and did not alter salt concentration preferences in a need-free state. In addition, we found that chemogenetic inhibition of dopamine neurons had no effect on salt intake. Lastly, optogenetic excitation of dopamine neurons reduced consumption of sucrose following an overnight fast, suggesting a more general role of VTA dopamine neuron excitation in organizing motivated behaviors.

Physiological Measures of Dopaminergic and Noradrenergic Activity During Attentional Set Shifting and Reversal

Dopamine (DA) and noradrenaline (NA) are important neurotransmitters, which are suggested to play a vital role in modulating the neural circuitry involved in the executive control of cognition. One way to investigate the functions of these neurotransmitter systems is to assess physiological indices of DA and NA transmission. Here we examined how variations of spontaneous eye-blink rate and pupil size, as indirect measures of DA and NA activity, respectively, are related to performance in a hallmark aspect of executive control: attentional set shifting. We used the Intra/Extradimensional Set Shifting Task, where participants have to choose between different compound stimuli while the stimulus-reward contingencies change periodically. During such rule shifts, participants have to refresh their attentional set while they reassess which stimulus-features are relevant. We found that both eye-blink rate (EBR) and pupil size increased after rule shifts, when explorative processes are required to establish stimulus–reward contingencies. Furthermore, baseline pupil size was related to performance during the most difficult, extradimensional set shifting stage, whereas baseline EBR was associated with task performance prior to this stage. Our results support a range of neurobiological models suggesting that the activity of DA and NA neurotransmitter systems determines individual differences in executive functions (EF), possibly by regulating neurotransmission in prefrontal circuits. We also suggest that assessing specific, easily accessible indirect physiological markers, such as pupil size and blink rate, contributes to the comprehension of the relationship between neurotransmitter systems and EF.

To Do or Not to Do: Dopamine, Affordability and the Economics of Opportunity

Five years ago, we introduced the thrift hypothesis of dopamine (DA), suggesting that the primary role of DA in adaptive behavior is regulating behavioral energy expenditure to match the prevailing economic conditions of the environment. Here we elaborate that hypothesis with several new ideas. First, we introduce the concept of affordability, suggesting that costs must necessarily be evaluated with respect to the availability of resources to the organism, which computes a value not only for the potential reward opportunity, but also the value of resources expended. Placing both costs and benefits within the context of the larger economy in which the animal is functioning requires consideration of the different timescales against which to compute resource availability, or average reward rate. Appropriate windows of computation for tracking resources requires corresponding neural substrates that operate on these different timescales. In discussing temporal patterns of DA signaling, we focus on a neglected form of DA plasticity and adaptation, changes in the physical substrate of the DA system itself, such as up- and down-regulation of receptors or release probability. We argue that changes in the DA substrate itself fundamentally alter its computational function, which we propose mediates adaptations to longer temporal horizons and economic conditions. In developing our hypothesis, we focus on DA D2 receptors (D2R), arguing that D2R implements a form of “cost control” in response to the environmental economy, serving as the “brain’s comptroller”. We propose that the balance between the direct and indirect pathway, regulated by relative expression of D1 and D2 DA receptors, implements affordability. Finally, as we review data, we discuss limitations in current approaches that impede fully investigating the proposed hypothesis and highlight alternative, more semi-naturalistic strategies more conducive to neuroeconomic investigations on the role of DA in adaptive behavior.

Separating the effect of reward from corrective feedback during learning in patients with Parkinson's disease

Parkinson's disease (PD) is associated with procedural learning deficits. Nonetheless, studies have demonstrated that reward-related learning is comparable between patients with PD and controls (Bódi et al., Brain, 132(9), 2385-2395, 2009; Frank, Seeberger, & O'Reilly, Science, 306(5703), 1940-1943, 2004; Palminteri et al., Proceedings of the National Academy of Sciences of the United States of America, 106(45), 19179-19184, 2009). However, because these studies do not separate the effect of reward from the effect of practice, it is difficult to determine whether the effect of reward on learning is distinct from the effect of corrective feedback on learning. Thus, it is unknown whether these group differences in learning are due to reward processing or learning in general. Here, we compared the performance of medicated PD patients to demographically matched healthy controls (HCs) on a task where the effect of reward can be examined separately from the effect of practice. We found that patients with PD showed significantly less reward-related learning improvements compared to HCs. In addition, stronger learning of rewarded associations over unrewarded associations was significantly correlated with smaller skin-conductance responses for HCs but not PD patients. These results demonstrate that when separating the effect of reward from the effect of corrective feedback, PD patients do not benefit from reward.

Differential coding of reward and movement information in the dorsomedial striatal direct and indirect pathways

The direct and indirect pathways of the basal ganglia have long been thought to mediate behavioral promotion and inhibition, respectively. However, this classic dichotomous model has been recently challenged. To better understand neural processes underlying reward-based learning and movement control, we recorded from direct (dSPNs) and indirect (iSPNs) pathway spiny projection neurons in the dorsomedial striatum of D1-Cre and D2-Cre mice performing a probabilistic Pavlovian conditioning task. dSPNs tend to increase activity while iSPNs decrease activity as a function of reward value, suggesting the striatum represents value in the relative activity levels of dSPNs versus iSPNs. Lick offset-related activity increase is largely dSPN selective, suggesting dSPN involvement in suppressing ongoing licking behavior. Rapid responses to negative outcome and previous reward-related responses are more frequent among iSPNs than dSPNs, suggesting stronger contributions of iSPNs to outcome-dependent behavioral adjustment. These findings provide new insights into striatal neural circuit operations.

Classically the direct and indirect pathways of basal ganglia are understood to have opposing roles in movement and reward learning, but recent work suggests a more complicated view. Here the authors further study indirect and direct pathway neurons, in the context of a probabilistic reward task.

Genetic Architecture of Natural Variation Underlying Adult Foraging Behavior That Is Essential for Survival of Drosophila melanogaster

Foraging behavior is critical for the fitness of individuals. However, the genetic basis of variation in foraging behavior and the evolutionary forces underlying such natural variation have rarely been investigated. We developed a systematic approach to assay the variation in survival rate in a foraging environment for adult flies derived from a wild Drosophila melanogaster population. Despite being such an essential trait, there is substantial variation of foraging behavior among D. melanogaster strains. Importantly, we provided the first evaluation of the potential caveats of using inbred Drosophila strains to perform genome-wide association studies on life-history traits, and concluded that inbreeding depression is unlikely a major contributor for the observed large variation in adult foraging behavior. We found that adult foraging behavior has a strong genetic component and, unlike larval foraging behavior, depends on multiple loci. Identified candidate genes are enriched in those with high expression in adult heads and, demonstrated by expression knock down assay, are involved in maintaining normal functions of the nervous system. Our study not only identified candidate genes for foraging behavior that is relevant to individual fitness, but also shed light on the initial stage underlying the evolution of the behavior.

Dopamine transporter mutant animals: a translational perspective

The dopamine transporter (DAT) plays an important homeostatic role in the control of both the extracellular and intraneuronal concentrations of dopamine, thereby providing effective control over activity of dopaminergic transmission. Since brain dopamine is known to be involved in numerous neuropsychiatric disorders, investigations using mice with genetically altered DAT function and thus intensity of dopamine-mediated signaling have provided numerous insights into the pathology of these disorders and novel pathological mechanisms that could be targeted to provide new therapeutic approaches for these disorders. In this brief overview, we discuss recent investigations involving animals with genetically altered DAT function, particularly focusing on translational studies providing new insights into pathology and pharmacology of dopamine-related disorders. Perspective applications of these and newly developed models of DAT dysfunction are also discussed.

Attentional deficits and altered neuronal activation in medial prefrontal and posterior parietal cortices in mice with reduced dopamine transporter levels

The executive control function of attention is regulated by the dopaminergic (DA) system. Dopamine transporter (DAT) likely plays a role in controlling the influence of DA on cognitive processes. We examined the effects of DAT depletion on cognitive processes related to attention. Mice with the DAT gene genetically deleted (DAT+/- heterozygotes) were compared to wild type (WT) mice on the Attentional Set-Shifting Task (ASST). Changes in neuronal activity during the ASST were shown with early growth response genes 1 and 2 (egr-1 and egr-2) immunohistochemistry in the medial prefrontal cortex (mPFC) and in the posterior parietal cortex (PPC). Heterozygotes were impaired in tasks that tax reversal learning, attentional-set formation and set-shifting. Densities of egr-2 labeled cells in the mPFC were lower in mutant mice when compared with wild-types in intradimensional shift of attention (IDS), extradimensional shift of attention and extradimensional shift of attention-reversal phases of the ASST task, and in PPC in the IDS phase of the task. The results demonstrate impairments of the areas associated with attentional functions in DAT+/- mice and show that an imbalance of the dopaminergic system has an impact on the complex attention-related executive functions.

A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Foraging is a fundamental behavior, and many types of animals appear to have solved foraging problems using a shared set of mechanisms. Perhaps the most common foraging problem is the choice between exploiting a familiar option for a known reward and exploring unfamiliar options for unknown rewards-the so-called explore/exploit trade-off. This trade-off has been studied extensively in behavioral ecology and computational neuroscience, but is relatively new to the field of psychiatry. Explore/exploit paradigms can offer psychiatry research a new approach to studying motivation, outcome valuation, and effort-related processes, which are disrupted in many mental and emotional disorders. In addition, the explore/exploit trade-off encompasses elements of risk-taking and impulsivity-common behaviors in psychiatric disorders-and provides a novel framework for understanding these behaviors within an ecological context. Here we explain relevant concepts and some common paradigms used to measure explore/exploit decisions in the laboratory, review clinically relevant research on the neurobiology and neuroanatomy of explore/exploit decision making, and discuss how computational psychiatry can benefit from foraging theory.

Striatal GPR88 Modulates Foraging Efficiency

The striatum is anatomically and behaviorally implicated in behaviors that promote efficient foraging. To investigate this function, we studied instrumental choice behavior in mice lacking GPR88, a striatum-enriched orphan G-protein-coupled receptor that modulates striatal medium spiny neuron excitability. Our results reveal that hungry mice lacking GPR88 (KO mice) were slow to acquire food-reinforced lever press but could lever press similar to controls on a progressive ratio schedule. Both WT and KO mice discriminated between reward and no-reward levers; however, KO mice failed to discriminate based on relative quantity-reward (1 vs 3 food pellets) or effort (3 vs 9 lever presses). We also demonstrate preference for the high-reward (3 pellet) lever was selectively reestablished when GPR88 expression was restored to the striatum. We propose that GPR88 expression within the striatum is integral to efficient action-selection during foraging.SIGNIFICANCE STATEMENT Evolutionary pressure driving energy homeostasis favored detection and comparison of caloric value. In wild and laboratory settings, neural systems involved in energy homeostasis bias foraging to maximize energy efficiency. This is observed when foraging behaviors are guided by superior nutritional density or minimized caloric expenditure. The striatum is anatomically and functionally well placed to perform the sensory and motor integration necessary for efficient action selection during foraging. However, few studies have examined this behavioral phenomenon or elucidated underlying molecular mechanisms. Both humans and mice with nonfunctional GPR88 have been shown to present striatal dysfunctions and impaired learning. We demonstrate that GPR88 expression is necessary to efficiently integrate effort and energy density information guiding instrumental choice.

Dopaminergic Medication Increases Reliance on Current Information in Parkinson's Disease

The neurotransmitter dopamine is crucial for decision-making under uncertainty but its computational role is still a subject of intense debate. To test potential roles, we had patients with Parkinson's disease (PD), who have less internally-generated dopamine, participate in a visual decision-making task in which uncertainty in both prior and current sensory information was varied and where behavior is often predicted by Bayesian statistics. We found that many aspects of uncertainty processing were conserved in PD: they could learn the prior uncertainty and utilize both priors and current sensory information. As predicted by prominent theories, we found that dopaminergic medication influenced the weight given to sensory information. However, as PD patients learn, this bias disappeared. In addition, throughout the experiment the patients exhibited lower sensitivity to current sensory uncertainty. Our results provide empirical evidence for the idea that dopamine levels, which are affected by PD and the drugs used for its treatment, influence the reliance on new information.

Role of dopamine neurotransmission in the long-term effects of repeated social defeat on the conditioned rewarding effects of cocaine

Numerous studies report that social defeat stress alters dopamine (DA) neurotransmission in several areas of the brain. Alterations of the mesolimbic dopaminergic pathway are believed to be responsible for the increased vulnerability to drug use observed as a result of social stress. In the present study, we evaluated the influence of DA receptors on the long-term effect of repeated social defeat (RSD) on the conditioned rewarding and reinstating effects of cocaine. For this purpose, the D1R antagonist SCH 23390 and the D1R antagonist raclopride were administered 30min before each social defeat and a cocaine-induced CPP procedure was initiated three weeks later. The expression of the D1R and D2R was also measured in the cortex and hippocampus throughout the entire procedure. Mice exposed to RSD showed an increase in the conditioned rewarding effects of cocaine that was blocked by both DA receptors antagonists when a subthreshold dose of cocaine was employed. However, while the vulnerability to reinstatement of the preference induced by 25mg/kg cocaine-induced CPP was abolished by the D1R antagonist, it was practically unaffected by raclopride. Increases in D2R receptor levels were observed in the cortex of defeated animals after the first and fourth social defeats and in the hippocampus 3weeks later. Nevertheless, D1R receptor levels in the hippocampus decreased only after the last social defeat. Our results confirm that RSD enhances the conditioned rewarding effects of cocaine and that both DA receptors are involved in this enduring effect of social stress.

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

Dopamine (DA) has been suggested to have two reward-related roles: (1) representing reward-prediction-error (RPE), and (2) providing motivational drive. Role(1) is based on the physiological results that DA responds to unpredicted but not predicted reward, whereas role(2) is supported by the pharmacological results that blockade of DA signaling causes motivational impairments such as slowdown of self-paced behavior. So far, these two roles are considered to be played by two different temporal patterns of DA signals: role(1) by phasic signals and role(2) by tonic/sustained signals. However, recent studies have found sustained DA signals with features indicative of both roles (1) and (2), complicating this picture. Meanwhile, whereas synaptic/circuit mechanisms for role(1), i.e., how RPE is calculated in the upstream of DA neurons and how RPE-dependent update of learned-values occurs through DA-dependent synaptic plasticity, have now become clarified, mechanisms for role(2) remain unclear. In this work, we modeled self-paced behavior by a series of ‘Go’ or ‘No-Go’ selections in the framework of reinforcement-learning assuming DA's role(1), and demonstrated that incorporation of decay/forgetting of learned-values, which is presumably implemented as decay of synaptic strengths storing learned-values, provides a potential unified mechanistic account for the DA's two roles, together with its various temporal patterns.

Frontostriatal Contribution to the Interplay of Flexibility and Stability in Serial Prediction

Surprising events may be relevant or irrelevant for behavior, requiring either flexible adjustment or stabilization of our model of the world and according response strategies. Cognitive flexibility and stability in response to environmental demands have been described as separable cognitive states, associated with activity of striatal and lateral prefrontal regions, respectively. It so far remains unclear, however, whether these two states act in an antagonistic fashion and which neural mechanisms mediate the selection of respective responses, on the one hand, and a transition between these states, on the other. In this study, we tested whether the functional dichotomy between striatal and prefrontal activity applies for the separate functions of updating (in response to changes in the environment, i.e., switches) and shielding (in response to chance occurrences of events violating expectations, i.e., drifts) of current predictions. We measured brain activity using fMRI while 20 healthy participants performed a task that required to serially predict upcoming items. Switches between predictable sequences had to be indicated via button press while sequence omissions (drifts) had to be ignored. We further varied the probability of switches and drifts to assess the neural network supporting the transition between flexible and stable cognitive states as a function of recent performance history in response to environmental demands. Flexible switching between models was associated with activation in medial pFC (BA 9 and BA 10), whereas stable maintenance of the internal model corresponded to activation in the lateral pFC (BA 6 and inferior frontal gyrus). Our findings extend previous studies on the interplay of flexibility and stability, suggesting that different prefrontal regions are activated by different types of prediction errors, dependent on their behavioral requirements. Furthermore, we found that striatal activation in response to switches and drifts was modulated by participants' successful behavior toward these events, suggesting the striatum to be responsible for response selections following unpredicted stimuli. Finally, we observed that the dopaminergic midbrain modulates the transition between different cognitive states, thresholded by participants' individual performance history in response to temporal environmental demands.

Spontaneous eye blink rate as predictor of dopamine-related cognitive function-A review

An extensive body of research suggests the spontaneous eye blink rate (EBR) is a non-invasive indirect marker of central dopamine (DA) function, with higher EBR predicting higher DA function. In the present review we provide a comprehensive overview of this literature. We broadly divide the available research in studies that aim to disentangle the dopaminergic underpinnings of EBR, investigate its utility in diagnosis of DA-related disorders and responsivity to drug treatment, and, lastly, investigate EBR as predictor of individual differences in DA-related cognitive performance. We conclude (i) EBR can reflect both DA receptor subtype D1 and D2 activity, although baseline EBR might be most strongly related to the latter, (ii) EBR can predict hypo- and hyperdopaminergic activity as well as normalization of this activity following treatment, and (iii) EBR can reliably predict individual differences in performance on many cognitive tasks, in particular those related to reward-driven behavior and cognitive flexibility. In sum, this review establishes EBR as a useful predictor of DA in a wide variety of contexts.

Generalized Anxiety Disorder and Social Anxiety Disorder, but Not Panic Anxiety Disorder, Are Associated with Higher Sensitivity to Learning from Negative Feedback: Behavioral and Computational Investigation

Anxiety disorders, including generalized anxiety disorder (GAD), social anxiety disorder (SAD), and panic anxiety disorder (PAD), are a group of common psychiatric conditions. They are characterized by excessive worrying, uneasiness, and fear of future events, such that they affect social and occupational functioning. Anxiety disorders can alter behavior and cognition as well, yet little is known about the particular domains they affect. In this study, we tested the cognitive correlates of medication-free patients with GAD, SAD, and PAD, along with matched healthy participants using a probabilistic category-learning task that allows the dissociation between positive and negative feedback learning. We also fitted all participants' data to a Q-learning model and various actor-critic models that examine learning rate parameters from positive and negative feedback to investigate effects of valence vs. action on performance. SAD and GAD patients were more sensitive to negative feedback than either PAD patients or healthy participants. PAD, SAD, and GAD patients did not differ in positive-feedback learning compared to healthy participants. We found that Q-learning models provide the simplest fit of the data in comparison to other models. However, computational analysis revealed that groups did not differ in terms of learning rate or exploration values. These findings argue that (a) not all anxiety spectrum disorders share similar cognitive correlates, but are rather different in ways that do not link them to the hallmark of anxiety (higher sensitivity to negative feedback); and (b) perception of negative consequences is the core feature of GAD and SAD, but not PAD. Further research is needed to examine the similarities and differences between anxiety spectrum disorders in other cognitive domains and potential implementation of behavioral therapy to remediate cognitive deficits.

Phasic Dopamine Transmission Reflects Initiation Vigor and Exerted Effort in an Action- and Region-Specific Manner

Initiating a reward-seeking behavior involves deciding on an action, how fast to initiate the action (initiation vigor), as well as how much effort to exert. These processes are thought to involve the mesolimbic dopamine system. Dopamine levels in the ventral striatum rise before initiating a reliably reinforced behavior. However, it is unknown whether dopamine is similarly involved with unreinforced actions (inactive lever presses, premature food port entries, insufficient number of active lever presses). Furthermore, does the dopamine response when initiating an action reflect specific aspects of motivated behavior, such as initiation vigor and exerted effort? Here, we analyzed voltammetry recordings of dopamine levels in the nucleus accumbens (NAcc) core and shell in rats working for food under a progressive ratio reinforcement schedule. We examined dopamine levels when rats initiated distinct actions (active lever presses, inactive lever presses, food port entries) that were temporally separated from cue- and reward-evoked dopamine release. Active lever pressing bouts were preceded by elevated dopamine release in the NAcc shell, as well as in the NAcc core, although only when rats exhibited high initiation vigor. Dopamine levels were transiently reduced in the NAcc core following an unreinforced food port entry and were unchanged throughout the NAcc when initiating inactive lever presses. The effort exerted and vigor to initiate a bout of active lever presses were signaled by dopamine transmission in the NAcc core, but not in the NAcc shell. These results demonstrate that the dopamine response when initiating a behavior is both region- and action-specific.

SIGNIFICANCE STATEMENT

Exogenous activation of the mesolimbic dopamine system facilitates motivated behavior. However, a direct relationship has not been established between endogenous phasic dopamine transmission and measures of motivation, such as the vigor to initiate an action and the effort exerted in a bout of activity. The present work demonstrates that the dopamine response when initiating an action depends both upon where dopamine is released and what action is performed. Furthermore, dopamine reflects measures of motivated behavior selectively within the nucleus accumbens core.

Cognitive Neurostimulation: Learning to Volitionally Sustain Ventral Tegmental Area Activation

Activation of the ventral tegmental area (VTA) and mesolimbic networks is essential to motivation, performance, and learning. Humans routinely attempt to motivate themselves, with unclear efficacy or impact on VTA networks. Using fMRI, we found untrained participants' motivational strategies failed to consistently activate VTA. After real-time VTA neurofeedback training, however, participants volitionally induced VTA activation without external aids, relative to baseline, Pre-test, and control groups. VTA self-activation was accompanied by increased mesolimbic network connectivity. Among two comparison groups (no neurofeedback, false neurofeedback) and an alternate neurofeedback group (nucleus accumbens), none sustained activation in target regions of interest nor increased VTA functional connectivity. The results comprise two novel demonstrations: learning and generalization after VTA neurofeedback training and the ability to sustain VTA activation without external reward or reward cues. These findings suggest theoretical alignment of ideas about motivation and midbrain physiology and the potential for generalizable interventions to improve performance and learning.

Humans Integrate Monetary and Liquid Incentives to Motivate Cognitive Task Performance

It is unequivocal that a wide variety of incentives can motivate behavior. However, few studies have explicitly examined whether and how different incentives are integrated in terms of their motivational influence. The current study examines the combined effects of monetary and liquid incentives on cognitive processing, and whether appetitive and aversive incentives have distinct influences. We introduce a novel task paradigm, in which participants perform cued task-switching for monetary rewards that vary parametrically across trials, with liquid incentives serving as post-trial performance feedback. Critically, the symbolic meaning of the liquid was held constant (indicating successful reward attainment), while liquid valence was blocked. In the first experiment, monetary rewards combined additively with appetitive liquid feedback to improve subject task performance. Aversive liquid feedback counteracted monetary reward effects in low monetary reward trials, particularly in a subset of participants who tended to avoid responding under these conditions. Self-report motivation ratings predicted behavioral performance above and beyond experimental effects. A follow-up experiment replicated the predictive power of motivation ratings even when only appetitive liquids were used, suggesting that ratings reflect idiosyncratic subjective values of, rather than categorical differences between, the liquid incentives. Together, the findings indicate an integrative relationship between primary and secondary incentives and potentially dissociable influences in modulating motivational value, while informing hypotheses regarding candidate neural mechanisms.

Development and function of the midbrain dopamine system: what we know and what we need to

The past two decades have seen an explosion in our understanding of the origin and development of the midbrain dopamine system. Much of this work has been focused on the aspects of dopamine neuron development related to the onset of movement disorders such as Parkinson's disease, with the intent of hopefully delaying, preventing or fixing symptoms. While midbrain dopamine degeneration is a major focus for treatment and research, many other human disorders are impacted by abnormal dopamine, including drug addiction, autism and schizophrenia. Understanding dopamine neuron ontogeny and how dopamine connections and circuitry develops may provide us with key insights into potentially important avenues of research for other dopamine-related disorders. This review will provide a brief overview of the major molecular and genetic players throughout the development of midbrain dopamine neurons and what we know about the behavioral- and disease-related implications associated with perturbations to midbrain dopamine neuron development. We intend to combine the knowledge of two broad fields of neuroscience, both developmental and behavioral, with the intent on fostering greater discussion between branches of neuroscience in the service of addressing complex cognitive questions from a developmental perspective and identifying important gaps in our knowledge for future study.

Dopamine Release Impairment in Striatum after Different Levels of Cerebral Cortical Fluid Percussion Injury

To investigate the role of dopamine release in cognitive impairment and motor learning deficits after brain injury, different levels of traumatic brain injury (TBI) were made in rats by using fluid percussion at two different atmospheres (2 Psi and 6 Psi). Tonic and phasic bursting dopamine release and behavior tests followed at several time points. We used in vitro fast-scan cyclic voltammetry to survey dopamine release in the striatum and analyzed the rats’ behavior using novel object recognition (NOR) and rotarod tests. Both tonic and bursting dopamine release were greatly depressed in the severely (6 Psi) injured group, which persisted up to 8 weeks later. However, in the 2 Psi-injured group, the suppression of bursting dopamine release occurred at 1~2 weeks after injury, but there were no significant differences after 4 weeks. Tonic dopamine release was also diminished significantly at 1~2 weeks after the injury; partial recovery could then be seen 4 weeks after injury. A significant deficiency in the fixed speed rotarod test and NOR test were noted in both 2 Psi and 6 Psi groups initially; however, the changes recovered in the 2 Psi group 2 weeks after injury while persisting in the 6 Psi group. In conclusion, striatal evoked dopamine release was affected by fluid percussion injury, with behavioral deficits showing differences as a function of injury severity. The severe fluid percussion injury (6 Psi) group showed more dopamine release defects, as well as cognitive and motor deficiencies. Recovery of dopamine release and improvement in behavioral impairment were better in the mild TBI group.

The dopamine antagonist flupentixol does not alter ghrelin-induced food intake in rats

It has been shown that dopamine antagonists suppress the ghrelin-induced increased motivation to work for food. The aim of this study was to investigate the influence of the dopamine antagonist flupentixol on ghrelin-induced food intake. Ad libitum fed male Sprague-Dawley (SD) rats were injected intraperitoneally (ip) with vehicle plus vehicle, vehicle plus ghrelin (13 μg/kg), 0.25mg/kg or 0.5mg/kg flupentixol plus ghrelin, or 0.25mg/kg or 0.5 mg/kg flupentixol plus vehicle. In a second experiment, intracerebroventricularly (icv) cannulated rats received an ip injection of vehicle (0.15M NaCl) or flupentixol (0.25mg/kg) and 20 min later an icv injection of vehicle or ghrelin (1 μg/rat). Both experiments were performed twice: first, rats were offered only standard chow, while in the second experiment they could choose between standard chow and a palatable/preferred chow. Cumulative light phase food intake was assessed for 7h. Ip as well as icv injected ghrelin reliably increased intake of standard chow. Flupentixol did not affect ghrelin-induced intake of standard chow. Ip injected ghrelin failed to increase the intake of palatable chow, whereas icv injected ghrelin did. This effect was not blocked by ip flupentixol. In summary, ip administered ghrelin did not increase the intake of chow the rats preferred; whereas icv injected ghrelin further stimulated the intake of preferred chow suggesting a direct central mediation of this effect. Our results show that the dopamine antagonist flupentixol does not influence ghrelin-induced feeding in our choice paradigm.

Effects of fictive reward on rat's choice behavior

Choices of humans and non-human primates are influenced by both actually experienced and fictive outcomes. To test whether this is also the case in rodents, we examined rat's choice behavior in a binary choice task in which variable magnitudes of actual and fictive rewards were delivered. We found that the animal's choice was significantly influenced by the magnitudes of both actual and fictive rewards in the previous trial. A model-based analysis revealed, however, that the effect of fictive reward was more transient and influenced mostly the choice in the next trial, whereas the effect of actual reward was more sustained, consistent with incremental learning of action values. Our results suggest that the capacity to modify future choices based on fictive outcomes might be shared by many different animal species, but fictive outcomes are less effective than actual outcomes in the incremental value learning system.

Nicotinic alteration of decision-making

Addiction to nicotine is characterized by impulses, urges and lack of self-control towards cigarettes. A key element in the process of addiction is the development of habits oriented towards nicotine consumption that surpass flexible systems as a consequence of a gradual adaptation to chronic drug exposure. However, the long-term effects of nicotine on brain circuits also induce wide changes in decision-making processes, affecting behaviors unrelated to cigarettes. This review aims at providing an update on the implications of nicotine on general decision-making processes, with an emphasis on impulsivity and risk-taking. As impulsivity is a rather ambiguous behavioral trait, we build on economic and normative theories to better characterize these nicotine-induced alterations in decision-making. Nonetheless, experimental data are sparse and often contradictory. We will discuss how the latest findings on the neurobiological basis of choice behavior may help disentangling these issues. We focus on the role of nicotine acetylcholine receptors and their different subunits, and on the spatio-temporal dynamics (i.e. diversity of the neural circuits, short- and long-term effects) of both endogenous acetylcholine and nicotine action. Finally, we try to link these neurobiological results with neuro-computational models of attention, valuation and action, and of the role of acetylcholine in these decision processes. This article is part of the Special Issue entitled 'The Nicotinic Acetylcholine Receptor: From Molecular Biology to Cognition'.

Exploration versus exploitation in space, mind, and society

Search is a ubiquitous property of life. Although diverse domains have worked on search problems largely in isolation, recent trends across disciplines indicate that the formal properties of these problems share similar structures and, often, similar solutions. Moreover, internal search (e.g., memory search) shows similar characteristics to external search (e.g., spatial foraging), including shared neural mechanisms consistent with a common evolutionary origin across species. Search problems and their solutions also scale from individuals to societies, underlying and constraining problem solving, memory, information search, and scientific and cultural innovation. In summary, search represents a core feature of cognition, with a vast influence on its evolution and processes across contexts and requiring input from multiple domains to understand its implications and scope.

Role of dopamine D2 receptors in optimizing choice strategy in a dynamic and uncertain environment

In order to investigate roles of dopamine receptor subtypes in reward-based learning, we examined choice behavior of dopamine D1 and D2 receptor-knockout (D1R-KO and D2R-KO, respectively) mice in an instrumental learning task with progressively increasing reversal frequency and a dynamic two-armed bandit task. Performance of D2R-KO mice was progressively impaired in the former as the frequency of reversal increased and profoundly impaired in the latter even with prolonged training, whereas D1R-KO mice showed relatively minor performance deficits. Choice behavior in the dynamic two-armed bandit task was well explained by a hybrid model including win-stay-lose-switch and reinforcement learning terms. A model-based analysis revealed increased win-stay, but impaired value updating and decreased value-dependent action selection in D2R-KO mice, which were detrimental to maximizing rewards in the dynamic two-armed bandit task. These results suggest an important role of dopamine D2 receptors in learning from past choice outcomes for rapid adjustment of choice behavior in a dynamic and uncertain environment.

Transcranial direct current stimulation over the left prefrontal cortex increases randomness of choice in instrumental learning

INTRODUCTION

There is growing evidence from neuro-computational studies that instrumental learning involves the dynamic interaction of a computationally rigid, low-level striatal and a more flexible, high-level prefrontal component.

METHODS

To evaluate the role of the prefrontal cortex in instrumental learning, we applied anodal transcranial direct current stimulation (tDCS) optimized for the left dorsolateral prefrontal cortex, by using realistic MR-derived finite element model-based electric field simulations. In a study with a double-blind, sham-controlled, repeated-measures design, sixteen male participants performed a probabilistic learning task while receiving anodal and sham tDCS in a counterbalanced order.

RESULTS

Compared to sham tDCS, anodal tDCS significantly increased the amount of maladaptive shifting behavior after optimal outcomes during learning when reward probabilities were highly dissociable. Derived parameters of the Q-learning computational model further revealed a significantly increased model parameter that was sensitive to random action selection in the anodal compared to the sham tDCS session, whereas the learning rate parameter was not influenced significantly by tDCS.

CONCLUSION

These results congruently indicate that prefrontal tDCS during instrumental learning increased randomness of choice, possibly reflecting the influence of the cognitive prefrontal component.

Bursts and heavy tails in temporal and sequential dynamics of foraging decisions

A fundamental understanding of behavior requires predicting when and what an individual will choose. However, the actual temporal and sequential dynamics of successive choices made among multiple alternatives remain unclear. In the current study, we tested the hypothesis that there is a general bursting property in both the timing and sequential patterns of foraging decisions. We conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. Regarding when choices were made, we found bursts of rapidly occurring actions, separated by time-varying inactive periods, partially based on a circadian rhythm. Regarding what was chosen, we found sequential dynamics in affective choices characterized by two key features: (a) a highly biased choice distribution; and (b) preferential attachment, in which the animals were more likely to choose what they had previously chosen. To capture the temporal dynamics, we propose a dual-state model consisting of active and inactive states. We also introduce a satiation-attainment process for bursty activity, and a non-homogeneous Poisson process for longer inactivity between bursts. For the sequential dynamics, we propose a dual-control model consisting of goal-directed and habit systems, based on outcome valuation and choice history, respectively. This study provides insights into how the bursty nature of behavior emerges from the interaction of different underlying systems, leading to heavy tails in the distribution of behavior over time and choices.

To understand spontaneous animal behavior, two key elements must be explained: when an action is made and what is chosen. Here, we conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. With respect to when, we found bursts of rapidly occurring responses separated by long inactive periods. With respect to what, we found biased choice behavior toward the favorite items as well as repetitive behavior, reflecting goal-directed and habitual responding, respectively. We account for the when and what components with two distinct computational mechanisms, each composed of two processes: (a) active and inactive states for the temporal dynamics, and (b) goal-directed and habitual control for the sequential dynamics. This study provides behavioral and computational insights into the dynamical properties of decision-making that determine both when an animal will act and what the animal will choose. Our findings provide an integrated framework for describing the temporal and sequential structure of everyday choices among, for example, food, music, books, brands, web-browsing and social interaction.

Dopamine modulates novelty seeking behavior during decision making

Novelty seeking refers to the tendency of humans and animals to explore novel and unfamiliar stimuli and environments. The idea that dopamine modulates novelty seeking is supported by evidence that novel stimuli excite dopamine neurons and activate brain regions receiving dopaminergic input. In addition, dopamine is shown to drive exploratory behavior in novel environments. It is not clear whether dopamine promotes novelty seeking when it is framed as the decision to explore novel options versus the exploitation of familiar options. To test this hypothesis, we administered systemic injections of saline or GBR-12909, a selective dopamine transporter (DAT) inhibitor, to monkeys and assessed their novelty seeking behavior during a probabilistic decision making task. The task involved pseudorandom introductions of novel choice options. This allowed monkeys the opportunity to explore novel options or to exploit familiar options that they had already sampled. We found that DAT blockade increased the monkeys' preference for novel options. A reinforcement learning (RL) model fit to the monkeys' choice data showed that increased novelty seeking after DAT blockade was driven by an increase in the initial value the monkeys assigned to novel options. However, blocking DAT did not modulate the rate at which the monkeys learned which cues were most predictive of reward or their tendency to exploit that knowledge. These data demonstrate that dopamine enhances novelty-driven value and imply that excessive novelty seeking-characteristic of impulsivity and behavioral addictions-might be caused by increases in dopamine, stemming from less reuptake.

Role of nicotinic acetylcholine receptors in regulating dopamine neuron activity

Midbrain dopamine (DA) neurons play a central role in a wide range of behaviors, from attention and motivation to motor control and reinforcement. The release of DA is modulated by a number of factors, and its deregulation has been implicated in multiple psychiatric disorders, such as addiction. In particular, nicotinic acetylcholine receptors (nAChRs) are key modulators of DA cells. Nicotine, the main addictive component in tobacco, strongly interacts with these receptors in the midbrain DA systems, resulting in reinforcing effects that are at the core of tobacco addiction. nAChRs are virtually expressed on every cell of the DA system, both at pre-, post- and extra-synaptic locations. The complex issue of interpreting the role of the large portfolio of different nAChR subtypes expressed on ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) neurons, and especially their role in defining functional DAergic subpopulations, is far from being solved. In this review we will try to provide the reader with an integrative view of the nicotinic modulation of DA neurons and its influence at the cellular, systemic and behavioral levels (exploratory behavior), as well as its implication in the reinforcing effects of nicotine.

Ghrelin acts as an interface between physiological state and phasic dopamine signaling

Brief, high-concentration (phasic) spikes in nucleus accumbens dopamine critically participate in aspects of food reward. Although physiological state (e.g., hunger, satiety) and associated hormones are known to affect dopamine tone in general, whether they modulate food-evoked, phasic dopamine specifically is unknown. Here, we used fast-scan cyclic voltammetry in awake, behaving rats to record dopamine spikes evoked by delivery of sugar pellets while pharmacologically manipulating central receptors for the gut "hunger" hormone ghrelin. Lateral ventricular (LV) ghrelin increased, while LV ghrelin receptor antagonism suppressed the magnitude of dopamine spikes evoked by food. Ghrelin was effective when infused directly into the lateral hypothalamus (LH), but not the ventral tegmental area (VTA). LH infusions were made in close proximity to orexin neurons, which are regulated by ghrelin and project to the VTA. Thus, we also investigated and found potentiation of food-evoked dopamine spikes by intra-VTA orexin-A. Importantly, intra-VTA blockade of orexin receptors attenuated food intake induced by LV ghrelin, thus establishing a behaviorally relevant connection between central ghrelin and VTA orexin. Further analysis revealed that food restriction increased the magnitude of dopamine spikes evoked by food independent of any pharmacological manipulations. The results support the regulation of food-evoked dopamine spikes by physiological state with endogenous fluctuations in ghrelin as a key contributor. Our data highlight a novel mechanism by which signals relating physiological state could influence food reinforcement and food-directed behavior.

Value learning and arousal in the extinction of probabilistic rewards: the role of dopamine in a modified temporal difference model

Because most rewarding events are probabilistic and changing, the extinction of probabilistic rewards is important for survival. It has been proposed that the extinction of probabilistic rewards depends on arousal and the amount of learning of reward values. Midbrain dopamine neurons were suggested to play a role in both arousal and learning reward values. Despite extensive research on modeling dopaminergic activity in reward learning (e.g. temporal difference models), few studies have been done on modeling its role in arousal. Although temporal difference models capture key characteristics of dopaminergic activity during the extinction of deterministic rewards, they have been less successful at simulating the extinction of probabilistic rewards. By adding an arousal signal to a temporal difference model, we were able to simulate the extinction of probabilistic rewards and its dependence on the amount of learning. Our simulations propose that arousal allows the probability of reward to have lasting effects on the updating of reward value, which slows the extinction of low probability rewards. Using this model, we predicted that, by signaling the prediction error, dopamine determines the learned reward value that has to be extinguished during extinction and participates in regulating the size of the arousal signal that controls the learning rate. These predictions were supported by pharmacological experiments in rats.

Frontopolar cortex and decision-making efficiency: comparing brain activity of experts with different professional background during an exploration-exploitation task

An optimal balance between efficient exploitation of available resources and creative exploration of alternatives is critical for adaptation and survival. Previous studies associated these behavioral drives with, respectively, the dopaminergic mesocorticolimbic system and frontopolar-intraparietal networks. We study the activation of these systems in two age and gender-matched groups of experienced decision-makers differing in prior professional background, with the aim to understand the neural bases of individual differences in decision-making efficiency (performance divided by response time). We compare brain activity of entrepreneurs (who currently manage the organization they founded based on their venture idea) and managers (who are constantly involved in making strategic decisions but have no venture experience) engaged in a gambling-task assessing exploitative vs. explorative decision-making. Compared with managers, entrepreneurs showed higher decision-making efficiency, and a stronger activation in regions of frontopolar cortex (FPC) previously associated with explorative choice. Moreover, activity across a network of regions previously linked to explore/exploit tradeoffs explained individual differences in choice efficiency. These results suggest new avenues for the study of individual differences in the neural antecedents of efficient decision-making.

Good riddance to dopamine: roles for the dopamine transporter in synaptic function and dopamine-associated brain disorders

The neurotransmitter dopamine (DA) plays a critical role in CNS circuits that provide for attention, executive function, reward responses, motivation and movement. DA is inactivated by the cocaine- and amphetamine-sensitive DA transporter (DAT), a protein that also provides a pathway for non-vesicular DA release. After a brief review of DAT function and psychostimulant actions, we consider the importance DAT in relation to the distinct firing patterns of DA neurons that permit awareness of novelty and reward. Finally, we review recent efforts to gather direct support for DAT-linked disorders, with a specific focus on DAT mutations recently identified in subjects with ADHD.

DAT1 polymorphism determines L-DOPA effects on learning about others' prosociality

Despite that a wealth of evidence links striatal dopamine to individualś reward learning performance in non-social environments, the neurochemical underpinnings of such learning during social interaction are unknown. Here, we show that the administration of 300 mg of the dopamine precursor L-DOPA to 200 healthy male subjects influences learning about a partners' prosocial preferences in a novel social interaction task, which is akin to a repeated trust game. We found learning to be modulated by a well-established genetic marker of striatal dopamine levels, the 40-bp variable number tandem repeats polymorphism of the dopamine transporter (DAT1 polymorphism). In particular, we found that L-DOPA improves learning in 10/10R genoype subjects, who are assumed to have lower endogenous striatal dopamine levels and impairs learning in 9/10R genotype subjects, who are assumed to have higher endogenous dopamine levels. These findings provide first evidence for a critical role of dopamine in learning whether an interaction partner has a prosocial or a selfish personality. The applied pharmacogenetic approach may open doors to new ways of studying psychiatric disorders such as psychosis, which is characterized by distorted perceptions of others' prosocial attitudes.

A healthy fear of the unknown: perspectives on the interpretation of parameter fits from computational models in neuroscience

Fitting models to behavior is commonly used to infer the latent computational factors responsible for generating behavior. However, the complexity of many behaviors can handicap the interpretation of such models. Here we provide perspectives on problems that can arise when interpreting parameter fits from models that provide incomplete descriptions of behavior. We illustrate these problems by fitting commonly used and neurophysiologically motivated reinforcement-learning models to simulated behavioral data sets from learning tasks. These model fits can pass a host of standard goodness-of-fit tests and other model-selection diagnostics even when the models do not provide a complete description of the behavioral data. We show that such incomplete models can be misleading by yielding biased estimates of the parameters explicitly included in the models. This problem is particularly pernicious when the neglected factors are unknown and therefore not easily identified by model comparisons and similar methods. An obvious conclusion is that a parsimonious description of behavioral data does not necessarily imply an accurate description of the underlying computations. Moreover, general goodness-of-fit measures are not a strong basis to support claims that a particular model can provide a generalized understanding of the computations that govern behavior. To help overcome these challenges, we advocate the design of tasks that provide direct reports of the computational variables of interest. Such direct reports complement model-fitting approaches by providing a more complete, albeit possibly more task-specific, representation of the factors that drive behavior. Computational models then provide a means to connect such task-specific results to a more general algorithmic understanding of the brain.