Dopaminergic Modulation of Human Intertemporal Choice: A Diffusion Model Analysis Using the D2-Receptor Antagonist Haloperidol

Dopamine and acetylcholine have distinct roles in delay- and effort-based decision-making in humans

In everyday life, we encounter situations that require tradeoffs between potential rewards and associated costs, such as time and (physical) effort. The literature indicates a prominent role for dopamine in discounting of both delay and effort, with mixed findings for delay discounting in humans. Moreover, the reciprocal antagonistic interaction between dopaminergic and cholinergic transmission in the striatum suggests a potential opponent role of acetylcholine in these processes. We found opposing effects of dopamine D2 (haloperidol) and acetylcholine M1 receptor (biperiden) antagonism on specific components of effort-based decision-making in healthy humans: haloperidol decreased, whereas biperiden increased the willingness to exert physical effort. In contrast, delay discounting was reduced under haloperidol, but not affected by biperiden. Together, our data suggest that dopamine, acting at D2 receptors, modulates both effort and delay discounting, while acetylcholine, acting at M1 receptors, appears to exert a more specific influence on effort discounting only.

Differences in Discounting Behavior and Brain Responses for Food and Money Reward

Most neuroeconomic research seeks to understand how value influences decision-making. The influence of reward type is less well understood. We used functional magnetic resonance imaging (fMRI) to investigate delay discounting of primary (i.e., food) and secondary rewards (i.e., money) in 28 healthy, normal-weighted participants (mean age = 26.77; 18 females). To decipher differences in discounting behavior between reward types, we compared how well-different option-based statistical models (exponential, hyperbolic discounting) and attribute-wise heuristic choice models (intertemporal choice heuristic, dual reasoning and implicit framework theory, trade-off model) captured the reward-specific discounting behavior. Contrary to our hypothesis of different strategies for different rewards, we observed comparable discounting behavior for money and food (i.e., exponential discounting). Higher k values for food discounting suggest that individuals decide more impulsive if confronted with food. The fMRI revealed that money discounting was associated with enhanced activity in the right dorsolateral prefrontal cortex, involved in executive control; the right dorsal striatum, associated with reward processing; and the left hippocampus, involved in memory encoding/retrieval. Food discounting, instead, was associated with higher activity in the left temporoparietal junction suggesting social reinforcement of food decisions. Although our findings do not confirm our hypothesis of different discounting strategies for different reward types, they are in line with the notion that reward types have a significant influence on impulsivity with primary rewards leading to more impulsive choices.

Decomposition of Reinforcement Learning Deficits in Disordered Gambling via Drift Diffusion Modeling and Functional Magnetic Resonance Imaging

Gambling disorder is associated with deficits in reward-based learning, but the underlying computational mechanisms are still poorly understood. Here, we examined this issue using a stationary reinforcement learning task in combination with computational modeling and functional resonance imaging (fMRI) in individuals that regular participate in gambling (n = 23, seven fulfilled one to three DSM 5 criteria for gambling disorder, sixteen fulfilled four or more) and matched controls (n = 23). As predicted, the gambling group exhibited substantially reduced accuracy, whereas overall response times (RTs) were not reliably different between groups. We then used comprehensive modeling using reinforcement learning drift diffusion models (RLDDMs) in combination with hierarchical Bayesian parameter estimation to shed light on the computational underpinnings of this performance deficit. In both groups, an RLDDM in which both non-decision time and decision threshold (boundary separation) changed over the course of the experiment accounted for the data best. The model showed good parameter and model recovery, and posterior predictive checks revealed that, in both groups, the model accurately reproduced the evolution of accuracies and RTs over time. Modeling revealed that, compared to controls, the learning impairment in the gambling group was linked to a more rapid reduction in decision thresholds over time, and a reduced impact of value-differences on the drift rate. The gambling group also showed shorter non-decision times. FMRI analyses replicated effects of prediction error coding in the ventral striatum and value coding in the ventro-medial prefrontal cortex, but there was no credible evidence for group differences in these effects. Taken together, our findings show that reinforcement learning impairments in disordered gambling are linked to both maladaptive decision threshold adjustments and a reduced consideration of option values in the choice process.

Dopamine Alters the Effect of Brain Stimulation on Decision-Making

Noninvasive brain stimulation techniques, such as transcranial direct current stimulation (tDCS), show promise in treating a range of psychiatric and neurologic conditions. However, optimization of such applications requires a better understanding of how tDCS alters cognition and behavior. Existing evidence implicates dopamine in tDCS alterations of brain activity and plasticity; however, there is as yet no causal evidence for a role of dopamine in tDCS effects on cognition and behavior. Here, in a preregistered, double-blinded study, we examined how pharmacologically manipulating dopamine altered the effect of tDCS on the speed-accuracy trade-off, which taps ubiquitous strategic operations. Cathodal tDCS was delivered over the left prefrontal cortex and the superior medial frontal cortex before participants (N = 62, 24 males, 38 females) completed a dot-motion task, making judgments on the direction of a field of moving dots under instructions to emphasize speed, accuracy, or both. We leveraged computational modeling to uncover how our interventions altered latent decisional processes driving the speed-accuracy trade-off. We show that dopamine in combination with tDCS (but not tDCS alone nor dopamine alone) not only impaired decision accuracy but also impaired discriminability, which suggests that these manipulations altered the encoding or representation of discriminative evidence. This is, to the best of our knowledge, the first direct evidence implicating dopamine in the way tDCS affects cognition and behavior.SIGNIFICANCE STATEMENT tDCS can improve cognitive and behavioral impairments in clinical conditions; however, a better understanding of its mechanisms is required to optimize future clinical applications. Here, using a pharmacological approach to manipulate brain dopamine levels in healthy adults, we demonstrate a role for dopamine in the effects of tDCS in the speed-accuracy trade-off, a strategic cognitive process ubiquitous in many contexts. In doing so, we provide direct evidence implicating dopamine in the way tDCS affects cognition and behavior.

Dopamine regulates decision thresholds in human reinforcement learning in males

Dopamine fundamentally contributes to reinforcement learning, but recent accounts also suggest a contribution to specific action selection mechanisms and the regulation of response vigour. Here, we examine dopaminergic mechanisms underlying human reinforcement learning and action selection via a combined pharmacological neuroimaging approach in male human volunteers (n = 31, within-subjects; Placebo, 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist Haloperidol). We found little credible evidence for previously reported beneficial effects of L-dopa vs. Haloperidol on learning from gains and altered neural prediction error signals, which may be partly due to differences experimental design and/or drug dosages. Reinforcement learning drift diffusion models account for learning-related changes in accuracy and response times, and reveal consistent decision threshold reductions under both drugs, in line with the idea that lower dosages of D2 receptor antagonists increase striatal DA release via an autoreceptor-mediated feedback mechanism. These results are in line with the idea that dopamine regulates decision thresholds during reinforcement learning, and may help to bridge action selection and response vigor accounts of dopamine.

Pharmacological Modulation of Temporal Discounting: A Systematic Review

Temporal discounting is a phenomenon where a reward loses its value as a function of time (e.g., a reward is more valuable immediately than when it delays in time). This is a type of intertemporal decision-making that has an association with impulsivity and self-control. Many pathologies exhibit higher discounting rates, meaning they discount more the values of rewards, such as addictive behaviors, bipolar disorder, attention-deficit/hyperactivity disorders, social anxiety disorders, and major depressive disorder, among others; thus, many studies look for the mechanism and neuromodulators of these decisions. This systematic review aims to investigate the association between pharmacological administration and changes in temporal discounting. A search was conducted in PubMed, Scopus, Web of Science, Science Direct and Cochrane. We used the PICO strategy: healthy humans (P-Participants) that received a pharmacological administration (I-Intervention) and the absence of a pharmacological administration or placebo (C-Comparison) to analyze the relationship between the pharmacological administration and the temporal discounting (O-outcome). Nineteen studies fulfilled the inclusion criteria. The most important findings were the involvement of dopamine modulation in a U-shape for choosing the delayed outcome (metoclopradime, haloperidol, and amisulpride). Furthermore, administration of tolcapone and high doses of d-amphetamine produced a preference for the delayed option. There was a time-dependent hydrocortisone effect in the preference for the immediate reward. Thus, it can be concluded that dopamine is a crucial modulator for temporal discounting, especially the D2 receptor, and cortisol also has an important time-dependent role in this type of decision. One of the limitations of this systematic review is the heterogeneity of the drugs used to assess the effect of temporal discounting.

A process model account of the role of dopamine in intertemporal choice

Theoretical accounts disagree on the role of dopamine in intertemporal choice and assume that dopamine either promotes delay of gratification by increasing the preference for larger rewards or that dopamine reduces patience by enhancing the sensitivity to waiting costs. Here, we reconcile these conflicting accounts by providing empirical support for a novel process model according to which dopamine contributes to two dissociable components of the decision process, evidence accumulation and starting bias. We re-analyzed a previously published data set where intertemporal decisions were made either under the D2 antagonist amisulpride or under placebo by fitting a hierarchical drift diffusion model that distinguishes between dopaminergic effects on the speed of evidence accumulation and the starting point of the accumulation process. Blocking dopaminergic neurotransmission not only strengthened the sensitivity to whether a reward is perceived as worth the delay costs during evidence accumulation (drift rate) but also attenuated the impact of waiting costs on the starting point of the evidence accumulation process (bias). In contrast, re-analyzing data from a D1 agonist study provided no evidence for a causal involvement of D1R activation in intertemporal choices. Taken together, our findings support a novel, process-based account of the role of dopamine for cost-benefit decision making, highlight the potential benefits of process-informed analyses, and advance our understanding of dopaminergic contributions to decision making.

The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting

Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.

Gambling Environment Exposure Increases Temporal Discounting but Improves Model-Based Control in Regular Slot-Machine Gamblers

Gambling disorder is a behavioral addiction that negatively impacts personal finances, work, relationships and mental health. In this pre-registered study (https://osf.io/5ptz9/) we investigated the impact of real-life gambling environments on two computational markers of addiction, temporal discounting and model-based reinforcement learning. Gambling disorder is associated with increased temporal discounting and reduced model-based learning. Regular gamblers (n = 30, DSM-5 score range 3-9) performed both tasks in a neutral (café) and a gambling-related environment (slot-machine venue) in counterbalanced order. Data were modeled using drift diffusion models for temporal discounting and reinforcement learning via hierarchical Bayesian estimation. Replicating previous findings, gamblers discounted rewards more steeply in the gambling-related context. This effect was positively correlated with gambling related cognitive distortions (pre-registered analysis). In contrast to our pre-registered hypothesis, model-based reinforcement learning was improved in the gambling context. Here we show that temporal discounting and model-based reinforcement learning are modulated in opposite ways by real-life gambling cue exposure. Results challenge aspects of habit theories of addiction, and reveal that laboratory-based computational markers of psychopathology are under substantial contextual control.

Motor response vigour and visual fixation patterns reflect subjective valuation during intertemporal choice

Value-based decision-making is of central interest in cognitive neuroscience and psychology, as well as in the context of neuropsychiatric disorders characterised by decision-making impairments. Studies examining (neuro-)computational mechanisms underlying choice behaviour typically focus on participants’ decisions. However, there is increasing evidence that option valuation might also be reflected in motor response vigour and eye movements, implicit measures of subjective utility. To examine motor response vigour and visual fixation correlates of option valuation in intertemporal choice, we set up a task where the participants selected an option by pressing a grip force transducer, simultaneously tracking fixation shifts between options. As outlined in our preregistration (https://osf.io/k6jct), we used hierarchical Bayesian parameter estimation to model the choices assuming hyperbolic discounting, compared variants of the softmax and drift diffusion model, and assessed the relationship between response vigour and the estimated model parameters. The behavioural data were best explained by a drift diffusion model specifying a non-linear scaling of the drift rate by the subjective value differences. Replicating previous findings, we found a magnitude effect for temporal discounting, such that higher rewards were discounted less. This magnitude effect was further reflected in motor response vigour, such that stronger forces were exerted in the high vs. the low magnitude condition. Bayesian hierarchical linear regression further revealed higher grip forces, faster response times and a lower number of fixation shifts for trials with higher subjective value differences. An exploratory analysis revealed that subjective value sums across options showed an even more pronounced association with trial-wise grip force amplitudes. Our data suggest that subjective utility or implicit valuation is reflected in motor response vigour and visual fixation patterns during intertemporal choice. Taking into account response vigour might thus provide deeper insight into decision-making, reward valuation and maladaptive changes in these processes, e.g. in the context of neuropsychiatric disorders.

Towards a Unifying Account of Dopamine’s Role in Cost-Benefit Decision Making

Dopamine is thought to play a crucial role in cost-benefit decision making, but so far there is no consensus on the precise role of dopamine in decision making. Here, we review the literature on dopaminergic manipulations of cost-benefit decision making in humans and evaluate how well different theoretical accounts explain the existing body of evidence. Reduced D₂ stimulation tends to increase the willingness to bear delay and risk costs (i.e., wait for later rewards, take riskier options), while increased D₁ and D₂ receptor stimulation increases willingness to bear effort costs. We argue that the empirical findings can best be explained by combining the strengths of two theoretical accounts: in cost-benefit decision making, dopamine may play a dual role both in promoting the pursuit of psychologically close options (e.g., sooner and safer rewards) and in computing which costs are acceptable for a reward at stake. Moreover, we identify several limiting factors in the study designs of previous investigations that prevented a fuller understanding of dopamine's role in value-based choice. Together, the proposed theoretical framework and the methodological suggestions for future studies may bring us closer to a unifying account of dopamine in healthy and impaired cost-benefit decision making.

Trial-wise exposure to visual emotional cues increases physiological arousal but not temporal discounting

Humans and many animals devalue future rewards as a function of time (temporal discounting). Increased discounting has been linked to various psychiatric conditions, including substance-use-disorders, behavioral addictions, and obesity. Despite its high intra-individual stability, temporal discounting is partly under contextual control. One prominent manipulation that has been linked to increases in discounting is the exposure to highly arousing appetitive cues. However, results from trial-wise cue exposure studies appear highly mixed, and changes in physiological arousal were not adequately controlled. Here we tested the effects of appetitive (erotic), aversive, and neutral visual cues on temporal discounting in 35 healthy male participants. The contribution of single-trial physiological arousal was assessed using comprehensive monitoring of autonomic activity (pupil size, heart rate, electrodermal activity). Physiological arousal was elevated following aversive and in particular erotic cues. In contrast to our pre-registered hypothesis, steepness of temporal discounting was not significantly affected by emotional cues of either valence. Aversive cues tended to increase decision noise. Computational modeling revealed that trial-wise arousal only accounted for minor variance over and above aversive and erotic condition effects, arguing against a general effect of physiological arousal on temporal discounting.

Temporal discounting in adolescents and adults with Tourette syndrome

Tourette syndrome is a neurodevelopmental disorder associated with hyperactivity in dopaminergic networks. Dopaminergic hyperactivity in the basal ganglia has previously been linked to increased sensitivity to positive reinforcement and increases in choice impulsivity. In this study, we examine whether this extends to changes in temporal discounting, where impulsivity is operationalized as an increased preference for smaller-but-sooner over larger-but-later rewards. We assessed intertemporal choice in two studies including nineteen adolescents (age: mean[sd] = 14.21[±2.37], 13 male subjects) and twenty-five adult patients (age: mean[sd] = 29.88 [±9.03]; 19 male subjects) with Tourette syndrome and healthy age- and education matched controls. Computational modeling using exponential and hyperbolic discounting models via hierarchical Bayesian parameter estimation revealed reduced temporal discounting in adolescent patients, and no evidence for differences in adult patients. Results are discussed with respect to neural models of temporal discounting, dopaminergic alterations in Tourette syndrome and the developmental trajectory of temporal discounting. Specifically, adolescents might show attenuated discounting due to improved inhibitory functions that also affect choice impulsivity and/or the developmental trajectory of executive control functions. Future studies would benefit from a longitudinal approach to further elucidate the developmental trajectory of these effects.

Reliability assessment of temporal discounting measures in virtual reality environments

In recent years the emergence of high-performance virtual reality (VR) technology has opened up new possibilities for the examination of context effects in psychological studies. The opportunity to create ecologically valid stimulation in a highly controlled lab environment is especially relevant for studies of psychiatric disorders, where it can be problematic to confront participants with certain stimuli in real life. However, before VR can be confidently applied widely it is important to establish that commonly used behavioral tasks generate reliable data within a VR surrounding. One field of research that could benefit greatly from VR-applications are studies assessing the reactivity to addiction related cues (cue-reactivity) in participants suffering from gambling disorder. Here we tested the reliability of a commonly used temporal discounting task in a novel VR set-up designed for the concurrent assessment of behavioral and psychophysiological cue-reactivity in gambling disorder. On 2 days, thirty-four healthy non-gambling participants explored two rich and navigable VR-environments (neutral: café vs. gambling-related: casino and sports-betting facility), while their electrodermal activity was measured using remote sensors. In addition, participants completed the temporal discounting task implemented in each VR environment. On a third day, participants performed the task in a standard lab testing context. We then used comprehensive computational modeling using both standard softmax and drift diffusion model (DDM) choice rules to assess the reliability of discounting model parameters assessed in VR. Test-retest reliability estimates were good to excellent for the discount rate log(k), whereas they were poor to moderate for additional DDM parameters. Differences in model parameters between standard lab testing and VR, reflecting reactivity to the different environments, were mostly numerically small and of inconclusive directionality. Finally, while exposure to VR generally increased tonic skin conductance, this effect was not modulated by the neutral versus gambling-related VR-environment. Taken together this proof-of-concept study in non-gambling participants demonstrates that temporal discounting measures obtained in VR are reliable, suggesting that VR is a promising tool for applications in computational psychiatry, including studies on cue-reactivity in addiction.