What Is the Relationship between Dopamine and Effort?

Modulating the Neuromodulators: Dopamine, Serotonin, and the Endocannabinoid System

Dopamine (DA), serotonin (5-hydroxytryptamine, 5-HT), and endocannabinoids (ECs) are key neuromodulators involved in many aspects of motivated behavior, including reward processing, reinforcement learning, and behavioral flexibility. Among the longstanding views about possible relationships between these neuromodulators is the idea of DA and 5-HT acting as opponents. This view has been challenged by emerging evidence that 5-HT supports reward seeking via activation of DA neurons in the ventral tegmental area. Adding an extra layer of complexity to these interactions, the endocannabinoid system is uniquely placed to influence dopaminergic and serotonergic neurotransmission. In this review we discuss how these three neuromodulatory systems interact at the cellular and circuit levels. Technological advances that facilitate precise identification and control of genetically targeted neuronal populations will help to achieve a better understanding of the complex relationship between these essential systems, and the potential relevance for motivated behavior.

GLP-1 and hunger modulate incentive motivation depending on insulin sensitivity in humans

OBJECTIVE

To regulate food intake, our brain constantly integrates external cues, such as the incentive value of a potential food reward, with internal state signals, such as hunger feelings. Incentive motivation refers to the processes that translate an expected reward into the effort spent to obtain the reward; the magnitude and probability of a reward involved in prompting motivated behaviour are encoded by the dopaminergic (DA) midbrain and its mesoaccumbens DA projections. This type of reward circuity is particularly sensitive to the metabolic state signalled by peripheral mediators, such as insulin or glucagon-like peptide 1 (GLP-1). While in rodents the modulatory effect of metabolic state signals on motivated behaviour is well documented, evidence of state-dependent modulation and the role of incentive motivation underlying overeating in humans is lacking.

METHODS

In a randomised, placebo-controlled, crossover design, 21 lean (body mass index [BMI] < 25 kg/m²) and 16 obese (BMI³ 30 kg/m²) volunteer participants received either liraglutide as a GLP-1 analogue or placebo on two separate testing days. Incentive motivation was measured using a behavioural task in which participants were required to exert physical effort using a handgrip to win different amounts of food and monetary rewards. Hunger levels were measured using visual analogue scales; insulin, glucose, and systemic insulin resistance as assessed by the homeostasis model assessment of insulin resistance (HOMA-IR) were quantified at baseline.

RESULTS

In this report, we demonstrate that incentive motivation increases with hunger in lean humans (F_(1,42) = 5.31, p = 0.026, β = 0.19) independently of incentive type (food and non-food reward). This effect of hunger is not evident in obese humans (F_(1,62) = 1.93, p = 0.17, β = -0.12). Motivational drive related to hunger is affected by peripheral insulin sensitivity (two-way interaction, F_{(1, 35)} = 6.23, p = 0.017, β = -0.281). In humans with higher insulin sensitivity, hunger increases motivation, while poorer insulin sensitivity dampens the motivational effect of hunger. The GLP-1 analogue application blunts the interaction effect of hunger on motivation depending on insulin sensitivity (three-way interaction, F_{(1, 127)} = 5.11, p = 0.026); no difference in motivated behaviour could be found between humans with normal or impaired insulin sensitivity under GLP-1 administration.

CONCLUSION

We report a differential effect of hunger on motivation depending on insulin sensitivity. We further revealed the modulatory role of GLP-1 in adaptive, motivated behaviour in humans and its interaction with peripheral insulin sensitivity and hunger. Our results suggest that GLP-1 might restore dysregulated processes of midbrain DA function and hence motivational behaviour in insulin-resistant humans.

Human complex exploration strategies are enriched by noradrenaline-modulated heuristics

An exploration-exploitation trade-off, the arbitration between sampling a lesser-known against a known rich option, is thought to be solved using computationally demanding exploration algorithms. Given known limitations in human cognitive resources, we hypothesised the presence of additional cheaper strategies. We examined for such heuristics in choice behaviour where we show this involves a value-free random exploration, that ignores all prior knowledge, and a novelty exploration that targets novel options alone. In a double-blind, placebo-controlled drug study, assessing contributions of dopamine (400 mg amisulpride) and noradrenaline (40 mg propranolol), we show that value-free random exploration is attenuated under the influence of propranolol, but not under amisulpride. Our findings demonstrate that humans deploy distinct computationally cheap exploration strategies and that value-free random exploration is under noradrenergic control.

Dissociating the effects of dopamine D2 receptors on effort-based versus value-based decision making using a novel behavioral approach

Cost-benefit decision making is essential for organisms to adapt to their ever-changing environment. Most studies of cost-benefit decision making involve choice conditions in which effort and value are varied simultaneously. This prevents identification of the aspects of cost-benefit decision making that are affected by experimental manipulations. We developed operant assays to isolate the individual impacts of effort and value manipulations on cost-benefit decision making. In the concurrent effort choice (CEC) task, mice choose between exerting two distinct types of effort: the number of responses and the duration of a response, to earn the same reward. By parametrically varying response cost, psychometric functions are obtained that reflect how the two types of effort scale against one another. Direct manipulations of effort shift the functions. Because reward value is held constant in this task, differences in scaling of the two response types must be related to the effort manipulations. In the concurrent value choice (CVC) task, mice make the same type of response to earn rewards of different value (e.g., pellets vs. sucrose solutions). Here the effort required to earn one reward type is parametrically varied to obtain the psychometric function that scales the value of the two rewards into the number of responses subjects will pay to earn one reward over the other. Direct value manipulations shift these functions. We tested the effect of the dopamine D2 receptor antagonist, haloperidol, on performance in the CEC and CVC assays and found that D2R signaling is important for effort-based, but not value-based decision making. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Pharmacological evidence for the implication of noradrenaline in effort

The trade-off between effort and reward is one of the main determinants of behavior, and its alteration is at the heart of major disorders such as depression or Parkinson's disease. Monoaminergic neuromodulators are thought to play a key role in this trade-off, but their relative contribution remains unclear. Rhesus monkeys (Macaca mulatta) performed a choice task requiring a trade-off between the volume of fluid reward and the amount of force to be exerted on a grip. In line with a causal role of noradrenaline in effort, decreasing noradrenaline levels with systemic clonidine injections (0.01 mg/kg) decreased exerted force and enhanced the weight of upcoming force on choices, without any effect on reward sensitivity. Using computational modeling, we showed that a single variable ("effort") could capture the amount of resources necessary for action and control both choices (as a variable for decision) and force production (as a driving force). Critically, the multiple effects of noradrenaline manipulation on behavior could be captured by a specific modulation of this single variable. Thus, our data strongly support noradrenaline's implication in effort processing.

Pupillary Dilations of Mice Performing a Vibrotactile Discrimination Task Reflect Task Engagement and Response Confidence

Pupillometry, the measure of pupil size and reactivity, has been widely used to assess cognitive processes. Changes in pupil size have been shown to correlate with various behavioral states, both externally and internally induced such as locomotion, arousal, cortical state, and decision-making processes. Besides, these pupillary responses have also been linked to the activity of neuromodulatory systems that modulate attention and perception such as the noradrenergic and cholinergic systems. Due to the extent of processes the pupil reflects, we aimed at further resolving pupillary responses in the context of behavioral state and task performance while recording pupillary transients of mice performing a vibrotactile two-alternative forced-choice task (2-AFC). We show that before the presentation of task-relevant information, pre-stimulus, pupil size differentiates between states of disengagement from task performance vs. engagement. Also, when subjects have to attend to task stimuli to attain a reward, post-stimulus, pupillary dilations exhibit a difference between correct and error responses with this difference reflecting an internal decision variable. We hypothesize that this internal decision variable relates to response confidence, the internal perception of the confidence the subject has in its choice. As opposed to this, we show that in a condition of passive performance, when the stimulus has no more task relevance due to reward being provided automatically, pupillary dilations reflect the occurrence of stimulation and reward provision but not decisional variables as under active performance. Our results provide evidence that in addition to reflecting attentiveness under task performance rather than arousal per se, pupil dilations also reflect the confidence of the subject in his ensuing response. This confidence coding is overlaid within a more pronounced pupil dilation that reflects post-decision components that are related to the response itself but not to the decision. We also provide evidence as to how different behavioral states, imposed by task demands, modulate what the pupil is reflecting, presumably showing what the underlying cognitive network is coding for.

Context-Dependent Multiplexing by Individual VTA Dopamine Neurons

Dopamine (DA) neurons of the VTA track cues and rewards to generate a reward prediction error signal during Pavlovian conditioning. Here we explored how these neurons respond to a self-paced, operant task in freely moving mice. The animal could trigger a reward-predicting cue by remaining in a specific location of an operant box for a brief time before moving to a spout for reward collection. VTA DA neurons were identified using DAT-Cre male mice that carried an optrode with minimal impact on the behavioral task. In vivo single-unit recordings revealed transient fast spiking responses to the cue and reward in correct trials, while for incorrect ones the activity paused, reflecting positive and negative error signals of a reward prediction. In parallel, a majority of VTA DA neurons simultaneously encoded multiple actions (e.g., movement velocity, acceleration, distance to goal, and licking) in sustained slow firing modulation. Applying a GLM, we show that such multiplexed encoding of rewarding and motor variables by individual DA neurons was only apparent while the mouse was engaged in the task. Downstream targets may exploit such goal-directed multiplexing of VTA DA neurons to adjust actions to optimize the task's outcome.SIGNIFICANCE STATEMENT VTA DA neurons code for multiple functions, including the reward prediction error but also motivation and locomotion. Here we show that about half of the recorded VTA DA neurons perform multiplexing: they exploit the phasic and tonic activity modes to encode, respectively, the cue/reward responses and motor parameters, most prominently when the mouse engages in a self-paced operand task. VTA non-DA neurons, by contrast, encode motor parameters regardless of task engagement.

Dopaminergic Signaling in the Nucleus Accumbens Modulates Stress-Coping Strategies during Inescapable Stress

Maladaptation to stress is a critical risk factor in stress-related disorders, such as major depression and post-traumatic stress disorder (PTSD). Dopamine signaling in the nucleus accumbens (NAc) has been shown to modulate behavior by reinforcing learning and evading aversive stimuli, which are important for the survival of animals under environmental challenges such as stress. However, the mechanisms through which dopaminergic transmission responds to stressful events and subsequently regulates its downstream neuronal activity during stress remain unknown. To investigate how dopamine signaling modulates stress-coping behavior, we measured the subsecond fluctuation of extracellular dopamine concentration and pH using fast scanning cyclic voltammetry (FSCV) in the NAc, a postsynaptic target of midbrain dopaminergic neurons, in male mice engaged in a tail suspension test (TST). The results revealed a transient decrease in dopamine concentration and an increase in pH levels when the animals changed behaviors, from being immobile to struggling. Interestingly, optogenetic inhibition of dopamine release in NAc, potentiated the struggling behavior in animals under the TST. We then addressed the causal relationship of such a dopaminergic transmission with behavioral alterations by knocking out both the dopamine receptors, i.e., D1 and D2, in the NAc using viral vector-mediated genome editing. Behavioral analyses revealed that male D1 knock-out mice showed significantly more struggling bouts and longer struggling durations during the TST, while male D2 knock-out mice did not. Our results therefore indicate that D1 dopaminergic signaling in the NAc plays a pivotal role in the modulation of stress-coping behaviors in animals under tail suspension stress.SIGNIFICANCE STATEMENT The tail suspension test (TST) has been widely used as a despair-based behavioral assessment to screen the antidepressant so long. Despite its prevalence in the animal studies, the neural substrate underlying the changes of behavior during the test remains unclear. This study provides an evidence for a role of dopaminergic transmission in the modulation of stress-coping behavior during the TST, a despair test widely used to screen the antidepressants in rodents. Taking into consideration the fact that the dopamine metabolism is upregulated by almost all antidepressants, a part of which acts directly on the dopaminergic transmission, current results would uncover the molecular mechanism through which the dopaminergic signaling mediates antidepressant effect with facilitation of the recovery from the despair-like behavior in the TST.

The role of norepinephrine in the pathophysiology of schizophrenia

Several lines of evidence have suggested for decades a role for norepinephrine (NE) in the pathophysiology and treatment of schizophrenia. Recent experimental findings reveal anatomical and physiological properties of the locus coeruleus-norepinephrine (LC-NE) system and its involvement in brain function and cognition. Here, we integrate these two lines of evidence. First, we review the functional and structural properties of the LC-NE system and its impact on functional brain networks, cognition, and stress, with special emphasis on recent experimental and theoretical advances. Subsequently, we present an update about the role of LC-associated functions for the pathophysiology of schizophrenia, focusing on the cognitive and motivational deficits. We propose that schizophrenia phenomenology, in particular cognitive symptoms, may be explained by an abnormal interaction between genetic susceptibility and stress-initiated LC-NE dysfunction. This in turn, leads to imbalance between LC activity modes, dysfunctional regulation of brain network integration and neural gain, and deficits in cognitive functions. Finally, we suggest how recent development of experimental approaches can be used to characterize LC function in schizophrenia.

The role of dopamine in dynamic effort-reward integration

When deciding to act, the neurotransmitter dopamine is implicated in a valuation of prospective effort and reward. However, its role in dynamic effort-reward integration during action, a process central to everyday behaviour, remains unclear. In a placebo-controlled, within-subject, study, we probed the impact of increasing brain dopamine levels (150 mg of levodopa) and blocking dopamine receptors (1.5 mg of haloperidol) in the context of a novel dynamic effort task in healthy human subjects. We show that modulating homoeostatic dopamine balance distinctly alters implicit and explicit effort allocation as a function of instantaneous reward. Pharmacologically boosting dopamine enhanced motor vigour, reflected in an implicit increase in effort allocation for high rewards. Conversely, pharmacological blockade of dopamine attenuated sensitivity to differences in reward context, reflected in reduced strategic effort discounting. These findings implicate dopamine in an integration of momentary physical experience and instantaneous reward, suggesting a key role of dopamine in acting to maximise reward on the fly.

Accumbal Dopamine Release Tracks the Expectation of Dopamine Neuron-Mediated Reinforcement

Dopamine (DA) transmission in the nucleus accumbens (NAc) facilitates cue-reward associations and appetitive action. Reward-related accumbal DA release dynamics are traditionally ascribed to ventral tegmental area (VTA) DA neurons. Activation of VTA to NAc DA signaling is thought to reinforce action and transfer reward-related information to predictive cues, allowing cues to guide behavior and elicit dopaminergic activity. Here, we use optogenetics to control DA neuron activity and voltammetry to simultaneously record accumbal DA release in order to quantify how reinforcer-evoked dopaminergic activity shapes conditioned mesolimbic DA transmission. We find that cues predicting access to DA neuron self-stimulation elicit conditioned responding and NAc DA release. However, cue-evoked DA release does not reflect the cost or magnitude of DA neuron activation. Accordingly, conditioned accumbal DA release selectively tracks the expected availability of DA-neuron-mediated reinforcement. This work provides insight into how mesolimbic DA transmission drives and encodes appetitive action.

Noradrenergic But Not Dopaminergic Neurons Signal Task State Changes and Predict Reengagement After a Failure

The two catecholamines, noradrenaline and dopamine, have been shown to play comparable roles in behavior. Both noradrenergic and dopaminergic neurons respond to cues predicting reward availability and novelty. However, even though both are thought to be involved in motivating actions, their roles in motivation have seldom been directly compared. We therefore examined the activity of putative noradrenergic neurons in the locus coeruleus and putative midbrain dopaminergic neurons in monkeys cued to perform effortful actions for rewards. The activity in both regions correlated with engagement with a presented option. By contrast, only noradrenaline neurons were also (i) predictive of engagement in a subsequent trial following a failure to engage and (ii) more strongly activated in nonrepeated trials, when cues indicated a new task condition. This suggests that while both catecholaminergic neurons are involved in promoting action, noradrenergic neurons are sensitive to task state changes, and their influence on behavior extends beyond the immediately rewarded action.

From sign-tracking to attentional bias: Implications for gambling and substance use disorders

Sign-tracking behavior in Pavlovian autoshaping is known to be a relevant index of the incentive salience attributed to reward-related cues. Evidence has accumulated to suggest that animals that exhibit a sign-tracker phenotype are especially vulnerable to addiction and relapse due to their proneness to attribute incentive salience to drug cues, and their relatively weak cognitive and attentional control over their behavior. Interestingly, sign-tracking is also influenced by reward uncertainty in a way that may promote gambling disorder. Research indicates that reward uncertainty sensitizes sign-tracking responses and favors the development of a sign-tracker phenotype, compatible with the conditioned attractiveness of lights and sounds in casinos for problem gamblers. The study of attentional biases in humans (an effect akin to sign-tracking in animals) leads to similar observations, notably that the propensity to develop attraction for conditioned stimuli (CSs) is predictive of addictive behavior. Here we review the literature on drug addiction and gambling disorder, highlighting the similarities between studies of sign-tracking and attentional biases.

Amphetamine disrupts haemodynamic correlates of prediction errors in nucleus accumbens and orbitofrontal cortex

In an uncertain world, the ability to predict and update the relationships between environmental cues and outcomes is a fundamental element of adaptive behaviour. This type of learning is typically thought to depend on prediction error, the difference between expected and experienced events and in the reward domain that has been closely linked to mesolimbic dopamine. There is also increasing behavioural and neuroimaging evidence that disruption to this process may be a cross-diagnostic feature of several neuropsychiatric and neurological disorders in which dopamine is dysregulated. However, the precise relationship between haemodynamic measures, dopamine and reward-guided learning remains unclear. To help address this issue, we used a translational technique, oxygen amperometry, to record haemodynamic signals in the nucleus accumbens (NAc) and orbitofrontal cortex (OFC), while freely moving rats performed a probabilistic Pavlovian learning task. Using a model-based analysis approach to account for individual variations in learning, we found that the oxygen signal in the NAc correlated with a reward prediction error, whereas in the OFC it correlated with an unsigned prediction error or salience signal. Furthermore, an acute dose of amphetamine, creating a hyperdopaminergic state, disrupted rats' ability to discriminate between cues associated with either a high or a low probability of reward and concomitantly corrupted prediction error signalling. These results demonstrate parallel but distinct prediction error signals in NAc and OFC during learning, both of which are affected by psychostimulant administration. Furthermore, they establish the viability of tracking and manipulating haemodynamic signatures of reward-guided learning observed in human fMRI studies by using a proxy signal for BOLD in a freely behaving rodent.

Dopaminergic neurons can influence heat-box place learning in Drosophila

Dopamine provides crucial neuromodulatory functions in several insect and rodent learning and memory paradigms. However, an early study suggested that dopamine may be dispensable for aversive place memory in Drosophila. Here we tested the involvement of particular dopaminergic neurons in place learning and memory. We used the thermogenetic tool Gr28bD to activate protocerebral anterior medial (PAM) cluster and non-PAM dopaminergic neurons in an operant way in heat-box place learning. We show that activation of PAM neurons influences performance during place learning, but not during memory testing. These findings provide a gateway to explore how dopamine influences place learning.

Differential modulation of cognitive control networks by monetary reward and punishment

Incentives are primary determinants of if and how well an organism will perform a given behavior. Here, we examined how incentive valence and magnitude influence task switching, a critical cognitive control process, and test the predictions that the anterior cingulate cortex (ACC) and the ventral striatum (vStr) function as key nodes linking motivation and control systems in the brain. Our results indicate that reward and punishment incentives have both common and distinct effects on cognitive control at the behavioral and neurobiological levels. For example, reward incentives led to greater activity in the ACC during the engagement of control relative to punishments. Furthermore, the neural responses to reward and punishment differed as a function of individual sensitivity to each incentive valence. Functional connectivity analyses suggest a role for vStr in signaling motivational value during cognitive control and as a potential link between motivation and control networks. Overall, our findings suggest that similar changes in observed behavior (e.g. response accuracy) under reward and punishment incentives are mediated by, at least partially, distinct neurobiological substrates.

Dopamine-Dependent Loss Aversion during Effort-Based Decision-Making

From psychology to economics, there has been substantial interest in how costs (e.g., delay, risk) are represented asymmetrically during decision-making when attempting to gain reward or avoid punishment. For example, in decision-making under risk, individuals show a tendency to prefer to avoid punishment rather than to acquire the equivalent reward (loss aversion). Although the cost of physical effort has recently received significant attention, it remains unclear whether loss aversion exists during effort-based decision-making. On the one hand, loss aversion may be hardwired due to asymmetric evolutionary pressure on losses and gains and therefore exists across decision-making contexts. On the other hand, distinct brain regions are involved with different decision costs, making it questionable whether similar asymmetries exist. Here, we demonstrate that young healthy human participants (females, 16; males, 6) exhibit loss aversion during effort-based decision-making by exerting more physical effort to avoid punishment than to gain a same-size reward. Next, we show that medicated Parkinson's disease (PD) patients (females, 9; males, 9) show a reduction in loss aversion compared with age-matched control subjects (females, 11; males, 9). Behavioral and computational analysis revealed that people with PD exerted similar physical effort in return for a reward but were less willing to produce effort to avoid punishment. Therefore, loss aversion is present during effort-based decision-making and can be modulated by altered dopaminergic state. This finding could have important implications for our understanding of clinical disorders that show a reduced willingness to exert effort in the pursuit of reward.SIGNIFICANCE STATEMENT Loss aversion-preferring to avoid punishment rather than to acquire equivalent reward-is an important concept in decision-making under risk. However, little is known about whether loss aversion also exists during decisions where the cost is physical effort. This is surprising given that motor cost shapes human behavior, and a reduced willingness to exert effort is a characteristic of many clinical disorders. Here, we show that healthy human individuals exert more effort to minimize punishment than to maximize reward (loss aversion). We also demonstrate that medicated Parkinson's disease patients exert similar effort to gain reward but less effort to avoid punishment when compared with healthy age-matched control subjects. This indicates that dopamine-dependent loss aversion is crucial for explaining effort-based decision-making.

Believing in dopamine

Midbrain dopamine signals are widely thought to report reward prediction errors that drive learning in the basal ganglia. However, dopamine has also been implicated in various probabilistic computations, such as encoding uncertainty and controlling exploration. Here, we show how these different facets of dopamine signalling can be brought together under a common reinforcement learning framework. The key idea is that multiple sources of uncertainty impinge on reinforcement learning computations: uncertainty about the state of the environment, the parameters of the value function and the optimal action policy. Each of these sources plays a distinct role in the prefrontal cortex-basal ganglia circuit for reinforcement learning and is ultimately reflected in dopamine activity. The view that dopamine plays a central role in the encoding and updating of beliefs brings the classical prediction error theory into alignment with more recent theories of Bayesian reinforcement learning.

Can't or Won't? Immunometabolic Constraints on Dopaminergic Drive

Inflammatory cytokines have been shown to have a direct effect on mesolimbic dopamine (DA) that is associated with a reduced willingness to expend effort for reward. To date, however, the broader implications of this communication between inflammation and mesolimbic DA have yet to be explored. Here, we suggest that the metabolic demands of chronic low-grade inflammation induce a reduction of striatal DA that in turn leads to a steeper effort-discounting curve because of reduced perceived ability (can't) versus preference (won't) for reward. This theoretical framework can inform how the mesolimbic DA system responds to increased immunometabolic demands during chronic inflammation, ultimately contributing to motivational impairments in psychiatric and other medical disorders.