Calculating utility: preclinical evidence for cost-benefit analysis by mesolimbic dopamine

Dopamine and acetylcholine have distinct roles in delay- and effort-based decision-making in humans

In everyday life, we encounter situations that require tradeoffs between potential rewards and associated costs, such as time and (physical) effort. The literature indicates a prominent role for dopamine in discounting of both delay and effort, with mixed findings for delay discounting in humans. Moreover, the reciprocal antagonistic interaction between dopaminergic and cholinergic transmission in the striatum suggests a potential opponent role of acetylcholine in these processes. We found opposing effects of dopamine D2 (haloperidol) and acetylcholine M1 receptor (biperiden) antagonism on specific components of effort-based decision-making in healthy humans: haloperidol decreased, whereas biperiden increased the willingness to exert physical effort. In contrast, delay discounting was reduced under haloperidol, but not affected by biperiden. Together, our data suggest that dopamine, acting at D2 receptors, modulates both effort and delay discounting, while acetylcholine, acting at M1 receptors, appears to exert a more specific influence on effort discounting only.

Brain stimulation over the left DLPFC enhances motivation for effortful rewards in patients with major depressive disorder

BACKGROUND

Amotivation is a typical feature in major depressive disorder (MDD), which produces reduced willingness to exert effort. The dorsolateral prefrontal cortex (DLPFC) is a crucial structure in goal-directed actions and therefore is a potential target in modulating effortful motivation. However, it remains unclear whether the intervention is effective for patients with MDD.

METHODS

We employed transcranial magnetic stimulation (TMS), computational modelling and event-related potentials (ERPs) to reveal the causal relationship between the left DLPFC and motivation for effortful rewards in MDD. Fifty patients underwent both active and sham TMS sessions, each followed by performing an Effort-Expenditure for Rewards Task, during which participants chose and implemented between low-effort/low-reward and high-effort/high-reward options.

RESULTS

The patients showed increased willingness to exert effort for rewards during the DLPFC facilitated session, compared with the sham session. They also had a trend in larger P3 amplitude for motivated attention toward chosen options, larger CNV during preparing for effort exertion, and larger SPN during anticipating a high reward. Besides, while behavior indexes for effortful choices were negatively related to depression severity in the sham session, this correlation was weakened in the active stimulation session.

CONCLUSIONS

These findings provide behavioral, computational, and neural evidence for the left DLPFC on effortful motivation for rewards. Facilitated DLPFC improves motor preparation and value anticipation after making decisions especially for highly effortful rewards in MDD. Facilitated DLPFC also has a potential function in enhancing motivated attention during cost-benefit trade-off. This neuromodulation effect provides a potential treatment for improving motivation in clinics.

Dopamine, activation of ingestion and evaluation of response efficacy: a focus on the within-session time-course of licking burst number

RATIONALE

Evidence on the effect of dopamine D1-like and D2-like receptor antagonists on licking microstructure and the forced swimming response led us to suggest that (i) dopamine on D1-like receptors plays a role in activating reward-directed responses and (ii) the level of response activation is reboosted based on a process of evaluation of response efficacy requiring dopamine on D2-like receptors. A main piece of evidence in support of this hypothesis is the observation that the dopamine D2-like receptor antagonist raclopride induces a within-session decrement of burst number occurring after the contact with the reward. The few published studies with a detailed analysis of the time-course of this measure were conducted in our laboratory.

OBJECTIVES

The aim of this review is to recapitulate and discuss the evidence in support of the analysis of the within-session burst number as a behavioural substrate for the study of the mechanisms governing ingestion, behavioural activation and the related evaluation processes, and its relevance in the analysis of drug effects on ingestion.

CONCLUSIONS

The evidence gathered so far suggests that the analysis of the within-session time-course of burst number provides an important behavioural substrate for the study of the mechanisms governing ingestion, behavioural activation and the related evaluation processes, and might provide decisive evidence in the analysis of the effects of drugs on ingestion. However, further evidence from independent sources is necessary to validate the use and the proposed interpretation of this measure.

Dopaminergic modulation of sensitivity to immediate and delayed punishment during decision-making

Effective decision-making involves careful consideration of all rewarding and aversive outcomes. Importantly, negative outcomes often occur later in time, leading to underestimation, or "discounting," of these consequences. Despite the frequent occurrence of delayed outcomes, little is known about the neurobiology underlying sensitivity to delayed punishment during decision-making. The Delayed Punishment Decision-making Task (DPDT) addresses this by assessing sensitivity to delayed versus immediate punishment in rats. Rats initially avoid punished reinforcers, then select this option more frequently when delay precedes punishment. We used DPDT to examine effects of acute systemic administration of catecholaminergic drugs on sensitivity to delayed punishment in male and female adult rats. Cocaine did not affect choice of rewards with immediate punishment but caused a dose-dependent reduction in choice of delayed punishment. Neither activation nor blockade of D1-like dopamine receptor affected decision-making, but activation of D2-like dopamine receptors reduced choice of delayed punishment. D2 blockade did not attenuate cocaine's effects on decision-making, suggesting that cocaine's effects are not dependent on D2 receptor activation. Increasing synaptic norepinephrine via atomoxetine also reduced choice of delayed (but not immediate) punishment. Notably, when DPDT was modified from ascending to descending pre-punishment delays, these drugs did not affect choice of delayed or immediate punishment, although high-dose quinpirole impaired behavioral flexibility. In summary, sensitivity to delayed punishment is regulated by both dopamine and norepinephrine transmission in task-specific fashion. Understanding the neurochemical modulation of decision-making with delayed punishment is a critical step toward treating disorders characterized by aberrant sensitivity to negative consequences.

Women compared with men work harder for small rewards

In cost-benefit decision-making, women and men often show different trade-offs. However, surprisingly little is known about sex differences in instrumental tasks, where physical effort is exerted to gain rewards. To this end, we tested 81 individuals (47 women) with an effort allocation task, where participants had to repeatedly press a button to collect food and money tokens. We analyzed the motivational phases of invigoration and effort maintenance with varying reward magnitude, difficulty, and reward type. Whereas women and men did not differ in invigoration, we found that women showed higher effort maintenance as well as higher subjective wanting and exertion ratings for small rewards compared with men. Notably, men increased their effort more than women for higher rewards to match women's levels of performance. Crucially, we found no sex differences depending on reward type or difficulty, indicating that sex differences were specific to the encoding of the magnitude of benefits, not costs. To summarize, women exerted higher physical effort for small rewards, which corresponded with an elevated subjective value in women compared with men. Therefore, sex differences in perceived reward magnitude may contribute to differential behavioral preferences highlighting the potential of cost-benefit decision-making to provide insights about potential mechanisms.

Examining tradeoffs between cognitive effort and relief among adults with self-injurious behavior

BACKGROUND

People engage in nonsuicidal self-injury (NSSI) to reduce negative affect, but it is not clear why they engage in this harmful type of behavior instead of using healthier strategies. The primary goal of this study was to evaluate whether people choose NSSI to reduce negative affect because they perceive it to be less cognitively costly than other available strategies.

METHOD

In experiment one, 43 adults completed a novel, relief-based effort discounting task designed to index preferences about exerting cognitive effort to achieve relief. In experiment two, 149 adults, 52 % with a history of NSSI, completed our effort discounting task.

RESULTS

Our main results suggest that people will accept less relief from an aversive experience if doing so requires expending less effort, i.e. they demonstrate effort discounting in the context of decisions about relief. We also found and that effort discounting is stronger among those with a history of NSSI, but this association became nonsignificant when simultaneously accounting for other conditions associated with aberrant effort tradeoffs.

LIMITATIONS

The use of a control group without NSSI or other potentially harmful relief-seeking behaviors limits our ability to draw specific conclusions about NSSI. The ecological validity of our task was limited by a modestly effective affect manipulation, and because participants made hypothetical choices.

CONCLUSIONS

This study demonstrates that preferences about exerting cognitive effort may be a barrier to using healthier affect regulation strategies. Further, the preference not to exert cognitive effort, though present in NSSI, is likely not unique to NSSI. Instead, effort discounting may be a transdiagnostic mechanism promoting an array of harmful relief-seeking behaviors.

Accumbal adenosine A2A receptor inactivation biases for large and costly rewards in the effort- but not delay-based decision making

The cost-benefit decision-making (CBDM) is critical to normal human activity and a diminished willingness to expend effort to obtain rewards is a prevalent/noted characteristic of neuropsychiatric disorders such as schizophrenia, Parkinson's disease. Numerous studies have identified nucleus accumbens (NAc) as an important locus for CBDM control but their neuromodulatory and behavioral mechanisms remain largely under-explored. Adenosine A_2A receptors (A_2ARs), which are highly concentrated in the striatopallidal neurons, can integrate glutamate and dopamine signals for controlling effort-related choice behaviors. While the involvement of A_2ARs in effort-based decision making is well documented, the role of other decision variables (reward discrimination) in effort-based decision making and the role of A_2AR in delay-based decision making are less clear. In this study, we have developed a well-controlled CBDM behavioral paradigm to manipulate effort/cost and reward independently or in combination, allowing a dissection of four behavioral elements: effort-based CBDM (E-CBDM), delay-based CBDM (D-CBDM), reward discrimination (RD), effort discrimination (ED), and determined the effect of genetic knockdown (KD) of NAc A_2AR on the four behavioral elements. We found that A_2AR KD in NAc increased the choice for larger, more costly reward in the E-CBDM, but not D-CBDM. Furthermore, this high-effort/high-reward bias was attributable to the increased willingness to engage in effort but not the effect of discrimination of reward magnitude. Our findings substantiate an important role of the NAc A_2AR in control of E-CBDM and support that pharmacologically targeting NAc A_2ARs would be a useful strategy for treating the aberrant effort-based decision making in neuropsychiatric disorders.

Who Is Satisfied With Effort?

Abstract. The effort required to obtain certain rewards may influence the level of satisfaction with the following reward. Since people differ in beliefs about the availability of willpower resources required to pursue effortful actions, we investigated how willpower beliefs affect the perception of effort and satisfaction with reward. We hypothesized that people with limited willpower beliefs (i.e., believing that exerting effort leads to depletion of their inner resources) will perceive cognitive tasks as more effortful and will be less satisfied with the subsequent reward than those with non-limited beliefs (i.e., believing that exerting effort is invigorating rather than depleting). We tested this hypothesis by manipulating effort with different difficulty levels of the N-back task and measuring participants’ perception of effort expenditure and subjective satisfaction with a reward depending on their willpower beliefs. In accordance with the predictions, we found that those with limited willpower beliefs perceived the task as more effortful than those with non-limited willpower beliefs. Furthermore, when asked to subjectively rate their satisfaction with the reward gained for the task, limited believers rated their satisfaction lower than non-limited believers. These findings suggest that people take their willpower capacities into effort-satisfaction calculations. Results are discussed within the context of other models of effort, and practical implications of the findings are suggested.

Critical periods when dopamine controls behavioral responding during Pavlovian learning

RATIONALE

Learning the association between rewards and predictive cues is critical for appetitive behavioral responding. The mesolimbic dopamine system is thought to play an integral role in establishing these cue-reward associations. The dopamine response to cues can signal differences in reward value, though this emerges only after significant training. This suggests that the dopamine system may differentially regulate behavioral responding depending on the phase of training.

OBJECTIVES

The purpose of this study was to determine whether antagonizing dopamine receptors elicited different effects on behavior depending on the phase of training or the type of Pavlovian task.

METHODS

Separate groups of male rats were trained on Pavlovian tasks in which distinct audio cues signaled either differences in reward size or differences in reward rate. The dopamine receptor antagonist flupenthixol was systemically administered prior to either the first ten sessions of training (acquisition phase) or the second ten sessions of training (expression phase), and we monitored the effect of these manipulations for an additional ten training sessions.

RESULTS

We identified acute effects of dopamine receptor antagonism on conditioned responding, the latency to respond, and post-reward head entries in both Pavlovian tasks. Interestingly, dopamine receptor antagonism during the expression phase produced persistent deficits in behavioral responding only in rats trained on the reward size Pavlovian task.

CONCLUSIONS

Together, our results illustrate that dopamine's control over behavior in Pavlovian tasks depends upon one's prior training experience and the information signaled by the cues.

Prenatal dexamethasone exposure alters effort decision making and triggers nucleus accumbens and anterior cingulate cortex functional changes in male rats

Daily, individuals select actions based on cost-benefit to allocate resources into goal-directed actions. Different brain regions coordinate this complex decision, including the nucleus accumbens (NAc), anterior cingulate cortex (ACC), and ventral tegmental area (VTA). In utero exposure to synthetic glucocorticoids (iuGC), such as dexamethasone, triggers prominent motivation deficits but the impact of this exposure in the ACC-NAc and/or ACC-VTA circuits is unknown. Here, we show that iuGC exposure causes decreased motivation for natural rewards (food) and impaired effort-based decision-making. Importantly, reduced neuronal activation (number of c-fos⁺ neurons) was observed in the NAc core and ACC of iuGC rats in comparison to CTR rats after performing the effort-based decision-making task. In addition, iuGC treatment led to increased NAc and ACC basal neuronal activity. Electrophysiological recordings during optogenetic modulation of ACC terminals in the NAc revealed that the ACC-NAc circuit is dysfunctional in iuGC animals. These data suggest that iuGC animals present motivational and effort-based decision-making deficits that can be associated with the observed ACC-NAc dysfunction.

Cholinergic and dopaminergic-mediated motivated behavior in healthy states and in substance use and mood disorders

Acetylcholine is an important neuromodulator of the mesolimbic dopamine (DA) system, which itself is a mediator of motivated behavior. Motivated behavior can be described by two primary components, termed directional and activational motivation, both of which can be examined and dissociated using effort-choice tasks. The directional component refers to motivated behavior directed towards reinforcing stimuli and away from aversive stimuli. Behaviors characterized by increased vigor, persistence, and work output are considered to reflect activational components of motivation. Disruption of DA signaling has been shown to decrease activational components of motivation, while leaving directional features intact. Facilitation of DA release promotes the activational aspects of motivated behavior. In this review, we discuss cholinergic and DA regulation of motivated behaviors. We place emphasis on effort-choice processes and the ability of effort-choice tasks to examine and dissociate changes of motivated behavior in the context of substance use and mood disorders. Furthermore, we consider how altered cholinergic transmission impacts motivated behavior across disease states, and the possible role of cholinergic dysregulation in the etiology of these illnesses. Finally, we suggest that treatments targeting cholinergic activity may be useful in ameliorating motivational disruptions associated with substance use and comorbid substance use and mood disorders.

The Cost of Imagined Actions in a Reward-Valuation Task

Growing evidence suggests that humans and other animals assign value to a stimulus based not only on its inherent rewarding properties, but also on the costs of the action required to obtain it, such as the cost of time. Here, we examined whether such cost also occurs for mentally simulated actions. Healthy volunteers indicated their subjective value for snack foods while the time to imagine performing the action to obtain the different stimuli was manipulated. In each trial, the picture of one food item and a home position connected through a path were displayed on a computer screen. The path could be either large or thin. Participants first rated the stimulus, and then imagined moving the mouse cursor along the path from the starting position to the food location. They reported the onset and offset of the imagined movements with a button press. Two main results emerged. First, imagery times were significantly longer for the thin than the large path. Second, participants liked significantly less the snack foods associated with the thin path (i.e., with longer imagery time), possibly because the passage of time strictly associated with action imagery discounts the value of the reward. Importantly, such effects were absent in a control group of participants who performed an identical valuation task, except that no action imagery was required. Our findings hint at the idea that imagined actions, like real actions, carry a cost that affects deeply how people assign value to the stimuli in their environment.

Sex Differences in Behavioral Responding and Dopamine Release during Pavlovian Learning

Learning associations between cues and rewards require the mesolimbic dopamine system. The dopamine response to cues signals differences in reward value in well trained animals. However, these value-related dopamine responses are absent during early training sessions when cues signal differences in the reward rate. These findings suggest cue-evoked dopamine release conveys differences between outcomes only after extensive training, though it is unclear whether this is unique to when cues signal differences in reward rate, or whether this is also evident when cues signal differences in other value-related parameters such as reward size. To address this, we used a Pavlovian conditioning task in which one audio cue was associated with a small reward (one pellet) and another audio cue was associated with a large reward (three pellets). We performed fast-scan cyclic voltammetry to record changes in dopamine release in the nucleus accumbens of male and female rats throughout learning. While female rats exhibited higher levels of conditioned responding, a faster latency to respond, and elevated post-reward head entries relative to male rats, there were no sex differences in the dopamine response to cues. Multiple training sessions were required before cue-evoked dopamine release signaled differences in reward size. Reward-evoked dopamine release scaled with reward size, though females displayed lower reward-evoked dopamine responses relative to males. Conditioned responding related to the decrease in the peak reward-evoked dopamine response and not to cue-evoked dopamine release. Collectively, these data illustrate sex differences in behavioral responding as well as in reward-evoked dopamine release during Pavlovian learning.

How changes in dopamine D2 receptor levels alter striatal circuit function and motivation

It was first posited, more than five decades ago, that the etiology of schizophrenia involves overstimulation of dopamine receptors. Since then, advanced clinical research methods, including brain imaging, have refined our understanding of the relationship between striatal dopamine and clinical phenotypes as well as disease trajectory. These studies point to striatal dopamine D2 receptors, the main target for all current antipsychotic medications, as being involved in both positive and negative symptoms. Simultaneously, animal models have been central to investigating causal relationships between striatal dopamine D2 receptors and behavioral phenotypes relevant to schizophrenia. We begin this article by reviewing the circuit, cell-type and subcellular locations of dopamine D2 receptors and their downstream signaling pathways. We then summarize results from several mouse models in which D2 receptor levels were altered in various brain regions, cell-types and developmental periods. Behavioral, electrophysiological and anatomical consequences of these D2 receptor perturbations are reviewed with a selective focus on striatal circuit function and alterations in motivated behavior, a core negative symptom of schizophrenia. These studies show that D2 receptors serve distinct physiological roles in different cell types and at different developmental time points, regulating motivated behaviors in sometimes opposing ways. We conclude by considering the clinical implications of this complex regulation of striatal circuit function by D2 receptors.

Locus coeruleus neurons encode the subjective difficulty of triggering and executing actions

The brain stem noradrenergic nucleus locus coeruleus (LC) is involved in various costly processes: arousal, stress, and attention. Recent work has pointed toward an implication in physical effort, and indirect evidence suggests that the LC could be also involved in cognitive effort. To assess the dynamic relation between LC activity, effort production, and difficulty, we recorded the activity of 193 LC single units in 5 monkeys performing 2 discounting tasks (a delay discounting task and a force discounting task), as well as a simpler target detection task where conditions were matched for difficulty and only differed in terms of sensory-motor processes. First, LC neurons displayed a transient activation both when monkeys initiated an action and when exerting force. Second, the magnitude of the activation scaled with the associated difficulty, and, potentially, the corresponding amount of effort produced, both for decision and force production. Indeed, at action initiation in both discounting tasks, LC activation increased in conditions associated with lower average engagement rate, i.e., those requiring more cognitive control to trigger the response. Decision-related activation also scaled with response time (RT), over and above task parameters, in line with the idea that it reflects the amount of resources (here time) spent on the decision process. During force production, LC activation only scaled with the amount of force produced in the force discounting task, but not in the control target detection task, where subjective difficulty was equivalent across conditions. Our data show that LC neurons dynamically track the amount of effort produced to face both cognitive and physical challenges with a subsecond precision. This works provides key insight into effort processing and the contribution of the noradrenergic system, which is affected in several pathologies where effort is impaired, including Parkinson disease and depression.

Subthalamic stimulation breaks the balance between distal and axial signs in Parkinson's disease

In Parkinson’s disease (PD), the effects of both L_dopa and subthalamic deep brain stimulation (STN-DBS) are known to change cost-valuation. However, this was mostly studied through reward-effort task involving distal movements, while axial effort, less responsive to treatments, have been barely studied. Thus, our objective was to compare the influence of both L_dopa and STN-DBS on cost-valuation between two efforts modalities: vowel production (as an example of axial movement) and hand squeezing (as an example of distal movement). Twelve PD patients were recruited to participate in this study. The task consisted in deciding whether to accept or reject trials based on a reward-effort trade-off. Participants performed two blocks with hand squeezing, and two with vowel production, in the four treatment conditions (L_dopaOn/Off; STN-DBS On/Off). We found that STN-DBS changed the ratio difference between hand and phonation efforts. Vowel production effort was estimated easier to perform with STN-DBS alone, and harder when associated with L_dopa. The difference between hand and phonation efforts was correlated with quality of life in Off/Off and On L_dopa alone conditions, and with impulsive assessment On STN-DBS alone. We highlighted that STN-DBS could introduce an imbalance between the actual motor impairments and their subjective costs. With this finding, we also suggest paying particular attention to the different treatment effects that should be expected for axial and distal movement dysfunctions.

Carrots for the donkey: Influence of evaluative conditioning and training on self-paced exercise intensity and delay discounting of exercise in healthy adults

To choose exercise over alternative behaviours, subjective reward evaluation of the potential choices is a principal step in decision making. However, the selection of exercise intensity might integrate acute visceral responses (i.e. pleasant or unpleasant feelings) and motives related to goals (i.e. enjoyment, competition, health). To understand the factors determining the selection of exercise in its intensity and evaluation as a modality, we conducted a study combining exercise training and evaluative conditioning. Evaluative conditioning was performed by using a novel technique using a primary reinforcer (sweetness) as the unconditioned stimulus and physical strain i.e. heart rate elevation as the conditioned stimulus during interval training, using a randomized control design (N = 58). Pre, post-three weeks interval training w/o conditioning, and after 4 weeks follow-up, participants were tested on self-paced speed selection on treadmill measuring heart rate, subjective pleasantness, and effort levels, as well as delay-discounting of exercise and food rewards. Results revealed that the selection of exercise intensity was significantly increased by adaptation to training and evaluative conditioning, revealing the importance of visceral factors as well as learned expected rewards. Delay discounting rates of self-paced exercise were transiently reduced by training but not affected by evaluative conditioning. In conclusion, exercise decisions are suggested to separate the decision-making process into a modality-specific cognitive evaluation of exercise, and an exercise intensity selection based on acute visceral experience integrating effort, pleasantness, and learned rewards.

Delays to Reward Delivery Enhance the Preference for an Initially Less Desirable Option: Role for the Basolateral Amygdala and Retrosplenial Cortex

Temporal costs influence reward-based decisions. This is commonly studied in temporal discounting tasks that involve choosing between cues signaling an imminent reward option or a delayed reward option. However, it is unclear whether the temporal delay before a reward can alter the value of that option. To address this, we identified the relative preference between different flavored rewards during a free-feeding test using male and female rats. Animals underwent training where either the initial preferred or the initial less preferred reward was delivered noncontingently. By manipulating the intertrial interval during training sessions, we could determine whether temporal delays impact reward preference in a subsequent free-feeding test. Rats maintained their initial preference if the same delays were used across all training sessions. When the initial less preferred option was delivered after short delays (high reward rate) and the initial preferred option was delivered after long delays (low reward rate), rats expectedly increased their preference for the initial less desirable option. However, rats also increased their preference for the initial less desirable option under the opposite training contingencies: delivering the initial less preferred reward after long delays and the initial preferred reward after short delays. These data suggest that sunk temporal costs enhance the preference for a less desirable reward option. Pharmacological and lesion experiments were performed to identify the neural systems responsible for this behavioral phenomenon. Our findings demonstrate the basolateral amygdala and retrosplenial cortex are required for temporal delays to enhance the preference for an initially less desirable reward.SIGNIFICANCE STATEMENT The goal of this study was to determine how temporal delays influence reward preference. We demonstrate that delivering an initially less desirable reward after long delays subsequently increases the consumption and preference for that reward. Furthermore, we identified the basolateral amygdala and the retrosplenial cortex as essential nuclei for mediating the change in reward preference elicited by sunk temporal costs.

Dopamine release and its control over early Pavlovian learning differs between the NAc core and medial NAc shell

Dopamine neurons respond to cues to reflect the value of associated outcomes. These cue-evoked dopamine responses can encode the relative rate of reward in rats with extensive Pavlovian training. Specifically, a cue that always follows the previous reward by a short delay (high reward rate) evokes a larger dopamine response in the nucleus accumbens (NAc) core relative to a distinct cue that always follows the prior reward by a long delay (low reward rate). However, it was unclear if these reward rate dopamine signals are evident during early Pavlovian training sessions and across NAc subregions. To address this, we performed fast-scan cyclic voltammetry recordings of dopamine levels to track the pattern of cue- and reward-evoked dopamine signals in the NAc core and medial NAc shell. We identified regional differences in the progression of cue-evoked dopamine signals across training. However, the dopamine response to cues did not reflect the reward rate in either the NAc core or the medial NAc shell during early training sessions. Pharmacological experiments found that dopamine-sensitive conditioned responding emerged in the NAc core before the medial NAc shell. Together, these findings illustrate regional differences in NAc dopamine release and its control over behavior during early Pavlovian learning.

Anhedonia as a central factor in depression: Neural mechanisms revealed from preclinical to clinical evidence

Anhedonia is one of the core symptoms of major depressive disorder (MDD), which is often inadequately treated by traditional antidepressants. The modern framework of anhedonia extends the definition from impaired consummatory pleasure or interest in rewards to a broad spectrum of deficits that impact functions such as reward anticipation, approach motivation, effort expenditure, reward valuation, expectation, and reward-cue association learning. Substantial preclinical and clinical research has explored the neural basis of reward deficits in the context of depression, and has implicated mesocorticolimbic reward circuitry comprising the nucleus accumbens, ventral pallidum, ventral tegmental area, amygdala, hippocampus, anterior cingulate, insula, orbitofrontal cortex, and other prefrontal cortex regions. Dopamine modulates several reward facets including anticipation, motivation, effort, and learning. As well, serotonin, norepinephrine, opioids, glutamate, Gamma aminobutyric acid (GABA), and acetylcholine are also involved in anhedonia, and medications targeting these systems may also potentially normalize reward processing in depression. Unfortunately, whereas reward anticipation and reward outcome are extensively explored by both preclinical and clinical studies, translational gaps remain in reward motivation, effort, valuation, and learning, where clinical neuroimaging studies are in the early stages. This review aims to synthesize the neurobiological mechanisms underlying anhedonia in MDD uncovered by preclinical and clinical research. The translational difficulties in studying the neural basis of reward are also discussed.

Neural Representation of Costs and Rewards in Decision Making

Decision making is crucial for animal survival because the choices they make based on their current situation could influence their future rewards and could have potential costs. This review summarises recent developments in decision making, discusses how rewards and costs could be encoded in the brain, and how different options are compared such that the most optimal one is chosen. The reward and cost are mainly encoded by the forebrain structures (e.g., anterior cingulate cortex, orbitofrontal cortex), and their value is updated through learning. The recent development on dopamine and the lateral habenula's role in reporting prediction errors and instructing learning will be emphasised. The importance of dopamine in powering the choice and accounting for the internal state will also be discussed. While the orbitofrontal cortex is the place where the state values are stored, the anterior cingulate cortex is more important when the environment is volatile. All of these structures compare different attributes of the task simultaneously, and the local competition of different neuronal networks allows for the selection of the most appropriate one. Therefore, the total value of the task is not encoded as a scalar quantity in the brain but, instead, as an emergent phenomenon, arising from the computation at different brain regions.

Using metabolic energy to quantify the subjective value of physical effort

Economists have known for centuries that to understand an individual's decisions, we must consider not only the objective value of the goal at stake, but its subjective value as well. However, achieving that goal ultimately requires expenditure of effort. Surprisingly, despite the ubiquitous role of effort in decision-making and movement, we currently do not understand how effort is subjectively valued in daily movements. Part of the difficulty arises from the lack of an objective measure of effort. Here, we use a physiological approach to address this knowledge gap. We quantified objective effort costs by measuring metabolic cost via expired gas analysis as participants performed a reaching task against increasing resistance. We then used neuroeconomic methods to quantify each individual's subjective valuation of effort. Rather than the diminishing sensitivity observed in reward valuation, effort was valued objectively, on average. This is significantly less than the near-quadratic sensitivity to effort observed previously in force-based motor tasks. Moreover, there was significant inter-individual variability with many participants undervaluing or overvaluing effort. These findings demonstrate that in contrast with monetary decisions in which subjective value exhibits diminishing marginal returns, effort costs are valued more objectively in low-effort reaching movements common in daily life.

Dopamine D2R upregulation in ventral striatopallidal neurons does not affect Pavlovian or go/no-go learning

Ventral striatal dopamine is thought to be important for associative learning. Dopamine exerts its role via activation of dopamine D1 and D2 receptors in the ventral striatum. Upregulation of dopamine D2R in ventral striatopallidal neurons impairs incentive motivation via inhibiting synaptic transmission to the ventral pallidum. Here, we determined whether upregulation of D2Rs and the resulting impairment in ventral striatopallidal pathway function modulates associative learning in an auditory Pavlovian reward learning task as well as Go/No-Go learning in an operant based reward driven Go/No-Go task. We found that upregulation of D2Rs did not affect Pavlovian learning or the extinction of Pavlovian responses, and neither did it alter No-Go learning. We however observed a delay in the Go component of the task which may indicate a deficit in learning though it could also be attributed to the established locomotor hyperactivity of the mice. In combination with previously published findings, our data suggest that D2Rs in ventral striatopallidal neurons play a specific role in regulating motivation by balancing cost/benefit computations but do not necessarily affect associative learning. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Organizational principles of amygdalar input-output neuronal circuits

The amygdala, one of the most studied brain structures, integrates brain-wide heterogeneous inputs and governs multidimensional outputs to control diverse behaviors central to survival, yet how amygdalar input-output neuronal circuits are organized remains unclear. Using a simplified cell-type- and projection-specific retrograde transsynaptic tracing technique, we scrutinized brain-wide afferent inputs of four major output neuronal groups in the amygdalar basolateral complex (BLA) that project to the bed nucleus of the stria terminals (BNST), ventral hippocampus (vHPC), medial prefrontal cortex (mPFC) and nucleus accumbens (NAc), respectively. Brain-wide input-output quantitative analysis unveils that BLA efferent neurons receive a diverse array of afferents with varied input weights and predominant contextual representation. Notably, the afferents received by BNST-, vHPC-, mPFC- and NAc-projecting BLA neurons exhibit virtually identical origins and input weights. These results indicate that the organization of amygdalar BLA input-output neuronal circuits follows the input-dependent and output-independent principles, ideal for integrating brain-wide diverse afferent stimuli to control parallel efferent actions. The data provide the objective basis for improving the virtual reality exposure therapy for anxiety disorders and validate the simplified cell-type- and projection-specific retrograde transsynaptic tracing method.

Identity pathology and borderline personality disorder: an empirical overview

Pathological identity-related functioning is a core feature of borderline personality disorder (BPD) that is grievously underrepresented in the empirical literature. Burgeoning evidence indicates that identity pathology is multifaceted, influenced by transactions between social and endogenous processes, and associated with a wide variety of maladaptive outcomes. Although historically defined by patterns of instability (e.g. frequent and rapid changes to career goals, religious beliefs, core values, friend groups, etc.), accumulating research indicates that identity pathology also manifests as distress and functional impairment related to excessive rigidity (i.e. unrelenting negative self-evaluations; overidentification with a restricted role or group membership) and/or incongruous behavior (i.e. simultaneously holding discordant beliefs/values/attitudes, actions that are grossly contradictory with values, incoherent sense of self, etc.). Despite the complex nature of this construct, and its centrality to BPD, researchers often assess identity pathology indirectly, inadequately, or fail to measure it entirely. In this article, we briefly describe how identity pathology may fit into etiological models of BPD, highlight emerging themes from recent research, and identify important gaps in our knowledge. We also discuss limitations associated with current assessment practices, recommend alternatives, and articulate a research agenda for the future.

Amphetamine reduces reward encoding and stabilizes neural dynamics in rat anterior cingulate cortex

Psychostimulants such as d-amphetamine (AMPH) often have behavioral effects that appear paradoxical within the framework of optimal choice theory. AMPH typically increases task engagement and the effort animals exert for reward, despite decreasing reward valuation. We investigated neural correlates of this phenomenon in the anterior cingulate cortex (ACC), a brain structure implicated in signaling cost-benefit utility. AMPH decreased signaling of reward, but not effort, in the ACC of freely-moving rats. Ensembles of simultaneously recorded neurons generated task-specific trajectories of neural activity encoding past, present, and future events. Low-dose AMPH contracted these trajectories and reduced their variance, whereas high-dose AMPH expanded both. We propose that under low-dose AMPH, increased network stability balances moderately increased excitability, which promotes accelerated unfolding of a neural 'script' for task execution, despite reduced reward valuation. Noise from excessive excitability at high doses overcomes stability enhancement to drive frequent deviation from the script, impairing task execution.

Vagus nerve stimulation boosts the drive to work for rewards

Interoceptive feedback transmitted via the vagus nerve plays a vital role in motivation by tuning actions according to physiological needs. Whereas vagus nerve stimulation (VNS) reinforces actions in animals, motivational effects elicited by VNS in humans are still largely elusive. Here, we applied non-invasive transcutaneous auricular VNS (taVNS) on the left or right ear while participants exerted effort to earn rewards using a randomized cross-over design (vs. sham). In line with preclinical studies, acute taVNS enhances invigoration of effort, and stimulation on the left side primarily facilitates invigoration for food rewards. In contrast, we do not find conclusive evidence that acute taVNS affects effort maintenance or wanting ratings. Collectively, our results suggest that taVNS enhances reward-seeking by boosting invigoration, not effort maintenance and that the stimulation side affects generalization beyond food reward. Thus, taVNS may enhance the pursuit of prospective rewards which may pave avenues to treat motivational deficiencies.

The vagus nerve transmits signals between the gut and the brain thereby tuning motivated behavior to physiological needs. Here, the authors show that acute non-invasive stimulation of the vagus nerve via the ear enhances the invigoration of effort for rewards.

A low dosage of the dopamine D2-receptor antagonist sulpiride affects effort allocation for reward regardless of trait extraversion

Dopamine (DA) is known to be involved in various aspects of reward processing and goal-directed behavior. The present preregistered study aims at directly accessing the causal influence of DA activity on reward motivation in humans, while also accounting for trait extraversion. Therefore, we examined the effect of a single dose of the DA D2 receptor antagonist sulpiride (200 mg) on effort allocation in a modified version of the Effort-Expenditure for Reward Task (EEfRT). Based on its presumably DA increasing action, we expected the low dose of sulpiride to increase participants' willingness to allocate effort during the modified EEfRT relative to placebo, especially in trials with low probability of reward attainment. Further, we expected a moderating effect of trait extraversion on the effects of sulpiride. Two hundred and three healthy male participants were tested in a randomized, double-blind between-subjects design. Contrary to our expectations, sulpiride reduced the average number of clicks within the modified EEfRT and did not interact with reward attributes, suggesting a more global and not reward-specific effect of sulpiride. Furthermore, trait extraversion did not moderate the effect of sulpiride. Our results provide initial support for the validity of the modified version of the EEfRT, suggesting a possible inhibiting effect of a low dose of sulpiride on approach motivation regardless of trait extraversion. However, given the mixed pattern of findings and the possible confounding role of motoric abilities, further studies examining these effects are clearly warranted.

Modeling Dynamic Allocation of Effort in a Sequential Task Using Discounting Models

Most rewards in our lives require effort to obtain them. It is known that effort is seen by humans as carrying an intrinsic disutility which devalues the obtainable reward. Established models for effort discounting account for this by using participant-specific discounting parameters inferred from experiments. These parameters offer only a static glance into the bigger picture of effort exertion. The mechanism underlying the dynamic changes in a participant's willingness to exert effort is still unclear and an active topic of research. Here, we modeled dynamic effort exertion as a consequence of effort- and probability-discounting mechanisms during goal reaching, sequential behavior. To do this, we developed a novel sequential decision-making task in which participants made binary choices to reach a minimum number of points. Importantly, the time points and circumstances of effort allocation were decided by participants according to their own preferences and not imposed directly by the task. Using the computational model to analyze participants' choices, we show that the dynamics of effort exertion arise from a combination of changing task needs and forward planning. In other words, the interplay between a participant's inferred discounting parameters is sufficient to explain the dynamic allocation of effort during goal reaching. Using formal model comparison, we also inferred the forward-planning strategy used by participants. The model allowed us to characterize a participant's effort exertion in terms of only a few parameters. Moreover, the model can be adapted to a number of tasks used in establishing the neural underpinnings of forward-planning behavior and meta-control, allowing for the characterization of behavior in terms of model parameters.

Signals of anticipation of reward and of mean reward rates in the human brain

Theoretical models of dopamine function stemming from reinforcement learning theory have emphasized the importance of prediction errors, which signal changes in the expectation of impending rewards. Much less is known about the effects of mean reward rates, which may be of motivational significance due to their role in computing the optimal effort put into exploiting reward opportunities. Here, we used a reinforcement learning model to design three functional neuroimaging studies and disentangle the effects of changes in reward expectations and mean reward rates, showing recruitment of specific regions in the brainstem regardless of prediction errors. While changes in reward expectations activated ventral striatal areas as in previous studies, mean reward rates preferentially modulated the substantia nigra/ventral tegmental area, deep layers of the superior colliculi, and a posterior pontomesencephalic region. These brainstem structures may work together to set motivation and attentional efforts levels according to perceived reward opportunities.

Voluntary exercise is motivated by ghrelin, possibly related to the central reward circuit

We previously reported that voluntary exercise contributed to the amelioration of abnormal feeding behavior with a concomitant restoration of ghrelin production in a rat model of obesity, suggesting a possible relationship between exercise and appetite-regulating hormones. Ghrelin is known to be involved in the brain reward circuits via dopamine neurons related to motivational properties. We investigated the relevance of ghrelin as an initiator of voluntary exercise as well as feeding behavior. The plasma ghrelin concentration fluctuates throughout the day with its peak at the beginning of the dark period in the wild-type (WT) mice with voluntary exercise. Although predominant increases in wheel running activity were observed accordant to the peak of plasma ghrelin concentration in the WT mice, those were severely attenuated in the ghrelin-knockout (GKO) mice under either ad libitum or time-restricted feeding. A single injection of ghrelin receptor agonist brought about and reproduced a marked enhancement of wheel running activity, in contrast to no effect by the continuous administration of the same drug. Brain dopamine levels (DAs) were enhanced after food consumption in the WT mice under voluntary exercise. Although the acceleration of DAs were apparently blunted in the GKO mice, they were dramatically revived after the administration of ghrelin receptor agonist, suggesting the relevance of ghrelin in the reward circuit under voluntary exercise. These findings emphasize that the surge of ghrelin plays a crucial role in the formation of motivation for the initiation of voluntary exercise possibly related to the central dopamine system.

Missing motoric manipulations: rethinking the imaging of the ventral striatum and dopamine in human reward

Human neuroimaging studies of natural rewards and drugs of abuse frequently assay the brain's response to stimuli that, through Pavlovian learning, have come to be associated with a drug's rewarding properties. This might be characterized as a 'sensorial' view of the brain's reward system, insofar as the paradigms are designed to elicit responses to a reward's (drug's) sight, aroma, or flavor. A different field of research nevertheless suggests that the mesolimbic dopamine system may also be critically involved in the motor behaviors provoked by such stimuli. This brief review and commentary surveys some of the preclinical data supporting this more "efferent" (motoric) view of the brain's reward system, and discusses what such findings might mean for how human brain imaging studies of natural rewards and drugs of abuse are designed.

A Reward-Based Framework of Perceived Control

Perceived control can be broadly defined as the belief in one's ability to exert control over situations or events. It has long been known that perceived control is a major contributor toward mental and physical health as well as a strong predictor of achievements in life. However, one issue that limits a mechanistic understanding of perceived control is the heterogeneity of how the term is defined in models in psychology and neuroscience, and used in experimental settings across a wide spectrum of studies. Here, we propose a framework for studying perceived control by integrating the ideas from traditionally separate work on perceived control. Specifically, we discuss key properties of perceived control from a reward-based framework, including choice opportunity, instrumental contingency, and success/reward rate. We argue that these separate reward-related processes are integral to fostering an enhanced perception of control and influencing an individual's behavior and well-being. We draw on select studies to elucidate how these reward-related elements are implicated separately and collectively in the investigation of perceived control. We highlight the role of dopamine within corticostriatal pathways shared by reward-related processes and perceived control. Finally, through the lens of this reward-based framework of perceived control, we consider the implications of perceived control in clinical deficits and how these insights could help us better understand psychopathology and treatment options.

Effects of subthalamic nucleus stimulation and levodopa on decision-making in Parkinson's disease

BACKGROUND

Parkinson's disease (PD) is frequently associated with behavioral disorders, particularly within the spectrum of motivated behaviors such as apathy or impulsivity. Both pharmacological and neurosurgical treatments have an impact on these impairments. However, there still is controversy as to whether subthalamic nucleus deep brain stimulation (STN-DBS) can cause or reduce impulsive behaviors.

OBJECTIVES

We aimed to identify the influence of functional surgery on decision-making processes in PD.

METHODS

We studied 13 PD patients and 13 healthy controls. The experimental task involved squeezing a dynamometer with variable force to obtain rewards of various values under four conditions: without treatment, with l-dopa or subthalamic stimulation alone, and with both l-dopa and subthalamic stimulation. Statistical analyses consisted of generalized linear mixed models including treatment condition, reward value, level of effort, and their interactions. We analyzed acceptance rate (the percentage of accepted trials), decision time, and force applied.

RESULTS

Comparatively to controls, patients without treatment exhibited lower acceptance rate and force applied. Patients under l-dopa alone did not exhibit increased acceptance rate. With subthalamic stimulation, either with or without added l-dopa, all measures were improved so that patients' behaviors were undistinguishable from healthy controls'.

CONCLUSIONS

Our study shows that l-dopa administration does not fully restore cost-benefit decision-making processes, whereas STN-DBS fully normalizes patients' behaviors. These findings suggest that dopamine is partly involved in cost-benefit valuation, and that STN-DBS can have a beneficial effect on motivated behaviors in PD and may improve certain forms of impulsive behaviors. © 2019 International Parkinson and Movement Disorder Society.

What Is the Relationship between Dopamine and Effort?

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remains poorly understood, both at the behavioral and neurophysiological levels. This is important because unwillingness to overcome effort to gain reward is a common feature of many neuropsychiatric and neurological disorders. A recent surge in interest in the neurobiological basis of effort has led to seemingly conflicting results regarding the role of dopamine. We argue here that, upon closer examination, there is actually striking consensus across studies: dopamine primarily codes for future reward but is less sensitive to anticipated effort cost. This strong association between dopamine and the incentive effects of rewards places dopamine in a key position to promote reward-directed action.

Biological and Behavioral Mechanisms of Identity Pathology Development: An Integrative Review

Although identity disturbance is a transdiagnostic mental health problem, modern explanatory models for its emergence are limited. To date, the social, developmental, clinical, and neuropsychological literatures exploring identity processes are also largely disconnected. Existing theories have laid the foundation for understanding important components of identity pathology, yet many overlook biological, behavioral, and interactive processes by which these difficulties may emerge. In this integrative review, we explore how broad transdiagnostic vulnerabilities for psychopathology and more specific risky behavioral processes may reciprocally interact and be refined over time into an identity disturbance profile. Our primary purpose is to review behavioral and biosocial theories and derive a testable conceptual framework for how identity disturbance emerges over the course of development. We aim to describe and integrate several disparate lines of theory and research in order to illuminate potential etiological pathways to identity pathology.

The anatomy of apathy: A neurocognitive framework for amotivated behaviour

Apathy is a debilitating syndrome associated with many neurological disorders, including several common neurodegenerative diseases such as Parkinson's disease and Alzheimer's disease, and focal lesion syndromes such as stroke. Here, we review neuroimaging studies to identify anatomical correlates of apathy, across brain disorders. Our analysis reveals that apathy is strongly associated with disruption particularly of dorsal anterior cingulate cortex (dACC), ventral striatum (VS) and connected brain regions. Remarkably, these changes are consistent across clinical disorders and imaging modalities. Review of the neuroimaging findings allows us to develop a neurocognitive framework to consider potential mechanisms underlying apathy. According to this perspective, an interconnected group of brain regions - with dACC and VS at its core - plays a crucial role in normal motivated behaviour. Specifically we argue that motivated behaviour requires a willingness to work, to keep working, and to learn what is worth working for. We propose that deficits in any one or more of these processes can lead to the clinical syndrome of apathy, and outline specific approaches to test this hypothesis. A richer neurobiological understanding of the mechanisms underlying apathy should ultimately facilitate development of effective therapies for this disabling condition.

Reward circuitry activation reflects social preferences in the face of cognitive effort

Research at the intersection of social neuroscience and cognitive effort is an interesting new area for exploration. There is great potential to broaden our understanding of how social context and cognitive effort processes, currently addressed in disparate literatures, interact with one another. In this paper, we briefly review the literature on cognitive effort, focusing on effort-linked valuation and the gap in the literature regarding cognitive effort in the social domain. Next, we present a study designed to explore valuation processes linked to cognitive effort within the social context of an inequality manipulation. More specifically, we created monetary inequality among the participant (SELF, endowed with $50) and two confederates: one also endowed with $50 (OTHER HIGH) and another with only $5 (OTHER LOW). We then scanned participants using fMRI as they attempted to earn bonus payments for themselves and others through a cognitively effortful feedback-based learning task. Positive feedback produced significantly greater activation than negative feedback in key valuation regions, the ventral striatum (VS) and ventromedial prefrontal cortex (vmPFC), both when participants were performing the task on their own behalf and when earning rewards for others. While reward-related activity in the VS was exaggerated for SELF compared to OTHER HIGH for both positive and negative feedback, activity in the vmPFC did not distinguish between recipients in the group-level results. Furthermore, participants naturally fell into two groups: those most engaged when playing for themselves and those who reported engagement for others. While Self-Engaged participants showed differences between the SELF and both OTHER conditions in the VS and vmPFC, Other-Engaged participants only showed an attenuated response to negative feedback for OTHER HIGH compared to SELF in the VS and no differences between recipient conditions in the vmPFC. Together, this work shows the importance of individual differences and the fragility of advantageous inequality aversion in the face of cognitive effort, highlighting the need to study cognitive effort in the social domain.

Corticoinsular circuits encode subjective value expectation and violation for effortful goal-directed behavior

The ability to form value estimates is crucial for optimal decision making, especially when not all features of a choice option are known. To date, however, the neural mechanisms for expectation processes under conditions of incomplete information are unknown. Using computational fMRI, we show that ventromedial prefrontal cortex encodes the expected value of a trial. We also observe a distinct network composed of dorsal anterior cingulate, anterior insula, and dorsomedial caudate that encodes an expectation violation or prediction error signal, based on previous trial history. These findings highlight how the brain computes and monitors value-based predictions during effortful goal-directed behavior when choice-relevant information is not fully available.

We are presented with choices each day about how to invest our effort to achieve our goals. Critically, these decisions must frequently be made under conditions of incomplete information, where either the effort required or possible reward to be gained is uncertain. Such choices therefore require the development of potential value estimates to guide effortful goal-directed behavior. To date, however, the neural mechanisms for this expectation process are unknown. Here, we used computational fMRI during an effort-based decision-making task where trial-wise information about effort costs and reward magnitudes was presented separately over time, thereby allowing us to model distinct effort/reward computations as choice-relevant information unfolded. We found that ventromedial prefrontal cortex (vmPFC) encoded expected subjective value. Further, activity in dorsal anterior cingulate (dACC) and anterior insula (aI) reflected both effort discounting as well as a subjective value prediction error signal derived from trial history. While prior studies have identified these regions as being involved in effort-based decision making, these data demonstrate their specific role in the formation and maintenance of subjective value estimates as relevant information becomes available.

Dopamine, Effort-Based Choice, and Behavioral Economics: Basic and Translational Research

Operant behavior is not only regulated by factors related to the quality or quantity of reinforcement, but also by the work requirements inherent in performing instrumental actions. Moreover, organisms often make effort-related decisions involving economic choices such as cost/benefit analyses. Effort-based decision making is studied using behavioral procedures that offer choices between high-effort options leading to relatively preferred reinforcers vs. low effort/low reward choices. Several neural systems, including the mesolimbic dopamine (DA) system and other brain circuits, are involved in regulating effort-related aspects of motivation. Considerable evidence indicates that mesolimbic DA transmission exerts a bi-directional control over exertion of effort on instrumental behavior tasks. Interference with DA transmission produces a low-effort bias in animals tested on effort-based choice tasks, while increasing DA transmission with drugs such as DA transport blockers tends to enhance selection of high-effort options. The results from these pharmacology studies are corroborated by the findings from recent articles using optogenetic, chemogenetic and physiological techniques. In addition to providing important information about the neural regulation of motivated behavior, effort-based choice tasks are useful for developing animal models of some of the motivational symptoms that are seen in people with various psychiatric and neurological disorders (e.g., depression, schizophrenia, Parkinson’s disease). Studies of effort-based decision making may ultimately contribute to the development of novel drug treatments for motivational dysfunction.

Accumbens dopamine D2 receptors increase motivation by decreasing inhibitory transmission to the ventral pallidum

Dopamine D2 receptors (D2Rs) in the nucleus accumbens (NAc) regulate motivated behavior, but the underlying neurobiological mechanisms remain unresolved. Here, we show that selective upregulation of D2Rs in the indirect pathway of the adult NAc enhances the willingness to work for food. Mechanistic studies in brain slices reveal that D2R upregulation attenuates inhibitory transmission at two main output projections of the indirect pathway, the classical long-range projections to the ventral pallidum (VP), as well as local collaterals to direct pathway medium spiny neurons. In vivo physiology confirms the reduction in indirect pathway inhibitory transmission to the VP, and inhibition of indirect pathway terminals to VP is sufficient to enhance motivation. In contrast, D2R upregulation in the indirect pathway does not disinhibit neuronal activity of the direct pathway in vivo. These data suggest that D2Rs in ventral striatal projection neurons promote motivation by weakening the canonical output to the ventral pallidum.

A Neural Circuit Mechanism for the Involvements of Dopamine in Effort-Related Choices: Decay of Learned Values, Secondary Effects of Depletion, and Calculation of Temporal Difference Error

Dopamine has been suggested to be crucially involved in effort-related choices. Key findings are that dopamine depletion (i) changed preference for a high-cost, large-reward option to a low-cost, small-reward option, (ii) but not when the large-reward option was also low-cost or the small-reward option gave no reward, (iii) while increasing the latency in all the cases but only transiently, and (iv) that antagonism of either dopamine D1 or D2 receptors also specifically impaired selection of the high-cost, large-reward option. The underlying neural circuit mechanisms remain unclear. Here we show that findings i–iii can be explained by the dopaminergic representation of temporal-difference reward-prediction error (TD-RPE), whose mechanisms have now become clarified, if (1) the synaptic strengths storing the values of actions mildly decay in time and (2) the obtained-reward-representing excitatory input to dopamine neurons increases after dopamine depletion. The former is potentially caused by background neural activity–induced weak synaptic plasticity, and the latter is assumed to occur through post-depletion increase of neural activity in the pedunculopontine nucleus, where neurons representing obtained reward exist and presumably send excitatory projections to dopamine neurons. We further show that finding iv, which is nontrivial given the suggested distinct functions of the D1 and D2 corticostriatal pathways, can also be explained if we additionally assume a proposed mechanism of TD-RPE calculation, in which the D1 and D2 pathways encode the values of actions with a temporal difference. These results suggest a possible circuit mechanism for the involvements of dopamine in effort-related choices and, simultaneously, provide implications for the mechanisms of TD-RPE calculation.

To Do or Not to Do: Dopamine, Affordability and the Economics of Opportunity

Five years ago, we introduced the thrift hypothesis of dopamine (DA), suggesting that the primary role of DA in adaptive behavior is regulating behavioral energy expenditure to match the prevailing economic conditions of the environment. Here we elaborate that hypothesis with several new ideas. First, we introduce the concept of affordability, suggesting that costs must necessarily be evaluated with respect to the availability of resources to the organism, which computes a value not only for the potential reward opportunity, but also the value of resources expended. Placing both costs and benefits within the context of the larger economy in which the animal is functioning requires consideration of the different timescales against which to compute resource availability, or average reward rate. Appropriate windows of computation for tracking resources requires corresponding neural substrates that operate on these different timescales. In discussing temporal patterns of DA signaling, we focus on a neglected form of DA plasticity and adaptation, changes in the physical substrate of the DA system itself, such as up- and down-regulation of receptors or release probability. We argue that changes in the DA substrate itself fundamentally alter its computational function, which we propose mediates adaptations to longer temporal horizons and economic conditions. In developing our hypothesis, we focus on DA D2 receptors (D2R), arguing that D2R implements a form of “cost control” in response to the environmental economy, serving as the “brain’s comptroller”. We propose that the balance between the direct and indirect pathway, regulated by relative expression of D1 and D2 DA receptors, implements affordability. Finally, as we review data, we discuss limitations in current approaches that impede fully investigating the proposed hypothesis and highlight alternative, more semi-naturalistic strategies more conducive to neuroeconomic investigations on the role of DA in adaptive behavior.

How do memory systems detect and respond to novelty?

The efficiency of the memory system lies not only in its readiness to detect and retrieve old stimuli but also in its ability to detect and integrate novel information. In this review, we discuss recent evidence suggesting that the neural substrates sensitive to detecting familiarity and novelty are not entirely overlapping. Instead, these partially distinct familiarity and novelty signals are integrated to support recognition memory decisions. We propose here that the mediodorsal thalamus is critical for familiarity detection, and for combining novelty signals from the medial temporal lobe cortex with the relative familiarity outputs of computations performed in other cortical structures, especially the prefrontal cortex. Importantly, we argue that the anterior hippocampus has a prominent role in detecting novelty and in communicating this with midbrain and striatal structures. We argue that different types of novelty (absolute or contextual) engage different neurotransmitter systems that converge in the hippocampus. We suggest that contextual or unexpected novelty triggers dopaminergic hippocampal-midbrain coupling and noradrenergic-mediated pupil dilation. In contrast, absolute novelty triggers cholinergic-mediated hippocampal encoding accompanied by diminished pupil dilation. These two, distinct hippocampal encoding mechanisms both lead to later recollection but are sensitive to different types of novelty. We conclude that this neurotransmitter-mediated hippocampal encoding establishes the hippocampus in an encoding mode that briefly prevents the engagement of retrieval.

An Interaction between Serotonin Receptor Signaling and Dopamine Enhances Goal-Directed Vigor and Persistence in Mice

The functionally selective 5-HT2C receptor ligand SB242084 can increase motivation and have rapid onset anti-depressant-like effects. We sought to identify the specific behavioral effects of SB242084 treatment and elucidate the mechanism in female and male mice. Using a quantitative behavioral approach, we determined that SB242084 increases the vigor and persistence of goal-directed activity across different types of physical work, particularly when work requirements are demanding. We found this influence of SB242084 on effort, rather than reward to be reflected in striatal DA measured during behavior. Using in vivo fast scan cyclic voltammetry, we found that SB242084 has no effect on reward-related phasic DA release in the NAc. Using in vivo microdialysis to measure tonic changes in extracellular DA, we also found no changes in the NAc. In contrast, SB242084 treatment increases extracellular DA in the dorsomedial striatum, an area that plays a key role in response vigor. These findings have several implications. At the behavioral level, this work shows that the capacity to work in demanding situations can be increased, without a generalized increase in motor activity or reward value. At the circuit level, we identified a pathway restricted potentiation of DA release and showed that this was the reason for the increased response vigor. At the cellular level, we show that a specific serotonin receptor cross talks to the DA system. Together, this information provides promise for the development of treatments for apathy, a serious clinical condition that can afflict patients with psychiatric and neurological disorders.SIGNIFICANCE STATEMENT Motivated behaviors are modulated by reward value, effort demands, and cost-benefit computations. This information drives the decision to act, which action to select, and the intensity with which the selected action is performed. Because these behavioral processes are all regulated by DA signaling, it is very difficult to influence selected aspects of motivated behavior without affecting others. Here we identify a pharmacological treatment that increases the vigor and persistence of responding in mice, without increasing generalized activity or changing reactions to rewards. We show that the 5-HT2C-selective ligand boosts motivation by potentiating activity-dependent DA release in the dorsomedial striatum. These results reveal a novel strategy for treating patients with motivational deficits, avolition, or apathy.

Individual differences in empathy are associated with apathy-motivation

Empathy - the capacity to understand and resonate with the experiences of other people - is considered an essential aspect of social cognition. However, although empathy is often thought to be automatic, recent theories have argued that there is a key role for motivation in modulating empathic experiences. Here we administered self-report measures of empathy and apathy-motivation to a large sample of healthy people (n = 378) to test whether people who are more empathic are also more motivated. We then sought to replicate our findings in an independent sample (n = 198) that also completed a behavioural task to measure state affective empathy and emotion recognition. Cognitive empathy was associated with higher levels of motivation generally across behavioural, social and emotional domains. In contrast, affective empathy was associated with lower levels of behavioural motivation, but higher levels of emotional motivation. Factor analyses showed that empathy and apathy are distinct constructs, but that affective empathy and emotional motivation are underpinned by the same latent factor. These results have potentially important clinical applications for disorders associated with reduced empathy and motivation as well as the understanding of these processes in healthy people.

Anticipating cognitive effort: roles of perceived error-likelihood and time demands

Why are some actions evaluated as effortful? In the present set of experiments we address this question by examining individuals' perception of effort when faced with a trade-off between two putative cognitive costs: how much time a task takes vs. how error-prone it is. Specifically, we were interested in whether individuals anticipate engaging in a small amount of hard work (i.e., low time requirement, but high error-likelihood) vs. a large amount of easy work (i.e., high time requirement, but low error-likelihood) as being more effortful. In between-subject designs, Experiments 1 through 3 demonstrated that individuals anticipate options that are high in perceived error-likelihood (yet less time consuming) as more effortful than options that are perceived to be more time consuming (yet low in error-likelihood). Further, when asked to evaluate which of the two tasks was (a) more effortful, (b) more error-prone, and (c) more time consuming, effort-based and error-based choices closely tracked one another, but this was not the case for time-based choices. Utilizing a within-subject design, Experiment 4 demonstrated overall similar pattern of judgments as Experiments 1 through 3. However, both judgments of error-likelihood and time demand similarly predicted effort judgments. Results are discussed within the context of extant accounts of cognitive control, with considerations of how error-likelihood and time demands may independently and conjunctively factor into judgments of cognitive effort.

Dopamine Manipulation Affects Response Vigor Independently of Opportunity Cost

Dopamine is known to be involved in regulating effort investment in relation to reward, and the disruption of this mechanism is thought to be central in some pathological situations such as Parkinson's disease, addiction, and depression. According to an influential model, dopamine plays this role by encoding the opportunity cost, i.e., the average value of forfeited actions, which is an important parameter to take into account when making decisions about which action to undertake and how fast to execute it. We tested this hypothesis by asking healthy human participants to perform two effort-based decision-making tasks, following either placebo or levodopa intake in a double blind within-subject protocol. In the effort-constrained task, there was a trade-off between the amount of force exerted and the time spent in executing the task, such that investing more effort decreased the opportunity cost. In the time-constrained task, the effort duration was constant, but exerting more force allowed the subject to earn more substantial reward instead of saving time. Contrary to the model predictions, we found that levodopa caused an increase in the force exerted only in the time-constrained task, in which there was no trade-off between effort and opportunity cost. In addition, a computational model showed that dopamine manipulation left the opportunity cost factor unaffected but altered the ratio between the effort cost and reinforcement value. These findings suggest that dopamine does not represent the opportunity cost but rather modulates how much effort a given reward is worth.

SIGNIFICANCE STATEMENT

Dopamine has been proposed in a prevalent theory to signal the average reward rate, used to estimate the cost of investing time in an action, also referred to as opportunity cost. We contrasted the effect of dopamine manipulation in healthy participants in two tasks, in which increasing response vigor (i.e., the amount of effort invested in an action) allowed either to save time or to earn more reward. We found that levodopa-a synthetic precursor of dopamine-increases response vigor only in the latter situation, demonstrating that, rather than the opportunity cost, dopamine is involved in computing the expected value of effort.