Ratio and time requirements on operant schedules: effort-related effects of nucleus accumbens dopamine depletions

Apathy and Motivation: Biological Basis and Drug Treatment

Apathy is a disabling syndrome associated with poor functional outcomes that is common across a broad range of neurological and psychiatric conditions. Currently, there are no established therapies specifically for the condition, and safe and effective treatments are urgently needed. Advances in the understanding of motivation and goal-directed behavior in humans and animals have shed light on the cognitive and neurobiological mechanisms contributing to apathy, providing an important foundation for the development of new treatments. Here, we review the cognitive components, neural circuitry, and pharmacology of apathy and motivation, highlighting converging evidence of shared transdiagnostic mechanisms. Though no pharmacological treatments have yet been licensed, we summarize trials of existing and novel compounds to date, identifying several promising candidates for clinical use and avenues of future drug development.

Hyperdopaminergia in rats is associated with reverse effort-cost dependent performance

BACKGROUND

Dopamine is implicated in the effort-based control of motivational processes; however, whether tonic dopamine regulates the effort-cost impact on motivation, is still debated.

AIMS

The rats lacking the dopamine transporter (DAT), which have dramatically increased levels of the synaptic dopamine, were used in the present study to elucidate the role of the synaptic dopamine in motivational processes.

METHODS

To study the reward-related processes, the progressive ratio 3 (PR3) operant schedule of food reinforcement (the ratio increases by 3 after each earned reinforcer) was performed in adult male rats (DAT knockouts (DAT-KO), heterozygotes (DAT-HT) and wild-types (DAT-WT)).

RESULTS

During the PR3 session, the response rate of DAT-KO rats was gradually increased following the augmented required number of responses. In contrast, the local response rate of DAT-WT and DAT-HT decreased. d-Amphetamine sulfate salt (3 mg/kg, i.p.) altered the local response rate dynamics in DAT-WT, which became similar to that of DAT-KO. Interestingly, the reduction in response rate at low effort demands was associated with decreased rate of entries into the magazine tray in DAT-WT rats treated with amphetamine (3 mg/kg) but not in DAT-KO rats.

CONCLUSIONS

Our results suggest that the elevated tonic synaptic dopamine can strongly affect motivation/effort-cost relation in rodents.

K v 7 Channel Opener Retigabine Reduces Self-Administration of Cocaine but Not Sucrose in Rats

The increasing rates of drug misuse highlight the urgency of identifying improved therapeutics for treatment. Most drug-seeking behaviors that can be modeled in rodents utilize the repeated intravenous self-administration (SA) of drugs. Recent studies examining the mesolimbic pathway suggest that K v 7/KCNQ channels may contribute in the transition from recreational to chronic drug use. However, to date, all such studies used noncontingent, experimenter-delivered drug model systems, and the extent to which this effect generalizes to rats trained to self-administer drug is not known. Here, we tested the ability of retigabine (ezogabine), a K v 7 channel opener, to regulate instrumental behavior in male Sprague Dawley rats. We first validated the ability of retigabine to target experimenter-delivered cocaine in a CPP assay and found that retigabine reduced the acquisition of place preference. Next, we trained rats for cocaine-SA under a fixed-ratio or progressive-ratio reinforcement schedule and found that retigabine-pretreatment attenuated the self-administration of low to moderate doses of cocaine. This was not observed in parallel experiments, with rats self-administering sucrose, a natural reward. Compared to sucrose-SA, cocaine-SA was associated with reductions in the expression of the K v 7.5 subunit in the nucleus accumbens, without alterations in K v 7.2 and K v 7.3. Therefore, these studies reveal a reward specific reduction in SA behavior considered relevant for the study of long-term compulsive-like behavior and supports the notion that K v 7 is a potential therapeutic target for human psychiatric diseases with dysfunctional reward circuitry.

Critical review of RDoC approaches to the study of motivation with animal models: effort valuation/willingness to work

The NIMH research domain criteria (RDoC) approach was instigated to refocus mental health research on the neural circuits that mediate psychological functions, with the idea that this would foster an understanding of the neural basis of specific psychiatric dysfunctions (i.e. 'symptoms and circuits') and ultimately facilitate treatment. As a general idea, this attempt to go beyond traditional diagnostic categories and focus on neural circuit dysfunctions related to specific symptoms spanning multiple disorders has many advantages. For example, motivational dysfunctions are present in multiple disorders, including depression, schizophrenia, Parkinson's disease, and other conditions. A critical aspect of motivation is effort valuation/willingness to work, and several clinical studies have identified alterations in effort-based decision making in various patient groups. In parallel, formal animal models focusing on the exertion of effort and effort-based decision making have been developed. This paper reviews the literature on models of effort-based motivational function in the context of a discussion of the RDoC approach, with an emphasis on the dissociable nature of distinct aspects of motivation. For example, conditions associated with depression and schizophrenia blunt the selection of high-effort activities as measured by several tasks in animal models (e.g. lever pressing, barrier climbing, wheel running). Nevertheless, these manipulations also leave fundamental aspects of hedonic reactivity, food motivation, and reinforcement intact. This pattern of effects demonstrates that the general emphasis of the RDoC on the specificity of the neural circuits mediating behavioral pathologies, and the dissociative nature of these dysfunctions, is a valid concept. Nevertheless, the specific placement of effort-related processes as simply a 'sub-construct' of 'reward processing' is empirically and conceptually problematic. Thus, while the RDoC is an excellent general framework for new ways to approach research and therapeutics, it still needs further refinement.

Vigor, Effort-Related Aspects of Motivation and Anhedonia

In this chapter we provide an overview of the pharmacological and circuit mechanisms that determine the willingness to expend effort in pursuit of rewards. A particular focus will be on the role of the mesolimbic dopamine system, as well the contributing roles of limbic and cortical brains areas involved in the evaluation, selection, and invigoration of goal-directed actions. We begin with a review of preclinical studies, which have provided key insights into the brain systems that are necessary and sufficient for effort-based decision-making and have characterized novel compounds that enhance selection of high-effort activities. Next, we summarize translational studies identifying and expanding this circuitry in humans. Finally, we discuss the relevance of this work for understanding common motivational impairments as part of the broader anhedonia symptom domain associated with mental illness, and the identification of new treatment targets within this circuitry to improve motivation and effort-expenditure.

Cholinergic and dopaminergic-mediated motivated behavior in healthy states and in substance use and mood disorders

Acetylcholine is an important neuromodulator of the mesolimbic dopamine (DA) system, which itself is a mediator of motivated behavior. Motivated behavior can be described by two primary components, termed directional and activational motivation, both of which can be examined and dissociated using effort-choice tasks. The directional component refers to motivated behavior directed towards reinforcing stimuli and away from aversive stimuli. Behaviors characterized by increased vigor, persistence, and work output are considered to reflect activational components of motivation. Disruption of DA signaling has been shown to decrease activational components of motivation, while leaving directional features intact. Facilitation of DA release promotes the activational aspects of motivated behavior. In this review, we discuss cholinergic and DA regulation of motivated behaviors. We place emphasis on effort-choice processes and the ability of effort-choice tasks to examine and dissociate changes of motivated behavior in the context of substance use and mood disorders. Furthermore, we consider how altered cholinergic transmission impacts motivated behavior across disease states, and the possible role of cholinergic dysregulation in the etiology of these illnesses. Finally, we suggest that treatments targeting cholinergic activity may be useful in ameliorating motivational disruptions associated with substance use and comorbid substance use and mood disorders.

The influence of reinforcement schedule on experience-dependent changes in motivation

The progressive ratio procedure is used across fields to assess motivation for different reinforcers, define the effects of experimental interventions on motivation, and determine experience-dependent changes in motivation. However, less is known about how operant training schedules affect performance on this widely utilized task. Here we designed an experiment to examine the effect of variable ratio versus fixed ratio training schedules of reinforcement on progressive ratio performance while holding other performance variables constant between groups. We found a robust increase in maximum ratio completed between the pretest and posttraining test highlighting a robust training effect on progressive ratio performance. However, it did not matter if the training was under a fixed or variable ratio schedule. Additionally, we show that neither individual rates during training nor extinction responding correlated with maximum ratio achieved during the sessions. Finally, we show that rates during the training sessions do correlate with extinction performance, suggesting that these variables measure a different aspect of performance that does not predict motivation.

Dopamine D3 Receptors: A Potential Target to Treat Motivational Deficits in Parkinson's Disease

Parkinson's disease (PD), which is traditionally viewed as a motor disorder involving the degeneration of dopaminergic (DA) neurons, has recently been identified as a quintessential neuropsychiatric condition. Indeed, a plethora of non-motor symptoms may occur in PD, including apathy. Apathy can be defined as a lack of motivation or a deficit of goal-directed behaviors and results in a pathological decrease of self-initiated voluntary behavior. Apathy in PD appears to fluctuate with the DA state of the patients, suggesting a critical role of DA neurotransmission in the pathophysiology of this neuropsychiatric syndrome. Using a lesion-based approach, we developed a rodent model which exhibits specific alteration in the preparatory component of motivational processes, reminiscent to apathy in PD. We found a selective decrease of DA D₃ receptors (D₃R) expression in the dorsal striatum of lesioned rats. Next, we showed that inhibition of D₃R neurotransmission in non-lesioned animals was sufficient to reproduce the motivational deficit observed in our model. Interestingly, we also found that pharmacologically targeting D₃R efficiently reversed the motivational deficit induced by the lesion. Our findings, among other recent data, suggest a critical role of D₃R in parkinsonian apathy and highlight this receptor as a promising target for treating motivational deficits.

Disruptions in effort-based decision-making following acute optogenetic stimulation of ventral tegmental area dopamine cells

The dopamine system has been implicated in decision-making particularly when associated with effortful behavior. We examined acute optogenetic stimulation of dopamine cells in the ventral tegmental area (VTA) as mice engaged in an effort-based decision-making task. Tyrosine hydroxylase-Cre mice were injected with Cre-dependent ChR2 or eYFP control virus in the VTA. While eYFP control mice showed effortful discounting, stimulation of dopamine cells in ChR2 mice disrupted effort-based decision-making by reducing choice toward the lever associated with a preferred outcome and greater effort. Surprisingly, disruptions in effortful discounting were observed in subsequent test sessions conducted in the absence of optogenetic stimulation, however during these sessions ChR2 mice displayed enhanced high choice responding across trial blocks. These findings suggest increases in VTA dopamine cell activity can disrupt effort-based decision-making in distinct ways dependent on the timing of optogenetic stimulation.

Rational inattention and tonic dopamine

Slow-timescale (tonic) changes in dopamine (DA) contribute to a wide variety of processes in reinforcement learning, interval timing, and other domains. Furthermore, changes in tonic DA exert distinct effects depending on when they occur (e.g., during learning vs. performance) and what task the subject is performing (e.g., operant vs. classical conditioning). Two influential theories of tonic DA-the average reward theory and the Bayesian theory in which DA controls precision-have each been successful at explaining a subset of empirical findings. But how the same DA signal performs two seemingly distinct functions without creating crosstalk is not well understood. Here we reconcile the two theories under the unifying framework of 'rational inattention,' which (1) conceptually links average reward and precision, (2) outlines how DA manipulations affect this relationship, and in so doing, (3) captures new empirical phenomena. In brief, rational inattention asserts that agents can increase their precision in a task (and thus improve their performance) by paying a cognitive cost. Crucially, whether this cost is worth paying depends on average reward availability, reported by DA. The monotonic relationship between average reward and precision means that the DA signal contains the information necessary to retrieve the precision. When this information is needed after the task is performed, as presumed by Bayesian inference, acute manipulations of DA will bias behavior in predictable ways. We show how this framework reconciles a remarkably large collection of experimental findings. In reinforcement learning, the rational inattention framework predicts that learning from positive and negative feedback should be enhanced in high and low DA states, respectively, and that DA should tip the exploration-exploitation balance toward exploitation. In interval timing, this framework predicts that DA should increase the speed of the internal clock and decrease the extent of interference by other temporal stimuli during temporal reproduction (the central tendency effect). Finally, rational inattention makes the new predictions that these effects should be critically dependent on the controllability of rewards, that post-reward delays in intertemporal choice tasks should be underestimated, and that average reward manipulations should affect the speed of the clock-thus capturing empirical findings that are unexplained by either theory alone. Our results suggest that a common computational repertoire may underlie the seemingly heterogeneous roles of DA.

Effort Displayed During Appetitive Phase of Feeding Behavior Requires Infralimbic Cortex Activity and Histamine H1 Receptor Signaling

The chances to succeed in goal-directed behaviors, such as food or water-seeking, improve when the subject is in an increased arousal state. The appetitive phase of these motivated behaviors is characterized by high levels of behavioral and vegetative excitation. The key decision of engaging in those particular behaviors depends primarily on prefrontal cortical areas, such as the ventromedial prefrontal cortex. We propose that the infralimbic cortex (ILC) located in the medial prefrontal cortex induces an increase in arousal during the appetitive phase of motivated behavior, and that this increase in arousal is, in turn, mediated by the activation of the brain histaminergic system, resulting in higher motivation for getting food rewards. To test this hypothesis, we conduct a progressive ratio operant conditioning to test the degree of motivation for food, while simultaneously manipulating the histaminergic system through pharmacologic interventions. We found that the behavioral responses to obtain food in hungry rats were disrupted when the ILC was inhibited through muscimol infusion, blocking brain H1 histamine receptors by intracerebroventricular infusion of pyrilamine or by satiety. In contrast, the consummatory behavior was not affected by ILC inhibition. The extracellular histamine levels in the ILC were increased in direct correlation with the degree of motivation measured in the progressive ratio test. ILC inhibition also prevented this increase in histamine levels. The rise in extracellular histamine levels during the progressive ratio test was similar (ca. 200%) during the active or the resting period of the day. However, different basal levels are observed for these two periods. Our findings suggest that increased histamine levels during this behavior are not simply explained by the awaked state, but instead, there is a motivation-related release of histamine, suggestive of a specific form of brain activation. Serotonin (another critical component of the ascending arousal system) was also tested. Interestingly, changes in levels of this neuromodulator were not detected during the progressive ratio test. In conclusion, our results suggest that ILC activation and subsequent increase in brain histamine release are both necessary for the normal performance of a motivated behavior such as feeding.

The Psychopharmacology of Effort-Related Decision Making: Dopamine, Adenosine, and Insights into the Neurochemistry of Motivation

Effort-based decision making is studied using tasks that offer choices between high-effort options leading to more highly valued reinforcers versus low-effort/low-reward options. These tasks have been used to study the involvement of neural systems, including mesolimbic dopamine and related circuits, in effort-related aspects of motivation. Moreover, such tasks are useful as animal models of some of the motivational symptoms that are seen in people with depression, schizophrenia, Parkinson's disease, and other disorders. The present review will discuss the pharmacology of effort-related decision making and will focus on the use of these tasks for the development of drug treatments for motivational dysfunction. Research has identified pharmacological conditions that can alter effort-based choice and serve as models for depression-related symptoms (e.g., the vesicular monoamine transport-2 inhibitor tetrabenazine and proinflammatory cytokines). Furthermore, tests of effort-based choice have identified compounds that are particularly useful for stimulating high-effort work output and reversing the deficits induced by tetrabenazine and cytokines. These studies indicate that drugs that act by facilitating dopamine transmission, as well as adenosine A_2A antagonists, are relatively effective at reversing effort-related impairments. Studies of effort-based choice may lead to the identification of drug targets that could be useful for treating motivational treatments that are resistant to commonly used antidepressants such as serotonin transport inhibitors.

What Is the Relationship between Dopamine and Effort?

The trade-off between reward and effort is at the heart of most behavioral theories, from ecology to economics. Compared to reward, however, effort remains poorly understood, both at the behavioral and neurophysiological levels. This is important because unwillingness to overcome effort to gain reward is a common feature of many neuropsychiatric and neurological disorders. A recent surge in interest in the neurobiological basis of effort has led to seemingly conflicting results regarding the role of dopamine. We argue here that, upon closer examination, there is actually striking consensus across studies: dopamine primarily codes for future reward but is less sensitive to anticipated effort cost. This strong association between dopamine and the incentive effects of rewards places dopamine in a key position to promote reward-directed action.

Inhibiting Mesolimbic Dopamine Neurons Reduces the Initiation and Maintenance of Instrumental Responding

Mesolimbic dopamine perturbations modulate performance of reward-seeking behavior, with tasks requiring high effort being especially vulnerable to disruption of dopamine signaling. Previous work primarily investigated long-term perturbations such as receptor antagonism and dopamine depletion, which constrain the ability to assess dopamine contributions to effort expenditure in isolation from other behavior events, such as reward consumption. Also unclear is if dopamine is required for both initiation and maintenance when a sequence of multiple instrumental responses is required. Here we used optogenetic inhibition of midbrain TH+  neurons to probe the role of dopamine neuron activity during instrumental responding for reward by varying the time epoch of neural inhibition relative to the time of response initiation. Within a fixed-ratio procedure, requiring eight nosepoke responses per reinforcer delivery, or a progressive ratio (PR) procedure, in which within-session response requirements increased exponentially, inhibiting dopamine neurons while mice were engaged in response bouts decreased the probability of continued responding. If inhibition occurred during each attempted bout, the effect was to decrease total responses, and thus amount of rewards earned, over a session. In contrast, if inhibition was applied only during some bouts, mice increased the number of bouts initiated to earn control levels of reward. Inhibiting dopamine neurons while mice were not responding decreased the probability of initiating an instrumental response but had no effect on the amount of effort exerted over the entire session. We conclude that midbrain dopamine signaling promotes initiation of instrumental responding and maintains motivation to continue ongoing bouts of effortful responses.

Compromised Dopaminergic Encoding of Reward Accompanying Suppressed Willingness to Overcome High Effort Costs Is a Prominent Prodromal Characteristic of the Q175 Mouse Model of Huntington's Disease

Huntington's disease (HD) is a heritable neurodegenerative disorder caused by expansion of CAG (glutamine) repeats in the HTT gene. A prodromal stage characterized by psychiatric disturbances normally precedes primary motor symptoms and suppressed motivation represents one of the earliest and most common psychiatric symptoms. Although dopamine in the nucleus accumbens (NAc) critically regulates motivation and altered dopamine signaling is implicated in HD, the nature of dopaminergic deficits and contribution to symptoms in HD is poorly understood. We therefore tested whether altered NAc dopamine release accompanies motivational deficits in the Q175 knock-in HD mouse model. Q175 mice express a CAG expansion of the human mutant huntingtin allele in the native mouse genome and gradually manifest symptoms late in life, closely mimicking the genotypic context and disease progression in human HD. Sub-second extracellular dopamine release dynamics were monitored using fast-scan cyclic voltammetry, whereas motivation was assessed using a progressive ratio reinforcement schedule. As the response ratio (lever presses per reward) escalated, Q175 mice exerted less effort to earn fewer rewards versus wild-type (WT). Moreover, dopamine released at reward delivery dynamically encoded increasing reward cost in WT but not Q175 mice. Deficits were specific to situations of high effortful demand as no difference was observed in locomotion, free feeding, hedonic processing, or reward seeking when the response requirement was low. This compromised dopaminergic encoding of reward delivery coincident with suppressed motivation to work for reward in Q175 mice provides novel, neurobiological insight into an established and clinically relevant endophenotype of prodromal HD.

SIGNIFICANCE STATEMENT

Psychiatric impairments in Huntington's disease (HD) typically manifest early in disease progression, before motor deficits. However, the neurobiological factors contributing to psychiatric symptoms are poorly understood. We used a mouse HD model and assessed whether impaired dopamine release in the nucleus accumbens (NAc), a brain region critical to goal-directed behaviors, accompanies motivational deficits, one of the most common early HD symptoms. HD mice exhibited blunted motivation to work for food reward coincident with diminished dopamine release to reward receipt. Motivational and NAc dopaminergic deficits were not associated with gross motor deficits or impaired food seeking when effortful demands were low. This work identifies a specific prodromal HD phenotype associated with a prominent and previously unidentified neurobiological impairment.

Networks of VTA Neurons Encode Real-Time Information about Uncertain Numbers of Actions Executed to Earn a Reward

Multiple and unpredictable numbers of actions are often required to achieve a goal. In order to organize behavior and allocate effort so that optimal behavioral policies can be selected, it is necessary to continually monitor ongoing actions. Real-time processing of information related to actions and outcomes is typically assigned to the prefrontal cortex and basal ganglia, but also depends on midbrain regions, especially the ventral tegmental area (VTA). We were interested in how individual VTA neurons, as well as networks within the VTA, encode salient events when an unpredictable number of serial actions are required to obtain a reward. We recorded from ensembles of putative dopamine and non-dopamine neurons in the VTA as animals performed multiple cued trials in a recording session where, in each trial, serial actions were randomly rewarded. While averaging population activity did not reveal a response pattern, we observed that different neurons were selectively tuned to low, medium, or high numbered actions in a trial. This preferential tuning of putative dopamine and non-dopamine VTA neurons to different subsets of actions in a trial allowed information about binned action number to be decoded from the ensemble activity. At the network level, tuning curve similarity was positively associated with action-evoked noise correlations, suggesting that action number selectivity reflects functional connectivity within these networks. Analysis of phasic responses to cue and reward revealed that the requirement to execute multiple and uncertain numbers of actions weakens both cue-evoked responses and cue-reward response correlation. The functional connectivity and ensemble coding scheme that we observe here may allow VTA neurons to cooperatively provide a real-time account of ongoing behavior. These computations may be critical to cognitive and motivational functions that have long been associated with VTA dopamine neurons.

Implication of dorsostriatal D3 receptors in motivational processes: a potential target for neuropsychiatric symptoms in Parkinson's disease

Beyond classical motor symptoms, motivational and affective deficits are frequently observed in Parkinson’s disease (PD), dramatically impairing the quality of life of patients. Using bilateral 6-hydroxydopamine (6-OHDA) lesions of the substantia nigra pars compacta (SNc) in rats, we have been able to reproduce these neuropsychiatric/non-motor impairments. The present study describes how bilateral 6-OHDA SNc lesions affect the function of the main striatal dopaminergic (DA) receptor subtypes. Autoradiography was used to measure the levels of striatal DA receptors, and operant sucrose self-administration and neuropharmacological approaches were combined to investigate the causal implication of specific DA receptors subtypes in the motivational deficits induced by a dorsostriatal DA denervation. We found that D3 receptors (D₃R) exclusively are down-regulated within the dorsal striatum of lesioned rats. We next showed that infusion of a D₃R antagonist (SB-277011A) in non-lesioned animals specifically disrupts preparatory, but not consummatory behaviors. Our findings reveal an unexpected involvement of dorsostriatal D₃R in motivational processes. They strongly suggest an implication of dorsostriatal D₃R in the neuropsychiatric symptoms observed in PD, highlighting this receptor as a potential target for pharmacological treatment.

Insights About Striatal Circuit Function and Schizophrenia From a Mouse Model of Dopamine D2 Receptor Upregulation

The dopamine hypothesis of schizophrenia is supported by a large number of imaging studies that have identified an increase in dopamine binding at the D₂ receptor selectively in the striatum. We review a decade of work using a regionally restricted and temporally regulated transgenic mouse model to investigate the behavioral, molecular, electrophysiological, and anatomical consequences of selective D₂ receptor upregulation in the striatum. These studies have identified new and potentially important biomarkers at the circuit and molecular level that can now be explored in patients with schizophrenia. They provide an example of how animal models and their detailed level of neurobiological analysis allow a deepening of our understanding of the relationship between neuronal circuit function and symptoms of schizophrenia, and as a consequence generate new hypotheses that are testable in patients.

Activational and effort-related aspects of motivation: neural mechanisms and implications for psychopathology

Motivation has been defined as the process that allows organisms to regulate their internal and external environment, and control the probability, proximity and availability of stimuli. As such, motivation is a complex process that is critical for survival, which involves multiple behavioural functions mediated by a number of interacting neural circuits. Classical theories of motivation suggest that there are both directional and activational aspects of motivation, and activational aspects (i.e. speed and vigour of both the instigation and persistence of behaviour) are critical for enabling organisms to overcome work-related obstacles or constraints that separate them from significant stimuli. The present review discusses the role of brain dopamine and related circuits in behavioural activation, exertion of effort in instrumental behaviour, and effort-related decision-making, based upon both animal and human studies. Impairments in behavioural activation and effort-related aspects of motivation are associated with psychiatric symptoms such as anergia, fatigue, lassitude and psychomotor retardation, which cross multiple pathologies, including depression, schizophrenia, and Parkinson's disease. Therefore, this review also attempts to provide an interdisciplinary approach that integrates findings from basic behavioural neuroscience, behavioural economics, clinical neuropsychology, psychiatry, and neurology, to provide a coherent framework for future research and theory in this critical field. Although dopamine systems are a critical part of the brain circuitry regulating behavioural activation, exertion of effort, and effort-related decision-making, mesolimbic dopamine is only one part of a distributed circuitry that includes multiple neurotransmitters and brain areas. Overall, there is a striking similarity between the brain areas involved in behavioural activation and effort-related processes in rodents and in humans. Animal models of effort-related decision-making are highly translatable to humans, and an emerging body of evidence indicates that alterations in effort-based decision-making are evident in several psychiatric and neurological disorders. People with major depression, schizophrenia, and Parkinson's disease show evidence of decision-making biases towards a lower exertion of effort. Translational studies linking research with animal models, human volunteers, and clinical populations are greatly expanding our knowledge about the neural basis of effort-related motivational dysfunction, and it is hoped that this research will ultimately lead to improved treatment for motivational and psychomotor symptoms in psychiatry and neurology.

Adenosine 2A receptors modulate reward behaviours for methamphetamine

Addiction to methamphetamine (METH) is a global health problem for which there are no approved pharmacotherapies. The adenosine 2A (A2 A ) receptor presents a potential therapeutic target for METH abuse due to its modulatory effects on striatal dopamine and glutamate transmission. Notably, A2 A receptor signalling has been implicated in the rewarding effects of alcohol, cocaine and opiates; yet, the role of this receptor in METH consumption and seeking is essentially unknown. Therefore, the current study used A2 A knockout (KO) mice to assess the role of A2 A in behaviours relevant to METH addiction. METH conditioned place preference was absent in A2 A KO mice compared with wild-type (WT) littermates. Repeated METH treatment produced locomotor sensitization in both genotypes; however, sensitization was attenuated in A2 A KO mice in a dose-related manner. METH intravenous self-administration was intact in A2 A KO mice over a range of doses and schedules of reinforcement. However, the motivation to self-administer was reduced in A2 A KO mice. Regression analysis further supported the observation that the motivation to self-administer METH was reduced in A2 A KO mice even when self-administration was similar to WT mice. Sucrose self-administration was also reduced in A2 A KO mice but only at higher schedules of reinforcement. Collectively, these data suggest that A2 A signalling is critically required to integrate rewarding and motivational properties of both METH and natural rewards.

The pharmacology of effort-related choice behavior: Dopamine, depression, and individual differences

This review paper is focused upon the involvement of mesolimbic dopamine (DA) and related brain systems in effort-based processes. Interference with DA transmission affects instrumental behavior in a manner that interacts with the response requirements of the task, such that rats with impaired DA transmission show a heightened sensitivity to ratio requirements. Impaired DA transmission also affects effort-related choice behavior, which is assessed by tasks that offer a choice between a preferred reinforcer that has a high work requirement vs. less preferred reinforcer that can be obtained with minimal effort. Rats and mice with impaired DA transmission reallocate instrumental behavior away from food-reinforced tasks with high response costs, and show increased selection of low reinforcement/low cost options. Tests of effort-related choice have been developed into models of pathological symptoms of motivation that are seen in disorders such as depression and schizophrenia. These models are being employed to explore the effects of conditions associated with various psychopathologies, and to assess drugs for their potential utility as treatments for effort-related symptoms. Studies of the pharmacology of effort-based choice may contribute to the development of treatments for symptoms such as psychomotor slowing, fatigue or anergia, which are seen in depression and other disorders.

Neurobiological basis of motivational deficits in psychopathology

In recent years, there has been increasing emphasis on the importance of motivational symptoms in depression, schizophrenia and other disorders. The present review discusses the conceptual background related to the construct of motivation, and provides a framework that for research on both physiological and pathological aspects of motivation. Particular emphasis is placed on what is known about the neurobiological basis of activational aspects of motivation, including studies from animal models. The role of limbic/prefrontal/striatal circuitry in behavioral activation and effort-related functions is examined, and the utility of behavioral tasks of effort-based decision making as models of motivational symptoms is discussed. We also review the neurobiology of motivational symptoms in relation to psychopathology, and issues related to the language used to characterize motivational dysfunctions are considered. The literature suggests that research on the neurobiology of motivational dysfunction in psychopathology, at both clinical and preclinical levels, could inform the development of novel and more effective treatments for a range of CNS disorders.

Behavioral modeling of human choices reveals dissociable effects of physical effort and temporal delay on reward devaluation

There has been considerable interest from the fields of biology, economics, psychology, and ecology about how decision costs decrease the value of rewarding outcomes. For example, formal descriptions of how reward value changes with increasing temporal delays allow for quantifying individual decision preferences, as in animal species populating different habitats, or normal and clinical human populations. Strikingly, it remains largely unclear how humans evaluate rewards when these are tied to energetic costs, despite the surge of interest in the neural basis of effort-guided decision-making and the prevalence of disorders showing a diminished willingness to exert effort (e.g., depression). One common assumption is that effort discounts reward in a similar way to delay. Here we challenge this assumption by formally comparing competing hypotheses about effort and delay discounting. We used a design specifically optimized to compare discounting behavior for both effort and delay over a wide range of decision costs (Experiment 1). We then additionally characterized the profile of effort discounting free of model assumptions (Experiment 2). Contrary to previous reports, in both experiments effort costs devalued reward in a manner opposite to delay, with small devaluations for lower efforts, and progressively larger devaluations for higher effort-levels (concave shape). Bayesian model comparison confirmed that delay-choices were best predicted by a hyperbolic model, with the largest reward devaluations occurring at shorter delays. In contrast, an altogether different relationship was observed for effort-choices, which were best described by a model of inverse sigmoidal shape that is initially concave. Our results provide a novel characterization of human effort discounting behavior and its first dissociation from delay discounting. This enables accurate modelling of cost-benefit decisions, a prerequisite for the investigation of the neural underpinnings of effort-guided choice and for understanding the deficits in clinical disorders characterized by behavioral inactivity.

One of the main functions of the brain is to select sequences of actions that lead to rewarding outcomes (e.g., food). However, such rewards are often not readily available; instead physical effort may be required to obtain them, or their arrival may be delayed. The ability to integrate the costs and benefits of potential courses of action is severely impaired in several common disorders, such as depression and schizophrenia. Mathematical models can describe how individuals depreciate rewards based on the costs associated with them. For example, models of how a reward loses appeal with increasing temporal delays can provide individual impulsivity scores, and can serve as a predictor of financial mismanagement. To date, there is no established model to describe accurately how humans depreciate rewards when obtaining them requires physical effort. This is surprising given the prevalence of disorders related to a diminished willingness to exert effort. Here we derive a biologically plausible mathematical model that can describe how healthy humans make decisions tied to physical efforts. We show that effort and delay influence reward valuation in different ways, contrary to common assumptions. Our model will be important for characterizing decision-making deficits in clinical disorders characterized by behavioral inactivity.

Neurons in the nucleus accumbens promote selection bias for nearer objects

Both animals and humans often prefer rewarding options that are nearby over those that are distant, but the neural mechanisms underlying this bias are unclear. Here we present evidence that a proximity signal encoded by neurons in the nucleus accumbens drives proximate reward bias by promoting impulsive approach to nearby reward-associated objects. On a novel decision-making task, rats chose the nearer option even when it resulted in greater effort expenditure and delay to reward; therefore, proximate reward bias was unlikely to be caused by effort or delay discounting. The activity of individual neurons in the nucleus accumbens did not consistently encode the reward or effort associated with specific alternatives, suggesting that it does not participate in weighing the values of options. In contrast, proximity encoding was consistent and did not depend on the subsequent choice, implying that accumbens activity drives approach to the nearest rewarding option regardless of its specific associated reward size or effort level.

Mesolimbic Dopamine and the Regulation of Motivated Behavior

It has been known for some time that nucleus accumbens dopamine (DA) is involved in aspects of motivation , but theoretical approaches to understanding the functions of DA have continued to evolve based upon emerging data and novel concepts. Although it has become traditional to label DA neurons as "reward" neurons, the actual findings are more complicated than that, because they indicate that DA neurons can respond to a variety of motivationally significant stimuli. Moreover, it is important to distinguish between aspects of motivation that are differentially affected by dopaminergic manipulations. Studies that involve nucleus accumbens DA antagonism or depletion indicate that accumbens DA does not mediate primary food motivation or appetite. Nevertheless, DA is involved in appetitive and aversive motivational processes including behavioral activation , exertion of effort, sustained task engagement, and Pavlovian-to-instrumental transfer. Interference with accumbens DA transmission affects instrumental behavior in a manner that interacts with the response requirements of the task and also shifts effort-related choice behavior, biasing animals toward low-effort alternatives. Dysfunctions of mesolimbic DA may contribute to motivational symptoms seen in various psychopathologies, including depression , schizophrenia, parkinsonism, and other disorders.

Nucleus accumbens core and shell inactivation differentially affects impulsive behaviours in rats

Impulsivity is a multifactorial phenomenon, determined by deficits in decision-making (impulsive choice) and impulse control (impulsive action). Recent findings indicate that impulsive behaviour is not only top-down controlled by cortical areas, but also modulated at subcortical level. The nucleus accumbens (NAc) might be a key substrate in cortico-limbic-striatal circuits involved in impulsive behaviour. Dissociable effects of the NAc subregions in various behavioural paradigms point to a potential functional distinction between NAc core and shell concerning different types of impulsivity. The present study used reversible inactivation of the rats' NAc core and shell via bilateral microinfusion of the GABAA receptor agonist muscimol (0.05μg/0.3μl) and fluorophore-conjugated muscimol (FCM, 0.27μg/0.3μl) in order to study their contribution to different aspects of impulse control in a 5-choice serial reaction time task (5-CSRTT) and impulsive choice in a delay-based decision-making T-maze task. Acute inactivation of NAc core as well as shell by muscimol increased impulsive choice, with higher impairments of the rats' waiting capacity in the T-maze following core injections compared to shell. Intra-NAc shell infusion of muscimol also induced specific impulse control deficits in the 5-CSRTT, while deactivation of the core caused severe general impairments in task performance. FCM did not affect animal behaviour. Our findings reveal clear involvement of NAc shell in both forms of impulsivity. Both subareas play a key role in the regulation of impulsive decision-making, but show functional dichotomy regarding impulse control with the core being more implicated in motivational and motor aspects.

Lesions of the nucleus accumbens core modulate development of matching behavior

Background

The development of choice is a crucial determinant in the performance of appetitive responses. Given two options with different reinforcement rates, animals match their relative rate of responding to the relative rates of reinforcement (i.e., matching behavior). A previous study has shown that the nucleus accumbens core (AcbC) is involved in the performance of matching behavior in trained animals. However, the role of the AcbC in the acquisition of matching behavior has not been addressed.

Results

We conducted a series of experimental sessions to examine the role of the AcbC on the development of matching behavior. Instrumental responding was measured in rats with excitotoxic lesions of the AcbC. Rats were given two options that differed in the relative rate of reinforcement under concurrent variable-interval schedules. The locations of the more frequently reinforced option and the alternative option were randomly switched between sessions. Lesions of the AcbC accelerated the development of matching behavior compared to the sham-operated group. The AcbC-lesioned rats exhibited closer conformity to the matching law than shams when the options were in the same positions as in the previous session (the same condition), but not when the option locations had been switched (the different condition). The AcbC rats showed smaller probabilities of switching behavior between alternatives than shams. Post-reinforcement pausing was not affected by the AcbC lesion. Neither numbers of rewards obtained nor number of lever presses were different between the AcbC-lesioned rats and shams over session blocks.

Conclusions

Our results suggest that the AcbC plays a regulatory role in the development of matching behavior through switching probabilities rather than perception of reward magnitude. The differential effect of AcbC lesions on the matching behavior between the same and different conditions suggests influence of the spontaneous recovery, that is, reversion to a previously reinforced choice at the beginning of the next session, on the development of matching behavior in the AcbC-lesioned rats.

Measuring reinforcement learning and motivation constructs in experimental animals: relevance to the negative symptoms of schizophrenia

The present review article summarizes and expands upon the discussions that were initiated during a meeting of the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS; http://cntrics.ucdavis.edu) meeting. A major goal of the CNTRICS meeting was to identify experimental procedures and measures that can be used in laboratory animals to assess psychological constructs that are related to the psychopathology of schizophrenia. The issues discussed in this review reflect the deliberations of the Motivation Working Group of the CNTRICS meeting, which included most of the authors of this article as well as additional participants. After receiving task nominations from the general research community, this working group was asked to identify experimental procedures in laboratory animals that can assess aspects of reinforcement learning and motivation that may be relevant for research on the negative symptoms of schizophrenia, as well as other disorders characterized by deficits in reinforcement learning and motivation. The tasks described here that assess reinforcement learning are the Autoshaping Task, Probabilistic Reward Learning Tasks, and the Response Bias Probabilistic Reward Task. The tasks described here that assess motivation are Outcome Devaluation and Contingency Degradation Tasks and Effort-Based Tasks. In addition to describing such methods and procedures, the present article provides a working vocabulary for research and theory in this field, as well as an industry perspective about how such tasks may be used in drug discovery. It is hoped that this review can aid investigators who are conducting research in this complex area, promote translational studies by highlighting shared research goals and fostering a common vocabulary across basic and clinical fields, and facilitate the development of medications for the treatment of symptoms mediated by reinforcement learning and motivational deficits.

Dopamine modulates reward-related vigor

Subjects routinely control the vigor with which they emit motoric responses. However, the bulk of formal treatments of decision-making ignores this dimension of choice. A recent theoretical study suggested that action vigor should be influenced by experienced average reward rate and that this rate is encoded by tonic dopamine in the brain. We previously examined how average reward rate modulates vigor as exemplified by response times and found a measure of agreement with the first suggestion. In the current study, we examined the second suggestion, namely the potential influence of dopamine signaling on vigor. Ninety healthy subjects participated in a double-blind experiment in which they received one of the following: placebo, L-DOPA (which increases dopamine levels in the brain), or citalopram (which has a selective, if complex, effect on serotonin levels). Subjects performed multiple trials of a rewarded odd-ball discrimination task in which we varied the potential reward over time in order to exercise the putative link between vigor and average reward rate. Replicating our previous findings, we found that a significant fraction of the variance in subjects' responses could be explained by our experimentally manipulated changes in average reward rate. Crucially, this relationship was significantly stronger under L-Dopa than under Placebo, suggesting that the impact of average reward levels on action vigor is indeed subject to a dopaminergic influence.

Nucleus accumbens neurotransmission and effort-related choice behavior in food motivation: effects of drugs acting on dopamine, adenosine, and muscarinic acetylcholine receptors

Mesolimbic dopamine (DA) is a critical component of the brain circuitry regulating behavioral activation and effort-related processes. Although nucleus accumbens (NAc) DA depletions or antagonism leave aspects of appetite and primary food motivation intact, rats with impaired DA transmission reallocate their instrumental behavior away from food-reinforced tasks with high response requirements, and instead select less effortful food-seeking behaviors. Previous work showed that adenosine A2A antagonists can reverse the effects of DA D2 antagonists on effort-related choice, and that stimulation of adenosine A2A receptors produces behavioral effects that are similar to those induced by DA antagonism. The present review summarizes the literature on the role of NAc DA and adenosine in effort-related processes, and also presents original data on the effects of local stimulation of muscarinic acetylcholine receptors in NAc core. Local injections of the muscarinic agonist pilocarpine directly into NAc core produces shifts in effort-related choice behavior similar to those induced by DA antagonism or A2A receptor stimulation, decreasing lever pressing but increasing chow intake in rats responding on a concurrent fixed ratio/chow feeding choice task. In contrast, injections into a neostriatal control site dorsal to the NAc were ineffective. The actions of pilocarpine on this task were attenuated by co-administration of the muscarinic antagonist scopolamine. Thus, drugs that act on DA, adenosine A2A, and muscarinic receptors regulate effort-related choice behavior, which may have implications for the treatment of psychiatric symptoms such as psychomotor slowing, fatigue or anergia that can be observed in depression and other disorders.

The mysterious motivational functions of mesolimbic dopamine

Nucleus accumbens dopamine is known to play a role in motivational processes, and dysfunctions of mesolimbic dopamine may contribute to motivational symptoms of depression and other disorders, as well as features of substance abuse. Although it has become traditional to label dopamine neurons as "reward" neurons, this is an overgeneralization, and it is important to distinguish between aspects of motivation that are differentially affected by dopaminergic manipulations. For example, accumbens dopamine does not mediate primary food motivation or appetite, but is involved in appetitive and aversive motivational processes including behavioral activation, exertion of effort, approach behavior, sustained task engagement, Pavlovian processes, and instrumental learning. In this review, we discuss the complex roles of dopamine in behavioral functions related to motivation.

Relative response cost determines the sensitivity of instrumental reward seeking to dopamine receptor blockade

Dopamine is a critical mediator of instrumental reward seeking behavior and appears to have a particularly important role in motivating actions that require considerable effort. As with rewards, response costs can be evaluated in both absolute and relative terms. The current study investigated whether the extent to which instrumental performance is dependent on dopamine transmission is influenced by relative or absolute response cost. Three groups of rats were rewarded for lever pressing on different fixed ratio (FR) schedules that required 1 (FR-1), 10 (FR-10), or 20 (FR-20) presses for each food reward. Rats were then injected systemically with flupentixol, a dopamine receptor antagonist, or vehicle before testing all groups on an intermediate-cost (FR-10) schedule, such that only the relative cost of responding differed across groups. Rats experiencing an upshift in cost (group FR-1/FR-10) showed greater response suppression following flupentixol administration than rats experiencing no shift in cost (group FR-10/FR-10), whereas flupentixol treatment had no effect on rats experiencing a downshift in cost (group FR-20/FR-10). A second round of flupentixol tests was conducted using the rats' maintenance schedules, such that only absolute response costs differed across groups. Here, the pattern was reversed among the groups, in line with previous reports. Specifically, flupentixol had a stronger suppressive effect in group FR-20/FR-20 than in group FR-10/FR-10, and had no detectable effect in group FR-1/FR-1. These findings suggest that response costs are evaluated in both absolute and relative terms and that dopamine has a role in overcoming both kinds of cost.

Dopaminergic modulation of effort-related choice behavior as assessed by a progressive ratio chow feeding choice task: pharmacological studies and the role of individual differences

Mesolimbic dopamine (DA) is involved in behavioral activation and effort-related processes. Rats with impaired DA transmission reallocate their instrumental behavior away from food-reinforced tasks with high response requirements, and instead select less effortful food-seeking behaviors. In the present study, the effects of several drug treatments were assessed using a progressive ratio (PROG)/chow feeding concurrent choice task. With this task, rats can lever press on a PROG schedule reinforced by a preferred high-carbohydrate food pellet, or alternatively approach and consume the less-preferred but concurrently available laboratory chow. Rats pass through each ratio level 15 times, after which the ratio requirement is incremented by one additional response. The DA D₂ antagonist haloperidol (0.025–0.1 mg/kg) reduced number of lever presses and highest ratio achieved but did not reduce chow intake. In contrast, the adenosine A_2A antagonist MSX-3 increased lever presses and highest ratio achieved, but decreased chow consumption. The cannabinoid CB1 inverse agonist and putative appetite suppressant AM251 decreased lever presses, highest ratio achieved, and chow intake; this effect was similar to that produced by pre-feeding. Furthermore, DA-related signal transduction activity (pDARPP-32(Thr34) expression) was greater in nucleus accumbens core of high responders (rats with high lever pressing output) compared to low responders. Thus, the effects of DA antagonism differed greatly from those produced by pre-feeding or reduced CB1 transmission, and it appears unlikely that haloperidol reduces PROG responding because of a general reduction in primary food motivation or the unconditioned reinforcing properties of food. Furthermore, accumbens core signal transduction activity is related to individual differences in work output.

The behavioral pharmacology of effort-related choice behavior: dopamine, adenosine and beyond

For many years, it has been suggested that drugs that interfere with dopamine (DA) transmission alter the "rewarding" impact of primary reinforcers such as food. Research and theory related to the functions of mesolimbic DA are undergoing a substantial conceptual restructuring, with the traditional emphasis on hedonia and primary reward yielding to other concepts and lines of inquiry. The present review is focused upon the involvement of nucleus accumbens DA in effort-related choice behavior. Viewed from the framework of behavioral economics, the effects of accumbens DA depletions and antagonism on food-reinforced behavior are highly dependent upon the work requirements of the instrumental task, and DA-depleted rats show a heightened sensitivity to response costs, especially ratio requirements. Moreover, interference with accumbens DA transmission exerts a powerful influence over effort-related choice behavior. Rats with accumbens DA depletions or antagonism reallocate their instrumental behavior away from food-reinforced tasks that have high response requirements, and show increased selection of low reinforcement/low cost options. Nucleus accumbens DA and adenosine interact in the regulation of effort-related functions, and other brain structures (anterior cingulate cortex, amygdala, ventral pallidum) also are involved. Studies of the brain systems regulating effort-based processes may have implications for understanding drug abuse, as well as symptoms such as psychomotor slowing, fatigue or anergia in depression and other neurological disorders.

Food reward in the obese and after weight loss induced by calorie restriction and bariatric surgery

Increased availability of tasty, energy-dense foods has been blamed as a major factor in the alarmingly high prevalence of obesity, diabetes, and metabolic disease, even in young age. A heated debate has started as to whether some of these foods should be considered addictive, similar to drugs and alcohol. One of the main arguments for food addiction is the similarity of the neural mechanisms underlying reward generation by foods and drugs. Here, we will discuss how food intake can generate reward and how behavioral and neural reward functions are different in obese subjects. Because most studies simply compare lean and obese subjects, it is not clear whether predisposing differences in reward functions cause overeating and weight gain, or whether repeated exposure or secondary effects of the obese state alter reward functions. While studies in both rodents and humans demonstrate preexisting differences in reward functions in the obese, studies in rodent models using calorie restriction and gastric bypass surgery show that some differences are reversible by weight loss and are therefore secondary to the obese state.

Regulator of calmodulin signaling knockout mice display anxiety-like behavior and motivational deficits

Regulator of calmodulin (CaM) signaling (RCS), when phosphorylated by protein kinase A (PKA) on Ser55, binds to CaM and inhibits CaM-dependent signaling. RCS expression is high in the dorsal striatum, nucleus accumbens and amygdala, suggesting that the protein is involved in limbic-striatal function. To test this hypothesis, we examined RCS knockout (KO) mice in behavioral models dependent on these brain areas. Mice were tested for food-reinforced instrumental conditioning and responding under a progressive ratio (PR) schedule of reinforcement and in models of anxiety (elevated plus maze and open field). While RCS KO mice showed normal acquisition of a food-motivated instrumental response, they exhibited a lower breakpoint value when tested on responding under a PR schedule of reinforcement. RCS KO mice also displayed decreased exploration in both the open arms of an elevated plus maze and in the center region of an open field, suggesting an enhanced anxiety response. Biochemical studies revealed a reduction in the levels of dopamine and cAMP-regulated phosphoprotein (DARPP-32) in the striatum of RCS KO mice. DARPP-32 is important in reward-mediated behavior, suggestive of a possible role for DARPP-32 in mediating some of the effects of RCS. Together these results implicate a novel PKA-regulated phosphoprotein, RCS, in the etiology of motivational deficits and anxiety.

Poor impulse control predicts inelastic demand for nicotine but not alcohol in rats

Tobacco and alcohol dependence are characterized by continued use despite deleterious health, social and occupational consequences, implying that addicted individuals pay a high price for their use. In behavioral economic terms, such persistent consumption despite increased costs can be conceptualized as inelastic demand. Recent animal studies demonstrated that high-impulsive individuals are more willing to work for nicotine or cocaine infusions than their low-impulsive counterparts, indicating that this trait might be causally related to inelastic drug demand. By employing progressive ratio schedules of reinforcement combined with a behavioral economics approach of analysis, we determined whether trait impulsivity is associated with an insensitivity of nicotine or alcohol consumption to price increments. Rats were trained on a delayed discounting task, measuring impulsive choice. Hereafter, high- and low-impulsive rats were selected and trained to nose poke for intravenous nicotine or oral alcohol. Upon stable self-administration on a continuous reinforcement schedule, the price (i.e. response requirement) was increased. Demand curves, depicting the relationship between price and consumption, were produced using Hursh's exponential demand equation. Similar to human observations, nicotine and alcohol consumption in rats fitted this equation, thereby demonstrating the validity of our model. Moreover, high-impulsive rats displayed inelastic nicotine demand, as their nicotine consumption was less sensitive to price increments as compared with that in low-impulsive rats. Impulsive choice was not related to differences in alcohol demand elasticity. Our model seems well suited for studying nicotine and alcohol demand in rats and, as such, might contribute to our understanding of tobacco and alcohol dependence.

Speed/accuracy trade-off between the habitual and the goal-directed processes

Instrumental responses are hypothesized to be of two kinds: habitual and goal-directed, mediated by the sensorimotor and the associative cortico-basal ganglia circuits, respectively. The existence of the two heterogeneous associative learning mechanisms can be hypothesized to arise from the comparative advantages that they have at different stages of learning. In this paper, we assume that the goal-directed system is behaviourally flexible, but slow in choice selection. The habitual system, in contrast, is fast in responding, but inflexible in adapting its behavioural strategy to new conditions. Based on these assumptions and using the computational theory of reinforcement learning, we propose a normative model for arbitration between the two processes that makes an approximately optimal balance between search-time and accuracy in decision making. Behaviourally, the model can explain experimental evidence on behavioural sensitivity to outcome at the early stages of learning, but insensitivity at the later stages. It also explains that when two choices with equal incentive values are available concurrently, the behaviour remains outcome-sensitive, even after extensive training. Moreover, the model can explain choice reaction time variations during the course of learning, as well as the experimental observation that as the number of choices increases, the reaction time also increases. Neurobiologically, by assuming that phasic and tonic activities of midbrain dopamine neurons carry the reward prediction error and the average reward signals used by the model, respectively, the model predicts that whereas phasic dopamine indirectly affects behaviour through reinforcing stimulus-response associations, tonic dopamine can directly affect behaviour through manipulating the competition between the habitual and the goal-directed systems and thus, affect reaction time.

When confronted with different alternatives, animals can respond either based on their pre-established habits, or by considering the short- and long-term consequences of each option. Whereas habitual decision making is fast, goal-directed thinking is a time-consuming task. Instead, habits are inflexible after being consolidated, but goal-directed decision making can rapidly adapt the animal's strategy after a change in environmental conditions. Based on these features of the two decision making systems, we suggest a computational model using the reinforcement learning framework, that makes a balance between the speed of decision making and behavioural flexibility. The behaviour of the model is consistent with the observation that at the early stages of learning, animals behave in a goal-directed way (flexible, but slow), but after extensive learning, their responses become habitual (inflexible, but fast). Moreover, the model explains that the animal's reaction time must decrease through the course of learning, as the habitual system takes control over behaviour. The model also attributes a functional role to the tonic activity of dopamine neurons in balancing the competition between the habitual and the goal-directed systems.

The flexible approach hypothesis: unification of effort and cue-responding hypotheses for the role of nucleus accumbens dopamine in the activation of reward-seeking behavior

Dopamine released in the nucleus accumbens is thought to contribute to the decision to exert effort to seek reward. This hypothesis is supported by findings that performance of tasks requiring higher levels of effort is more susceptible to disruption by manipulations that reduce accumbens dopamine function than tasks that require less effort. However, performance of some low-effort cue-responding tasks is highly dependent on accumbens dopamine. To reconcile these disparate results, we made detailed behavioral observations of rats performing various operant tasks and determined how injection of dopamine receptor antagonists into the accumbens influenced specific aspects of the animals' behavior. Strikingly, once animals began a chain of operant responses, the antagonists did not affect the ability to continue the chain until reward delivery. Instead, when rats left the operandum, the antagonists severely impaired the ability to return. We show that this impairment is specific to situations in which the animal must determine a new set of approach actions on each approach occasion; this behavior is called "flexible approach." Both high-effort operant tasks and some low-effort cue-responding tasks require dopamine receptor activation in the accumbens because animals pause their responding and explore the chamber, and accumbens dopamine is required to terminate these pauses with flexible approach to the operandum. The flexible approach hypothesis provides a unified framework for understanding the contribution of the accumbens and its dopamine projection to reward-seeking behavior.

Nucleus accumbens neurons encode predicted and ongoing reward costs in rats

Efficient decision-making requires that animals consider both the benefits and the costs of potential actions, such as the amount of effort or temporal delay involved in reward seeking. The nucleus accumbens (NAc) has been implicated in the ability to choose between options with different costs and overcome high costs when necessary, but it is not clear how NAc processing contributes to this role. Here, neuronal activity in the rat NAc was monitored using multi-neuron electrophysiology during two cost-based decision tasks in which either reward effort or reward delay was manipulated. In each task, distinct visual cues predicted high-value (low effort/immediate) and low-value (high effort/delayed) rewards. After training, animals exhibited a behavioral preference for high-value rewards, yet overcame high costs when necessary to obtain rewards. Electrophysiological analysis indicated that a subgroup of NAc neurons exhibited phasic increases in firing rate during cue presentations. In the effort-based decision task (but not the delay-based task), this population reflected the cost-discounted value of the future response. In contrast, other subgroups of cells were activated during response initiation or reward delivery, but activity did not differ on the basis of reward cost. Finally, another population of cells exhibited sustained changes in firing rate while animals completed high-effort requirements or waited for delayed rewards. These findings are consistent with previous reports that implicate NAc function in reward prediction and behavioral allocation during reward-seeking behavior, and suggest a mechanism by which NAc activity contributes to both cost-based decisions and actual cost expenditure.

Tonic dopamine modulates exploitation of reward learning

The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a semi-naturalistic homecage environment. In this “closed economy” paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.

Lever pressing responses under a fixed-ratio schedule of mice with 6-hydroxydopamine-induced dopamine depletion in the nucleus accumbens

In order to investigate the relationship between dopamine transmission in the nucleus accumbens and operant behavior in mice, mice with 6-hydroxydopamine (6-OHDA)-induced dopamine depletion in the nucleus accumbens were tested for their performance in lever pressing tasks under FR schedules with 8 ratios from FR5 to FR120. The mice were given one 20-mg food pellet per completed FR schedule in FR5, FR10, and FR20; they were given 2 pellets in FR40, and one more cumulatively in the rest of the schedules. Before the 6-OHDA injection surgery, all mice were trained to press a lever under all FR schedules. Then, 6-OHDA or ascorbate was injected into the nucleus accumbens. Postoperatively, the mice were tested under each FR schedule, with 3 sessions per schedule. 6-OHDA-treated mice exhibited an increase in lever pressing latency, i.e., the time interval between the last presentation of the reward and the next lever press, and a decrease in inter-response intervals, i.e., the time interval between 2 lever presses excluding lever pressing latency, irrespective of the FR ratios. Furthermore, in these 6-OHDA-treated mice, the number of lever presses during the first 300s of the session decreased under FR schedules with low ratios (5, 10, and 20). Open field activity, food motivation, and the amount of food consumed were not affected by dopamine depletion in the nucleus accumbens. These results suggest that the dopamine system in the nucleus accumbens had an important role in the control of lever pressing latency and inter-response intervals under FR reinforcement schedules.

Reconsidering anhedonia in depression: lessons from translational neuroscience

Anhedonia is a core symptom of major depressive disorder (MDD), the neurobiological mechanisms of which remain poorly understood. Despite decades of speculation regarding the role of dopamine (DA) in anhedonic symptoms, empirical evidence has remained elusive, with frequent reports of contradictory findings. In the present review, we argue that this has resulted from an underspecified definition of anhedonia, which has failed to dissociate between consummatory and motivational aspects of reward behavior. Given substantial preclinical evidence that DA is involved primarily in motivational aspects of reward, we suggest that a refined definition of anhedonia that distinguishes between deficits in pleasure and motivation is essential for the purposes of identifying its neurobiological substrates. Moreover, bridging the gap between preclinical and clinical models of anhedonia may require moving away from the conceptualization of anhedonia as a steady-state, mood-like phenomena. Consequently, we introduce the term "decisional anhedonia" to address the influence of anhedonia on reward decision-making. These proposed modifications to the theoretical definition of anhedonia have implications for research, assessment and treatment of MDD.

Differential effects of selective adenosine antagonists on the effort-related impairments induced by dopamine D1 and D2 antagonism

Mesolimbic dopamine (DA) is a critical component of the brain circuitry regulating behavioral activation and effort-related processes. Rats with impaired DA transmission reallocate their instrumental behavior away from food-reinforced tasks with high response requirements, and instead select less effortful food-seeking behaviors. Previous work showed that adenosine A(2A) antagonists can reverse the effects of DA D(2) antagonists on effort-related choice. However, less is known about the effects of adenosine A(1) antagonists. Despite anatomical data showing that A(1) and D(1) receptors are co-localized on the same striatal neurons, it is uncertain if A(1) antagonists can reverse the effects DA D(1) antagonists. The present work systematically compared the ability of adenosine A(1) and A(2A) receptor antagonists to reverse the effects of DA D(1) and D(2) antagonists on a concurrent lever pressing/feeding choice task. With this procedure, rats can choose between responding on a fixed ratio 5 lever-pressing schedule for a highly preferred food (i.e. high carbohydrate pellets) vs. approaching and consuming a less preferred rodent chow. The D(1) antagonist ecopipam (0.2 mg/kg i.p.) and the D(2) antagonist eticlopride (0.08 mg/kg i.p.) altered choice behavior, reducing lever pressing and increasing lab chow intake. Co-administration of the adenosine A(1) receptor antagonists 8-cyclopentyl-1,3-dipropylxanthine (DPCPX; 0.375, 0.75, and 1.5 mg/kg i.p.), and 8-cyclopentyltheophylline (CPT; 3.0, 6.0, 12.0 mg/kg i.p.) failed to reverse the effects of either the D(1) or D(2) antagonist. In contrast, the adenosine A(2A) antagonist KW-6002 (0.125, 0.25 and 0.5 mg/kg i.p.) was able to produce a robust reversal of the effects of eticlopride, as well as a mild partial reversal of the effects of ecopipam. Adenosine A(2A) and DA D(2) receptors interact to regulate effort-related choice behavior, which may have implications for the treatment of psychiatric symptoms such as psychomotor slowing, fatigue or anergia that can be observed in depression and other disorders.

Role of dopamine–adenosine interactions in the brain circuitry regulating effort-related decision making: insights into pathological aspects of motivation

Brain dopamine, particularly in the nucleus accumbens, has been implicated in activational aspects of motivation and effort-related processes. Accumbens dopamine depletions reduce the tendency of rats to work for food, and alter effort-related decision making, but leave aspects of food motivation such as appetite intact. Recent evidence indicates that the purine neuromodulator adenosine, largely through actions on adenosine A2A receptors, also participates in regulating effort-related processes. Adenosine A2A antagonists can reverse the effects of dopamine D2 antagonists on effort-related choice, and intra-accumbens injections of adenosine A2A agonists produce effects that are similar to those induced by accumbens dopamine depletion or antagonism. These studies have implications for the understanding and treatment of energy-related disorders such as anergia and fatigue in psychiatry and neurology.