Modeling the effects of motivation on choice and learning in the basal ganglia

Striatal dopamine integrates cost, benefit, and motivation

Striatal dopamine (DA) release has long been linked to reward processing, but it remains controversial whether DA release reflects costs or benefits and how these signals vary with motivation. Here, we measure DA release in the nucleus accumbens (NAc) and dorsolateral striatum (DLS) while independently varying costs and benefits and apply behavioral economic principles to determine a mouse's level of motivation. We reveal that DA release in both structures incorporates both reward magnitude and sunk cost. Surprisingly, motivation was inversely correlated with reward-evoked DA release. Furthermore, optogenetically evoked DA release was also heavily dependent on sunk cost. Our results reconcile previous disparate findings by demonstrating that striatal DA release simultaneously encodes cost, benefit, and motivation but in distinct manners over different timescales. Future work will be necessary to determine whether the reduction in phasic DA release in highly motivated animals is due to changes in tonic DA levels.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Liraglutide restores impaired associative learning in individuals with obesity

Survival under selective pressure is driven by the ability of our brain to use sensory information to our advantage to control physiological needs. To that end, neural circuits receive and integrate external environmental cues and internal metabolic signals to form learned sensory associations, consequently motivating and adapting our behaviour. The dopaminergic midbrain plays a crucial role in learning adaptive behaviour and is particularly sensitive to peripheral metabolic signals, including intestinal peptides, such as glucagon-like peptide 1 (GLP-1). In a single-blinded, randomized, controlled, crossover basic human functional magnetic resonance imaging study relying on a computational model of the adaptive learning process underlying behavioural responses, we show that adaptive learning is reduced when metabolic sensing is impaired in obesity, as indexed by reduced insulin sensitivity (participants: N = 30 with normal insulin sensitivity; N = 24 with impaired insulin sensitivity). Treatment with the GLP-1 receptor agonist liraglutide normalizes impaired learning of sensory associations in men and women with obesity. Collectively, our findings reveal that GLP-1 receptor activation modulates associative learning in people with obesity via its central effects within the mesoaccumbens pathway. These findings provide evidence for how metabolic signals can act as neuromodulators to adapt our behaviour to our body's internal state and how GLP-1 receptor agonists work in clinics.

Gambling on an empty stomach: Hunger modulates preferences for learned but not described risks

INTRODUCTION

We assess risks differently when they are explicitly described, compared to when we learn directly from experience, suggesting dissociable decision-making systems. Our needs, such as hunger, could globally affect our risk preferences, but do they affect described and learned risks equally? On one hand, decision-making from descriptions is often considered flexible and context sensitive, and might therefore be modulated by metabolic needs. On the other hand, preferences learned through reinforcement might be more strongly coupled to biological drives.

METHOD

Thirty-two healthy participants (females: 20, mean age: 25.6 ± 6.5 years) with a normal weight (Body Mass Index: 22.9 ± 3.2 kg/m2 ) were tested in a within-subjects counterbalanced, randomized crossover design for the effects of hunger on two separate risk-taking tasks. We asked participants to choose between two options with different risks to obtain monetary outcomes. In one task, the outcome probabilities were described numerically, whereas in a second task, they were learned.

RESULT

In agreement with previous studies, we found that rewarding contexts induced risk-aversion when risks were explicitly described (F1,31  = 55.01, p < .0001, ηp 2  = .64), but risk-seeking when they were learned through experience (F1,31  = 10.28, p < .003, ηp 2  = .25). Crucially, hunger attenuated these contextual biases, but only for learned risks (F1,31  = 8.38, p < .007, ηp 2  = .21).

CONCLUSION

The results suggest that our metabolic state determines risk-taking biases when we lack explicit descriptions.

Impact of insulin and insulin resistance on brain dopamine signalling and reward processing- an underexplored mechanism in the pathophysiology of depression?

Type 2 diabetes and major depressive disorder (MDD) are the leading causes of disability worldwide and have a high comorbidity rate with fatal outcomes. Despite the long-established association between these conditions, the underlying molecular mechanisms remain unknown. Since the discovery of insulin receptors in the brain and the brain's reward system, evidence has accumulated indicating that insulin modulates dopaminergic (DA) signalling and reward behaviour. Here, we review the evidence from rodent and human studies, that insulin resistance directly alters central DA pathways, which may result in motivational deficits and depressive symptoms. Specifically, we first elaborate on the differential effects of insulin on DA signalling in the ventral tegmental area (VTA) - the primary DA source region in the midbrain - and the striatum as well as its effects on behaviour. We then focus on the alterations induced by insulin deficiency and resistance. Finally, we review the impact of insulin resistance in DA pathways in promoting depressive symptoms and anhedonia on a molecular and epidemiological level and discuss its relevance for stratified treatment strategies.

The computational roots of positivity and confirmation biases in reinforcement learning

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Obesity - A Matter of Motivation?

Excessive food intake and reduced physical activity have long been established as primary causes of obesity. However, the underlying mechanisms causing this unhealthy behavior characterized by heightened motivation for food but not for physical effort are unclear. Despite the common unjustified stigmatization that obesity is a result of laziness and lack of discipline, it is becoming increasingly clear that high-fat diet feeding and obesity cause alterations in brain circuits that are critical for the control of motivational behavior.

In this mini-review, we provide a comprehensive overview of incentive motivation, its neural encoding in the dopaminergic mesolimbic system as well as its metabolic modulation with a focus on derangements of incentive motivation in obesity. We further discuss the emerging field of metabolic interventions to counteract motivational deficits and their potential clinical implications.

Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks

Computational models of the basal ganglia (BG) provide a mechanistic account of different phenomena observed during reinforcement learning tasks performed by healthy individuals, as well as by patients with various nervous or mental disorders. The aim of the present work was to develop a BG model that could represent a good compromise between simplicity and completeness. Based on more complex (fine-grained neural network, FGNN) models, we developed a new (coarse-grained neural network, CGNN) model by replacing layers of neurons with single nodes that represent the collective behavior of a given layer while preserving the fundamental anatomical structures of BG. We then compared the functionality of both the FGNN and CGNN models with respect to several reinforcement learning tasks that are based on BG circuitry, such as the Probabilistic Selection Task, Probabilistic Reversal Learning Task and Instructed Probabilistic Selection Task. We showed that CGNN still has a functionality that mirrors the behavior of the most often used reinforcement learning tasks in human studies. The simplification of the CGNN model reduces its flexibility but improves the readability of the signal flow in comparison to more detailed FGNN models and, thus, can help to a greater extent in the translation between clinical neuroscience and computational modeling.

GLP-1 and hunger modulate incentive motivation depending on insulin sensitivity in humans

OBJECTIVE

To regulate food intake, our brain constantly integrates external cues, such as the incentive value of a potential food reward, with internal state signals, such as hunger feelings. Incentive motivation refers to the processes that translate an expected reward into the effort spent to obtain the reward; the magnitude and probability of a reward involved in prompting motivated behaviour are encoded by the dopaminergic (DA) midbrain and its mesoaccumbens DA projections. This type of reward circuity is particularly sensitive to the metabolic state signalled by peripheral mediators, such as insulin or glucagon-like peptide 1 (GLP-1). While in rodents the modulatory effect of metabolic state signals on motivated behaviour is well documented, evidence of state-dependent modulation and the role of incentive motivation underlying overeating in humans is lacking.

METHODS

In a randomised, placebo-controlled, crossover design, 21 lean (body mass index [BMI] < 25 kg/m²) and 16 obese (BMI³ 30 kg/m²) volunteer participants received either liraglutide as a GLP-1 analogue or placebo on two separate testing days. Incentive motivation was measured using a behavioural task in which participants were required to exert physical effort using a handgrip to win different amounts of food and monetary rewards. Hunger levels were measured using visual analogue scales; insulin, glucose, and systemic insulin resistance as assessed by the homeostasis model assessment of insulin resistance (HOMA-IR) were quantified at baseline.

RESULTS

In this report, we demonstrate that incentive motivation increases with hunger in lean humans (F_(1,42) = 5.31, p = 0.026, β = 0.19) independently of incentive type (food and non-food reward). This effect of hunger is not evident in obese humans (F_(1,62) = 1.93, p = 0.17, β = -0.12). Motivational drive related to hunger is affected by peripheral insulin sensitivity (two-way interaction, F_{(1, 35)} = 6.23, p = 0.017, β = -0.281). In humans with higher insulin sensitivity, hunger increases motivation, while poorer insulin sensitivity dampens the motivational effect of hunger. The GLP-1 analogue application blunts the interaction effect of hunger on motivation depending on insulin sensitivity (three-way interaction, F_{(1, 127)} = 5.11, p = 0.026); no difference in motivated behaviour could be found between humans with normal or impaired insulin sensitivity under GLP-1 administration.

CONCLUSION

We report a differential effect of hunger on motivation depending on insulin sensitivity. We further revealed the modulatory role of GLP-1 in adaptive, motivated behaviour in humans and its interaction with peripheral insulin sensitivity and hunger. Our results suggest that GLP-1 might restore dysregulated processes of midbrain DA function and hence motivational behaviour in insulin-resistant humans.

Hypothalamic Interactions with Large-Scale Neural Circuits Underlying Reinforcement Learning and Motivated Behavior

Biological agents adapt behavior to support the survival needs of the individual and the species. In this review we outline the anatomical, physiological, and computational processes that support reinforcement learning (RL). We describe two circuits in the primate brain that are linked to specific aspects of learning and goal-directed behavior. The ventral circuit, that includes the amygdala, ventral medial prefrontal cortex, and ventral striatum, has substantial connectivity with the hypothalamus. The dorsal circuit, that includes inferior parietal cortex, dorsal lateral prefrontal cortex, and the dorsal striatum, has minimal connectivity with the hypothalamus. The hypothalamic connectivity suggests distinct roles for these circuits. We propose that the ventral circuit defines behavioral goals, and the dorsal circuit orchestrates behavior to achieve those goals.

Dopamine role in learning and action inference

This paper describes a framework for modelling dopamine function in the mammalian brain. It proposes that both learning and action planning involve processes minimizing prediction errors encoded by dopaminergic neurons. In this framework, dopaminergic neurons projecting to different parts of the striatum encode errors in predictions made by the corresponding systems within the basal ganglia. The dopaminergic neurons encode differences between rewards and expectations in the goal-directed system, and differences between the chosen and habitual actions in the habit system. These prediction errors trigger learning about rewards and habit formation, respectively. Additionally, dopaminergic neurons in the goal-directed system play a key role in action planning: They compute the difference between a desired reward and the reward expected from the current motor plan, and they facilitate action planning until this difference diminishes. Presented models account for dopaminergic responses during movements, effects of dopamine depletion on behaviour, and make several experimental predictions.

In the brain, chemicals such as dopamine allow nerve cells to ‘talk’ to each other and to relay information from and to the environment. Dopamine, in particular, is released when pleasant surprises are experienced: this helps the organism to learn about the consequences of certain actions. If a new flavour of ice-cream tastes better than expected, for example, the release of dopamine tells the brain that this flavour is worth choosing again.

However, dopamine has an additional role in controlling movement. When the cells that produce dopamine die, for instance in Parkinson’s disease, individuals may find it difficult to initiate deliberate movements. Here, Rafal Bogacz aimed to develop a comprehensive framework that could reconcile the two seemingly unrelated roles played by dopamine.

The new theory proposes that dopamine is released when an outcome differs from expectations, which helps the organism to adjust and minimise these differences. In the ice-cream example, the difference is between how good the treat is expected to taste, and how tasty it really is. By learning to select the same flavour repeatedly, the brain aligns expectation and the result of the choice. This ability would also apply when movements are planned. In this case, the brain compares the desired reward with the predicted results of the planned actions. For example, while planning to get a spoonful of ice-cream, the brain compares the pleasure expected from the movement that is currently planned, and the pleasure of eating a full spoon of the treat. If the two differ, for example because no movement has been planned yet, the brain releases dopamine to form a better version of the action plan. The theory was then tested using a computer simulation of nerve cells that release dopamine; this showed that the behaviour of the virtual cells closely matched that of their real-life counterparts.

This work offers a comprehensive description of the fundamental role of dopamine in the brain. The model now needs to be verified through experiments on living nerve cells; ultimately, it could help doctors and researchers to develop better treatments for conditions such as Parkinson’s disease or ADHD, which are linked to a lack of dopamine.