Dopamine and performance in a reinforcement learning task: evidence from Parkinson's disease

Clinically established early Parkinson's disease patients do not show impaired use of priors in conditions of perceptual uncertainty

The ability to use past learned experiences to guide decisions is an important component of adaptive behavior, especially when decision-making is performed under time pressure or when perceptual information is unreliable. Previous studies using visual discrimination tasks have shown that this prior-informed decision-making ability is impaired in Parkinson's disease (PD), but the mechanisms underlying this deficit and the precise impact of dopaminergic denervation within cortico-basal circuits remain unclear. To shed light on this problem, we evaluated prior-informed decision-making under various conditions of perceptual uncertainty in a sample of 13 clinically established early PD patients, and compared behavioral performance with healthy control (HC) subjects matched in age, sex and education. PD patients and HC subjects performed a random dot motion task in which they had to decide the net direction (leftward vs. rightward) of a field of moving dots and communicate their choices through manual button presses. We manipulated prior knowledge by modulating the probability of occurrence of leftward vs. rightward motion stimuli between blocks of trials, and by explicitly giving these probabilities to subjects at the beginning of each block. We further manipulated stimulus discriminability by varying the proportion of dots moving coherently in the signal direction and speed-accuracy instructions. PD patients used choice probabilities to guide perceptual decisions in both speed and accuracy conditions, and their performance did not significantly differ from that of HC subjects. An additional analysis of the data with the diffusion decision model confirmed this conclusion. These results suggest that the impaired use of priors during visual discrimination observed at more advanced stages of PD is independent of dopaminergic denervation, though additional studies with larger sample sizes are needed to more firmly establish this conclusion.

Impaired value-based decision-making in Parkinson's disease apathy

Apathy is a common and disabling complication of Parkinson's disease characterized by reduced goal-directed behaviour. Several studies have reported dysfunction within prefrontal cortical regions and projections from brainstem nuclei whose neuromodulators include dopamine, serotonin and noradrenaline. Work in animal and human neuroscience have confirmed contributions of these neuromodulators on aspects of motivated decision-making. Specifically, these neuromodulators have overlapping contributions to encoding the value of decisions, and influence whether to explore alternative courses of action or persist in an existing strategy to achieve a rewarding goal. Building upon this work, we hypothesized that apathy in Parkinson's disease should be associated with an impairment in value-based learning. Using a four-armed restless bandit reinforcement learning task, we studied decision-making in 75 volunteers; 53 patients with Parkinson's disease, with and without clinical apathy, and 22 age-matched healthy control subjects. Patients with apathy exhibited impaired ability to choose the highest value bandit. Task performance predicted an individual patient's apathy severity measured using the Lille Apathy Rating Scale (R = -0.46, P < 0.001). Computational modelling of the patient's choices confirmed the apathy group made decisions that were indifferent to the learnt value of the options, consistent with previous reports of reward insensitivity. Further analysis demonstrated a shift away from exploiting the highest value option and a reduction in perseveration, which also correlated with apathy scores (R = -0.5, P < 0.001). We went on to acquire functional MRI in 59 volunteers; a group of 19 patients with and 20 without apathy and 20 age-matched controls performing the Restless Bandit Task. Analysis of the functional MRI signal at the point of reward feedback confirmed diminished signal within ventromedial prefrontal cortex in Parkinson's disease, which was more marked in apathy, but not predictive of their individual apathy severity. Using a model-based categorization of choice type, decisions to explore lower value bandits in the apathy group activated prefrontal cortex to a similar degree to the age-matched controls. In contrast, Parkinson's patients without apathy demonstrated significantly increased activation across a distributed thalamo-cortical network. Enhanced activity in the thalamus predicted individual apathy severity across both patient groups and exhibited functional connectivity with dorsal anterior cingulate cortex and anterior insula. Given that task performance in patients without apathy was no different to the age-matched control subjects, we interpret the recruitment of this network as a possible compensatory mechanism, which compensates against symptomatic manifestation of apathy in Parkinson's disease.

Impulse control disorder in Parkinson's disease is associated with abnormal frontal value signalling

Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson's disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson's disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression. One hundred and ninety-nine patients with Parkinson's disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered. These data substantiate the hypothesis that dopamine's effects on reinforcement learning in Parkinson's disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum. See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.

Reliability of gamified reinforcement learning in densely sampled longitudinal assessments

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

A Brief Review on the Potential of Psychedelics for Treating Alzheimer's Disease and Related Depression

Alzheimer's disease (AD), the most common form of senile dementia, is poised to place an even greater societal and healthcare burden as the population ages. With few treatment options for the symptomatic relief of the disease and its unknown etiopathology, more research into AD is urgently needed. Psychedelic drugs target AD-related psychological pathology and symptoms such as depression. Using microdosing, psychedelic drugs may prove to help combat this devastating disease by eliciting psychiatric benefits via acting through various mechanisms of action such as serotonin and dopamine pathways. Herein, we review the studied benefits of a few psychedelic compounds that may show promise in treating AD and attenuating its related depressive symptoms. We used the listed keywords to search through PubMed for relevant preclinical, clinical research, and review articles. The putative mechanism of action (MOA) for psychedelics is that they act mainly as serotonin receptor agonists and induce potential beneficial effects for treating AD and related depression.

Decreased sensitivity to loss of options in patients with Parkinson's disease

Humans prefer to have many options when making decisions. When there is a threat of options disappearing, humans invest more to keep these options available, indicating that they are sensitive to the loss of options. This study examined whether patients with Parkinson's disease (PD), a disease characterized by dopamine depletion, try to keep options available when options are disappearing. Twenty-seven PD patients without dementia and 27 healthy controls (HCs) performed the door game, in which participants were presented with multiple alternatives in the form of three doors, each associated with a different point distribution. The participants were asked to maximize their point earnings by finding the best door. The task included two conditions. In the shutter condition, shutters gradually closed on doors that were not chosen; once the shutters completely closed, the door was no longer available. There were no shutters in the control condition. The results revealed that the HCs switched doors more often in the shutter condition than in the control condition, indicating a tendency to keep options available. However, the PD patients did not show such differences between the two conditions. The difference in the number of switches between the shutter and control conditions in the PD patients was significantly positively correlated with the distribution of dopamine transporters in the left striatum, as measured by ¹²³I-ioflupane-SPECT (DaTSCAN) images. These results suggest that PD patients are less sensitive to the loss of options, and this decreased sensitivity may be caused by a decline in dopaminergic neurotransmission.

Role of dopamine and clinical heterogeneity in cognitive dysfunction in Parkinson's disease

Parkinson's disease (PD) is commonly treated with dopaminergic medication, which enhances some, while impairing other cognitive functions. It can even contribute to impulse control disorder and addiction. We describe the history of research supporting the dopamine overdose hypothesis, which accounts for the large within-patient variability in dopaminergic medication effects across different tasks by referring to the spatially non-uniform pattern of dopamine depletion in dorsal versus ventral striatum. However, there is tremendous variability in dopaminergic medication effects not just within patients across distinct tasks, but also across different patients. In the second part of this chapter we review recent studies addressing the large individual variability in the negative side effects of dopaminergic medication on functions that implicate dopamine, such as value-based learning and choice. These studies begin to unravel the mechanisms of dopamine overdosing, thus revising the strict version of the overdose hypothesis. For example, the work shows that the canonical boosting of reward-versus punishment-based choice by medication is greater in patients with depression and a non-tremor phenotype, which both implicate, among other pathology, more rather than less severe dysregulation of the mesolimbic dopamine system. Future longitudinal cohort studies are needed to identify how to optimally combine different clinical, personality, cognitive, neural, genetic and molecular predictors of detrimental medication effects in order to account for as much of the relevant variability as possible. This will provide a useful tool for precision neurology, allowing individual and contextual tailoring of (the dose of) dopaminergic medication in order to maximize its cognitive benefits, yet minimize its side effects.

Advances in clinical basic research: Performance, treatments, and mechanisms of Parkinson disease

The loss of neuronal in the substantia nigra of the elderly contributes to striatal damage and plays a critical part in the common forms of neurodegenerative diseases such as Parkinson disease (PD). The deficit of dopamine is one of the most familiar neuropathological features of PD as well as α-Synuclein aggregation. The peripheral autonomic nervous system is also affected negatively during the course of the disease, although the subsistent of dyskinesias and else major motor characteristic deficits take significant role in the diagnostic methods during clinical practice, which is related to a number of non-motor symptoms that might increase aggregate risks. Multiple pathways and mechanisms are involved in the molecular pathogenesis: α-Synuclein, neuronal homeostasis, mitochondrial function, oxidative stress, as well as neuroinflammation. Investigations in the last few years for diagnostic biomarkers used neuroimaging, including single photon emission computed tomography  as well as cutting-edge magnetic resonance imaging techniques, which has been presented to facilitate discrepant diagnosis. Pharmacological treatment is also important and efficient in equal measure. In addition to reliance on striatal dopamine replacement therapy, many solutions that are used for motor or nonmotor symptoms in these patients are available.

What Is Social about Autism? The Role of Allostasis-Driven Learning

Scientific research on neuro-cognitive mechanisms of autism often focuses on circuits that support social functioning. However, autism is a heterogeneous developmental variation in multiple domains, including social communication, but also language, cognition, and sensory-motor control. This suggests that the underlying mechanisms of autism share a domain-general foundation that impacts all of these processes. In this Perspective Review, we propose that autism is not a social deficit that results from an atypical "social brain". Instead, typical social development relies on learning. In social animals, infants depend on their caregivers for survival, which makes social information vitally salient. The infant must learn to socially interact in order to survive and develop, and the most prominent learning in early life is crafted by social interactions. Therefore, the most prominent outcome of a learning variation is atypical social development. To support the hypothesis that autism results from a variation in learning, we first review evidence from neuroscience and developmental science, demonstrating that typical social development depends on two domain-general processes that determine learning: (a) motivation, guided by allostatic regulation of the internal milieu; and (b) multi-modal associations, determined by the statistical regularities of the external milieu. These two processes are basic ingredients of typical development because they determine allostasis-driven learning of the social environment. We then review evidence showing that allostasis and learning are affected among individuals with autism, both neurally and behaviorally. We conclude by proposing a novel domain-general framework that emphasizes allostasis-driven learning as a key process underlying autism. Guided by allostasis, humans learn to become social, therefore, the atypical social profile seen in autism can reflect a domain-general variation in allostasis-driven learning. This domain-general view raises novel research questions in both basic and clinical research and points to targets for clinical intervention that can lower the age of diagnosis and improve the well-being of individuals with autism.

Application of Three Ecological Assessment Tools in Examining Chromatographic Methods for the Green Analysis of a Mixture of Dopamine, Serotonin, Glutamate and GABA: A Comparative Study

The assessment of greenness of analytical protocols is of great importance now to preserve the environment. Some studies have analyzed either only the neurotransmitters, dopamine, serotonin, glutamate, and gamma-aminobutyric acid (GABA), together or with other neurotransmitters and biomarkers. However, these methods have not been investigated for their greenness and were not compared with each other to find the optimum one. Therefore, this study aims to compare seven published chromatographic methods that analyzed the four neurotransmitters and their mixtures using the National Environmental Method Index, Analytical Eco-Scale Assessment (ESA), and Green Analytical Procedure Index (GAPI). As these methods cover both qualitative and quantitative aspects, they offer better transparency. Overall, GAPI showed maximum greenness throughout the analysis. Method 6 was proven to be the method of choice for analyzing the mixture, owing to its greenness, according to NEMI, ESA, and GAPI. Additionally, method 6 has a wide scope of application (13 components can be analyzed), high sensitivity (low LOQ values), and fast analysis (low retention times, especially for glutamate and GABA).

Rational inattention and tonic dopamine

Slow-timescale (tonic) changes in dopamine (DA) contribute to a wide variety of processes in reinforcement learning, interval timing, and other domains. Furthermore, changes in tonic DA exert distinct effects depending on when they occur (e.g., during learning vs. performance) and what task the subject is performing (e.g., operant vs. classical conditioning). Two influential theories of tonic DA-the average reward theory and the Bayesian theory in which DA controls precision-have each been successful at explaining a subset of empirical findings. But how the same DA signal performs two seemingly distinct functions without creating crosstalk is not well understood. Here we reconcile the two theories under the unifying framework of 'rational inattention,' which (1) conceptually links average reward and precision, (2) outlines how DA manipulations affect this relationship, and in so doing, (3) captures new empirical phenomena. In brief, rational inattention asserts that agents can increase their precision in a task (and thus improve their performance) by paying a cognitive cost. Crucially, whether this cost is worth paying depends on average reward availability, reported by DA. The monotonic relationship between average reward and precision means that the DA signal contains the information necessary to retrieve the precision. When this information is needed after the task is performed, as presumed by Bayesian inference, acute manipulations of DA will bias behavior in predictable ways. We show how this framework reconciles a remarkably large collection of experimental findings. In reinforcement learning, the rational inattention framework predicts that learning from positive and negative feedback should be enhanced in high and low DA states, respectively, and that DA should tip the exploration-exploitation balance toward exploitation. In interval timing, this framework predicts that DA should increase the speed of the internal clock and decrease the extent of interference by other temporal stimuli during temporal reproduction (the central tendency effect). Finally, rational inattention makes the new predictions that these effects should be critically dependent on the controllability of rewards, that post-reward delays in intertemporal choice tasks should be underestimated, and that average reward manipulations should affect the speed of the clock-thus capturing empirical findings that are unexplained by either theory alone. Our results suggest that a common computational repertoire may underlie the seemingly heterogeneous roles of DA.

The Ties That Bind: Aberrant Plasticity and Networks Dysfunction in Movement Disorders-Implications for Rehabilitation

Background: Movement disorders encompass various conditions affecting the nervous system. The pathological processes underlying movement disorders lead to aberrant synaptic plastic changes, which in turn alter the functioning of large-scale brain networks. Therefore, clinical phenomenology does not only entail motor symptoms but also cognitive and motivational disturbances. The result is the disruption of motor learning and motor behavior. Due to this complexity, the responsiveness to standard therapies could be disappointing. Specific forms of rehabilitation entailing goal-based practice, aerobic training, and the use of noninvasive brain stimulation techniques could "restore" neuroplasticity at motor-cognitive circuitries, leading to clinical gains. This is probably associated with modulations occurring at both molecular (synaptic) and circuitry levels (networks). Several gaps remain in our understanding of the relationships among plasticity and neural networks and how neurorehabilitation could promote clinical gains is still unclear. Purposes: In this review, we outline first the networks involved in motor learning and behavior and analyze which mechanisms link the pathological synaptic plastic changes with these networks' disruption in movement disorders. Therefore, we provide theoretical and practical bases to be applied for treatment in rehabilitation.

Tremor in Parkinson's disease inverts the effect of dopamine on reinforcement

This scientific commentary refers to ‘Effects of dopamine on reinforcement learning in Parkinson’s disease depend on motor phenotype’ by van Nuland et al. (doi:10.1093/brain/awaa335).

Effects of dopamine on reinforcement learning in Parkinson's disease depend on motor phenotype

Parkinson's disease is clinically defined by bradykinesia, along with rigidity and tremor. However, the severity of these motor signs is greatly variable between individuals, particularly the presence or absence of tremor. This variability in tremor relates to variation in cognitive/motivational impairment, as well as the spatial distribution of neurodegeneration in the midbrain and dopamine depletion in the striatum. Here we ask whether interindividual heterogeneity in tremor symptoms could account for the puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning, a fundamental cognitive function known to rely on dopamine. Given that tremor-dominant and non-tremor Parkinson's disease patients have different dopaminergic phenotypes, we hypothesized that effects of dopaminergic medication on reinforcement learning differ between tremor-dominant and non-tremor patients. Forty-three tremor-dominant and 20 non-tremor patients with Parkinson's disease were recruited to be tested both OFF and ON dopaminergic medication (200/50 mg levodopa-benserazide), while 22 age-matched control subjects were recruited to be tested twice OFF medication. Participants performed a reinforcement learning task designed to dissociate effects on learning rate from effects on motivational choice (i.e. the tendency to 'Go/NoGo' in the face of reward/threat of punishment). In non-tremor patients, dopaminergic medication improved reward-based choice, replicating previous studies. In contrast, in tremor-dominant patients, dopaminergic medication improved learning from punishment. Formal modelling showed divergent computational effects of dopaminergic medication as a function of Parkinson's disease motor phenotype, with a modulation of motivational choice bias and learning rate in non-tremor and tremor patients, respectively. This finding establishes a novel cognitive/motivational difference between tremor and non-tremor Parkinson's disease patients, and highlights the importance of considering motor phenotype in future work.

Corticostriatal White Matter Integrity and Dopamine D1 Receptor Availability Predict Age Differences in Prefrontal Value Signaling during Reward Learning

Probabilistic reward learning reflects the ability to adapt choices based on probabilistic feedback. The dopaminergically innervated corticostriatal circuit in the brain plays an important role in supporting successful probabilistic reward learning. Several components of the corticostriatal circuit deteriorate with age, as it does probabilistic reward learning. We showed previously that D1 receptor availability in NAcc predicts the strength of anticipatory value signaling in vmPFC, a neural correlate of probabilistic learning that is attenuated in older participants and predicts probabilistic reward learning performance. We investigated how white matter integrity in the pathway between nucleus accumbens (NAcc) and ventromedial prefrontal cortex (vmPFC) relates to the strength of anticipatory value signaling in vmPFC in younger and older participants. We found that in a sample of 22 old and 23 young participants, fractional anisotropy in the pathway between NAcc and vmPFC predicted the strength of value signaling in vmPFC independently from D1 receptor availability in NAcc. These findings provide tentative evidence that integrity in the dopaminergic and white matter pathways of corticostriatal circuitry supports the expression of value signaling in vmPFC which supports reward learning, however, the limited sample size calls for independent replication. These and future findings could add to the improved understanding of how corticostriatal integrity contributes to reward learning ability.

Dopamine is associated with prioritization of reward-associated memories in Parkinson's disease

Patients with Parkinson's disease have reduced reward sensitivity related to dopaminergic neuron loss, which is associated with impairments in reinforcement learning. Increasingly, however, dopamine-dependent reward signals are recognized to play an important role beyond reinforcement learning. In particular, it has been shown that reward signals mediated by dopamine help guide the prioritization of events for long-term memory consolidation. Meanwhile, studies of memory in patients with Parkinson's disease have focused on overall memory capacity rather than what is versus what isn't remembered, leaving open questions about the effect of dopamine replacement on the prioritization of memories by reward and the time-dependence of this effect. The current study sought to fill this gap by testing the effect of reward and dopamine on memory in patients with Parkinson's disease. We tested the effect of dopamine modulation and reward on two forms of long-term memory: episodic memory for neutral objects and memory for stimulus-value associations. We measured both forms of memory in a single task, adapting a standard task of reinforcement learning with incidental episodic encoding events of trial-unique objects. Objects were presented on each trial at the time of feedback, which was either rewarding or not. Memory for the trial-unique images and for the stimulus-value associations, and the influence of reward on both, was tested immediately after learning and 2 days later. We measured performance in Parkinson's disease patients tested either ON or OFF their dopaminergic medications and in healthy older control subjects. We found that dopamine was associated with a selective enhancement of memory for reward-associated images, but that it did not influence overall memory capacity. Contrary to predictions, this effect did not differ between the immediate and delayed memory tests. We also found that while dopamine had an effect on reward-modulated episodic memory, there was no effect of dopamine on memory for stimulus-value associations. Our results suggest that impaired prioritization of cognitive resource allocation may contribute to the early cognitive deficits of Parkinson's disease.

Solid-Contact Potentiometric Sensors Based on Stimulus-Responsive Imprinted Polymers for Reversible Detection of Neutral Dopamine

Herein, we present for the first time a novel potentiometric sensor based on the stimulus-responsive molecularly imprinted polymer (MIP) as a selective receptor for neutral dopamine determination. This smart receptor can change its capabilities to recognize according to external environmental stimuli. Therefore, MIP-binding sites can be regenerated in the polymeric membrane by stimulating with stimulus after each measurement. Based on this effect, reversible detection of the analyte via potentiometric transduction can be achieved. MIPs based on 4-vinylphenylboronic acid as the functional monomer were prepared as the selective receptor. This monomer can successfully bind to dopamine via covalent binding and forming a five- or six-membered cyclic ester in a weakly alkaline aqueous solution. In acidic medium, the produced ester dissociates and regenerates new binding sites in the polymeric membrane. The proposed smart sensor exhibited fast response and good sensitivity towards dopamine with a limit of detection 0.15 µM over the linear range 0.2–10 µM. The selectivity pattern of the proposed ISEs was also evaluated and revealed an enhanced selectivity towards dopamine over several phenolic compounds. Constant-current chronopotentiometry is used for evaluating the short-term potential stability of the proposed ISEs. The obtained results confirm that the stimulus-responsive MIPs provide an attractive way towards reversible MIP-based electrochemical sensors designation.

Motor-cognitive approach and aerobic training: a synergism for rehabilitative intervention in Parkinson's disease

Parkinson's disease (PD) results in a complex deterioration of motor behavior. Effective pharmacological or surgical treatments addressing the whole spectrum of both motor and cognitive symptoms are lacking. The cumulative functional impairment may have devastating socio-economic consequences on both patients and caregivers. Comprehensive models of care based on multidisciplinary approaches may succeed in better addressing the overall complexity of PD. Neurorehabilitation is a highly promising non-pharmacological intervention for managing PD. The scientific rationale beyond rehabilitation and its practical applicability remain to be established. In the present perspective, we aim to discuss the current evidence supporting integrated motor-cognitive and aerobic rehabilitation approaches for patients with PD while suggesting a practical framework to optimize this intervention in the next future.

Spontaneous eye blink rate predicts individual differences in exploration and exploitation during reinforcement learning

Spontaneous eye blink rate (sEBR) has been linked to striatal dopamine function and to how individuals make value-based choices after a period of reinforcement learning (RL). While sEBR is thought to reflect how individuals learn from the negative outcomes of their choices, this idea has not been tested explicitly. This study assessed how individual differences in sEBR relate to learning by focusing on the cognitive processes that drive RL. Using Bayesian latent mixture modelling to quantify the mapping between RL behaviour and its underlying cognitive processes, we were able to differentiate low and high sEBR individuals at the level of these cognitive processes. Further inspection of these cognitive processes indicated that sEBR uniquely indexed explore-exploit tendencies during RL: lower sEBR predicted exploitative choices for high valued options, whereas higher sEBR predicted exploration of lower value options. This relationship was additionally supported by a network analysis where, notably, no link was observed between sEBR and how individuals learned from negative outcomes. Our findings challenge the notion that sEBR predicts learning from negative outcomes during RL, and suggest that sEBR predicts individual explore-exploit tendencies. These then influence value sensitivity during choices to support successful performance when facing uncertain reward.

Dopamine-Dependent Loss Aversion during Effort-Based Decision-Making

From psychology to economics, there has been substantial interest in how costs (e.g., delay, risk) are represented asymmetrically during decision-making when attempting to gain reward or avoid punishment. For example, in decision-making under risk, individuals show a tendency to prefer to avoid punishment rather than to acquire the equivalent reward (loss aversion). Although the cost of physical effort has recently received significant attention, it remains unclear whether loss aversion exists during effort-based decision-making. On the one hand, loss aversion may be hardwired due to asymmetric evolutionary pressure on losses and gains and therefore exists across decision-making contexts. On the other hand, distinct brain regions are involved with different decision costs, making it questionable whether similar asymmetries exist. Here, we demonstrate that young healthy human participants (females, 16; males, 6) exhibit loss aversion during effort-based decision-making by exerting more physical effort to avoid punishment than to gain a same-size reward. Next, we show that medicated Parkinson's disease (PD) patients (females, 9; males, 9) show a reduction in loss aversion compared with age-matched control subjects (females, 11; males, 9). Behavioral and computational analysis revealed that people with PD exerted similar physical effort in return for a reward but were less willing to produce effort to avoid punishment. Therefore, loss aversion is present during effort-based decision-making and can be modulated by altered dopaminergic state. This finding could have important implications for our understanding of clinical disorders that show a reduced willingness to exert effort in the pursuit of reward.SIGNIFICANCE STATEMENT Loss aversion-preferring to avoid punishment rather than to acquire equivalent reward-is an important concept in decision-making under risk. However, little is known about whether loss aversion also exists during decisions where the cost is physical effort. This is surprising given that motor cost shapes human behavior, and a reduced willingness to exert effort is a characteristic of many clinical disorders. Here, we show that healthy human individuals exert more effort to minimize punishment than to maximize reward (loss aversion). We also demonstrate that medicated Parkinson's disease patients exert similar effort to gain reward but less effort to avoid punishment when compared with healthy age-matched control subjects. This indicates that dopamine-dependent loss aversion is crucial for explaining effort-based decision-making.

Dopaminergic medication reduces striatal sensitivity to negative outcomes in Parkinson's disease

Reduced levels of dopamine in Parkinson's disease contribute to changes in learning, resulting from the loss of midbrain neurons that transmit a dopaminergic teaching signal to the striatum. Dopamine medication used by patients with Parkinson's disease has previously been linked to behavioural changes during learning as well as to adjustments in value-based decision-making after learning. To date, however, little is known about the specific relationship between dopaminergic medication-driven differences during learning and subsequent changes in approach/avoidance tendencies in individual patients. Twenty-four Parkinson's disease patients ON and OFF dopaminergic medication and 24 healthy controls subjects underwent functional MRI while performing a probabilistic reinforcement learning experiment. During learning, dopaminergic medication reduced an overemphasis on negative outcomes. Medication reduced negative (but not positive) outcome learning rates, while concurrent striatal blood oxygen level-dependent responses showed reduced prediction error sensitivity. Medication-induced shifts in negative learning rates were predictive of changes in approach/avoidance choice patterns after learning, and these changes were accompanied by systematic striatal blood oxygen level-dependent response alterations. These findings elucidate the role of dopamine-driven learning differences in Parkinson's disease, and show how these changes during learning impact subsequent value-based decision-making.

Moral decision making under modafinil: a randomized placebo-controlled double-blind crossover fMRI study

RATIONALE

Modafinil is increasingly used by healthy humans as a neuroenhancer in order to improve cognitive functioning. Research on the effects of modafinil on cognition yielded most consistent findings for complex tasks relying on the prefrontal cortex (PFC).

OBJECTIVES

The present randomized placebo-controlled double-blind crossover study aimed to investigate the effect of a single dose of modafinil (200 mg) on everyday moral decision making and its neural correlates, which have been linked to the ventro- and dorsomedial PFC.

METHODS

Healthy male study participants were presented with short stories describing everyday moral or neutral dilemmas. Each moral dilemma required a decision between a personal desire and a moral standard, while the neutral dilemmas required decisions between two personal desires. The participants underwent this task twice, once under the influence of modafinil and once under placebo. Brain activity associated with the processing of the dilemmas was assessed by means of functional magnetic resonance imaging.

RESULTS

For the processing of moral vs. neutral dilemmas, activations were found in a network of brain regions linked to social cognitive processes including, among others, the bilateral medial PFC, the insula, and the precuneus. Modafinil was found to increase the number of moral decisions and had no effect on brain activity associated with dilemma processing. Exploratory analyses revealed reduced response-locked activity in the dorsomedial PFC for moral compared to neutral dilemmas under modafinil, but not under placebo.

CONCLUSIONS

The results are discussed in terms of altered predictions of others' emotional states under modafinil, possibly due to higher processing efficiency.

Adapting the flow of time with dopamine

The modulation of interval timing by dopamine (DA) has been well established over decades of research. The nature of this modulation, however, has remained controversial: Although the pharmacological evidence has largely suggested that time intervals are overestimated with higher DA levels, more recent optogenetic work has shown the opposite effect. In addition, a large body of work has asserted DA's role as a "reward prediction error" (RPE), or a teaching signal that allows the basal ganglia to learn to predict future rewards in reinforcement learning tasks. Whether these two seemingly disparate accounts of DA may be related has remained an open question. By taking a reinforcement learning-based approach to interval timing, we show here that the RPE interpretation of DA naturally extends to its role as a modulator of timekeeping and furthermore that this view reconciles the seemingly conflicting observations. We derive a biologically plausible, DA-dependent plasticity rule that can modulate the rate of timekeeping in either direction and whose effect depends on the timing of the DA signal itself. This bidirectional update rule can account for the results from pharmacology and optogenetics as well as the behavioral effects of reward rate on interval timing and the temporal selectivity of striatal neurons. Hence, by adopting a single RPE interpretation of DA, our results take a step toward unifying computational theories of reinforcement learning and interval timing. NEW & NOTEWORTHY How does dopamine (DA) influence interval timing? A large body of pharmacological evidence has suggested that DA accelerates timekeeping mechanisms. However, recent optogenetic work has shown exactly the opposite effect. In this article, we relate DA's role in timekeeping to its most established role, as a critical component of reinforcement learning. This allows us to derive a neurobiologically plausible framework that reconciles a large body of DA's temporal effects, including pharmacological, behavioral, electrophysiological, and optogenetic.

Levodopa does not affect expression of reinforcement learning in older adults

Dopamine has been implicated in learning from rewards and punishment, and in the expression of this learning. However, many studies do not fully separate retrieval and decision mechanisms from learning and consolidation. Here, we investigated the effects of levodopa (dopamine precursor) on choice performance (isolated from learning or consolidation). We gave 31 healthy older adults 150 mg of levodopa or placebo (double-blinded, randomised) 1 hour before testing them on stimuli they had learned the value of the previous day. We found that levodopa did not affect the overall accuracy of choices, nor the relative expression of positively or negatively reinforced values. This contradicts several studies and suggests that overall dopamine levels may not play a role in the choice performance for values learned through reinforcement learning in older adults.

The Role of Frontostriatal Systems in Instructed Reinforcement Learning: Evidence From Genetic and Experimentally-Induced Variation

Instructions have a powerful effect on learning and decision-making, biasing choice even in the face of disconfirming feedback. Detrimental biasing effects have been reported in a number of studies in which instruction was given prior to trial-and-error learning. Previous work has attributed individual differences in instructional bias to variations in prefrontal and striatal dopaminergic genes, suggesting a role for prefrontally-mediated cognitive control processes in biasing learning. The current study replicates and extends these findings. Human subjects performed a probabilistic reinforcement learning task after receiving inaccurate instructions about the quality of one of the options. In order to establish a causal relationship between prefrontal cortical mechanisms and instructional bias, we applied transcranial direct current stimulation over dorsolateral prefrontal cortex (anodal, cathodal, or sham) while subjects performed the task. We additionally genotyped subjects for the COMT Val158Met genetic polymorphism, which influences the breakdown of prefrontal dopamine, and for the DAT1/SLC6A3 variable number tandem repeat, which affects expression of striatal dopamine transporter. We replicated the finding that the COMT Met allele is associated with increased instructional bias and further demonstrated that variation in DAT1 has similar effects to variation in COMT, with 9-repeat carriers demonstrating increased bias relative to 10-repeat homozygotes. Consistent with increased top-down regulation of reinforcement learning, anodal subjects demonstrated greater bias relative to sham, though this effect was present only early in training. In contrast, there was no effect of cathodal stimulation. Finally, we fit computational models to subjects' data to better characterize the mechanisms underlying instruction bias. A novel choice bias model, in which instructions influence decision-making rather than learning, was found to best account for subjects' behavior. Overall, these data provide further evidence for the role of frontostriatal interactions in biasing instructed reinforcement learning, which adds to the growing literature documenting both costs and benefits of cognitive control.

How pupil responses track value-based decision-making during and after reinforcement learning

Cognition can reveal itself in the pupil, as latent cognitive processes map onto specific pupil responses. For instance, the pupil dilates when we make decisions and these pupil size fluctuations reflect decision-making computations during and after a choice. Surprisingly little is known, however, about how pupil responses relate to decisions driven by the learned value of stimuli. This understanding is important, as most real-life decisions are guided by the outcomes of earlier choices. The goal of this study was to investigate which cognitive processes the pupil reflects during value-based decision-making. We used a reinforcement learning task to study pupil responses during value-based decisions and subsequent decision evaluations, employing computational modeling to quantitatively describe the underlying cognitive processes. We found that the pupil closely tracks reinforcement learning processes independently across participants and across trials. Prior to choice, the pupil dilated as a function of trial-by-trial fluctuations in value beliefs about the to-be chosen option and predicted an individual's tendency to exploit high value options. After feedback a biphasic pupil response was observed, the amplitude of which correlated with participants' learning rates. Furthermore, across trials, early feedback-related dilation scaled with value uncertainty, whereas later constriction scaled with signed reward prediction errors. These findings show that pupil size fluctuations can provide detailed information about the computations underlying value-based decisions and the subsequent updating of value beliefs. As these processes are affected in a host of psychiatric disorders, our results indicate that pupillometry can be used as an accessible tool to non-invasively study the processes underlying ongoing reinforcement learning in the clinic.

Dorsal striatum does not mediate feedback-based, stimulus-response learning: An event-related fMRI study in patients with Parkinson's disease tested on and off dopaminergic therapy

Learning associations between stimuli and responses is essential to everyday life. Dorsal striatum (DS) has long been implicated in stimulus-response learning, though recent results challenge this contention. We have proposed that discrepant findings arise because stimulus-response learning methodology generally confounds learning and response selection processes. In 19 patients with Parkinson's disease (PD) and 18 age-matched controls, we found that dopaminergic therapy decreased the efficiency of stimulus-response learning, with corresponding attenuation of ventral striatum (VS) activation. In contrast, exogenous dopamine improved response selection accuracy related to enhanced DS BOLD signal. Contrasts between PD patients and controls fully support these within-subject patterns. These double dissociations in terms of behaviour and neural activity related to VS and DS in PD and in response to dopaminergic therapy, strongly refute the view that DS mediates stimulus-response learning through feedback. Our findings integrate with a growing literature favouring a role for DS in decision making rather than learning, and unite two literature that have been evolving independently.

Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner

Optimal decision-making is based on integrating information from several dimensions of decisional space (e.g., reward expectation, cost estimation, effort exertion). Despite considerable empirical and theoretical efforts, the computational and neural bases of such multidimensional integration have remained largely elusive. Here we propose that the current theoretical stalemate may be broken by considering the computational properties of a cortical-subcortical circuit involving the dorsal anterior cingulate cortex (dACC) and the brainstem neuromodulatory nuclei: ventral tegmental area (VTA) and locus coeruleus (LC). From this perspective, the dACC optimizes decisions about stimuli and actions, and using the same computational machinery, it also modulates cortical functions (meta-learning), via neuromodulatory control (VTA and LC). We implemented this theory in a novel neuro-computational model–the Reinforcement Meta Learner (RML). We outline how the RML captures critical empirical findings from an unprecedented range of theoretical domains, and parsimoniously integrates various previous proposals on dACC functioning.

A major challenge for all organisms is selecting optimal behaviour to obtain resources while minimizing energetic and other expenses. Evolution provided mammals with exceptional decision-making capabilities to face this challenge. Even though neuroscientists have identified a heterogeneous and distributed set of brain structures to be involved, a comprehensive theory about the biological and computational basis of such decision-making is yet to be formulated. We propose that the interaction between the medial prefrontal cortex (a part of the frontal lobes) and the subcortical nuclei releasing catecholaminergic neuromodulators will be key to such a theory. We argue that this interaction allows both the selection of optimal behaviour and, more importantly, the optimal modulation of the very brain circuits that drive such behavioral selection (i.e., meta-learning). We implemented this theory in a novel neuro-computational model, the Reinforcement Meta-Learner (RML). By means of computer simulations we showed that the RML provides a biological and computational account for a set of neuroscientific data with unprecedented scope, thereby suggesting a critical mechanism of decision-making in the mammalian brain.

Paradoxical Decision-Making: A Framework for Understanding Cognition in Parkinson's Disease

People with Parkinson's disease (PD) show impaired decision-making when sensory and memory information must be combined. This recently identified impairment results from an inability to accumulate the proper amount of information needed to make a decision and appears to be independent of dopamine tone and reinforcement learning mechanisms. Although considerable work focuses on PD and decisions involving risk and reward, in this Opinion article we propose that the emerging findings in perceptual decision-making highlight the multisystem nature of PD, and that unraveling the neuronal circuits underlying perceptual decision-making impairment may help in understanding other cognitive impairments in people with PD. We also discuss how a decision-making framework may be extended to gain insights into mechanisms of motor impairments in PD.

Selective Effects of the Loss of NMDA or mGluR5 Receptors in the Reward System on Adaptive Decision-Making

Selecting the most advantageous actions in a changing environment is a central feature of adaptive behavior. The midbrain dopamine (DA) neurons along with the major targets of their projections, including dopaminoceptive neurons in the frontal cortex and basal ganglia, play a key role in this process. Here, we investigate the consequences of a selective genetic disruption of NMDA receptor and metabotropic glutamate receptor 5 (mGluR5) in the DA system on adaptive choice behavior in mice. We tested the effects of the mutation on performance in the probabilistic reinforcement learning and probability-discounting tasks. In case of the probabilistic choice, both the loss of NMDA receptors in dopaminergic neurons or the loss mGluR5 receptors in D1 receptor-expressing dopaminoceptive neurons reduced the probability of selecting the more rewarded alternative and lowered the likelihood of returning to the previously rewarded alternative (win-stay). When observed behavior was fitted to reinforcement learning models, we found that these two mutations were associated with a reduced effect of the expected outcome on choice (i.e., more random choices). None of the mutations affected probability discounting, which indicates that all animals had a normal ability to assess probability. However, in both behavioral tasks animals with targeted loss of NMDA receptors in dopaminergic neurons or mGluR5 receptors in D1 neurons were significantly slower to perform choices. In conclusion, these results show that glutamate receptor-dependent signaling in the DA system is essential for the speed and accuracy of choices, but at the same time probably is not critical for correct estimation of probable outcomes.

Effects of feedback delay on learning from positive and negative feedback in patients with Parkinson's disease off medication

Phasic dopamine (DA) signals conveyed from the substantia nigra to the striatum and the prefrontal cortex crucially affect learning from feedback, with DA bursts facilitating learning from positive feedback and DA dips facilitating learning from negative feedback. Consequently, diminished nigro-striatal dopamine levels as in unmedicated patients suffering from Parkinson's Disease (PD) have been shown to lead to a negative learning bias. Recent studies suggested a diminished striatal contribution to feedback processing when the outcome of an action is temporally delayed. This study investigated whether the bias towards negative feedback learning induced by a lack of DA in PD patients OFF medication is modulated by feedback delay. To this end, PD patients OFF medication and healthy controls completed a probabilistic selection task, in which feedback was given immediately (after 800 ms) or delayed (after 6800 ms). PD patients were impaired in immediate but not delayed feedback learning. However, differences in the preference for positive/negative learning between patients and controls were seen for both learning from immediate and delayed feedback, with evidence of stronger negative learning in patients than controls. A Bayesian analysis of the data supports the conclusion that feedback timing did not affect the learning bias in the patients. These results hint at reduced, but still relevant nigro-striatal contribution to feedback learning, when feedback is delayed.

What does dopamine mean?

Dopamine is a critical modulator of both learning and motivation. This presents a problem: how can target cells know whether increased dopamine is a signal to learn or to move? It is often presumed that motivation involves slow ('tonic') dopamine changes, while fast ('phasic') dopamine fluctuations convey reward prediction errors for learning. Yet recent studies have shown that dopamine conveys motivational value and promotes movement even on subsecond timescales. Here I describe an alternative account of how dopamine regulates ongoing behavior. Dopamine release related to motivation is rapidly and locally sculpted by receptors on dopamine terminals, independently from dopamine cell firing. Target neurons abruptly switch between learning and performance modes, with striatal cholinergic interneurons providing one candidate switch mechanism. The behavioral impact of dopamine varies by subregion, but in each case dopamine provides a dynamic estimate of whether it is worth expending a limited internal resource, such as energy, attention, or time.

The Protective Action Encoding of Serotonin Transients in the Human Brain

The role of serotonin in human brain function remains elusive due, at least in part, to our inability to measure rapidly the local concentration of this neurotransmitter. We used fast-scan cyclic voltammetry to infer serotonergic signaling from the striatum of 14 brains of human patients with Parkinson's disease. Here we report these novel measurements and show that they correlate with outcomes and decisions in a sequential investment game. We find that serotonergic concentrations transiently increase as a whole following negative reward prediction errors, while reversing when counterfactual losses predominate. This provides initial evidence that the serotonergic system acts as an opponent to dopamine signaling, as anticipated by theoretical models. Serotonin transients on one trial were also associated with actions on the next trial in a manner that correlated with decreased exposure to poor outcomes. Thus, the fluctuations observed for serotonin appear to correlate with the inhibition of over-reactions and promote persistence of ongoing strategies in the face of short-term environmental changes. Together these findings elucidate a role for serotonin in the striatum, suggesting it encodes a protective action strategy that mitigates risk and modulates choice selection particularly following negative environmental events.

Perceptual decisions based on previously learned information are independent of dopaminergic tone

Both cognitive and motor symptoms in people with Parkinson's disease (PD) arise from either too little or too much dopamine (DA). Akinesia stems from DA neuronal cell loss, and dyskinesia often stems from an overdose of DA medication. Cognitive behaviors typically associated with frontal cortical function, such as working memory and task switching, are also affected by too little or too much DA in PD. Whether motor and cognitive circuits overlap in PD is unknown. In this article, we show that whereas motor performance improves in people with PD when on dopaminergic medication compared with off medication, perceptual decision-making based on previously learned information (priors) remains impaired whether on or off medications. To rule out effects of long-term DA treatment and dopaminergic neuronal loss such as occur in PD, we also tested a group of people with dopa-unresponsive focal dystonia, a disease that involves the basal ganglia, like PD, but has motor symptoms that are insensitive to dopamine treatment and is not thought to involve frontal cortical DA circuits, unlike PD. We found that people with focal dystonia showed intact perceptual decision-making performance but impaired use of priors in perceptual decision-making, similar to people with PD. Together, the results show a dissociation between motor and cognitive performance in people with PD and reveal a novel cognitive impairment, independent of sensory and motor impairment, in people with focal dystonia. The combined results from people with PD and people with focal dystonia provide mechanistic insights into the role of basal ganglia non-dopaminergic circuits in perceptual decision-making based on priors.

Mechanisms Underlying Dopamine-Induced Risky Choice in Parkinson's Disease With and Without Depression (History)

Patients with Parkinson’s disease (PD) are often treated with dopaminergic medication. Dopaminergic medication is known to improve both motor and certain nonmotor symptoms, such as depression. However, it can contribute to behavioral impairment, for example, by enhancing risky choice. Here we characterize the computational mechanisms that contribute to dopamine-induced changes in risky choice in PD patients with and without a depression (history). We adopt a clinical–neuroeconomic approach to investigate the effects of dopaminergic medication on specific components of risky choice in PD. Twenty-three healthy controls, 21 PD patients with a depression (history), and 22 nondepressed PD patients were assessed using a well-established risky choice paradigm. Patients were tested twice: once after taking their normal dopaminergic medication and once after withdrawal of their medication. Dopaminergic medication increased a value-independent gambling propensity in nondepressed PD patients, while leaving loss aversion unaffected. By contrast, dopaminergic medication effects on loss aversion were associated with current depression severity and with drug effects on depression scores. The present findings demonstrate that dopaminergic medication increases a value-independent gambling bias in nondepressed PD patients. Moreover, the current study raises the hypothesis that dopamine-induced reductions in loss aversion might underlie previously observed comorbidity between depression and medication-related side effects in PD, such as impulse control disorder.

Novel reinforcement learning paradigm based on response patterning under interval schedules of reinforcement

There is a need to develop cognitive tasks that address valid neuropsychological constructs implicated in disease mechanisms and can be used in animals and humans to guide novel drug discovery. Present experiments aimed to characterize a novel reinforcement learning task based on a classical operant behavioral phenomenon observed in multiple species - differences in response patterning under variable (VI) vs fixed interval (FI) schedules of reinforcement. Wistar rats were trained to press a lever for food under VI30s and later weekly test sessions were introduced with reinforcement schedule switched to FI30s. During the FI30s test session, post-reinforcement pauses (PRPs) gradually grew towards the end of the session reaching 22-43% of the initial values. Animals could be retrained under VI30s conditions, and FI30s test sessions were repeated over a period of several months without appreciable signs of a practice effect. Administration of the non-competitive N-methyl-d-aspartate (NMDA) receptor antagonist MK-801 ((5S,10R)-(+)-5-Methyl-10,11-dihydro-5H-dibenzo[a,d]cyclohepten-5,10-imine maleate) prior to FI30s sessions prevented adjustment of PRPs associated with the change from VI to FI schedule. This effect was most pronounced at the highest tested dose of MK-801 and appeared to be independent of the effects of this dose on response rates. These results provide initial evidence for the possibility to use different response patterning under VI and FI schedules with equivalent reinforcement density for studying effects of drug treatment on reinforcement learning.

Dopaminergic Medication Increases Reliance on Current Information in Parkinson's Disease

The neurotransmitter dopamine is crucial for decision-making under uncertainty but its computational role is still a subject of intense debate. To test potential roles, we had patients with Parkinson's disease (PD), who have less internally-generated dopamine, participate in a visual decision-making task in which uncertainty in both prior and current sensory information was varied and where behavior is often predicted by Bayesian statistics. We found that many aspects of uncertainty processing were conserved in PD: they could learn the prior uncertainty and utilize both priors and current sensory information. As predicted by prominent theories, we found that dopaminergic medication influenced the weight given to sensory information. However, as PD patients learn, this bias disappeared. In addition, throughout the experiment the patients exhibited lower sensitivity to current sensory uncertainty. Our results provide empirical evidence for the idea that dopamine levels, which are affected by PD and the drugs used for its treatment, influence the reliance on new information.

Effects of dopamine on reinforcement learning and consolidation in Parkinson's disease

Emerging evidence suggests that dopamine may modulate learning and memory with important implications for understanding the neurobiology of memory and future therapeutic targeting. An influential hypothesis posits that dopamine biases reinforcement learning. More recent data also suggest an influence during both consolidation and retrieval. Eighteen Parkinson's disease patients learned through feedback ON or OFF medication, with memory tested 24 hr later ON or OFF medication (4 conditions, within-subjects design with matched healthy control group). Patients OFF medication during learning decreased in memory accuracy over the following 24 hr. In contrast to previous studies, however, dopaminergic medication during learning and testing did not affect expression of positive or negative reinforcement. Two further experiments were run without the 24 hr delay, but they too failed to reproduce effects of dopaminergic medication on reinforcement learning. While supportive of a dopaminergic role in consolidation, this study failed to replicate previous findings on reinforcement learning.

Decision Making and Sequential Sampling from Memory

Decisions take time, and as a rule more difficult decisions take more time. But this only raises the question of what consumes the time. For decisions informed by a sequence of samples of evidence, the answer is straightforward: more samples are available with more time. Indeed, the speed and accuracy of such decisions are explained by the accumulation of evidence to a threshold or bound. However, the same framework seems to apply to decisions that are not obviously informed by sequences of evidence samples. Here, we proffer the hypothesis that the sequential character of such tasks involves retrieval of evidence from memory. We explore this hypothesis by focusing on value-based decisions and argue that mnemonic processes can account for regularities in choice and decision time. We speculate on the neural mechanisms that link sampling of evidence from memory to circuits that represent the accumulated evidence bearing on a choice. We propose that memory processes may contribute to a wider class of decisions that conform to the regularities of choice-reaction time predicted by the sequential sampling framework.

Catecholaminergic challenge uncovers distinct Pavlovian and instrumental mechanisms of motivated (in)action

Catecholamines modulate the impact of motivational cues on action. Such motivational biases have been proposed to reflect cue-based, 'Pavlovian' effects. Here, we assess whether motivational biases may also arise from asymmetrical instrumental learning of active and passive responses following reward and punishment outcomes. We present a novel paradigm, allowing us to disentangle the impact of reward and punishment on instrumental learning from Pavlovian response biasing. Computational analyses showed that motivational biases reflect both Pavlovian and instrumental effects: reward and punishment cues promoted generalized (in)action in a Pavlovian manner, whereas outcomes enhanced instrumental (un)learning of chosen actions. These cue- and outcome-based biases were altered independently by the catecholamine enhancer melthylphenidate. Methylphenidate's effect varied across individuals with a putative proxy of baseline dopamine synthesis capacity, working memory span. Our study uncovers two distinct mechanisms by which motivation impacts behaviour, and helps refine current models of catecholaminergic modulation of motivated action.

Fatigue modulates dopamine availability and promotes flexible choice reversals during decision making

During decisions, animals balance goal achievement and effort management. Despite physical exercise and fatigue significantly affecting the levels of effort that an animal exerts to obtain a reward, their role in effort-based choice and the underlying neurochemistry are incompletely known. In particular, it is unclear whether fatigue influences decision (cost-benefit) strategies flexibly or only post-decision action execution and learning. To answer this question, we trained mice on a T-maze task in which they chose between a high-cost, high-reward arm (HR), which included a barrier, and a low-cost, low-reward arm (LR), with no barrier. The animals were parametrically fatigued immediately before the behavioural tasks by running on a treadmill. We report a sharp choice reversal, from the HR to LR arm, at 80% of their peak workload (PW), which was temporary and specific, as the mice returned to choose the HC when the animals were successively tested at 60% PW or in a two-barrier task. These rapid reversals are signatures of flexible choice. We also observed increased subcortical dopamine levels in fatigued mice: a marker of individual bias to use model-based control in humans. Our results indicate that fatigue levels can be incorporated in flexible cost-benefits computations that improve foraging efficiency.

Fidelity of the representation of value in decision-making

The ability to make optimal decisions depends on evaluating the expected rewards associated with different potential actions. This process is critically dependent on the fidelity with which reward value information can be maintained in the nervous system. Here we directly probe the fidelity of value representation following a standard reinforcement learning task. The results demonstrate a previously-unrecognized bias in the representation of value: extreme reward values, both low and high, are stored significantly more accurately and precisely than intermediate rewards. The symmetry between low and high rewards pertained despite substantially higher frequency of exposure to high rewards, resulting from preferential exploitation of more rewarding options. The observed variation in fidelity of value representation retrospectively predicted performance on the reinforcement learning task, demonstrating that the bias in representation has an impact on decision-making. A second experiment in which one or other extreme-valued option was omitted from the learning sequence showed that representational fidelity is primarily determined by the relative position of an encoded value on the scale of rewards experienced during learning. Both variability and guessing decreased with the reduction in the number of options, consistent with allocation of a limited representational resource. These findings have implications for existing models of reward-based learning, which typically assume defectless representation of reward value.

Many models of learning and decision-making assume that experienced rewards are stored without error. We examined this assumption experimentally: participants first learned an association between different options and rewards in a simple two-alternative choice task. We then asked them to report what reward they expected to receive for each of the options they had experienced. We checked that the reports we collected matched performance on the choice task, meaning that the values participants reported were the same as those they used to decide between options. The results showed that participants were both less precise (greater variability) and less accurate (greater bias) in their reports of middling reward values compared to either high- or low-valued options. Reports of high and low values were similar in quality even though participants had experienced the rewards associated with high-value options considerably more often. Whether an option’s value was stored well or poorly was not fixed, but instead depended on how the value compared to other options the participant had experienced. These results should lead to better models of how decisions are made based on experiences of reward.

Modafinil alters decision making based on feedback history - a randomized placebo-controlled double blind study in humans

Modafinil is becoming increasingly popular as a cognitive enhancer. Research on the effects of modafinil on cognitive function have yielded mixed results, with negative findings for simple memory and attention tasks and enhancing effects for more complex tasks. In the present study we examined whether modafinil, due to its known effect on the dopamine level in the striatum, alters feedback-related choice behaviour. We applied a task that separately tests the choice of previously rewarded behaviours (approach) and avoidance of previously punished behaviours. 18 participants received a single dose of 200 mg modafinil. Their performance was compared to a group of 22 participants who received placebo in a double-blind design. Modafinil but not placebo induced a significant bias towards approach behaviour as compared to the frequency of avoidance behaviour. General attention, overall feedback-based acquisition of choice behaviour and reaction times in high vs low conflict choices were not significantly affected by modafinil. This finding suggests that modafinil has a specific effect on dopamine-mediated choice behaviour based on the history of feedback, while a contribution of noradrenaline is also conceivable. The described change in decision making cannot be considered as cognitive enhancement, but might rather have detrimental effects on decisions in everyday life.

An Integrative Perspective on the Role of Dopamine in Schizophrenia

We propose that schizophrenia involves a combination of decreased phasic dopamine responses for relevant stimuli and increased spontaneous phasic dopamine release. Using insights from computational reinforcement-learning models and basic-science studies of the dopamine system, we show that each of these two disturbances contributes to a specific symptom domain and explains a large set of experimental findings associated with that domain. Reduced phasic responses for relevant stimuli help to explain negative symptoms and provide a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with negative symptoms: reduced learning from rewards; blunted activation of the ventral striatum, midbrain, and other limbic regions for rewards and positive prediction errors; blunted activation of the ventral striatum during reward anticipation; blunted autonomic responding for relevant stimuli; blunted neural activation for aversive outcomes and aversive prediction errors; reduced willingness to expend effort for rewards; and psychomotor slowing. Increased spontaneous phasic dopamine release helps to explain positive symptoms and provides a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with positive symptoms: aberrant learning for neutral cues (assessed with behavioral and autonomic responses), and aberrant, increased activation of the ventral striatum, midbrain, and other limbic regions for neutral cues, neutral outcomes, and neutral prediction errors. Taken together, then, these two disturbances explain many findings in schizophrenia. We review evidence supporting their co-occurrence and consider their differential implications for the treatment of positive and negative symptoms.