A role for dopamine in temporal decision making and reward maximization in parkinsonism

Exploration-Exploitation and Suicidal Behavior in Borderline Personality Disorder and Depression

Importance

Clinical theory and behavioral studies suggest that people experiencing suicidal crisis are often unable to find constructive solutions or incorporate useful information into their decisions, resulting in premature convergence on suicide and neglect of better alternatives. However, prior studies of suicidal behavior have not formally examined how individuals resolve the tradeoffs between exploiting familiar options and exploring potentially superior alternatives.

Objective

To investigate exploration and exploitation in suicidal behavior from the formal perspective of reinforcement learning.

Design, Setting, and Participants

Two case-control behavioral studies of exploration-exploitation of a large 1-dimensional continuous space and a 21-day prospective ambulatory study of suicidal ideation were conducted between April 2016 and March 2022. Participants were recruited from inpatient psychiatric units, outpatient clinics, and the community in Pittsburgh, Pennsylvania, and underwent laboratory and ambulatory assessments. Adults diagnosed with borderline personality disorder (BPD) and midlife and late-life major depressive disorder (MDD) were included, with each sample including demographically equated groups with a history of high-lethality suicide attempts, low-lethality suicide attempts, individuals with BPD or MDD but no suicide attempts, and control individuals without psychiatric disorders. The MDD sample also included a subgroup with serious suicidal ideation.

Main Outcomes and Measures

Behavioral (model-free and model-derived) indices of exploration and exploitation, suicide attempt lethality (Beck Lethality Scale), and prospectively assessed suicidal ideation.

Results

The BPD group included 171 adults (mean [SD] age, 30.55 [9.13] years; 135 [79%] female). The MDD group included 143 adults (mean [SD] age, 62.03 [6.82] years; 81 [57%] female). Across the BPD (χ23 = 50.68; P < .001) and MDD (χ24 = 36.34; P < .001) samples, individuals with high-lethality suicide attempts discovered fewer options than other groups as they were unable to shift away from unrewarded options. In contrast, those with low-lethality attempts were prone to excessive behavioral shifts after rewarded and unrewarded actions. No differences were seen in strategic early exploration or in exploitation. Among 84 participants with BPD in the ambulatory study, 56 reported suicidal ideation. Underexploration also predicted incident suicidal ideation (χ21 = 30.16; P < .001), validating the case-control results prospectively. The findings were robust to confounds, including medication exposure, affective state, and behavioral heterogeneity.

Conclusions and Relevance

The findings suggest that narrow exploration and inability to abandon inferior options are associated with serious suicidal behavior and chronic suicidal thoughts. By contrast, individuals in this study who engaged in low-lethality suicidal behavior displayed a low threshold for taking potentially disadvantageous actions.

Valence-dependent dopaminergic modulation during reversal learning in Parkinson's disease: A neurocomputational approach

Reinforcement learning, crucial for behavior in dynamic environments, is driven by rewards and punishments, modulated by dopamine (DA) changes. This study explores the dopaminergic system's influence on learning, particularly in Parkinson's disease (PD), where medication leads to impaired adaptability. Highlighting the role of tonic DA in signaling the valence of actions, this research investigates how DA affects response vigor and decision-making in PD. DA not only influences reward and punishment learning but also indicates the cognitive effort level and risk propensity in actions, which are essential for understanding and managing PD symptoms. In this work, we adapt our existing neurocomputational model of basal ganglia (BG) to simulate two reversal learning tasks proposed by Cools et al. We first optimized a Hebb rule for both probabilistic and deterministic reversal learning, conducted a sensitivity analysis (SA) on parameters related to DA effect, and compared performances between three groups: PD-ON, PD-OFF, and control subjects. In our deterministic task simulation, we explored switch error rates after unexpected task switches and found a U-shaped relationship between tonic DA levels and switch error frequency. Through SA, we classify these three groups. Then, assuming that the valence of the stimulus affects the tonic levels of DA, we were able to reproduce the results by Cools et al. As for the probabilistic task simulation, our results are in line with clinical data, showing similar trends with PD-ON, characterized by higher tonic DA levels that are correlated with increased difficulty in both acquisition and reversal tasks. Our study proposes a new hypothesis: valence, signaled by tonic DA levels, influences learning in PD, confirming the uncorrelation between phasic and tonic DA changes. This hypothesis challenges existing paradigms and opens new avenues for understanding cognitive processes in PD, particularly in reversal learning tasks.

Beyond discrete-choice options

While decision theories have evolved over the past five decades, their focus has largely been on choices among a limited number of discrete options, even though many real-world situations have a continuous-option space. Recently, theories have attempted to address decisions with continuous-option spaces, and several computational models have been proposed within the sequential sampling framework to explain how we make a decision in continuous-option space. This article aims to review the main attempts to understand decisions on continuous-option spaces, give an overview of applications of these types of decisions, and present puzzles to be addressed by future developments.

Meta-Analysis Reveals That Explore-Exploit Decisions are Dissociable by Activation in the Dorsal Lateral Prefrontal Cortex, Anterior Insula, and the Anterior Cingulate Cortex

Explore-exploit research faces challenges in generalizability due to a limited theoretical basis for exploration and exploitation. Neuroimaging can help identify whether explore-exploit decisions involve an opponent processing system to address this issue. Thus, we conducted a coordinate-based meta-analysis (N=23 studies) finding activation in the dorsal lateral prefrontal cortex, anterior insula, and anterior cingulate cortex during exploration versus exploitation, which provides some evidence for opponent processing. However, the conjunction of explore-exploit decisions was associated with activation in the dorsal anterior cingulate cortex and dorsal medial prefrontal cortex, suggesting that these brain regions do not engage in opponent processing. Furthermore, exploratory analyses revealed heterogeneity in brain responses between task types during exploration and exploitation respectively. Coupled with results suggesting that activation during exploration and exploitation decisions is generally more similar than it is different suggests, there remain significant challenges in characterizing explore-exploit decision making. Nonetheless, dlPFC, AI, and ACC activation differentiate explore and exploit decisions and identifying these responses can aid in targeted interventions aimed at manipulating these decisions.

Reward-based option competition in human dorsal stream and transition from stochastic exploration to exploitation in continuous space

Primates exploring and exploiting a continuous sensorimotor space rely on dynamic maps in the dorsal stream. Two complementary perspectives exist on how these maps encode rewards. Reinforcement learning models integrate rewards incrementally over time, efficiently resolving the exploration/exploitation dilemma. Working memory buffer models explain rapid plasticity of parietal maps but lack a plausible exploration/exploitation policy. The reinforcement learning model presented here unifies both accounts, enabling rapid, information-compressing map updates and efficient transition from exploration to exploitation. As predicted by our model, activity in human frontoparietal dorsal stream regions, but not in MT+, tracks the number of competing options, as preferred options are selectively maintained on the map, while spatiotemporally distant alternatives are compressed out. When valuable new options are uncovered, posterior β1/α oscillations desynchronize within 0.4 to 0.7 s, consistent with option encoding by competing β1-stabilized subpopulations. Together, outcomes matching locally cached reward representations rapidly update parietal maps, biasing choices toward often-sampled, rewarded options.

A computational neuroimaging study of reinforcement learning and goal-directed exploration in schizophrenia spectrum disorders

BACKGROUND

Prior evidence indicates that negative symptom severity and cognitive deficits, in people with schizophrenia (PSZ), relate to measures of reward-seeking and loss-avoidance behavior (implicating the ventral striatum/VS), as well as uncertainty-driven exploration (reliant on rostrolateral prefrontal cortex/rlPFC). While neural correlates of reward-seeking and loss-avoidance have been examined in PSZ, neural correlates of uncertainty-driven exploration have not. Understanding neural correlates of uncertainty-driven exploration is an important next step that could reveal insights to how this mechanism of cognitive and negative symptoms manifest at a neural level.

METHODS

We acquired fMRI data from 29 PSZ and 36 controls performing the Temporal Utility Integration decision-making task. Computational analyses estimated parameters corresponding to learning rates for both positive and negative reward prediction errors (RPEs) and the degree to which participates relied on representations of relative uncertainty. Trial-wise estimates of expected value, certainty, and RPEs were generated to model fMRI data.

RESULTS

Behaviorally, PSZ demonstrated reduced reward-seeking behavior compared to controls, and negative symptoms were positively correlated with loss-avoidance behavior. This finding of a bias toward loss avoidance learning in PSZ is consistent with previous work. Surprisingly, neither behavioral measures of exploration nor neural correlates of uncertainty in the rlPFC differed significantly between groups. However, we showed that trial-wise estimates of relative uncertainty in the rlPFC distinguished participants who engaged in exploratory behavior from those who did not. rlPFC activation was positively associated with intellectual function.

CONCLUSIONS

These results further elucidate the nature of reinforcement learning and decision-making in PSZ and healthy volunteers.

A practical introduction to using the drift diffusion model of decision-making in cognitive psychology, neuroscience, and health sciences

Recent years have seen a rapid increase in the number of studies using evidence-accumulation models (such as the drift diffusion model, DDM) in the fields of psychology and neuroscience. These models go beyond observed behavior to extract descriptions of latent cognitive processes that have been linked to different brain substrates. Accordingly, it is important for psychology and neuroscience researchers to be able to understand published findings based on these models. However, many articles using (and explaining) these models assume that the reader already has a fairly deep understanding of (and interest in) the computational and mathematical underpinnings, which may limit many readers' ability to understand the results and appreciate the implications. The goal of this article is therefore to provide a practical introduction to the DDM and its application to behavioral data - without requiring a deep background in mathematics or computational modeling. The article discusses the basic ideas underpinning the DDM, and explains the way that DDM results are normally presented and evaluated. It also provides a step-by-step example of how the DDM is implemented and used on an example dataset, and discusses methods for model validation and for presenting (and evaluating) model results. Supplementary material provides R code for all examples, along with the sample dataset described in the text, to allow interested readers to replicate the examples themselves. The article is primarily targeted at psychologists, neuroscientists, and health professionals with a background in experimental cognitive psychology and/or cognitive neuroscience, who are interested in understanding how DDMs are used in the literature, as well as some who may to go on to apply these approaches in their own work.

Humans adaptively resolve the explore-exploit dilemma under cognitive constraints: Evidence from a multi-armed bandit task

When navigating uncertain worlds, humans must balance exploring new options versus exploiting known rewards. Longer horizons and spatially structured option values encourage humans to explore, but the impact of real-world cognitive constraints such as environment size and memory demands on explore-exploit decisions is unclear. In the present study, humans chose between options varying in uncertainty during a multi-armed bandit task with varying environment size and memory demands. Regression and cognitive computational models of choice behavior showed that with a lower cognitive load, humans are more exploratory than a simulated value-maximizing learner, but under cognitive constraints, they adaptively scale down exploration to maintain exploitation. Thus, while humans are curious, cognitive constraints force people to decrease their strategic exploration in a resource-rational-like manner to focus on harvesting known rewards.

Daytime and season do not affect reinforcement learning capacity in a response time adjustment task

Seasonal and circadian rhythms have a broad impact on physiological aspects, such as dopamine neurotransmission, and may be involved in the etiology of mood disorders. Considering this, studies on the influence of season and daytime on cognitive function are rare. The present study aimed to assess the impact of seasonal and diurnal effects on the ability to maximize reward outcomes by optimizing response times adaptively. For this purpose, a reward-based learning task that required an adaptation of response time to either a fast or a slow response was used. Eighty German participants (mean age ± SD = 21.86 ± 1.89 years, 41 women) were examined twice, in the morning and in the evening. Half of the participants were tested during the summer, while the other half performed the test in the winter. No impact of daytime, season or of the external factors photoperiodicity and temperature on reinforcement learning could be found. However, a generally slower response speed in the morning compared to the evening appeared. Previously conducted tasks could not display behavioral differences in both times of season and daytime, although neurophysiological findings suggest it.

Reinforcement-based cognitive biases as vulnerability factors in alcohol addiction: From humans to animal models

Alcohol use disorder (AUD) is one of the most common, but still poorly treated, psychiatric conditions. Developing new treatments requires a better understanding of the aetiology of symptoms and evaluation of novel therapeutic targets in preclinical studies. Recent developments in our understanding of the reinforcement-based cognitive biases (RBCBs) that contribute to the development of AUD and its treatment offer new opportunities for both clinical and preclinical research. In this review, we first briefly describe psychological and cognitive theories that involve various aspects of reinforcement sensitivity in the development, maintenance, and recurrence of alcohol addiction. Furthermore, in separate sections, we describe studies investigating RBCBs and their neural, neurochemical, and pharmacological correlates, and we discuss possible interactions between RBCBs and trajectories of AUD. Finally, we describe how recent translational studies using state-of-the-art animal models can facilitate our understanding of the role of reinforcement sensitivity and RBCBs in various aspects of AUD.

The Role of User Behaviour in Improving Cyber Security Management

Information security has for long time been a field of study in computer science, software engineering, and information communications technology. The term 'information security' has recently been replaced with the more generic term cybersecurity. The goal of this paper is to show that, in addition to computer science studies, behavioural sciences focused on user behaviour can provide key techniques to help increase cyber security and mitigate the impact of attackers' social engineering and cognitive hacking methods (i.e., spreading false information). Accordingly, in this paper, we identify current research on psychological traits and individual differences among computer system users that explain vulnerabilities to cyber security attacks and crimes. Our review shows that computer system users possess different cognitive capabilities which determine their ability to counter information security threats. We identify gaps in the existing research and provide possible psychological methods to help computer system users comply with security policies and thus increase network and information security.

Reinforcement learning abnormalities in the attenuated psychosis syndrome and first episode psychosis

Prior studies indicate that chronic schizophrenia (SZ) is associated with a specific profile of reinforcement learning abnormalities. These impairments are characterized by: 1) reductions in learning rate, and 2) impaired Go learning and intact NoGo learning. Furthermore, each of these deficits are associated with greater severity of negative symptoms, consistent with theoretical perspectives positing that avolition and anhedonia are associated with impaired value representation. However, it is unclear whether these deficits extend to earlier phases of psychotic illness and when individuals are unmedicated. Two studies were conducted to examine reinforcement learning deficits in earlier phases of psychosis and in high risk patients. In study 1, participants included 35 participants with first episode psychosis (FEP) with limited antipsychotic medication exposure and 25 healthy controls (HC). Study 2 included 17 antipsychotic naïve individuals who were at clinical high-risk for psychosis (CHR) (i.e., attenuated psychosis syndrome) and 18 matched healthy controls (HC). In both studies, participants completed the Temporal Utility Integration Task, a measure of probabilistic reinforcement learning that contained Go and NoGo learning blocks. FEP displayed impaired Go and NoGo learning. In contrast, CHR did not display impairments in Go or NoGo learning. Impaired Go learning was not significantly associated with clinical outcomes in the CHR or FEP samples. Findings provide new evidence for areas of spared and impaired reinforcement learning in early phases of psychosis.

Neural Mechanisms of Human Decision-Making

We present a theory and neural network model of the neural mechanisms underlying human decision-making. We propose a detailed model of the interaction between brain regions, under a proposer-predictor-actor-critic​ ​framework. This theory is based on detailed animal data and theories of action-selection. Those theories are adapted to serial operation to bridge levels of analysis and explain human decision-making. Task-relevant areas of cortex propose a candidate plan using fast, model-free, parallel neural computations. Other areas of cortex and medial temporal lobe can then predict​ likely outcomes of that plan in this situation. This optional prediction- (or model-) based computation can produce better accuracy and generalization, at the expense of speed. Next, linked regions of basal ganglia act​ to accept or reject the proposed plan based on its reward history in similar contexts. If that plan is rejected, the process repeats to consider a new option. The reward-prediction system acts as a critic​ to determine the value of the outcome relative to expectations and produce dopamine as a training signal for cortex and basal ganglia. By operating sequentially and hierarchically, the same mechanisms previously proposed for animal action-selection could explain the most complex human plans and decisions. We discuss explanations of model-based decisions, habitization, and risky behavior based on the computational model.

Effects of dopamine on reinforcement learning in Parkinson's disease depend on motor phenotype

Parkinson's disease is clinically defined by bradykinesia, along with rigidity and tremor. However, the severity of these motor signs is greatly variable between individuals, particularly the presence or absence of tremor. This variability in tremor relates to variation in cognitive/motivational impairment, as well as the spatial distribution of neurodegeneration in the midbrain and dopamine depletion in the striatum. Here we ask whether interindividual heterogeneity in tremor symptoms could account for the puzzlingly large variability in the effects of dopaminergic medication on reinforcement learning, a fundamental cognitive function known to rely on dopamine. Given that tremor-dominant and non-tremor Parkinson's disease patients have different dopaminergic phenotypes, we hypothesized that effects of dopaminergic medication on reinforcement learning differ between tremor-dominant and non-tremor patients. Forty-three tremor-dominant and 20 non-tremor patients with Parkinson's disease were recruited to be tested both OFF and ON dopaminergic medication (200/50 mg levodopa-benserazide), while 22 age-matched control subjects were recruited to be tested twice OFF medication. Participants performed a reinforcement learning task designed to dissociate effects on learning rate from effects on motivational choice (i.e. the tendency to 'Go/NoGo' in the face of reward/threat of punishment). In non-tremor patients, dopaminergic medication improved reward-based choice, replicating previous studies. In contrast, in tremor-dominant patients, dopaminergic medication improved learning from punishment. Formal modelling showed divergent computational effects of dopaminergic medication as a function of Parkinson's disease motor phenotype, with a modulation of motivational choice bias and learning rate in non-tremor and tremor patients, respectively. This finding establishes a novel cognitive/motivational difference between tremor and non-tremor Parkinson's disease patients, and highlights the importance of considering motor phenotype in future work.

Differential reinforcement encoding along the hippocampal long axis helps resolve the explore-exploit dilemma

When making decisions, should one exploit known good options or explore potentially better alternatives? Exploration of spatially unstructured options depends on the neocortex, striatum, and amygdala. In natural environments, however, better options often cluster together, forming structured value distributions. The hippocampus binds reward information into allocentric cognitive maps to support navigation and foraging in such spaces. Here we report that human posterior hippocampus (PH) invigorates exploration while anterior hippocampus (AH) supports the transition to exploitation on a reinforcement learning task with a spatially structured reward function. These dynamics depend on differential reinforcement representations in the PH and AH. Whereas local reward prediction error signals are early and phasic in the PH tail, global value maximum signals are delayed and sustained in the AH body. AH compresses reinforcement information across episodes, updating the location and prominence of the value maximum and displaying goal cell-like ramping activity when navigating toward it.

Dopamine modulates individual differences in avoidance behavior: A pharmacological, immunohistochemical, neurochemical and volumetric investigation

Avoidance behavior is a hallmark in pathological anxiety disorders and results in impairment of daily activities. Individual differences in avoidance responses are critical in determining vulnerability or resistance to anxiety disorders. Dopaminergic activation is implicated in the processing of avoidance responses; however, the mechanisms underlying these responses are unknown. In this sense, we used a preclinical model of avoidance behavior to investigate the possibility of an intrinsic differential dopaminergic pattern between good and poor performers. The specific goal was to assess the participation of dopamine (DA) through pharmacological manipulation, and we further evaluated the effects of systemic injections of the dopaminergic receptor type 1 (D1 antagonist - SCH23390) and dopaminergic receptor type 2 (D2 antagonist - sulpiride) antagonists in the good performers. Additionally, we evaluated the effects of intra-amygdala microinjection of a D1 antagonist (SCH23390) and a D2 antagonist (sulpiride) in good performers as well as intra-amygdala microinjection of a D1 agonist (SKF38393) and D2 agonist (quinpirole) in poor performers. Furthermore, we quantified the contents of dopamine and metabolites (3,4-dihydroxyphenylacetic acid (DOPAC) and homovanillic acid (HVA)) in the amygdala, evaluated the basal levels of tyrosine hydroxylase expression (catecholamine synthesis enzyme) and measured the volume of the substantia nigra, ventral tegmental area and locus coeruleus. Our results showed that it could be possible to convert animals from good to poor performers, and vice versa, by intra-amygdala (basolateral and central nucleus) injections of D1 receptor antagonists in good performers or D2 receptor agonists in poor performers. Additionally, the good performers had lower levels of DOPAC and HVA in the amygdala, an increase in the total volume of the amygdala (AMG), substantia nigra (SN), ventral tegmental area (VTA) and locus coeruleus (LC), and an increase in the number of tyrosine hydroxylase-positive cells in SN, VTA and LC, which positively correlates with the avoidance behavior. Taken together, our data show evidence for a dopaminergic signature of avoidance performers, emphasizing the role of distinct dopaminergic receptors in individual differences in avoidance behavior based on pharmacological, immunohistochemical, neurochemical and volumetric analyses. Our findings provide a better understanding of the role of the dopaminergic system in the execution of avoidance behavior.

•

The role of dopamine in individual differences in avoidance behavior.

•

Dopamine modulates avoidance behavior.

•

Dopaminergic evidence of individual difference in avoidance behavior.

•

Good and poor avoiders distinction based on dopaminergic signature.

•

Dopaminergic signature of avoidance performers: poor versus good avoiders.

Mathematical modeling and parameter estimation of levodopa motor response in patients with parkinson disease

Parkinson disease (PD) is characterized by a clear beneficial motor response to levodopa (LD) treatment. However, with disease progression and longer LD exposure, drug-related motor fluctuations usually occur. Recognition of the individual relationship between LD concentration and its effect may be difficult, due to the complexity and variability of the mechanisms involved. This work proposes an innovative procedure for the automatic estimation of LD pharmacokinetics and pharmacodynamics parameters, by a biologically-inspired mathematical model. An original issue, compared with previous similar studies, is that the model comprises not only a compartmental description of LD pharmacokinetics in plasma and its effect on the striatal neurons, but also a neurocomputational model of basal ganglia action selection. Parameter estimation was achieved on 26 patients (13 with stable and 13 with fluctuating LD response) to mimic plasma LD concentration and alternate finger tapping frequency along four hours after LD administration, automatically minimizing a cost function of the difference between simulated and clinical data points. Results show that individual data can be satisfactorily simulated in all patients and that significant differences exist in the estimated parameters between the two groups. Specifically, the drug removal rate from the effect compartment, and the Hill coefficient of the concentration-effect relationship were significantly higher in the fluctuating than in the stable group. The model, with individualized parameters, may be used to reach a deeper comprehension of the PD mechanisms, mimic the effect of medication, and, based on the predicted neural responses, plan the correct management and design innovative therapeutic procedures.

Selective maintenance of value information helps resolve the exploration/exploitation dilemma

In natural environments with many options of uncertain value, one faces a difficult tradeoff between exploiting familiar, valuable options or searching for better alternatives. Reinforcement learning models of this exploration/exploitation dilemma typically modulate the rate of exploratory choices or preferentially sample uncertain options. The extent to which such models capture human behavior remains unclear, in part because they do not consider the constraints on remembering what is learned. Using reinforcement-based timing as a motivating example, we show that selectively maintaining high-value actions compresses the amount of information to be tracked in learning, as quantified by Shannon's entropy. In turn, the information content of the value representation controls the balance between exploration (high entropy) and exploitation (low entropy). Selectively maintaining preferred action values while allowing others to decay renders the choices increasingly exploitative across learning episodes. To adjudicate among alternative maintenance and sampling strategies, we developed a new reinforcement learning model, StrategiC ExPloration/ExPloitation of Temporal Instrumental Contingencies (SCEPTIC). In computational studies, a resource-rational selective maintenance approach was as successful as more resource-intensive strategies. Furthermore, human behavior was consistent with selective maintenance; information compression was most pronounced in subjects with superior performance and non-verbal intelligence, and in learnable vs. unlearnable contingencies. Cognitively demanding uncertainty-directed exploration recovered a more accurate representation in simulations with no foraging advantage and was strongly unsupported in our human study.

Interpersonal dysfunction in borderline personality: a decision neuroscience perspective

Borderline personality disorder (BPD) is characterized by disadvantageous decisions that are often expressed in close relationships and associated with intense negative emotions. Although functional neuroimaging studies of BPD have described regions associated with altered social cognition and emotion processing, these correlates do not inform an understanding of how brain activity leads to maladaptive choices. Drawing on recent research, we argue that formal models of decision-making are crucial to elaborating theories of BPD that bridge psychological constructs, behavior, and neural systems. We propose that maladaptive interactions between Pavlovian and instrumental influences play a crucial role in the expression of interpersonal problems. Finally, we articulate specific hypotheses about how clinical features of BPD may map onto neural systems that implement separable decision processes.

High monetary reward rates and caloric rewards decrease temporal persistence

Temporal persistence refers to an individual's capacity to wait for future rewards, while forgoing possible alternatives. This requires a trade-off between the potential value of delayed rewards and opportunity costs, and is relevant to many real-world decisions, such as dieting. Theoretical models have previously suggested that high monetary reward rates, or positive energy balance, may result in decreased temporal persistence. In our study, 50 fasted participants engaged in a temporal persistence task, incentivised with monetary rewards. In alternating blocks of this task, rewards were delivered at delays drawn randomly from distributions with either a lower or higher maximum reward rate. During some blocks participants received either a caloric drink or water. We used survival analysis to estimate participants' probability of quitting conditional on the delay distribution and the consumed liquid. Participants had a higher probability of quitting in blocks with the higher reward rate. Furthermore, participants who consumed the caloric drink had a higher probability of quitting than those who consumed water. Our results support the predictions from the theoretical models, and importantly, suggest that both higher monetary reward rates and physiologically relevant rewards can decrease temporal persistence, which is a crucial determinant for survival in many species.

DAT1-Genotype and Menstrual Cycle, but Not Hormonal Contraception, Modulate Reinforcement Learning: Preliminary Evidence

Hormone by genotype interactions have been widely ignored by cognitive neuroscience. Yet, the dependence of cognitive performance on both baseline dopamine (DA) and current 17ß-estradiol (E2) level argues for their combined effect also in the context of reinforcement learning. Here, we assessed how the interaction between the natural rise of E2 in the late follicular phase (FP) and the 40 base-pair variable number tandem repeat polymorphism of the dopamine transporter (DAT1) affects reinforcement learning capacity. 30 women with a regular menstrual cycle performed a probabilistic feedback learning task twice during the early and late FP. In addition, 39 women, who took hormonal contraceptives (HC) to suppress natural ovulation, were tested during the "pill break" and the intake phase of HC. The present data show that DAT1-genotype may interact with transient hormonal state, but only in women with a natural menstrual cycle. We found that carriers of the 9-repeat allele (9RP) experienced a significant decrease in the ability to avoid punishment from early to late FP. Neither homozygote subjects of the 10RP allele, nor subjects from the HC group showed a change in behavior between phases. These data are consistent with neurobiological studies that found that rising E2 may reverse DA transporter function and could enhance DA efflux, which would in turn reduce punishment sensitivity particularly in subjects with a higher transporter density to begin with. Taken together, the present results, although based on a small sample, add to the growing understanding of the complex interplay between different physiological modulators of dopaminergic transmission. They may not only point out the necessity to control for hormonal state in behavioral genetic research, but may offer new starting points for studies in clinical settings.

Reward learning deficits in Parkinson's disease depend on depression

BACKGROUND

Depression is one of the most common and debilitating non-motor symptoms of Parkinson's disease (PD). The neurocognitive mechanisms underlying depression in PD are unclear and treatment is often suboptimal.

METHODS

We investigated the role of striatal dopamine in reversal learning from reward and punishment by combining a controlled medication withdrawal procedure with functional magnetic resonance imaging in 22 non-depressed PD patients and 19 PD patients with past or present depression.

RESULTS

PD patients with a depression (history) exhibited impaired reward v. punishment reversal learning as well as reduced reward v. punishment-related BOLD signal in the striatum (putamen) compared with non-depressed PD patients. No effects of dopaminergic medication were observed.

CONCLUSIONS

The present findings demonstrate that impairments in reversal learning from reward v. punishment and associated striatal signalling depend on the presence of (a history of) depression in PD.

A Primer on Foraging and the Explore/Exploit Trade-Off for Psychiatry Research

Foraging is a fundamental behavior, and many types of animals appear to have solved foraging problems using a shared set of mechanisms. Perhaps the most common foraging problem is the choice between exploiting a familiar option for a known reward and exploring unfamiliar options for unknown rewards-the so-called explore/exploit trade-off. This trade-off has been studied extensively in behavioral ecology and computational neuroscience, but is relatively new to the field of psychiatry. Explore/exploit paradigms can offer psychiatry research a new approach to studying motivation, outcome valuation, and effort-related processes, which are disrupted in many mental and emotional disorders. In addition, the explore/exploit trade-off encompasses elements of risk-taking and impulsivity-common behaviors in psychiatric disorders-and provides a novel framework for understanding these behaviors within an ecological context. Here we explain relevant concepts and some common paradigms used to measure explore/exploit decisions in the laboratory, review clinically relevant research on the neurobiology and neuroanatomy of explore/exploit decision making, and discuss how computational psychiatry can benefit from foraging theory.

Comprehensive review: Computational modelling of schizophrenia

Computational modelling has been used to address: (1) the variety of symptoms observed in schizophrenia using abstract models of behavior (e.g. Bayesian models - top-down descriptive models of psychopathology); (2) the causes of these symptoms using biologically realistic models involving abnormal neuromodulation and/or receptor imbalance (e.g. connectionist and neural networks - bottom-up realistic models of neural processes). These different levels of analysis have been used to answer different questions (i.e. understanding behavioral vs. neurobiological anomalies) about the nature of the disorder. As such, these computational studies have mostly supported diverging hypotheses of schizophrenia's pathophysiology, resulting in a literature that is not always expanding coherently. Some of these hypotheses are however ripe for revision using novel empirical evidence. Here we present a review that first synthesizes the literature of computational modelling for schizophrenia and psychotic symptoms into categories supporting the dopamine, glutamate, GABA, dysconnection and Bayesian inference hypotheses respectively. Secondly, we compare model predictions against the accumulated empirical evidence and finally we identify specific hypotheses that have been left relatively under-investigated.

Dopamine Selectively Modulates the Outcome of Learning Unnatural Action-Valence Associations

Learning the contingencies between stimulus, action, and outcomes is disrupted in disorders associated with altered dopamine (DA) function in the BG, such as Parkinson disease (PD). Although the role of DA in learning to act has been extensively investigated in PD, the role of DA in "learning to withhold" (or inhibit) action to influence outcomes is not as well understood. The current study investigated the role of DA in learning to act or to withhold action to receive rewarding, or avoid punishing outcomes, in patients with PD tested "off" and "on" dopaminergic medication (n = 19) versus healthy controls (n = 30). Participants performed a reward-based learning task that orthogonalized action and outcome valence (action-reward, inaction-reward, action-punishment, inaction-punishment). We tested whether DA would bias learning toward action, toward reward, or to particular action-outcome interactions. All participants demonstrated inherent learning biases preferring action with reward and inaction to avoid punishment, and this was unaffected by medication. Instead, DA produced a complex modulation of learning less natural action-outcome associations. "Off" DA medication, patients demonstrated impairments in learning to withhold action to gain reward, suggesting a difficulty to overcome a bias toward associating inaction with punishment avoidance. On DA medication, these patterns changed, and patients showed a reduced ability to learn to act to avoid punishment, indicating a bias toward action and reward. The current findings suggest that DA in PD has a complex influence on the formation of action-outcome associations, particularly those involving less natural linkages between action and outcome valence.

Modeling Trait Anxiety: From Computational Processes to Personality

Computational methods are increasingly being applied to the study of psychiatric disorders. Often, this involves fitting models to the behavior of individuals with subclinical character traits that are known vulnerability factors for the development of psychiatric conditions. Anxiety disorders can be examined with reference to the behavior of individuals high in "trait" anxiety, which is a known vulnerability factor for the development of anxiety and mood disorders. However, it is not clear how this self-report measure relates to neural and behavioral processes captured by computational models. This paper reviews emerging computational approaches to the study of trait anxiety, specifying how interacting processes susceptible to analysis using computational models could drive a tendency to experience frequent anxious states and promote vulnerability to the development of clinical disorders. Existing computational studies are described in the light of this perspective and appropriate targets for future studies are discussed.

Levodopa impairs probabilistic reversal learning in healthy young adults

RATIONALE

Dopaminergic therapy improves some cognitive functions and worsens others in patients with Parkinson's disease (PD). These paradoxical effects are explained by the dopamine overdose hypothesis, which proposes that effects of dopaminergic therapy on a cognitive function is determined by the baseline dopamine levels in brain regions mediating that function.

OBJECTIVES

We directly tested this prevalent hypothesis, evaluating the effects of levodopa on stimulus-reward learning in healthy young adults, who presumably have optimal baseline dopamine levels and dopamine regulation.

METHODS

Twenty-six healthy, young adults completed a probabilistic reversal learning task in a randomized, double-blind, placebo-controlled, crossover design. Participants completed one session on levodopa 100 mg/carbidopa 25 mg and another session on placebo.

RESULTS

We found that levodopa impaired reversal learning relative to placebo. Further analyses revealed that levodopa impaired learning from both punishment and reward.

CONCLUSIONS

Exogenous dopamine impairs stimulus-reward learning, independent of PD pathology and prior to sensitization through repeated exposure, in healthy adults with normal cognition and baseline dopamine function. Our findings support the dopamine overdose hypothesis and caution clinicians about detrimental effects of levodopa in all clinical populations (e.g., early PD, restless leg syndrome) regardless of baseline cognitive and dopaminergic system function.

Working memory gating mechanisms explain developmental change in rule-guided behavior

Cognitive control requires choosing contextual information to update into working memory (input gating), maintaining it there (maintenance) stable against distraction, and then choosing which subset of maintained information to use in guiding action (output gating). Recent work has raised the possibility that the development of rule-guided behavior, in the transition from childhood to adolescence, is linked specifically to changes in the gating components of working memory (Amso, Haas, McShane, & Badre, 2014). Given the importance of effective rule-guided behavior for decision making in this developmental transition, we used hierarchical rule tasks to probe the precise developmental dynamics of working memory gating. This mechanistic precision informs ongoing efforts to train cognitive control and working memory operations across typical and atypical development. The results of Experiment 1 verified that the development of rule-guided behavior is uniquely linked to increasing hierarchical complexity but not to increasing maintenance demands across 1st, 2nd, and 3rd order rule tasks. Experiment 2 then investigated whether this developmental trajectory in rule-guided behavior is best explained by change in input gating or output gating. Further, as input versus output gating also tend to correlate with a more proactive versus reactive control strategy in these tasks, we assessed developmental change in the degree to which these two processes were deployed efficiently given the task. Experiment 2 shows that the developmental change observed in Experiment 1 and in Amso et al. (2014) is likely a result of increased efficacy of output gating processes, as well as greater strategic efficiency in that adolescents opt for this costly process less often than children.

Neural and neurochemical basis of reinforcement-guided decision making

Decision making is an adaptive behavior that takes into account several internal and external input variables and leads to the choice of a course of action over other available and often competing alternatives. While it has been studied in diverse fields ranging from mathematics, economics, ecology, and ethology to psychology and neuroscience, recent cross talk among perspectives from different fields has yielded novel descriptions of decision processes. Reinforcement-guided decision making models are based on economic and reinforcement learning theories, and their focus is on the maximization of acquired benefit over a defined period of time. Studies based on reinforcement-guided decision making have implicated a large network of neural circuits across the brain. This network includes a wide range of cortical (e.g., orbitofrontal cortex and anterior cingulate cortex) and subcortical (e.g., nucleus accumbens and subthalamic nucleus) brain areas and uses several neurotransmitter systems (e.g., dopaminergic and serotonergic systems) to communicate and process decision-related information. This review discusses distinct as well as overlapping contributions of these networks and neurotransmitter systems to the processing of decision making. We end the review by touching on neural circuitry and neuromodulatory regulation of exploratory decision making.

The Role of Informative and Ambiguous Feedback in Avoidance Behavior: Empirical and Computational Findings

Avoidance behavior is a critical component of many psychiatric disorders, and as such, it is important to understand how avoidance behavior arises, and whether it can be modified. In this study, we used empirical and computational methods to assess the role of informational feedback and ambiguous outcome in avoidance behavior. We adapted a computer-based probabilistic classification learning task, which includes positive, negative and no-feedback outcomes; the latter outcome is ambiguous as it might signal either a successful outcome (missed punishment) or a failure (missed reward). Prior work with this task suggested that most healthy subjects viewed the no-feedback outcome as strongly positive. Interestingly, in a later version of the classification task, when healthy subjects were allowed to opt out of (i.e. avoid) responding, some subjects (“avoiders”) reliably avoided trials where there was a risk of punishment, but other subjects (“non-avoiders”) never made any avoidance responses at all. One possible interpretation is that the “non-avoiders” valued the no-feedback outcome so positively on punishment-based trials that they had little incentive to avoid. Another possible interpretation is that the outcome of an avoided trial is unspecified and that lack of information is aversive, decreasing subjects’ tendency to avoid. To examine these ideas, we here tested healthy young adults on versions of the task where avoidance responses either did or did not generate informational feedback about the optimal response. Results showed that provision of informational feedback decreased avoidance responses and also decreased categorization performance, without significantly affecting the percentage of subjects classified as “avoiders.” To better understand these results, we used a modified Q-learning model to fit individual subject data. Simulation results suggest that subjects in the feedback condition adjusted their behavior faster following better-than-expected outcomes, compared to subjects in the no-feedback condition. Additionally, in both task conditions, “avoiders” adjusted their behavior faster following worse-than-expected outcomes, and treated the ambiguous no-feedback outcome as less rewarding, compared to non-avoiders. Together, results shed light on the important role of ambiguous and informative feedback in avoidance behavior.

A Biologically Inspired Computational Model of Basal Ganglia in Action Selection

The basal ganglia (BG) are a subcortical structure implicated in action selection. The aim of this work is to present a new cognitive neuroscience model of the BG, which aspires to represent a parsimonious balance between simplicity and completeness. The model includes the 3 main pathways operating in the BG circuitry, that is, the direct (Go), indirect (NoGo), and hyperdirect pathways. The main original aspects, compared with previous models, are the use of a two-term Hebb rule to train synapses in the striatum, based exclusively on neuronal activity changes caused by dopamine peaks or dips, and the role of the cholinergic interneurons (affected by dopamine themselves) during learning. Some examples are displayed, concerning a few paradigmatic cases: action selection in basal conditions, action selection in the presence of a strong conflict (where the role of the hyperdirect pathway emerges), synapse changes induced by phasic dopamine, and learning new actions based on a previous history of rewards and punishments. Finally, some simulations show model working in conditions of altered dopamine levels, to illustrate pathological cases (dopamine depletion in parkinsonian subjects or dopamine hypermedication). Due to its parsimonious approach, the model may represent a straightforward tool to analyze BG functionality in behavioral experiments.

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

Motivational Modulation of Self-Initiated and Externally Triggered Movement Speed Induced by Threat of Shock: Experimental Evidence for Paradoxical Kinesis in Parkinson's Disease

Background

Paradoxical kinesis has been observed in bradykinetic people with Parkinson’s disease. Paradoxical kinesis occurs in situations where an individual is strongly motivated or influenced by relevant external cues. Our aim was to induce paradoxical kinesis in the laboratory. We tested whether the motivation of avoiding a mild electric shock was sufficient to induce paradoxical kinesis in externally-triggered and self-initiated conditions in people with Parkinson’s disease tested on medication and in age-matched controls.

Methods

Participants completed a shock avoidance behavioural paradigm in which half of the trials could result in a mild electric shock if the participant did not move fast enough. Half of the trials of each type were self-initiated and half were externally-triggered. The criterion for avoiding shock was a maximum movement time, adjusted according to each participant’s performance on previous trials using a staircase tracking procedure.

Results

On trials with threat of shock, both patients with Parkinson’s disease and controls had faster movement times compared to no potential shock trials, in both self-initiated and externally-triggered conditions. The magnitude of improvement of movement time from no potential shock to potential shock trials was positively correlated with anxiety ratings.

Conclusions

When motivated to avoid mild electric shock, patients with Parkinson’s disease, similar to healthy controls, showed significant speeding of movement execution. This was observed in both self-initiated and externally-triggered versions of the task. Nevertheless, in the ET condition the improvement of reaction times induced by motivation to avoid shocks was greater for the PD patients than controls, highlighting the value of external cues for movement initiation in PD patients. The magnitude of improvement from the no potential shock to the potential shock trials was associated with the threat-induced anxiety. This demonstration of paradoxical kinesis in the laboratory under both self-initiated and externally-triggered conditions has implications for motivational and attentional enhancement of movement speed in Parkinson’s disease.

Be quick about it. Endogenous estradiol level, menstrual cycle phase and trait impulsiveness predict impulsive choice in the context of reward acquisition

This article is part of a Special Issue "Estradiol and Cognition". Variations in the steroid hormone 17ß-estradiol (E2) may promote intra-individual differences in reward seeking behavior and temporal decision-making (Reimers et al., 2014; Front. Neurosci. 8: 401). Yet, in humans the exact role of E2 in impulsive choice still needs to be determined. The present study assessed the effect of a cycle-dependent rise in endogenous E2 on temporal response adaptation across the follicular phase (FP). For this purpose a reward acquisition paradigm was employed that is sensitive to hormone-induced changes in central dopamine (DA) level. The present data show that women acted more impulsively in the early as opposed to the late FP. Early follicular E2 further correlated with an increased capacity to speed up for reward maximization, while simultaneously the ability to wait for higher reward was compromised. This correlation was most pronounced in women with low trait impulsiveness. In contrast, E2 and optimized response speed failed to correlate in women with high trait impulsiveness and in the late FP, despite a generally higher E2 level. Collectively, these findings support the theory that E2 may act as an endogenous DA agonist. The fact that the hormone-behavior relationship was restricted to women with low trait impulsiveness and thus supposedly lower central DA level provides indirect support for this idea. Yet, choices became relatively less impulsive in the state of heightened E2 (i.e., in the late FP), suggesting that the relationship between E2 and impulsive choice may not be linear, but might resemble an inverted U-function.

The influence of trial order on learning from reward vs. punishment in a probabilistic categorization task: experimental and computational analyses

Previous research has shown that trial ordering affects cognitive performance, but this has not been tested using category-learning tasks that differentiate learning from reward and punishment. Here, we tested two groups of healthy young adults using a probabilistic category learning task of reward and punishment in which there are two types of trials (reward, punishment) and three possible outcomes: (1) positive feedback for correct responses in reward trials; (2) negative feedback for incorrect responses in punishment trials; and (3) no feedback for incorrect answers in reward trials and correct answers in punishment trials. Hence, trials without feedback are ambiguous, and may represent either successful avoidance of punishment or failure to obtain reward. In Experiment 1, the first group of subjects received an intermixed task in which reward and punishment trials were presented in the same block, as a standard baseline task. In Experiment 2, a second group completed the separated task, in which reward and punishment trials were presented in separate blocks. Additionally, in order to understand the mechanisms underlying performance in the experimental conditions, we fit individual data using a Q-learning model. Results from Experiment 1 show that subjects who completed the intermixed task paradoxically valued the no-feedback outcome as a reinforcer when it occurred on reinforcement-based trials, and as a punisher when it occurred on punishment-based trials. This is supported by patterns of empirical responding, where subjects showed more win-stay behavior following an explicit reward than following an omission of punishment, and more lose-shift behavior following an explicit punisher than following an omission of reward. In Experiment 2, results showed similar performance whether subjects received reward-based or punishment-based trials first. However, when the Q-learning model was applied to these data, there were differences between subjects in the reward-first and punishment-first conditions on the relative weighting of neutral feedback. Specifically, early training on reward-based trials led to omission of reward being treated as similar to punishment, but prior training on punishment-based trials led to omission of reward being treated more neutrally. This suggests that early training on one type of trials, specifically reward-based trials, can create a bias in how neutral feedback is processed, relative to those receiving early punishment-based training or training that mixes positive and negative outcomes.

Cognitive correlates of psychosis in patients with Parkinson's disease

INTRODUCTION

Psychosis and hallucinations occur in 20-30% of patients with Parkinson's disease (PD). In the current study, we investigate cognitive functions in relation to the occurrence of psychosis in PD patients.

METHODS

We tested three groups of subjects - PD with psychosis, PD without psychosis and healthy controls - on working memory, learning and transitive inference tasks, which are known to assess prefrontal, basal ganglia and hippocampal functions.

RESULTS

In the working memory task, results show that patients with and without psychosis were more impaired than the healthy control group. In the transitive inference task, we did not find any difference among the groups in the learning phase performance. Importantly, PD patients with psychosis were more impaired than both PD patients without psychosis and controls at transitive inference. We also found that the severity of psychotic symptoms in PD patients [as measured by the Unified Parkinson Disease Rating Scale Thought Disorder (UPDRS TD) item] is directly associated with the severity of cognitive impairment [as measured by the mini-mental status exam (MMSE)], sleep disturbance [as measured by the Scales for Outcome in Parkinson Disease (SCOPA) sleep scale] and transitive inference (although the latter did not reach significance).

CONCLUSIONS

Although hypothetical, our data may suggest that the hippocampus is a neural substrate underlying the occurrence of psychosis, sleep disturbance and cognitive impairment in PD patients.

Linking neuroscience with modern concepts of impulse control disorders in Parkinson's disease

Patients with Parkinson's disease (PD) may experience impulse control disorders (ICDs) when on dopamine agonist therapy for their motor symptoms. In the last few years, a rapid growth of interest for the recognition of these aberrant behaviors and their neurobiological correlates has occurred. Recent advances in neuroimaging are helping to identify the neuroanatomical networks responsible for these ICDs, and together with psychopharmacological assessments are providing new insights into the brain status of impulsive behavior. The genetic associations that may be unique to ICDs in PD are also being identified. Complementing human studies, electrophysiological and biochemical studies in animal models are providing insights into neuropathological mechanisms associated with these disorders. New animal models of ICDs in PD patients are being implemented that should provide critical means to identify efficacious therapies for PD-related motor deficits while avoiding ICD side effects. Here, we provide an overview of these recent advances, with a particular emphasis on the neurobiological correlates reported in animal models and patients along with their genetic underpinnings.

Age moderates the effect of acute dopamine depletion on passive avoidance learning

Despite extensive links between reinforcement-based learning and dopamine (DA), studies to date have not found consistent effects of acute DA reduction on reinforcement learning in both men and women. Here, we tested the effects of reducing DA on reward- and punishment-based learning using the deterministic passive avoidance learning (PAL) task. We tested 16 (5 female) adults (ages 22-40) in a randomized, cross-over design to determine whether reducing global DA by administering an amino acid beverage deficient in the DA precursors, phenylalanine and tyrosine (P/T[-]), would affect PAL task performance. We found that P/T[-] beverage effects on PAL performance were modulated by age. Specifically, we found that P/T depletion significantly improved learning from punishment with increasing participant age. Participants committed 1.49 fewer passive avoidance errors per additional year of age (95% CI, -0.71 - -2.27, r=-0.74, p=0.001). Moreover, P/T depletion improved learning from punishment in adults (ages 26-40) while it impaired learning from punishment in emerging adults (ages 22-25). We observed similar, but non-significant trends in learning from reward. While there was no overall effect of P/T-depletion on reaction time (RT), there was a relationship between the effect of P/T depletion on PAL performance and RT; those who responded more slowly on the P/T[-] beverage also made more errors on the P/T[-] beverage. When P/T-depletion slowed RT after a correct response, there was a worsening of PAL task performance; there was no similar relationship for the RT after an incorrect response and PAL task performance. Moreover, among emerging adults, changes in mood on the P/T[-] beverage negatively correlated with learning from reward on the P/T[-] beverage. Together, we found that both reward- and punishment-based learning are sensitive to central catecholamine levels, and that these effects of acute DA reduction vary with age.

Striatal D1 and D2 signaling differentially predict learning from positive and negative outcomes

The extent to which we learn from positive and negative outcomes of decisions is modulated by the neurotransmitter dopamine. Dopamine neurons burst fire in response to unexpected rewards and pause following negative outcomes. This dual signaling mechanism is hypothesized to drive both approach and avoidance behavior. Here we test a prediction deriving from a computational reinforcement learning model, in which approach is mediated via activation of the direct cortico-striatal pathway due to striatal D1 receptor stimulation, while avoidance occurs via disinhibition of indirect pathway striatal neurons secondary to a reduction of D2 receptor stimulation. Using positron emission tomography with two separate radioligands, we demonstrate that individual differences in human approach and avoidance learning are predicted by variability in striatal D1 and D2 receptor binding, respectively. Moreover, transient dopamine precursor depletion improved learning from negative outcomes. These findings support a bidirectional modulatory role for striatal dopamine in reward and avoidance learning via segregated D1 and D2 cortico-striatal pathways.

Dopamine, locus of control, and the exploration-exploitation tradeoff

Whether to continue to exploit a source of reward, or to search for a new one of potentially greater value, is a fundamental and underconstrained decision. Recent computational studies of this exploration-exploitation tradeoff have found that variability in exploration across individuals is influenced by a functional polymorphism (Val158Met) in the catechol-O-methyltransferase (COMT) gene, whose protein product degrades synaptically released dopamine. However, these and other genotype-phenotype associations have rarely been causally tested. To directly test this association and to evaluate additional behavioral characteristics, including perceived locus of control (LOC), here we used the COMT inhibitor tolcapone in a randomized, double-blind, counterbalanced, within-subject study of 66 subjects genotyped for the Val158Met allele to assess the hypothesis that reducing COMT enzymatic activity interacts with genotype to increase uncertainty-driven exploration. In keeping with our initial hypothesis, tolcapone led to an increase in exploratory, but not exploitative, behavior in Met/Met rather than Val/Val subjects. Independent of genotype, those subjects with a more external LOC also showed increases in uncertainty-driven exploration on tolcapone relative to placebo. However, we did not replicate our previous finding that Met/Met subjects show greater exploration at baseline. Together these findings support a model in which exploration is hypothesized to have a dopaminergic basis. Moreover, in keeping with findings in other behavioral and cognitive domains, the response to an increase in presumptively frontal dopamine is dependent upon baseline dopamine tone.

Cognitive deficits in Parkinson's disease: a cognitive neuroscience perspective

Progress in characterization of the nature, neural basis, and treatment of cognitive deficits in Parkinson's disease is reviewed from the perspective of cognitive neuroscience. An initial emphasis on fronto-striatal executive deficits is surveyed along with the discoveries of disruption as well as remediation of certain impairments by dopaminergic mediation and their association with theories of reinforcement learning. Subsequent focus on large cohorts has revealed considerable heterogeneity in the cognitive impairments as well as a suggestion of at least two distinct syndromes, with the dopamine-dependent fronto-striatal deficits being somewhat independent of other signs commonly associated with Parkinson's disease dementia. The utility is proposed of a new, integrated cognitive neuroscience approach based on combining genetic and neuroimaging methodologies with neuropsychological and, ultimately, psychopharmacological approaches.