Dopamine, learning, and impulsivity: a biological account of attention-deficit/hyperactivity disorder

Working memory and inhibitory control deficits in children with ADHD: an experimental evaluation of competing model predictions

Introduction

Children with ADHD demonstrate difficulties on many different neuropsychological tests. However, it remains unclear whether this pattern reflects a large number of distinct deficits or a small number of deficit(s) that broadly impact test performance. The current study is among the first experiments to systematically manipulate demands on both working memory and inhibition, with implications for competing conceptual models of ADHD pathogenesis.

Method

A clinically evaluated, carefully phenotyped sample of 110 children with ADHD, anxiety disorders, or co-occurring ADHD+anxiety (Mage=10.35, 44 girls; 69% White Not Hispanic/Latino) completed a counterbalanced, double dissociation experiment, with two tasks each per inhibition (low vs. high) x working memory (low vs. high) condition.

Results

Bayesian and frequentist models converged in indicating that both manipulations successfully increased demands on their target executive function (BF10>5.33x108, p<.001). Importantly, occupying children's limited capacity working memory system produced slower response times and reduced accuracy on inhibition tasks (BF10>317.42, p<.001, d=0.67-1.53). It also appeared to differentially reduce inhibition (and non-inhibition) accuracy for children with ADHD relative to children with anxiety (BF10=2.03, p=.02, d=0.50). In contrast, there was strong evidence against models that view working memory deficits as secondary outcomes of underlying inhibition deficits in ADHD (BF01=18.52, p=.85).

Discussion

This pattern indicates that working memory broadly affects children's ability to inhibit prepotent tendencies and maintain fast/accurate performance, and may explain the errors that children with ADHD make on inhibition tests. These findings are broadly consistent with models describing working memory as a causal mechanism that gives rise to secondary impairments. In contrast, these findings provide evidence against models that view disinhibition as a cause of working memory difficulties or view working memory as a non-causal correlate or epiphenomenon in ADHD.

Using rodent data to elucidate dopaminergic mechanisms of ADHD: Implications for human personality

An altered behavioral response to positive reinforcement has been proposed to be a core deficit in attention deficit hyperactivity disorder (ADHD). The spontaneously hypertensive rat (SHR), a congenic animal strain, displays a similarly altered response to reinforcement. The presence of this genetically determined phenotype in a rodent model allows experimental investigation of underlying neural mechanisms. Behaviorally, the SHR displays increased preference for immediate reinforcement, increased sensitivity to individual instances of reinforcement relative to integrated reinforcement history, and a steeper delay of reinforcement gradient compared to other rat strains. The SHR also shows less development of incentive to approach sensory stimuli, or cues, that predict reward after repeated cue-reward pairing. We consider the underlying neural mechanisms for these characteristics. It is well known that midbrain dopamine neurons are initially activated by unexpected reward and gradually transfer their responses to reward-predicting cues. This finding has inspired the dopamine transfer deficit (DTD) hypothesis, which predicts certain behavioral effects that would arise from a deficient transfer of dopamine responses from actual rewards to reward-predicting cues. We argue that the DTD predicts the altered responses to reinforcement seen in the SHR and individuals with ADHD. These altered responses to reinforcement in turn predict core symptoms of ADHD. We also suggest that variations in the degree of dopamine transfer may underlie variations in personality dimensions related to altered reinforcement sensitivity. In doing so, we highlight the value of rodent models to the study of human personality.

Biological Predictors of Treatment Response in Adult Attention Deficit Hyperactivity Disorder (ADHD): A Systematic Review

Background: Attention-deficit/hyperactivity disorder (ADHD) is a highly prevalent condition with onset in childhood and in many cases persisting into adulthood. Even though an increasing number of studies have investigated the efficacy of pharmacotherapy in the management of adult ADHD, few authors have tried to identify the biological predictors of treatment response. Objectives: To summarize the available data about the biological markers of treatment response in adults affected by ADHD. Methods: A search on the main biomedical and psychological archives (PubMed, Embase, Scopus, and PsycINFO) was performed. Manuscripts in English, published up to May 2022 and having the biological predictors of treatment response in adults with ADHD as their main topic, were included. Results: A total of 3855 articles was screened. Twenty-two articles were finally included. Most of the manuscripts studied neuroimaging and electrophysiological factors as potential predictors of treatment response in adult ADHD patients. No reliable markers were identified until now. Promising findings on this topic regard genetic polymorphisms in snap receptor (SNARE) proteins and default mode network-striatum connectivity. Conclusions: Even though some biological markers seem promising for the prediction of treatment response in adults affected by ADHD, further studies are needed to confirm the available data in the context of precision medicine.

Neuroprotection in late life attention-deficit/hyperactivity disorder: A review of pharmacotherapy and phenotype across the lifespan

For decades, psychostimulants have been the gold standard pharmaceutical treatment for attention-deficit/hyperactivity disorder (ADHD). In the United States, an astounding 9% of all boys and 4% of girls will be prescribed stimulant drugs at some point during their childhood. Recent meta-analyses have revealed that individuals with ADHD have reduced brain volume loss later in life (>60 y.o.) compared to the normal aging brain, which suggests that either ADHD or its treatment may be neuroprotective. Crucially, these neuroprotective effects were significant in brain regions (e.g., hippocampus, amygdala) where severe volume loss is linked to cognitive impairment and Alzheimer's disease. Historically, the ADHD diagnosis and its pharmacotherapy came about nearly simultaneously, making it difficult to evaluate their effects in isolation. Certain evidence suggests that psychostimulants may normalize structural brain changes typically observed in the ADHD brain. If ADHD itself is neuroprotective, perhaps exercising the brain, then psychostimulants may not be recommended across the lifespan. Alternatively, if stimulant drugs are neuroprotective, then this class of medications may warrant further investigation for their therapeutic effects. Here, we take a bottom-up holistic approach to review the psychopharmacology of ADHD in the context of recent models of attention. We suggest that future studies are greatly needed to better appreciate the interactions amongst an ADHD diagnosis, stimulant treatment across the lifespan, and structure-function alterations in the aging brain.

Value-free random exploration is linked to impulsivity

Deciding whether to forgo a good choice in favour of exploring a potentially more rewarding alternative is one of the most challenging arbitrations both in human reasoning and in artificial intelligence. Humans show substantial variability in their exploration, and theoretical (but only limited empirical) work has suggested that excessive exploration is a critical mechanism underlying the psychiatric dimension of impulsivity. In this registered report, we put these theories to test using large online samples, dimensional analyses, and computational modelling. Capitalising on recent advances in disentangling distinct human exploration strategies, we not only demonstrate that impulsivity is associated with a specific form of exploration—value-free random exploration—but also explore links between exploration and other psychiatric dimensions.

The Stage 1 protocol for this Registered Report was accepted in principle on 19/03/2021. The protocol, as accepted by the journal, can be found at 10.6084/m9.figshare.14346506.v1.

Deciding between known rewarding options and exploring novel avenues is central to decision making. Humans show variability in their exploration. Here, the authors show that impulsivity is associated to an increased usage of a cognitively cheap (and sometimes sub-optimal) exploration strategy.

Self-Administration of Right Vagus Nerve Stimulation Activates Midbrain Dopaminergic Nuclei

Background: Left cervical vagus nerve stimulation (l-VNS) is an FDA-approved treatment for neurological disorders including epilepsy, major depressive disorder, and stroke, and l-VNS is increasingly under investigation for a range of other neurological indications. Traditional l-VNS is thought to induce therapeutic neuroplasticity in part through the coordinated activation of multiple broadly projecting neuromodulatory systems in the brain. Recently, it has been reported that striking lateralization exists in the anatomical and functional connectivity between the vagus nerves and the dopaminergic midbrain. These emerging findings suggest that VNS-driven activation of this important plasticity-promoting neuromodulatory system may be preferentially driven by targeting the right, rather than the left, cervical nerve.

Objective: To compare the effects of right cervical VNS (r-VNS) vs. traditional l-VNS on self-administration behavior and midbrain dopaminergic activation in rats.

Methods: Rats were implanted with a stimulating cuff electrode targeting either the right or left cervical vagus nerve. After surgical recovery, rats underwent a VNS self-administration assay in which lever pressing was paired with r-VNS or l-VNS delivery. Self-administration was followed by extinction, cue-only reinstatement, and stimulation reinstatement sessions. Rats were sacrificed 90 min after completion of behavioral training, and brains were removed for immunohistochemical analysis of c-Fos expression in the dopaminergic ventral tegmental area (VTA) and substantia nigra pars compacta (SNc), as well as in the noradrenergic locus coeruleus (LC).

Results: Rats in the r-VNS cohort performed significantly more lever presses throughout self-administration and reinstatement sessions than did rats in the l-VNS cohort. Moreover, this appetitive behavioral responding was associated with significantly greater c-Fos expression among neuronal populations within the VTA, SNc, and LC. Differential c-Fos expression following r-VNS vs. l-VNS was particularly prominent within dopaminergic midbrain neurons.

Conclusion: Our results support the existence of strong lateralization within vagal-mesencephalic signaling pathways, and suggest that VNS targeted to the right, rather than left, cervical nerve preferentially activates the midbrain dopaminergic system. These findings raise the possibility that r-VNS could provide a promising strategy for enhancing dopamine-dependent neuroplasticity, opening broad avenues for future research into the efficacy and safety of r-VNS in the treatment of neurological disease.

Adaptive aspects of impulsivity and interactions with effects of catecholaminergic agents in the 5-choice serial reaction time task: implications for ADHD

BACKGROUND

Work in humans has shown that impulsivity can be advantageous in certain settings. However, evidence for so-called functional impulsivity is lacking in experimental animals.

AIMS

This study investigated the contexts in which high impulsive (HI) rats show an advantage in performance compared with mid- (MI) and low impulsive (LI) rats. We also assessed the effects of dopaminergic and noradrenergic agents to investigate underlying neurotransmitter mechanisms.

METHODS

We tested rats on a variable inter-trial interval (ITI) version of the 5-choice serial reaction time task (5CSRTT). Rats received systemic injections of methylphenidate (MPH, 1 mg/kg and 3 mg/kg), atomoxetine (ATO, 0.3 mg/kg and 1 mg/kg), amphetamine (AMPH, 0.2 mg/kg), the alpha-2a adrenoceptor antagonist atipamezole (ATI, 0.3 mg/kg) and the alpha-1 adrenoceptor agonist phenylephrine (PHEN, 1 mg/kg) prior to behavioural testing.

RESULTS

Unlike LI rats, HI rats exhibited superior performance, earning more reinforcers, on short ITI trials, when the task required rapid responding. MPH, AMPH and ATI improved performance on short ITI trials and increased impulsivity in long ITI trials, recapitulating the behavioural profile of HI. In contrast, ATO and PHEN impaired performance on short ITI trials and decreased impulsivity, thus mimicking the behavioural profile of LI rats. The effects of ATO were greater on MI rats and LI rats.

CONCLUSIONS

These findings indicate that impulsivity can be advantageous when rapid focusing and actions are required, an effect that may depend on increased dopamine neurotransmission. Conversely, activation of the noradrenergic system, with ATO and PHEN, led to a general inhibition of responding.

Probabilistic Learning in Children With Attention-Deficit/Hyperactivity Disorder

Objective: The current study examined instrumental learning in ADHD. Method: A total of 58 children with ADHD and 58 typically developing (TD) children performed a probabilistic learning task using three reward probability conditions (100%, 85%, 70% reward). After a learning phase, application of what was learned was assessed in a test phase. Results: Results showed that children with ADHD performed less accurate compared with TD children during the learning phase, particularly in the 100% and 85% reward probability conditions. These findings were accompanied by a blunted learning rate in the first few task trials. Furthermore, children with ADHD showed poorer application of what was learned. Conclusion: To conclude, children with ADHD show initial learning problems, but increased performance in a similar manner as TD children independent of the probability of reward, although they fail to apply their knowledge. Findings are of clinical relevance as the application of knowledge is important to successfully adapt to daily challenges in life.

Opposing effects of impulsivity and mindset on sources of science self-efficacy and STEM interest in adolescents

Impulsivity has been linked to academic performance in the context of Attention Deficit Hyperactivity Disorder, though its influence on a wider spectrum of students remains largely unexplored, particularly in the context of STEM learning (i.e. science, technology, engineering, and math). STEM learning was hypothesized to be more challenging for impulsive students, since it requires the practice and repetition of tasks as well as concerted attention to task performance. Impulsivity was assessed in a cross-sectional sample of 2,476 students in grades 6-12. Results show impulsivity affects a larger population of students, not limited to students with learning disabilities. Impulsivity was associated with lower sources of self-efficacy for science (SSSE), interest in most STEM domains (particularly math), and self-reported STEM skills. The large negative effect size observed for impulsivity was opposed by higher mindset, which describes a student's belief in the importance of effort when learning is difficult. Mindset had a large positive effect size associated with greater SSSE, STEM interest, and STEM skills. When modeled together, results offer that mindset interventions may benefit impulsive students who struggle with STEM. Together, these data suggest important interconnected roles for impulsivity and mindset that can influence secondary students' STEM trajectories.

A neurocomputational account of reward and novelty processing and effects of psychostimulants in attention deficit hyperactivity disorder

Computational models of reinforcement learning have helped dissect discrete components of reward-related function and characterize neurocognitive deficits in psychiatric illnesses. Stimulus novelty biases decision-making, even when unrelated to choice outcome, acting as if possessing intrinsic reward value to guide decisions toward uncertain options. Heightened novelty seeking is characteristic of attention deficit hyperactivity disorder, yet how this influences reward-related decision-making is computationally encoded, or is altered by stimulant medication, is currently uncertain. Here we used an established reinforcement-learning task to model effects of novelty on reward-related behaviour during functional MRI in 30 adults with attention deficit hyperactivity disorder and 30 age-, sex- and IQ-matched control subjects. Each participant was tested on two separate occasions, once ON and once OFF stimulant medication. OFF medication, patients with attention deficit hyperactivity disorder showed significantly impaired task performance (P = 0.027), and greater selection of novel options (P = 0.004). Moreover, persistence in selecting novel options predicted impaired task performance (P = 0.025). These behavioural deficits were accompanied by a significantly lower learning rate (P = 0.011) and heightened novelty signalling within the substantia nigra/ventral tegmental area (family-wise error corrected P < 0.05). Compared to effects in controls, stimulant medication improved attention deficit hyperactivity disorder participants' overall task performance (P = 0.011), increased reward-learning rates (P = 0.046) and enhanced their ability to differentiate optimal from non-optimal novel choices (P = 0.032). It also reduced substantia nigra/ventral tegmental area responses to novelty. Preliminary cross-sectional evidence additionally suggested an association between long-term stimulant treatment and a reduction in the rewarding value of novelty. These data suggest that aberrant substantia nigra/ventral tegmental area novelty processing plays an important role in the suboptimal reward-related decision-making characteristic of attention deficit hyperactivity disorder. Compared to effects in controls, abnormalities in novelty processing and reward-related learning were improved by stimulant medication, suggesting that they may be disorder-specific targets for the pharmacological management of attention deficit hyperactivity disorder symptoms.

Computational modeling of interventions for developmental disorders

We evaluate the potential of connectionist models of developmental disorders to offer insights into the efficacy of interventions. Based on a range of computational simulation results, we assess factors that influence the effectiveness of interventions for reading, language, and other cognitive developmental disorders. The analysis provides a level of mechanistic detail that is generally lacking in behavioral approaches to intervention. We review an extended program of modeling work in four sections. In the first, we consider long-term outcomes and the possibility of compensated outcomes and resolution of early delays. In the second section, we address methods to remediate atypical development in a single network. In the third section, we address interventions to encourage compensation via alternative pathways. In the final section, we consider the key issue of individual differences in response to intervention. Together with advances in understanding the neural basis of developmental disorders and neural responses to training, formal computational approaches can spur theoretical progress to narrow the gap between the theory and practice of intervention.

Impulsivity and Active Inference

This paper characterizes impulsive behavior using a patch-leaving paradigm and active inference-a framework for describing Bayes optimal behavior. This paradigm comprises different environments (patches) with limited resources that decline over time at different rates. The challenge is to decide when to leave the current patch for another to maximize reward. We chose this task because it offers an operational characterization of impulsive behavior, namely, maximizing proximal reward at the expense of future gain. We use a Markov decision process formulation of active inference to simulate behavioral and electrophysiological responses under different models and prior beliefs. Our main finding is that there are at least three distinct causes of impulsive behavior, which we demonstrate by manipulating three different components of the Markov decision process model. These components comprise (i) the depth of planning, (ii) the capacity to maintain and process information, and (iii) the perceived value of immediate (relative to delayed) rewards. We show how these manipulations change beliefs and subsequent choices through variational message passing. Furthermore, we appeal to the process theories associated with this message passing to simulate neuronal correlates. In future work, we will use this scheme to identify the prior beliefs that underlie different sorts of impulsive behavior-and ask whether different causes of impulsivity can be inferred from the electrophysiological correlates of choice behavior.

Dorsal anterior cingulate-brainstem ensemble as a reinforcement meta-learner

Optimal decision-making is based on integrating information from several dimensions of decisional space (e.g., reward expectation, cost estimation, effort exertion). Despite considerable empirical and theoretical efforts, the computational and neural bases of such multidimensional integration have remained largely elusive. Here we propose that the current theoretical stalemate may be broken by considering the computational properties of a cortical-subcortical circuit involving the dorsal anterior cingulate cortex (dACC) and the brainstem neuromodulatory nuclei: ventral tegmental area (VTA) and locus coeruleus (LC). From this perspective, the dACC optimizes decisions about stimuli and actions, and using the same computational machinery, it also modulates cortical functions (meta-learning), via neuromodulatory control (VTA and LC). We implemented this theory in a novel neuro-computational model–the Reinforcement Meta Learner (RML). We outline how the RML captures critical empirical findings from an unprecedented range of theoretical domains, and parsimoniously integrates various previous proposals on dACC functioning.

A major challenge for all organisms is selecting optimal behaviour to obtain resources while minimizing energetic and other expenses. Evolution provided mammals with exceptional decision-making capabilities to face this challenge. Even though neuroscientists have identified a heterogeneous and distributed set of brain structures to be involved, a comprehensive theory about the biological and computational basis of such decision-making is yet to be formulated. We propose that the interaction between the medial prefrontal cortex (a part of the frontal lobes) and the subcortical nuclei releasing catecholaminergic neuromodulators will be key to such a theory. We argue that this interaction allows both the selection of optimal behaviour and, more importantly, the optimal modulation of the very brain circuits that drive such behavioral selection (i.e., meta-learning). We implemented this theory in a novel neuro-computational model, the Reinforcement Meta-Learner (RML). By means of computer simulations we showed that the RML provides a biological and computational account for a set of neuroscientific data with unprecedented scope, thereby suggesting a critical mechanism of decision-making in the mammalian brain.

Active Inference in OpenAI Gym: A Paradigm for Computational Investigations Into Psychiatric Illness

BACKGROUND

Artificial intelligence has recently attained humanlike performance in a number of gamelike domains. These advances have been spurred by brain-inspired architectures and algorithms such as hierarchical filtering and reinforcement learning. OpenAI Gym is an open-source platform in which to train, test, and benchmark algorithms-it provides a range of tasks, including those of classic arcade games such as Doom. Here we describe how the platform might be used as a simulation, test, and diagnostic paradigm for psychiatric conditions.

METHODS

To illustrate how active inference models of game play could be used to test mechanistic and algorithmic properties of psychiatric disorders, we provide two exemplar analyses. The first speaks to the impact of aging on cognition, examining game-play behaviors in a model of aging in which we compared age-dependent changes of younger (n = 9, 22 ± 1 years of age) and older (n = 7, 56 ± 5 years of age) adult players. The second is an illustration of a putative feature of anhedonia in which we simulated diminished sensitivity to reward.

RESULTS

These simulations demonstrate how active inference can be used to test predicted changes in both neurobiology and beliefs in psychiatric cohorts. We show that, as well as behavioral measures, putative neural correlates of active inference can be simulated, and hypothesized (model-based) differences in local field potentials and blood oxygen level-dependent responses can be produced.

CONCLUSIONS

We show that active inference, through epistemic and value-based goals, enables simulated subjects to actively develop detailed representations of gaming environments, and we demonstrate the use of a principled algorithmic and neurobiological framework for testing hypotheses in psychiatric illness.

Low Cognitive Impulsivity Is Associated with Better Gain and Loss Learning in a Probabilistic Decision-Making Task

Many advances have been made over the last decades in describing, on the one hand, the link between reward-based learning and decision-making, and on the other hand, the link between impulsivity and decision-making. However, the association between reward-based learning and impulsivity remains poorly understood. In this study, we evaluated the association between individual differences in loss-minimizing and gain-maximizing behavior in a learning-based probabilistic decision-making task and individual differences in cognitive impulsivity. We found that low cognitive impulsivity was associated both with a better performance minimizing losses and maximizing gains during the task. These associations remained significant after controlling for mathematical skills and gender as potential confounders. We discuss potential mechanisms through which cognitive impulsivity might interact with reward-based learning and decision-making.

Modelling ADHD: A review of ADHD theories through their predictions for computational models of decision-making and reinforcement learning

Attention deficit hyperactivity disorder (ADHD) is characterized by altered decision-making (DM) and reinforcement learning (RL), for which competing theories propose alternative explanations. Computational modelling contributes to understanding DM and RL by integrating behavioural and neurobiological findings, and could elucidate pathogenic mechanisms behind ADHD. This review of neurobiological theories of ADHD describes predictions for the effect of ADHD on DM and RL as described by the drift-diffusion model of DM (DDM) and a basic RL model. Empirical studies employing these models are also reviewed. While theories often agree on how ADHD should be reflected in model parameters, each theory implies a unique combination of predictions. Empirical studies agree with the theories' assumptions of a lowered DDM drift rate in ADHD, while findings are less conclusive for boundary separation. The few studies employing RL models support a lower choice sensitivity in ADHD, but not an altered learning rate. The discussion outlines research areas for further theoretical refinement in the ADHD field.

Computational Psychiatry of ADHD: Neural Gain Impairments across Marrian Levels of Analysis

Attention-deficit hyperactivity disorder (ADHD), one of the most common psychiatric disorders, is characterised by unstable response patterns across multiple cognitive domains. However, the neural mechanisms that explain these characteristic features remain unclear. Using a computational multilevel approach, we propose that ADHD is caused by impaired gain modulation in systems that generate this phenotypic increased behavioural variability. Using Marr's three levels of analysis as a heuristic framework, we focus on this variable behaviour, detail how it can be explained algorithmically, and how it might be implemented at a neural level through catecholamine influences on corticostriatal loops. This computational, multilevel, approach to ADHD provides a framework for bridging gaps between descriptions of neuronal activity and behaviour, and provides testable predictions about impaired mechanisms.

Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens

Substantial evidence suggests that the phasic activity of dopamine neurons represents reinforcement learning’s temporal difference prediction error. However, recent reports of ramp-like increases in dopamine concentration in the striatum when animals are about to act, or are about to reach rewards, appear to pose a challenge to established thinking. This is because the implied activity is persistently predictable by preceding stimuli, and so cannot arise as this sort of prediction error. Here, we explore three possible accounts of such ramping signals: (a) the resolution of uncertainty about the timing of action; (b) the direct influence of dopamine over mechanisms associated with making choices; and (c) a new model of discounted vigour. Collectively, these suggest that dopamine ramps may be explained, with only minor disturbance, by standard theoretical ideas, though urgent questions remain regarding their proximal cause. We suggest experimental approaches to disentangling which of the proposed mechanisms are responsible for dopamine ramps.

Dopamine has long been implicated in reward-motivated behaviour. Theory and experiments suggest that activity of dopamine-containing neurons resembles a temporally-sophisticated prediction error used to learn expectations of future reward. This account would appear to be inconsistent with recent observations of ‘ramps’, i.e., gradual increases in extracellular dopamine concentration prior to the execution of actions or the acquisition of rewards. We explore three different possible explanations of such ramping signals as arising: (a) when subjects experience uncertainty about when actions will be executed; (b) when dopamine itself influences the timecourse of choice; and (c) under a new model in which ‘quasi-tonic’ dopamine signals arise through a form of temporal discounting. We thereby show that dopamine ramps can be integrated with current theories, and also suggest experiments to clarify which mechanisms are involved.

Lack of neuronal nitric oxide synthase results in attention deficit hyperactivity disorder-like behaviors in mice

Nitric oxide (NO) is an important molecule for the proper development and function of the central nervous system. In this study, we investigated the behavioral alterations in the neuronal NO synthase knockout mice (NOS1 KO) with a deficient NO production mechanism in the brain, characterizing it as a potential rodent model for attention deficit hyperactivity disorder (ADHD). NOS1 KO exhibited higher locomotor activity than their wildtype counterparts in a novel environment, as measured by open field (OF) test. In a 2-way active avoidance paradigm (TWAA), we found sex-dependent effects, where male KO displayed deficits in avoidance and escape behavior, sustained higher incidences of shuttle crossings, and higher incidences of intertrial interval crossings, suggesting learning, and/or performance impairments. On the other hand, female KO demonstrated few deficits in TWAA. Molsidomine (MSD), a NO donor, rescued TWAA deficits in male KO when acutely administered before training. In a passive avoidance paradigm, KO of both sexes displayed significantly shorter step-through latencies after training. Further, abnormal spontaneous motor activity rhythms were found in the KO during the dark phase of the day, indicating dysregulation of rhythmic activities. These data indicate that NOS1 KO mimics certain ADHD-like behaviors and could potentially serve as a novel rodent model for ADHD.

Adaptive effort investment in cognitive and physical tasks: a neurocomputational model

Despite its importance in everyday life, the computational nature of effort investment remains poorly understood. We propose an effort model obtained from optimality considerations, and a neurocomputational approximation to the optimal model. Both are couched in the framework of reinforcement learning. It is shown that choosing when or when not to exert effort can be adaptively learned, depending on rewards, costs, and task difficulty. In the neurocomputational model, the limbic loop comprising anterior cingulate cortex (ACC) and ventral striatum in the basal ganglia allocates effort to cortical stimulus-action pathways whenever this is valuable. We demonstrate that the model approximates optimality. Next, we consider two hallmark effects from the cognitive control literature, namely proportion congruency and sequential congruency effects. It is shown that the model exerts both proactive and reactive cognitive control. Then, we simulate two physical effort tasks. In line with empirical work, impairing the model's dopaminergic pathway leads to apathetic behavior. Thus, we conceptually unify the exertion of cognitive and physical effort, studied across a variety of literatures (e.g., motivation and cognitive control) and animal species.

ADHD, altered dopamine neurotransmission, and disrupted reinforcement processes: implications for smoking and nicotine dependence

Attention deficit hyperactivity disorder (ADHD) is a common and impairing disorder affecting millions of children, adolescents, and adults. Individuals with ADHD smoke cigarettes at rates significantly higher than their non-diagnosed peers and the disorder also confers risk for a number of related adverse smoking outcomes including earlier age of initiation, faster progression to regular use, heavier smoking/greater dependence, and more difficulty quitting. Progress in our understanding of dopamine neurotransmission and basic behavioral reinforcement processes in ADHD may help increase our understanding of the ADHD-smoking comorbidity. This review will examine how these areas have been studied and how further work may aid in the development of better prevention and treatment for smoking in those with ADHD.

Elucidating poor decision-making in a rat gambling task

Although poor decision-making is a hallmark of psychiatric conditions such as attention deficit/hyperactivity disorder, pathological gambling or substance abuse, a fraction of healthy individuals exhibit similar poor decision-making performances in everyday life and specific laboratory tasks such as the Iowa Gambling Task. These particular individuals may provide information on risk factors or common endophenotypes of these mental disorders. In a rodent version of the Iowa gambling task--the Rat Gambling Task (RGT), we identified a population of poor decision makers, and assessed how these rats scored for several behavioral traits relevant to executive disorders: risk taking, reward seeking, behavioral inflexibility, and several aspects of impulsivity. First, we found that poor decision-making could not be well predicted by single behavioral and cognitive characteristics when considered separately. By contrast, a combination of independent traits in the same individual, namely risk taking, reward seeking, behavioral inflexibility, as well as motor impulsivity, was highly predictive of poor decision-making. Second, using a reinforcement-learning model of the RGT, we confirmed that only the combination of extreme scores on these traits could induce maladaptive decision-making. Third, the model suggested that a combination of these behavioral traits results in an inaccurate representation of rewards and penalties and inefficient learning of the environment. Poor decision-making appears as a consequence of the over-valuation of high-reward-high-risk options in the task. Such a specific psychological profile could greatly impair clinically healthy individuals in decision-making tasks and may predispose to mental disorders with similar symptoms.

Cue reactivity in smokers: an event-related potential study

Drugs-of-abuse may increase the salience of drug cues by sensitizing the dopaminergic (DA) system (Robinson and Berridge, 1993), leading to differential attention to smoking stimuli. Event-related potentials (ERPs) have been used to assess attention to smoking cues but not using an ERP component associated with DA-mediated salience evaluation. In this study the DA-related P2a and the P3, were compared in smokers (N = 21) and non-smokers (N = 21) during an attention selection cue exposure task including both cigarette and neutral images. We predicted that both the P2a and P3 would be larger to targets than non-targets, but larger to non-target cigarette images than non-target neutral images only in the smokers, reflecting smokers' evaluation of smoking stimuli as relevant even when they were not targets. Results indicated that smokers showed behavioral cue reactivity, with more false alarms to cigarette images (responding to cigarette images when they were not targets) than non-smokers; however, both smokers and non-smokers had a larger P2a and P3 to cigarette images. Thus, while smokers showed behavioral evidence of differential salience evaluation of the cigarette images, this group difference was not reflected in differential brain activity. These findings may reflect characteristics of the ERPs (both ERP components were smaller in the smokers), the smoking sample (they were not more impulsive, i.e. reward sensitive, than the non-smokers, in contrast to prior studies) and the design (all participants were aware that the aim of the study was related to smoking).

Deficient reinforcement learning in medial frontal cortex as a model of dopamine-related motivational deficits in ADHD

Attention Deficit/Hyperactivity Disorder (ADHD) is a pathophysiologically complex and heterogeneous condition with both cognitive and motivational components. We propose a novel computational hypothesis of motivational deficits in ADHD, drawing together recent evidence on the role of anterior cingulate cortex (ACC) and associated mesolimbic dopamine circuits in both reinforcement learning and ADHD. Based on findings of dopamine dysregulation and ACC involvement in ADHD we simulated a lesion in a previously validated computational model of ACC (Reward Value and Prediction Model, RVPM). We explored the effects of the lesion on the processing of reinforcement signals. We tested specific behavioral predictions about the profile of reinforcement-related deficits in ADHD in three experimental contexts; probability tracking task, partial and continuous reward schedules, and immediate versus delayed rewards. In addition, predictions were made at the neurophysiological level. Behavioral and neurophysiological predictions from the RVPM-based lesion-model of motivational dysfunction in ADHD were confirmed by data from previously published studies. RVPM represents a promising model of ADHD reinforcement learning suggesting that ACC dysregulation might play a role in the pathogenesis of motivational deficits in ADHD. However, more behavioral and neurophysiological studies are required to test core predictions of the model. In addition, the interaction with different brain networks underpinning other aspects of ADHD neuropathology (i.e., executive function) needs to be better understood.

Dopamine neurons learn to encode the long-term value of multiple future rewards

Midbrain dopamine neurons signal reward value, their prediction error, and the salience of events. If they play a critical role in achieving specific distant goals, long-term future rewards should also be encoded as suggested in reinforcement learning theories. Here, we address this experimentally untested issue. We recorded 185 dopamine neurons in three monkeys that performed a multistep choice task in which they explored a reward target among alternatives and then exploited that knowledge to receive one or two additional rewards by choosing the same target in a set of subsequent trials. An analysis of anticipatory licking for reward water indicated that the monkeys did not anticipate an immediately expected reward in individual trials; rather, they anticipated the sum of immediate and multiple future rewards. In accordance with this behavioral observation, the dopamine responses to the start cues and reinforcer beeps reflected the expected values of the multiple future rewards and their errors, respectively. More specifically, when monkeys learned the multistep choice task over the course of several weeks, the responses of dopamine neurons encoded the sum of the immediate and expected multiple future rewards. The dopamine responses were quantitatively predicted by theoretical descriptions of the value function with time discounting in reinforcement learning. These findings demonstrate that dopamine neurons learn to encode the long-term value of multiple future rewards with distant rewards discounted.

EEG oscillatory activity associated to monetary gain and loss signals in a learning task: effects of attentional impulsivity and learning ability

This study investigated the influence of individual differences in attentional impulsivity (Att-Imp), learning ability, and learning practice on oscillatory activity and phase synchrony responses to monetary gain and loss signals during an instrumental-learning task in healthy women. We used a trial-by-trial wavelet-based time-frequency analysis of the electroencephalographic (EEG) signal to provide amplitude and inter-site phase synchrony measures from 30 electrodes in theta (4-8 Hz, 350-500 ms), alpha (8-12 Hz, 100-200 ms), beta (13-25 Hz, 100-200 ms), and gamma (30-40 Hz, 350-450 ms) time-frequency ranges. Oscillatory amplitude and inter-site phase synchrony were both greater following loss signals as compared to gain signals in theta, beta, and gamma frequency bands. Low Att-Imp subjects had higher theta activity within a 350-500 ms time window over frontocentral, and centroparietal sites than high Att-Imp subjects. Monetary gain signals elicited higher theta and gamma activities in high Att-Imp individuals and loss signals elicited higher activities to loss signals in low Att-Imp individuals. Good learners showed enhanced intrahemispheric theta synchrony between frontoparietal, and fronto-occipital sites to monetary loss compared to gain signals. In good learners, monetary loss produced an increase of gamma synchrony that enhanced in the second stage of learning. In low Att-Imp individuals, there was a reduction of theta synchrony during the second stage, as compared with the first stage of learning, between temporal, parietal and fronto-parietal brain regions. These findings may offer valuable clues to understand outcome processing, attentional impulsivity, and learning ability. We propose that the punishment-related theta and gamma waves play a leading role in learning process.

Are computational models of any use to psychiatry?

Mathematically rigorous descriptions of key hypotheses and theories are becoming more common in neuroscience and are beginning to be applied to psychiatry. In this article two fictional characters, Dr. Strong and Mr. Micawber, debate the use of such computational models (CMs) in psychiatry. We present four fundamental challenges to the use of CMs in psychiatry: (a) the applicability of mathematical approaches to core concepts in psychiatry such as subjective experiences, conflict and suffering; (b) whether psychiatry is mature enough to allow informative modelling; (c) whether theoretical techniques are powerful enough to approach psychiatric problems; and (d) the issue of communicating clinical concepts to theoreticians and vice versa. We argue that CMs have yet to influence psychiatric practice, but that they help psychiatric research in two fundamental ways: (a) to build better theories integrating psychiatry with neuroscience; and (b) to enforce explicit, global and efficient testing of hypotheses through more powerful analytical methods. CMs allow the complexity of a hypothesis to be rigorously weighed against the complexity of the data. The paper concludes with a discussion of the path ahead. It points to stumbling blocks, like the poor communication between theoretical and medical communities. But it also identifies areas in which the contributions of CMs will likely be pivotal, like an understanding of social influences in psychiatry, and of the co-morbidity structure of psychiatric diseases.

Opponency revisited: competition and cooperation between dopamine and serotonin

Affective valence lies on a spectrum ranging from punishment to reward. The coding of such spectra in the brain almost always involves opponency between pairs of systems or structures. There is ample evidence for the role of dopamine in the appetitive half of this spectrum, but little agreement about the existence, nature, or role of putative aversive opponents such as serotonin. In this review, we consider the structure of opponency in terms of previous biases about the nature of the decision problems that animals face, the conflicts that may thus arise between Pavlovian and instrumental responses, and an additional spectrum joining invigoration to inhibition. We use this analysis to shed light on aspects of the role of serotonin and its interactions with dopamine.

Tonic dopamine modulates exploitation of reward learning

The impact of dopamine on adaptive behavior in a naturalistic environment is largely unexamined. Experimental work suggests that phasic dopamine is central to reinforcement learning whereas tonic dopamine may modulate performance without altering learning per se; however, this idea has not been developed formally or integrated with computational models of dopamine function. We quantitatively evaluate the role of tonic dopamine in these functions by studying the behavior of hyperdopaminergic DAT knockdown mice in an instrumental task in a semi-naturalistic homecage environment. In this “closed economy” paradigm, subjects earn all of their food by pressing either of two levers, but the relative cost for food on each lever shifts frequently. Compared to wild-type mice, hyperdopaminergic mice allocate more lever presses on high-cost levers, thus working harder to earn a given amount of food and maintain their body weight. However, both groups show a similarly quick reaction to shifts in lever cost, suggesting that the hyperdominergic mice are not slower at detecting changes, as with a learning deficit. We fit the lever choice data using reinforcement learning models to assess the distinction between acquisition and expression the models formalize. In these analyses, hyperdopaminergic mice displayed normal learning from recent reward history but diminished capacity to exploit this learning: a reduced coupling between choice and reward history. These data suggest that dopamine modulates the degree to which prior learning biases action selection and consequently alters the expression of learned, motivated behavior.

Focus on the positive: computational simulations implicate asymmetrical reward prediction error signals in childhood attention-deficit/hyperactivity disorder

A number of hypotheses have suggested that the principal neurological dysfunction responsible for the behavioural symptoms associated with Attention-Deficit/Hyperactive Disorder (ADHD) is likely rooted in abnormal phasic signals coded by the firing rate of midbrain dopamine neurons. We present a formal investigation of the impact atypical phasic dopamine signals have on behaviour by applying a TD(λ) reinforcement learning model to simulations of operant conditioning tasks that have been argued to quantify the hyperactive, inattentive and impulsive behaviour associated with ADHD. The results presented here suggest that asymmetrically effective dopamine signals encoded by a punctate increase or decrease in dopamine levels provide the best account for the behaviour of children with ADHD as well as an animal model of ADHD, the spontaneously hypertensive rat (SHR). The biological sources of this asymmetry are considered, as are other computational models of ADHD.

Temporal reward discounting in attention-deficit/hyperactivity disorder: the contribution of symptom domains, reward magnitude, and session length

BACKGROUND

Theoretical models have hypothesized that one core problem in attention-deficit/hyperactivity disorder (ADHD) is abnormal reward processing. Temporal reward discounting (decreases in subjective reward value due to prereward delay) is of interest because of its relation with a key symptom of ADHD-impulsivity. This study investigated 1) whether steep temporal reward discounting (TD) is associated with ADHD-combined type (ADHD-C)/symptoms of hyperactivity-impulsivity specifically; 2) the role of reward magnitude in TD in ADHD-C/participants with symptoms of hyperactivity-impulsivity; and 3) whether steep TD in ADHD-C/participants with symptoms of hyperactivity-impulsivity is affected by session length.

METHODS

Three TD tasks were administered to children and adolescents (aged 6-17) with ADHD-C (n = 25), ADHD-inattentive type (ADHD-I; n = 20), and matched typically developing participants (n = 37). Reward magnitude and session length were varied.

RESULTS

Steep TD was observed in participants with ADHD-C but not in those with ADHD-I, independent of reward magnitude and session length. Dimensional analyses revealed that steep TD was associated with hyperactivity-impulsivity (transcending the arbitrary cutoff for ADHD subtypes), especially when reward magnitude at the trial level was small.

CONCLUSIONS

These findings suggest that steep TD in ADHD is best thought of as a correlate of the symptom dimension of hyperactivity/impulsivity. Additionally, steep TD in ADHD is the result of a trade-off between delay and reward magnitude, with all factors contributing to choice preferences. These findings may help refine the delay aversion theory of ADHD, and provide evidence for the notion that unique reward processing is one mechanism associated with symptoms of hyperactivity-impulsivity.

Impaired behavior regulation under conditions of concurrent variable schedules of reinforcement in children with ADHD

OBJECTIVE

To bridge theory of response inhibition and learning in children with ADHD.

METHOD

Thirty ADHD and 30 non-ADHD children (ages 9-12) were compared under concurrent variable interval (VI-15 sec., VI-30 sec. and VI- 45 sec.) reinforcement schedules that required the child to switch between the three schedules under conditions of experimentally controlled inhibition (change over delay [COD] vs. No COD). Classical matching law was used to evaluate children's success in maximizing reinforcement opportunities.

RESULTS

Children with ADHD showed normal matching only when immediate reinforcement for responding was blocked by the presence of a 3-s COD. Without a COD, ADHD children failed to conform to the normal matching law. Non-ADHD children's behavior fit the matching law (i.e., rate of response was proportional to rate of reinforcement) whether a COD was present or absent.

CONCLUSIONS

Results supported other findings that response inhibition is a key mechanism in responses to reinforcement schedules by children with ADHD and that the absence of contingencies that inhibit impulsive responding might impair complex learning in which the child must choose between many different behavioral options, each with associated reinforcement schedules.

Identifying the neurobiology of altered reinforcement sensitivity in ADHD: a review and research agenda

ADHD is associated with altered reinforcement sensitivity, despite a number of inconsistent findings. This review focuses on the overlap and differences between seven neurobiologically valid models and lists 15 predictions assessing reinforcement sensitivity in ADHD. When comparing the models it becomes clear that there are great differences in the level of explanation. For example, some models try to explain a single core deficit in terms lower-level reinforcement systems, such as the dopamine transfer to reward back in time. Other models explain multiple deficits, by describing higher-level systems, such as impaired bottom-up prefrontal activation. When reviewing the available experimental evidence in support of the predictions, most experimental studies have been focusing on behavioral changes in the face of reward and response cost over no-reward, and on delay discounting. There is currently a lack in studies that focus on explaining underlying cognitive or neural mechanisms of altered reinforcement sensitivity in ADHD. Additionally, there is a lack in studies that try to understand what subgroup of children with ADHD shows alterations in reinforcement sensitivity. The scarcity in studies testing the neurobiological predictions is explained partly by a lack in knowledge how to test some of these predictions in humans. Nevertheless, we believe that these predictions can serve as a useful guide to the systematic evaluation of altered reinforcement sensitivity in ADHD.

Ventro-striatal reductions underpin symptoms of hyperactivity and impulsivity in attention-deficit/hyperactivity disorder

BACKGROUND

Models of attention-deficit/hyperactivity disorder (ADHD) classically emphasize the relevance of executive processes and, recently, reward circuits. The neural bases of reward processes have barely been explored in relation to this disorder, in contrast to extensive neuroimaging studies that examine executive functions in patients with ADHD. To our knowledge, no previous studies have analyzed the volume of the ventral striatum, a key region for reward processes in ADHD children.

METHODS

We used a manual region-of-interest approach to examine whether there were volumetric differences in the ventral striatum of ADHD children. Forty-two children/adolescents with ADHD (ages 6-18), and 42 healthy control subjects matched on age, gender, and handedness were selected for the study.

RESULTS

The ADHD children presented significant reductions in both right and left ventro-striatal volumes (t = 3.290, p = .001; and t = 3.486, p = .001, respectively). In addition, we found that the volume of the right ventral striatum negatively correlated with maternal ratings of hyperactivity/impulsivity (r = -.503, p = .003).

CONCLUSIONS

Our study provides neuroanatomical evidence of alterations in the ventral striatum of ADHD children. These findings coincide with previous explicative models as well as with recent reports in behavioral and functional neuroimaging studies. Furthermore, the negative correlations we observed strongly uphold the relation between the ventral striatum and symptoms of hyperactivity/impulsivity.

The evolution of hyperactivity, impulsivity and cognitive diversity

The evolutionary status of attention deficit/hyperactivity disorder (ADHD) is central to assessments of whether modern society has created it, either physically or socially; and is potentially useful in understanding its neurobiological basis and treatment. The high prevalence of ADHD (5-10%) and its association with the seven-repeat allele of DRD4, which is positively selected in evolution, raise the possibility that ADHD increases the reproductive fitness of the individual, and/or the group. However, previous suggestions of evolutionary roles for ADHD have not accounted for its confinement to a substantial minority. Because one of the key features of ADHD is its diversity, and many benefits of population diversity are well recognized (as in immunity), we study the impact of groups' behavioural diversity on their fitness. Diversity occurs along many dimensions, and for simplicity we choose unpredictability (or variability), excess of which is a well-established characteristic of ADHD.Simulations of the Changing Food group task show that unpredictable behaviour by a minority optimizes results for the group. Characteristics of such group exploration tasks are risk-taking, in which costs are borne mainly by the individual; and information-sharing, in which benefits accrue to the entire group. Hence, this work is closely linked to previous studies of evolved altruism.We conclude that even individually impairing combinations of genes, such as ADHD, can carry specific benefits for society, which can be selected for at that level, rather than being merely genetic coincidences with effects confined to the individual. The social benefits conferred by diversity occur both inside and outside the 'normal' range, and these may be distinct. This view has the additional merit of offering explanations for the prevalence, sex and age distribution, severity distribution and heterogeneity of ADHD.

Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning

Natural rewards and drugs of abuse can alter dopamine signaling, and ventral tegmental area (VTA) dopaminergic neurons are known to fire action potentials tonically or phasically under different behavioral conditions. However, without technology to control specific neurons with appropriate temporal precision in freely behaving mammals, the causal role of these action potential patterns in driving behavioral changes has been unclear. We used optogenetic tools to selectively stimulate VTA dopaminergic neuron action potential firing in freely behaving mammals. We found that phasic activation of these neurons was sufficient to drive behavioral conditioning and elicited dopamine transients with magnitudes not achieved by longer, lower-frequency spiking. These results demonstrate that phasic dopaminergic activity is sufficient to mediate mammalian behavioral conditioning.

Anomalous dopamine release associated with a human dopamine transporter coding variant

Dopamine (DA) signaling at synapses is tightly coordinated through opposing mechanisms of vesicular fusion-mediated DA release and transporter-mediated DA clearance. Altered brain DA signaling is suspected to underlie multiple brain disorders, including schizophrenia, Parkinson's disease, bipolar disorder, and attention-deficit hyperactivity disorder (ADHD). We identified a pedigree containing two male children diagnosed with ADHD who share a rare human DA transporter (DAT; SLC6A3) coding variant, Ala559Val. Among >1000 control and affected subjects, the Val559 variant has only been isolated once previously, in a female subject with bipolar disorder. Although hDAT Ala559Val supports normal DAT protein and cell surface expression, as well as normal DA uptake, the variant exhibits anomalous DA efflux from DA-loaded cells. We also demonstrate that hDAT Ala599Val exhibits increased sensitivity to intracellular Na(+), but not intracellular DA, and displays exaggerated DA efflux at depolarized potentials. Remarkably, the two most common ADHD medications, amphetamine and methylphenidate, both block hDAT Ala559Val-mediated DA efflux, whereas these drugs have opposite actions at wild-type hDAT. Our findings reveal that DA efflux, typically associated with amphetamine-like psychostimulants, can be produced through a heritable change in hDAT structure. Because multiple gene products are known to coordinate to support amphetamine-mediated DA efflux, the properties of hDAT Ala559Val may have broader significance in identifying a new mechanism through which DA signaling disorders arise. Additionally, they suggest that block of inappropriate neurotransmitter efflux may be an unsuspected mechanism supporting the therapeutic actions of existing transporter-directed medications.

Research review: dopamine transfer deficit: a neurobiological theory of altered reinforcement mechanisms in ADHD

This review considers the hypothesis that changes in dopamine signalling might account for altered sensitivity to positive reinforcement in children with ADHD. The existing evidence regarding dopamine cell activity in relation to positive reinforcement is reviewed. We focus on the anticipatory firing of dopamine cells brought about by a transfer of dopamine cell responses to cues that precede reinforcers. It is proposed that in children with ADHD there is diminished anticipatory dopamine cell firing, which we call the dopamine transfer deficit (DTD). The DTD theory leads to specific and testable predictions for human and animal research. The extent to which DTD explains symptoms of ADHD and effects of pharmacological interventions is discussed. We conclude by considering the neural changes underlying the etiology of DTD.

Working toward a neurobiological account of ADHD: commentary on Gail Tripp and Jeff Wickens, dopamine transfer deficit

The dopamine transfer deficit model of attention deficit hyperactivity disorder (ADHD) is compared and contrasted with the existing dynamic developmental theory and the extended temporal difference (TD) model. The first two both identify learning deficits as a key problem in ADHD, but this mechanism would seem at least as likely to cause other neurodevelopmental disorders. Learning deficits also do not provide a natural account of ADHD performance worsening in longer trials, nor of stimulant action, nor of the high rate of test-retest inconsistency. The extended TD model is based on the quite different premise that there are very many underlying causes for the disorder, and computational simulation of it has supported the existence of two subtypes that involve no dopamine transfer deficit, as well as several subtypes that do. The three models have markedly different ways of accounting for key aspects of ADHD, including intra-individual variability, stimulant effects, and extinction deficits. The most important shortcoming of all the models is their limited treatment of state-regulation processes, which are very important in ADHD; this is partially addressed by the extended TD model's proposed dopamine appetite.

Electrophysiological evidence of atypical motivation and reward processing in children with attention-deficit hyperactivity disorder

Behavioral and neurophysiological evidence suggest that attention-deficit hyperactivity disorder (ADHD) is characterized by the impact of abnormal reward prediction error signals carried by the midbrain dopamine system on frontal brain areas that implement cognitive control. To investigate this issue, we recorded the event-related brain potential (ERP) from typical children and children with ADHD as they navigated a "virtual maze" to find monetary rewards, and physically gave them their accumulated rewards halfway through the task and at the end of the experiment. We found that the amplitude of a reward-related ERP component decreased somewhat for typical children after they received their first payment, but increased for children with ADHD following the payment. This result indicates that children with ADHD are unusually sensitive to the salience of reward and suggests that such sensitivity may be mediated in part by the midbrain dopamine system.

A temporal difference account of avoidance learning

Aversive processing plays a central role in human phobic fears and may also be important in some symptoms of psychosis. We developed a temporal-difference model of the conditioned avoidance response, an important experimental model for aversive learning which is also a central pharmacological model of psychosis. In the model, dopamine neurons reported outcomes that were better than the learner expected, typically coming from reaching safety states, and thus controlled the acquisition of a suitable policy. The model accounts for normal conditioned avoidance learning, the persistence of responding in extinction, and critical effects of dopamine blockade, notably that subjects experiencing shocks under dopamine blockade, and hence failing to avoid them, nevertheless develop avoidance responses when both shocks and dopamine blockade are subsequently removed. These postulated roles of dopamine in aversive learning can thus account for many of the effects of dopaminergic modulation seen in laboratory models of psychopathological processes.