Striatal dopamine in motor activation and reward-mediated learning: steps towards a unifying model

Neurofeedback and neural self-regulation: a new perspective based on allostasis

The field of neurofeedback training (NFT) has seen growing interest and an expansion of scope, resulting in a steadily increasing number of publications addressing different aspects of NFT. This development has been accompanied by a debate about the underlying mechanisms and expected outcomes. Recent developments in the understanding of psychophysiological regulation have cast doubt on the validity of control systems theory, the principal framework traditionally used to characterize NFT. The present article reviews the theoretical and empirical aspects of NFT and proposes a predictive framework based on the concept of allostasis. Specifically, we conceptualize NFT as an adaptation to changing contingencies. In an allostasis four-stage model, NFT involves (a) perceiving relations between demands and set-points, (b) learning to apply collected patterns (experience) to predict future output, (c) determining efficient set-points, and (d) adapting brain activity to the desired ("set") state. This model also identifies boundaries for what changes can be expected from a neurofeedback intervention and outlines a time frame for such changes to occur.

Temporally restricted dopaminergic control of reward-conditioned movements

Midbrain dopamine (DA) neurons encode both reward- and movement-related events and are implicated in disorders of reward processing as well as movement. Consequently, disentangling the contribution of DA neurons in reinforcing versus generating movements is challenging and has led to lasting controversy. In this study, we dissociated these functions by parametrically varying the timing of optogenetic manipulations in a Pavlovian conditioning task and examining the influence on anticipatory licking before reward delivery. Inhibiting both ventral tegmental area and substantia nigra pars compacta DA neurons in the post-reward period had a significantly greater behavioral effect than inhibition in the pre-reward period of the task. Furthermore, the contribution of DA neurons to behavior decreased linearly as a function of elapsed time after reward. Together, the results indicate a temporally restricted role of DA neurons primarily related to reinforcing stimulus-reward associations and suggest that directly generating movements is a comparatively less important function.

A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum

Dopamine‐dependent synaptic plasticity is a candidate mechanism for reinforcement learning. A silent eligibility trace – initiated by synaptic activity and transformed into synaptic strengthening by later action of dopamine – has been hypothesized to explain the retroactive effect of dopamine in reinforcing past behaviour. We tested this hypothesis by measuring time‐dependent modulation of synaptic plasticity by dopamine in adult mouse striatum, using whole‐cell recordings. Presynaptic activity followed by postsynaptic action potentials (pre–post) caused spike‐timing‐dependent long‐term depression in D1‐expressing neurons, but not in D2 neurons, and not if postsynaptic activity followed presynaptic activity. Subsequent experiments focused on D1 neurons. Applying a dopamine D1 receptor agonist during induction of pre–post plasticity caused long‐term potentiation. This long‐term potentiation was hidden by long‐term depression occurring concurrently and was unmasked when long‐term depression blocked an L‐type calcium channel antagonist. Long‐term potentiation was blocked by a Ca²⁺‐permeable AMPA receptor antagonist but not by an NMDA antagonist or an L‐type calcium channel antagonist. Pre–post stimulation caused transient elevation of rectification – a marker for expression of Ca²⁺‐permeable AMPA receptors – for 2–4‐s after stimulation. To test for an eligibility trace, dopamine was uncaged at specific time points before and after pre‐ and postsynaptic conjunction of activity. Dopamine caused potentiation selectively at synapses that were active 2‐s before dopamine release, but not at earlier or later times. Our results provide direct evidence for a silent eligibility trace in the synapses of striatal neurons. This dopamine‐timing‐dependent plasticity may play a central role in reinforcement learning.

Paired-Associate and Feedback-Based Weather Prediction Tasks Support Multiple Category Learning Systems

It remains unclear whether probabilistic category learning in the feedback-based weather prediction task (FB-WPT) can be mediated by a non-declarative or procedural learning system. To address this issue, we compared the effects of training time and verbal working memory, which influence the declarative learning system but not the non-declarative learning system, in the FB and paired-associate (PA) WPTs, as the PA task recruits a declarative learning system. The results of Experiment 1 showed that the optimal accuracy in the PA condition was significantly decreased when the training time was reduced from 7 to 3 s, but this did not occur in the FB condition, although shortened training time impaired the acquisition of explicit knowledge in both conditions. The results of Experiment 2 showed that the concurrent working memory task impaired the optimal accuracy and the acquisition of explicit knowledge in the PA condition but did not influence the optimal accuracy or the acquisition of self-insight knowledge in the FB condition. The apparent dissociation results between the FB and PA conditions suggested that a non-declarative or procedural learning system is involved in the FB-WPT and provided new evidence for the multiple-systems theory of human category learning.

Acute effects of cocaine and cannabis on reversal learning as a function of COMT and DRD2 genotype

RATIONALE

Long-term cannabis and cocaine use has been associated with impairments in reversal learning. However, how acute cannabis and cocaine administration affect reversal learning in humans is not known.

OBJECTIVE

In this study, we aimed to establish the acute effects of administration of cannabis and cocaine on valence-dependent reversal learning as a function of DRD2 Taq1A (rs1800497) and COMT Val108/158Met (rs4680) genotype.

METHODS

A double-blind placebo-controlled randomized 3-way crossover design was used. Sixty-one regular poly-drug users completed a deterministic reversal learning task under the influence of cocaine, cannabis, and placebo that enabled assessment of both reward- and punishment-based reversal learning.

RESULTS

Proportion correct on the reversal learning task was increased by cocaine, but decreased by cannabis. Effects of cocaine depended on the DRD2 genotype, as increases in proportion correct were seen only in the A1 carriers, and not in the A2/A2 homozygotes. COMT genotype did not modulate drug-induced effects on reversal learning.

CONCLUSIONS

These data indicate that acute administration of cannabis and cocaine has opposite effects on reversal learning. The effects of cocaine, but not cannabis, depend on interindividual genetic differences in the dopamine D2 receptor gene.

Activity of Monoamine Oxidase in the Nigrostriatal System at Presymptomatic and Early Symptomatic Stages of Parkinsonism in Mice

Activities of monoamine oxidases A and B were examined on the models of presymptomatic and early symptomatic stages of Parkinson's disease developed in mice treated with MPTP, a specific neurotoxin affecting dopaminergic neurons. Activity of monoamine oxidases A, the key enzyme of dopamine degradation, is increased in neuronal somas during the symptomatic stage, and it is augmented in the axons during both stages. Neuronal activity of monoamine oxidases A is higher during the symptomatic stage than that during the presymptomatic stage, which can explain depletion of intercellular dopamine and appearance of motor disturbances. Activity of monoamine oxidase B in the striatum is reduced during the presymptomatic stage, but returns to the control level during the symptomatic stage. Variation in monoamine oxidase activity seems to reflect the compensatory mechanisms triggered in degrading nigrostriatal dopaminergic system.

Differential effects of dopamine-directed treatments on cognition

Dopamine, a prominent neuromodulator, is implicated in many neuropsychiatric disorders. It has wide-ranging effects on both cortical and subcortical brain regions and on many types of cognitive tasks that rely on a variety of different learning and memory systems. As neuroscience and behavioral evidence for the existence of multiple memory systems and their corresponding neural networks accumulated, so did the notion that dopamine's role is markedly different depending on which memory system is engaged. As a result, dopamine-directed treatments will have different effects on different types of cognitive behaviors. To predict what these effects will be, it is critical to understand: which memory system is mediating the behavior; the neural basis of the mediating memory system; the nature of the dopamine projections into that system; and the time course of dopamine after its release into the relevant brain regions. Consideration of these questions leads to different predictions for how changes in brain dopamine levels will affect automatic behaviors and behaviors mediated by declarative, procedural, and perceptual representation memory systems.

Adenosinergic regulation of binge-like ethanol drinking and associated locomotor effects in male C57BL/6J mice

We recently observed that the addition of caffeine (a nonselective adenosine receptor antagonist) to a 20% ethanol solution significantly altered the intoxication profile of male C57BL/6J (B6) mice induced by voluntary binge-like consumption in the 'Drinking-in-the-Dark' (DID) paradigm. In the current study, the roles of A1 and A2A adenosine receptor subtypes, specifically, in binge-like ethanol consumption and associated locomotor effects were explored. Adult male B6 mice (PND 60-70) were allowed to consume 20% ethanol (v/v) or 2% sucrose (w/v) for 6days via DID. On day 7, mice received a systemic administration (i.p.) of the A1 antagonist DPCPX (1, 3, 6mg/kg), the A2A antagonist MSX-3 (1, 2, 4mg/kg), or vehicle immediately prior to fluid access in DID. Antagonism of the A1 receptor via DPCPX was found to dose-dependently decrease binge-like ethanol intake and associated blood ethanol concentrations (p's<0.05), although no effect was observed on sucrose intake. Antagonism of A2A had no effect on ethanol or sucrose consumption, however, MSX-3 elicited robust locomotor stimulation in mice consuming either solution (p's<0.05). Together, these findings suggest unique roles for the A1 and A2A adenosine receptor subtypes in binge-like ethanol intake and its associated locomotor effects.

Increased brain activity to unpleasant stimuli in individuals with the 7R allele of the DRD4 gene

The aim of the study was to examine functional brain activity in response to unpleasant images in individuals with the 7-repeat (7R) allele compared to individuals with the 4-repeat (4R) allele of the dopamine receptor D4 (DRD4) gene (VNTR in exon 3). Based on the response ready hypothesis, individuals with the DRD4-4R/7R genotype were expected to show greater functional brain activity in response to unpleasant compared to neutral stimuli in specific regions of the frontal, temporal, parietal and limbic lobes, which form the networks involved in attentional, emotional, and preparatory responses. Functional Magnetic Resonance Imaging activity was studied in 26 young adults (13 with the DRD4-4R/7R genotype and 13 with the DRD4-4R/4R genotype). Participants were asked to look at and subjectively rate unpleasant and neutral images. Results showed increased brain activity in response to unpleasant images compared to neutral images in the right temporal lobe in participants with the DRD4-4R/7R genotype versus participants with the DRD4-4R/4R genotype. The increase in right temporal lobe activity in individuals with DRD4-4R/7R suggests greater involvement in processing negative emotional stimuli. Intriguingly, no differences were found between the two genotypes in the subjective ratings of the images. The findings corroborate the response ready hypothesis, which suggests that individuals with the 7R allele are more responsive to negative emotional stimuli compared to individuals with the 4R allele of the DRD4 gene.

Developmental but not adult cannabinoid treatments persistently alter axonal and dendritic morphology within brain regions important for zebra finch vocal learning

Prior work shows developmental cannabinoid exposure alters zebra finch vocal development in a manner associated with altered CNS physiology, including changes in patterns of CB1 receptor immunoreactivity, endocannabinoid concentrations and dendritic spine densities. These results raise questions about the selectivity of developmental cannabinoid effects: are they a consequence of a generalized developmental disruption, or are effects produced through more selective and distinct interactions with biochemical pathways that control receptor, endogenous ligand and dendritic spine dynamics? To begin to address this question we have examined effects of developmental cannabinoid exposure on the pattern and density of expression of proteins critical to dendritic (MAP2) and axonal (Nf-200) structure to determine the extent to which dendritic vs. axonal neuronal morphology may be altered. Results demonstrate developmental, but not adult cannabinoid treatments produce generalized changes in expression of both dendritic and axonal cytoskeletal proteins within brain regions and cells known to express CB1 cannabinoid receptors. Results clearly demonstrate that cannabinoid exposure during a period of sensorimotor development, but not adulthood, produce profound effects upon both dendritic and axonal morphology that persist through at least early adulthood. These findings suggest an ability of exogenous cannabinoids to alter general processes responsible for normal brain development. Results also further implicate the importance of endocannabinoid signaling to peri-pubertal periods of adolescence, and underscore potential consequences of cannabinoid abuse during periods of late-postnatal CNS development.

No learning where to go without first knowing where you're coming from: action discovery is trajectory, not endpoint based

Intrinsic motivations drive an agent to explore, providing essential data for linking behaviors with novel outcomes and so laying the foundation for future flexible action. We present experiments using a new behavioral task which allows us to interrogate the connection between exploration and action learning. Human participants used a joystick to search repeatedly for a target location, only receiving feedback on successful discovery. Feedback delay was manipulated, as was the starting position. Experiment 1 employed stable starting positions, so the task could be learnt with respect to a target location or a target trajectory. Participants were able to learn the correct movement under all delay conditions. Experiment 2 used a variable starting location, so the correct movement could only be learnt in terms of target location. Participants displayed little to no learning in this experiment. These results suggest that movements on this scale are stored as trajectories rather than in terms of target location. Overall the experiments demonstrate the potential of this task for uncovering the native representational substrates of action learning.

Cell-type specific increases in female hamster nucleus accumbens spine density following female sexual experience

Female sexual behavior is an established model of a naturally motivated behavior which is regulated by activity within the mesolimbic dopamine system. Repeated activation of the mesolimbic circuit by female sexual behavior elevates dopamine release and produces persistent postsynaptic alterations to dopamine D1 receptor signaling within the nucleus accumbens. Here we demonstrate that sexual experience in female Syrian hamsters significantly increases spine density and alters morphology selectively in D1 receptor-expressing medium spiny neurons within the nucleus accumbens core, with no corresponding change in dopamine receptor binding or protein expression. Our findings demonstrate that previous life experience with a naturally motivated behavior has the capacity to induce persistent structural alterations to the mesolimbic circuit that can increase reproductive success and are analogous to the persistent structural changes following repeated exposure to many drugs of abuse.

The discovery of novel actions is affected by very brief reinforcement delays and reinforcement modality

ABSTRACT The authors investigated the ability of human participants to discover novel actions under conditions of delayed reinforcement. Participants used a joystick to search for a target indicated by visual or auditory reinforcement. Reinforcement delays of 75-150 ms were found to significantly impair action acquisition. They also found an effect of modality, with acquisition superior with auditory feedback. The duration at which delay was found to impede action discovery is, to the authors' knowledge, shorter than that previously reported from work with operant and causal learning paradigms. The sensitivity to delay reported, and the difference between modalities, is consistent with accounts of action discovery that emphasize the importance of a time stamp in the motor record for solving the credit assignment problem.

Animal foraging and the evolution of goal-directed cognition

Foraging- and feeding-related behaviors across eumetazoans share similar molecular mechanisms, suggesting the early evolution of an optimal foraging behavior called area-restricted search (ARS), involving mechanisms of dopamine and glutamate in the modulation of behavioral focus. Similar mechanisms in the vertebrate basal ganglia control motor behavior and cognition and reveal an evolutionary progression toward increasing internal connections between prefrontal cortex and striatum in moving from amphibian to primate. The basal ganglia in higher vertebrates show the ability to transfer dopaminergic activity from unconditioned stimuli to conditioned stimuli. The evolutionary role of dopamine in the modulation of goal-directed behavior and cognition is further supported by pathologies of human goal-directed cognition, which have motor and cognitive dysfunction and organize themselves, with respect to dopaminergic activity, along the gradient described by ARS, from perseverative to unfocused. The evidence strongly supports the evolution of goal-directed cognition out of mechanisms initially in control of spatial foraging but, through increasing cortical connections, eventually used to forage for information.

Reinforcement, dopamine and rodent models in drug development for ADHD

Attention deficit hyperactivity disorder (ADHD) presents special challenges for drug development. Current treatment with psychostimulants and nonstimulants is effective, but their mechanism of action beyond the cellular level is incompletely understood. We review evidence suggesting that altered reinforcement mechanisms are a fundamental characteristic of ADHD. We show that a deficit in the transfer of dopamine signals from established positive reinforcers to cues that predict such reinforcers may underlie these altered reinforcement mechanisms, and in turn explain key symptoms of ADHD. We argue that the neural substrates controlling the excitation and inhibition of dopamine neurons during the transfer process are a promising target for future drug development. There is a need to develop animal models and behavioral paradigms that can be used to experimentally investigate these mechanisms and their effects on sensitivity to reinforcement. More specific and selective targeting of drug development may be possible through this approach.

Combining computational modeling and neuroimaging to examine multiple category learning systems in the brain

Considerable evidence has argued in favor of multiple neural systems supporting human category learning, one based on conscious rule inference and one based on implicit information integration. However, there have been few attempts to study potential system interactions during category learning. The PINNACLE (Parallel Interactive Neural Networks Active in Category Learning) model incorporates multiple categorization systems that compete to provide categorization judgments about visual stimuli. Incorporating competing systems requires inclusion of cognitive mechanisms associated with resolving this competition and creates a potential credit assignment problem in handling feedback. The hypothesized mechanisms make predictions about internal mental states that are not always reflected in choice behavior, but may be reflected in neural activity. Two prior functional magnetic resonance imaging (fMRI) studies of category learning were re-analyzed using PINNACLE to identify neural correlates of internal cognitive states on each trial. These analyses identified additional brain regions supporting the two types of category learning, regions particularly active when the systems are hypothesized to be in maximal competition, and found evidence of covert learning activity in the “off system” (the category learning system not currently driving behavior). These results suggest that PINNACLE provides a plausible framework for how competing multiple category learning systems are organized in the brain and shows how computational modeling approaches and fMRI can be used synergistically to gain access to cognitive processes that support complex decision-making machinery.

Mechanisms of antidepressant action: an integrated dopaminergic perspective

The molecular mechanisms that cause and maintain the major depressive disorder (MDD) are currently unknown. Consistently, antidepressant treatments are characterized by insufficient success rates. This causes high social costs and severe personal sufferings. In the present review we analyze some of the paradigms that are used to explain MDD, particularly from the perspective of the dopaminergic (DA) system. DA has been more classically associated with psychosis and substance abuse disorders, even though a role of DA in MDD has been proposed as well and some antidepressants with DA profile exist. In the present work, we review some of the molecular mechanisms that underpin MDD from the perspective of the dopaminergic system, in the hope of unifying some of the major theories of MDD - the monoaminergic, inflammatory, epigenetics, neurotrophin and anti-apoptotic theories. Several shared components of these theories are highlighted, partially accounted by the functions of the DA system (see supplementary video).

Sub-second dopamine detection in human striatum

Fast-scan cyclic voltammetry at carbon fiber microelectrodes allows rapid (sub-second) measurements of dopamine release in behaving animals. Herein, we report the modification of existing technology and demonstrate the feasibility of making sub-second measurements of dopamine release in the caudate nucleus of a human subject during brain surgery. First, we describe the modification of our electrodes that allow for measurements to be made in a human brain. Next, we demonstrate in vitro and in vivo, that our modified electrodes can measure stimulated dopamine release in a rat brain equivalently to previously determined rodent electrodes. Finally, we demonstrate acute measurements of dopamine release in the caudate of a human patient during DBS electrode implantation surgery. The data generated are highly amenable for future work investigating the relationship between dopamine levels and important decision variables in human decision-making tasks.

Timing is not Everything: Neuromodulation Opens the STDP Gate

Spike timing dependent plasticity (STDP) is a temporally specific extension of Hebbian associative plasticity that has tied together the timing of presynaptic inputs relative to the postsynaptic single spike. However, it is difficult to translate this mechanism to in vivo conditions where there is an abundance of presynaptic activity constantly impinging upon the dendritic tree as well as ongoing postsynaptic spiking activity that backpropagates along the dendrite. Theoretical studies have proposed that, in addition to this pre- and postsynaptic activity, a “third factor” would enable the association of specific inputs to specific outputs. Experimentally, the picture that is beginning to emerge, is that in addition to the precise timing of pre- and postsynaptic spikes, this third factor involves neuromodulators that have a distinctive influence on STDP rules. Specifically, neuromodulatory systems can influence STDP rules by acting via dopaminergic, noradrenergic, muscarinic, and nicotinic receptors. Neuromodulator actions can enable STDP induction or – by increasing or decreasing the threshold – can change the conditions for plasticity induction. Because some of the neuromodulators are also involved in reward, a link between STDP and reward-mediated learning is emerging. However, many outstanding questions concerning the relationship between neuromodulatory systems and STDP rules remain, that once solved, will help make the crucial link from timing-based synaptic plasticity rules to behaviorally based learning.

Acetylcholine-dopamine interactions in the pathophysiology and treatment of CNS disorders

Dopaminergic neurons in the substantia nigra pars compacta and ventral tegmental area of the midbrain form the nigrostriatal and mesocorticolimbic dopaminergic pathways that, respectively, project to dorsal and ventral striatum (including prefrontal cortex). These midbrain dopaminergic nuclei and their respective forebrain and cortical target areas are well established as serving a critical role in mediating voluntary motor control, as evidenced in Parkinson's disease, and incentive-motivated behaviors and cognitive functions, as exhibited in drug addiction and schizophrenia, respectively. Although it cannot be disputed that excitatory and inhibitory amino acid-based neurotransmitters, such as glutamate and GABA, play a vital role in modulating activity of midbrain dopaminergic neurons, recent evidence suggests that acetylcholine may be as important in regulating dopaminergic transmission. Midbrain dopaminergic cell tonic and phasic activity is closely dependent upon projections from hindbrain pedunculopontine and the laterodorsal tegmental nuclei, which comprises the only known cholinergic inputs to these neurons. In close coordination with glutamatergic and GABAergic activity, these excitatory cholinergic projections activate nicotinic and muscarinic acetylcholine receptors within the substantia nigra and ventral tegmental area to modulate dopamine transmission in the dorsal/ventral striatum and prefrontal cortex. Additionally, acetylcholine-containing interneurons in the striatum also constitute an important neural substrate to provide further cholinergic modulation of forebrain striatal dopaminergic transmission. In this review, we examine neurological and psychopathological conditions associated with dysfunctions in the interaction of acetylcholine and dopamine and conventional and new pharmacological approaches to treat these disorders.

Neurobiology of ADHD

Attention-deficit hyperactivity disorder (ADHD) is a prevalent and debilitating disorder diagnosed on the basis of persistent and developmentally-inappropriate levels of overactivity, inattention and impulsivity. The etiology and pathophysiology of ADHD is incompletely understood. There is evidence of a genetic basis for ADHD but it is likely to involve many genes of small individual effect. Differences in the dimensions of the frontal lobes, caudate nucleus, and cerebellar vermis have been demonstrated. Neuropsychological testing has revealed a number of well documented differences between children with and without ADHD. These occur in two main domains: executive function and motivation although neither of these is specific to ADHD. In view of the recent advances in the neurobiology of reinforcement, we concentrate in this review on altered reinforcement mechanisms. Among the motivational differences, many pieces of evidence indicate that an altered response to reinforcement may play a central role in the symptoms of ADHD. In particular, sensitivity to delay of reinforcement appears to be a reliable finding. We review neurobiological mechanisms of reinforcement and discuss how these may be altered in ADHD, with particular focus on the neurotransmitter dopamine and its actions at the cellular and systems level. We describe how dopamine cell firing activity is normally associated with reinforcing events, and transfers to earlier time-points in the behavioural sequence as reinforcement becomes more predictable. We discuss how a failure of this transfer may give rise to many symptoms of ADHD, and propose that methylphenidate might act to compensate for the proposed dopamine transfer deficit.

Altered dopaminergic profiles: implications for the regulation of voluntary physical activity

The biological regulating factors of physical activity in animals are not well understood. This study investigated differences in the central mRNA expression of seven dopamine genes (Drd1, Drd2, Drd3, Drd4, Drd5, TH, and DAT) between high active C57/LJ (n=17) male mice and low active C3H/HeJ (n=20) male mice, and between mice with access to a running wheel and without running wheel access within strain. Mice were housed with running wheels interfaced with a computer for 21 days with distance and duration recorded every 24 h. On day 21, the striatum and nucleus accumbens were removed during the active period (approximately 9 pm) for dopaminergic analysis. On average, the C57L/J mice with wheels ran significantly farther (10.25+/-1.37 km/day vs. 0.01+/-0.09 km/day, p<0.001), longer (329.73+/-30.52 min/day vs. 7.81+/-6.32 min/day, p<0.001), and faster (31.27+/-3.13 m/min vs. 11.81+/-1.08 m/min, p<0.001) than the C3H/HeJ mice with wheels over the 21 day period. No differences in gene expression were found between mice in either strain with wheels and those without wheels suggesting that access to running wheels did not alter dopaminergic expression. In contrast, relative expression for two dopamine genes was significantly lower in the C57L/J mice compared to the C3H/HeJ mice. These results indicate that decreased dopaminergic functioning is correlated with increased activity levels in C57L/J mice and suggests that D1-like receptors as well as tyrosine hydroxylase (an indicator of dopamine production), but not D2-like receptors may be associated with the regulation of physical activity in inbred mice.

Decision theory, reinforcement learning, and the brain

Decision making is a core competence for animals and humans acting and surviving in environments they only partially comprehend, gaining rewards and punishments for their troubles. Decision-theoretic concepts permeate experiments and computational models in ethology, psychology, and neuroscience. Here, we review a well-known, coherent Bayesian approach to decision making, showing how it unifies issues in Markovian decision problems, signal detection psychophysics, sequential sampling, and optimal exploration and discuss paradigmatic psychological and neural examples of each problem. We discuss computational issues concerning what subjects know about their task and how ambitious they are in seeking optimal solutions; we address algorithmic topics concerning model-based and model-free methods for making choices; and we highlight key aspects of the neural implementation of decision making.

Origins of altered reinforcement effects in ADHD

Attention-deficit/hyperactivity disorder (ADHD), characterized by hyperactivity, impulsiveness and deficient sustained attention, is one of the most common and persistent behavioral disorders of childhood. ADHD is associated with catecholamine dysfunction. The catecholamines are important for response selection and memory formation, and dopamine in particular is important for reinforcement of successful behavior. The convergence of dopaminergic mesolimbic and glutamatergic corticostriatal synapses upon individual neostriatal neurons provides a favorable substrate for a three-factor synaptic modification rule underlying acquisition of associations between stimuli in a particular context, responses, and reinforcers. The change in associative strength as a function of delay between key stimuli or responses, and reinforcement, is known as the delay of reinforcement gradient. The gradient is altered by vicissitudes of attention, intrusions of irrelevant events, lapses of memory, and fluctuations in dopamine function. Theoretical and experimental analyses of these moderating factors will help to determine just how reinforcement processes are altered in ADHD. Such analyses can only help to improve treatment strategies for ADHD.

Stimulus-response and response-outcome learning mechanisms in the striatum

While midbrain DA neurons show phasic activations in response to both reward-predicting and salient non-reward events, activation responses to primary and conditioned rewards are sustained for several hundreds of milliseconds beyond those elicited by salient non-reward-related stimuli. The longer-duration DA reward response and corresponding elevated DA release in striatal target sites may selectively strengthen currently-active corticostriatal synapses, i.e., those associated with the successful reward-procuring behavior. This paper describes how similar models of DA-mediated plasticity of corticostriatal synapses may describe both stimulus-response and response-outcome learning. DA-mediated strengthening of corticostriatal synapses in regions of the dorsolateral striatum receiving afferents from primary sensorimotor cortex is likely to bind corticostriatal inputs representing the previously-emitted movement to striatal outputs contributing to the selection of the next movement segment in a behavioral sequence. Within the striatum, more generally, inputs from distinct regions of the frontal cortex that code independently for movement direction and reward expectation send convergent projections to striatal output cells. DA-mediated strengthening of active corticostriatal synapses promotes the future output of the striatal cell under similar input conditions. This is postulated to promote persistence of neuronal activity in the very cortical cells that drive corticostriatal input, leading to the establishment of sustained reverberatory loops that permit cortical movement-related cells to maintain activity until the appropriate time of movement initiation.

Synaptic plasticity in the basal ganglia

Activity-dependent synaptic plasticity occurs in several parts of the basal ganglia. Increasing evidence supports the hypothesis that activity-dependent plasticity underlies the acquisition, maintenance, and extinction of certain types of learning in the basal ganglia. This review focuses on synaptic plasticity in the corticostriatal pathway. As in other systems, both long-term potentiation and long-term depression have been described, and intracellular calcium signalling plays an important role in the induction of plasticity. However, intracellular calcium levels do not appear to be the dominating control factor. Dopamine, via intracellular signalling cascades, also plays a crucial role in determining the magnitude and direction of plasticity, and in modulating the requirements for induction. Endocannabinoids also play an important role in mediating presynaptic expression of synaptic depression. Recent studies have highlighted spike-timing dependent plasticity phenomena, which also involve dopamine and endocannabinoid signalling. Despite significant progress in recent years, many important questions remain unanswered, especially in relation to long-term potentiation. Of particular interest is the question of how to link the molecular and cellular mechanisms of synaptic plasticity to learning operations at the systems level, which are expressed behaviourally as reinforcement-related learning.

Research review: dopamine transfer deficit: a neurobiological theory of altered reinforcement mechanisms in ADHD

This review considers the hypothesis that changes in dopamine signalling might account for altered sensitivity to positive reinforcement in children with ADHD. The existing evidence regarding dopamine cell activity in relation to positive reinforcement is reviewed. We focus on the anticipatory firing of dopamine cells brought about by a transfer of dopamine cell responses to cues that precede reinforcers. It is proposed that in children with ADHD there is diminished anticipatory dopamine cell firing, which we call the dopamine transfer deficit (DTD). The DTD theory leads to specific and testable predictions for human and animal research. The extent to which DTD explains symptoms of ADHD and effects of pharmacological interventions is discussed. We conclude by considering the neural changes underlying the etiology of DTD.

Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model

A critical problem in daily decision making is how to choose actions now in order to bring about rewards later. Indeed, many of our actions have long-term consequences, and it is important to not be myopic in balancing the pros and cons of different options, but rather to take into account both immediate and delayed consequences of actions. Failures to do so may be manifest as persistent, maladaptive decision-making, one example of which is addiction where behavior seems to be driven by the immediate positive experiences with drugs, despite the delayed adverse consequences. A recent study by Takahashi et al. (2007) investigated the effects of cocaine sensitization on decision making in rats and showed that drug use resulted in altered representations in the ventral striatum and the dorsolateral striatum, areas that have been implicated in the neural instantiation of a computational solution to optimal long-term actions selection called the Actor/Critic framework. In this Focus article we discuss their results and offer a computational interpretation in terms of drug-induced impairments in the Critic. We first survey the different lines of evidence linking the subparts of the striatum to the Actor/Critic framework, and then suggest two possible scenarios of breakdown that are suggested by Takahashi et al.'s (2007) data. As both are compatible with the current data, we discuss their different predictions and how these could be empirically tested in order to further elucidate (and hopefully inch towards curing) the neural basis of drug addiction.

Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity

Single action potentials (APs) backpropagate into the higher-order dendrites of striatal spiny projection neurons during cortically driven "up" states. The timing of these backpropagating APs relative to the arriving corticostriatal excitatory inputs determines changes in dendritic calcium concentration. The question arises to whether this spike-timing relative to cortical excitatory inputs can also induce synaptic plasticity at corticostriatal synapses. Here we show that timing of single postsynaptic APs relative to the cortically evoked EPSP determines both the direction and the strength of synaptic plasticity in spiny projection neurons. Single APs occurring 30 ms before the cortically evoked EPSP induced long-term depression (LTD), whereas APs occurring 10 ms after the EPSP induced long-term potentiation (LTP). The amount of plasticity decreased as the time between the APs and EPSPs was increased, with the resulting spike-timing window being broader for LTD than for LTP. In addition, we show that dopamine receptor activation is required for this spike-timing-dependent plasticity (STDP). Blocking dopamine D(1)/D(5) receptors prevented both LTD and LTP induction. In contrast, blocking dopamine D(2) receptors delayed, but did not prevent, LTD and sped induction of LTP. We conclude (1) that, in combination with cortical inputs, single APs evoked in spiny projection neurons can induce both LTP and LTD of the corticostriatal pathway; (2) that the strength and direction of these synaptic changes depend deterministically on the AP timing relative to the arriving cortical inputs; (3) that, whereas dopamine D(2) receptor activation modulates the initial phase of striatal STDP, dopamine D(1)/D(5) receptor activation is critically required for striatal STDP. Thus, the timing of APs relative to cortical inputs alone is not enough to induce corticostriatal plasticity, implying that ongoing activity does not affect synaptic strength unless dopamine receptors are activated.

Corticotropin releasing factor induces anxiogenic locomotion in trout and alters serotonergic and dopaminergic activity

Corticotropin releasing factor (CRF) and serotonin (5-HT) are strongly linked to stress and anxiety in vertebrates. As a neuromodulator in the brain, CRF has anxiogenic properties often characterized by increased locomotion and stereotyped behavior in familiar environments. We hypothesized that expression of anxiogenic behavior in response to CRF will also be exhibited in a teleost fish. Rainbow trout were treated with intracerebroventricular (icv) injections of artificial cerebrospinal fluid (aCSF), 500 or 2000 ng ovine CRF, or not injected. Treatment with either dose of CRF elicited greater locomotion and pronounced head shaking behavior but did not influence water column position. Locomotor and head shaking behaviors may be analogous to the increased stereotypy evoked by icv CRF in rats and may reflect the expression of stress/anxiety behavior. Injection with either aCSF or CRF produced significant increases in plasma cortisol. The absence of behavioral changes in aCSF-injected fish suggests that the behavioral responses following CRF were not due to cortisol. Treatment with 2000 ng CRF significantly increased serotonin, 5-HIAA and dopamine concentrations in the subpallium and raphé and increased 5-HIAA in the preoptic hypothalamus (POA). Concurrent effects of CRF on central monoamines, locomotion and head shaking in trout suggest that anxiogenic properties of CRF are evolutionarily conserved. In addition, positive linear correlations between locomotion and serotonergic and dopaminergic function in the subpallium, POA and raphé nuclei suggest a locomotory function for these monoamines.

A review of medial temporal lobe and caudate contributions to visual category learning

Here we review recent functional neuroimaging, neuropsychological and behavioral studies examining the role of the medial temporal lobe (MTL) and the caudate in learning visual categories either by verbalizeable rules or without awareness. The MTL and caudate are found to play dissociable roles in different types of category learning with successful rule-based (RB) categorization depending selectively on the MTL and non-verbalizeable information-integration (II) category learning depending on the posterior caudate. These studies utilize a combination of experimental cognitive psychology, mathematical modeling (Decision Bound Theory (DBT)) and cognitive computational modeling (the COVIS model of Ashby et al. [1998. A neuropsychological theory of multiple systems in category learning. Psychological Review 105, 442-481]) to enhance the understanding of data obtained via functional magnetic resonance imaging (fMRI). The combination of approaches is used to both test hypotheses of the cognitive model and also to incorporate hypotheses about the strategies used by participants to direct analysis of fMRI data. Examination of the roles of the MTL and caudate in visual category learning holds the promise of bridging between abstract cognitive models of behavior, systems neuroscience, neuropsychology, and the underlying neurophysiology of these brain regions.

Striatal Contributions to Reward and Decision Making: Making Sense of Regional Variations in a Reiterated Processing Matrix

The striatum is the major input nucleus of the basal ganglia. It is thought to play a key role in learning on the basis of positive reinforcement and in action selection. One view of the striatum conceives it as comprising a reiterated matrix of processing units that perform common operations in different striatal regions, namely synaptic plasticity according to a three-factor rule, and lateral inhibition. These operations are required for reinforcement learning and selection of previously reinforced actions. Analysis of the behavioral effects of circumscribed lesions of the striatum, however, suggests regional specialization of learning and decision-making operations. We consider how a basic processing unit may be modified by regional variations in neurochemical parameters, for example, by the gradient in density of dopamine terminals from dorsal to ventral striatum. These variations suggest subtle differences between dorsolateral and ventromedial striatal regions in the temporal properties of dopamine signaling, which are superimposed on regional differences in connectivity. We propose that these variations make sense in relation to the temporal structure of activity in striatal inputs from different regions, and the requirements of different learning operations. Dorsolateral striatal (DLS) regions may be subject to brief, precisely timed pulses of dopamine, whereas ventromedial striatal regions integrate dopamine signals over a longer time course. These differences may be important for understanding regional variations in the contribution to reinforcement of habits, versus incentive processes that are sensitive to the value of expected rewards.

Tonic dopamine: opportunity costs and the control of response vigor

RATIONALE

Dopamine neurotransmission has long been known to exert a powerful influence over the vigor, strength, or rate of responding. However, there exists no clear understanding of the computational foundation for this effect; predominant accounts of dopamine's computational function focus on a role for phasic dopamine in controlling the discrete selection between different actions and have nothing to say about response vigor or indeed the free-operant tasks in which it is typically measured.

OBJECTIVES

We seek to accommodate free-operant behavioral tasks within the realm of models of optimal control and thereby capture how dopaminergic and motivational manipulations affect response vigor.

METHODS

We construct an average reward reinforcement learning model in which subjects choose both which action to perform and also the latency with which to perform it. Optimal control balances the costs of acting quickly against the benefits of getting reward earlier and thereby chooses a best response latency.

RESULTS

In this framework, the long-run average rate of reward plays a key role as an opportunity cost and mediates motivational influences on rates and vigor of responding. We review evidence suggesting that the average reward rate is reported by tonic levels of dopamine putatively in the nucleus accumbens.

CONCLUSIONS

Our extension of reinforcement learning models to free-operant tasks unites psychologically and computationally inspired ideas about the role of tonic dopamine in striatum, explaining from a normative point of view why higher levels of dopamine might be associated with more vigorous responding.

Dopamine and incentive learning: a framework for considering antipsychotic medication effects

Hyperfunction of brain dopamine (DA) systems is associated with psychosis in schizophrenia and the medications used to treat schizophrenia are DA receptor blockers. DA also plays a critical role in incentive learning produced by rewarding stimuli. Using DA as the link, these results suggest that psychosis in schizophrenia can be understood from the point of view of excessive incentive learning. Incentive learning is mediated through the non-declarative memory system and may rely on the striatum or medial prefrontal cortex depending on the task. Typical and atypical antipsychotics differentially affect expression of the immediate early gene c-fos, producing greater activity in the striatum and medial prefrontal cortex, respectively. This led to the hypothesis that performance of schizophrenic patients on tasks that depend on the striatum or medial prefrontal cortex will be differentially affected by their antipsychotic medication. Results from a number of published papers supported this dissociation. Furthermore, the effects of two atypical drugs, clozapine and olanzapine, on c-fos expression were different from another atypical, risperidone that resembles the typical antipsychotics. Similarly, in tests of incentive learning, risperidone acted like the typical antipsychotics. Thus, typical and atypical antipsychotic drugs differed in the types of cognitive performance they affected and, furthermore, members of the atypical class differed in their effects on cognition. It remains the task of researchers and clinicians to sort out the symptoms associated with the endogenous illness from possible iatrogenic symptoms resulting from the antipsychotic medications used to treat schizophrenia.

A "good parent" function of dopamine: transient modulation of learning and performance during early stages of training

While extracellular dopamine (DA) concentrations are increased by a wide category of salient stimuli, there is evidence to suggest that DA responses to primary and conditioned rewards may be distinct from those elicited by other types of salient events. A reward-specific mode of neuronal responding would be necessary if DA acts to strengthen behavioral response tendencies under particular environmental conditions or to set current environmental inputs as goals that direct approach responses. As described in this review, DA critically mediates both the acquisition and expression of learned behaviors during early stages of training, however, during later stages, at least some forms of learned behavior become independent of (or less dependent upon) DA transmission for their expression.

Insights into the Role of Dopamine Receptor Systems in Learning and Memory

It is well established that learning and memory are complex processes involving and recruiting different brain modulatory neurotransmitter systems. Considerable evidence points to the involvement of dopamine in various aspects of cognition, and interest has been focused on investigating the clinical relevance of dopamine systems to age-related cognitive decline and manifestations of cognitive impairment in schizophrenia, Alzheimer's disease, Parkinson's disease and other neurodegenerative diseases. In the past decade or so, in spite of the molecular cloning of the five dopamine receptor subtypes, their specific roles in brain function remained inconclusive due to the lack of completely selective ligands that could distinguish between the members of the D1-like and D2-like dopamine receptor families. One of the most important advances in the field of dopamine research has been the generation of mutant mouse models permitting evaluation of the dopaminergic system using gene targeting technologies. These mouse models represent an important approach to explore the functional roles of closely related receptor subtypes. In this review, we present and discuss evidence on the role of dopamine receptors in different aspects of learning and memory at the cellular, molecular and behavioral levels. We compare evidence using conventional pharmacological, lesion or electrophysiological studies with results from mice with targeted deletions of different subtypes of dopamine receptor genes. We particularly focus on dopamine D1 and D2 receptors in an effort to delineate their specific roles in various aspects of cognitive function. We provide strong evidence, from our own recent work as well as others, that dopamine is part of the network that plays a very important role in cognitive function, and that although multiple dopamine receptor subtypes contribute to different aspects of learning and memory, the D1 receptor seems to play a more prominent role in mediating plasticity and specific aspects of cognitive function, including spatial learning and memory processes, reversal learning, extinction learning, and incentive learning.

Conditioned locomotion induced by unilateral intrastriatal administration of apomorphine: D(2) receptor activation is critical but not the expression of the unconditioned response

The present study examined the role of D(1) and D(2) receptors in the conditioning of apomorphine-induced locomotor behavior. A Pavlovian conditioning protocol was used in which rats received 5 daily intrastriatal apomorphine treatments paired or unpaired to an open-field environment followed, 2 days later, by a saline test for conditioning. In the conditioning induction phase, the intrastriatal apomorphine treatment increased locomotor activity expressed as an increased number of sectional crossings and rearings. In the conditioning test, the apomorphine-paired group had significantly higher locomotor activity than the unpaired and vehicle groups, consistent with the development of a conditioned locomotor response. The concomitant blockade of D(1) and D(2) receptors with D(1) (SCH23390) and D(2) (sulpiride) antagonists prevented the apomorphine-induced behavioral response during the induction phase and in the conditioning test. Pretreatment with the D(1) antagonist SCH 23390 also blocked the apomorphine-induced behavioral response during the induction phase but did not block the apomorphine conditioned response. Pretreatment with the selective D(2) antagonist sulpiride blocked the apomorphine behavioral response during the induction phase and in the conditioning test. Altogether, these results indicate that antagonism of either the D(1) or D(2) receptors in the dorsal striatum can block apomorphine-induced locomotor activation but that D(2) but not D(1) antagonism can prevent the development of the apomorphine conditioned response. Altogether, these findings indicate a key role for the D(2) receptor site in the mediation of apomorphine conditioned behavior; and, in addition, that apomorphine conditioned locomotor response can develop without the expression of the locomotor stimulant response during the induction phase of conditioning.