Predictive reward signal of dopamine neurons

How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues

After classically conditioned learning, dopaminergic cells in the substantia nigra pars compacta (SNc) respond immediately to unexpected conditioned stimuli (CS) but omit formerly seen responses to expected unconditioned stimuli, notably rewards. These cells play an important role in reinforcement learning. A neural model explains the key neurophysiological properties of these cells before, during, and after conditioning, as well as related anatomical and neurophysiological data about the pedunculopontine tegmental nucleus (PPTN), lateral hypothalamus, ventral striatum, and striosomes. The model proposes how two parallel learning pathways from limbic cortex to the SNc, one devoted to excitatory conditioning (through the ventral striatum, ventral pallidum, and PPTN) and the other to adaptively timed inhibitory conditioning (through the striosomes), control SNc responses. The excitatory pathway generates CS-induced excitatory SNc dopamine bursts. The inhibitory pathway prevents dopamine bursts in response to predictable reward-related signals. When expected rewards are not received, striosomal inhibition of SNc that is unopposed by excitation results in a phasic drop in dopamine cell activity. The adaptively timed inhibitory learning uses an intracellular spectrum of timed responses that is proposed to be similar to adaptively timed cellular mechanisms in the hippocampus and cerebellum. These mechanisms are proposed to include metabotropic glutamate receptor-mediated Ca(2+) spikes that occur with different delays in striosomal cells. A dopaminergic burst in concert with a Ca(2+) spike is proposed to potentiate inhibitory learning. The model provides a biologically predictive alternative to temporal difference conditioning models and explains substantially more data than alternative models.

The neurobiology of cognition

Perhaps the deepest mysteries facing the natural sciences concern the higher functions of the central nervous system. Understanding how the brain gives rise to mental experiences looms as one of the central challenges for science in the new millennium.

The Reward Signal of Midbrain Dopamine Neurons

Dopamine projections from the midbrain to striatum and frontal cortex play a major role in behavioral reactions controlled by rewards. Recent experiments have shown that dopamine neurons code the discrepancy between the prediction and occurrence of rewards and in this way signal a crucial learning term for approach behavior.

The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking

Studies addressing behavioral functions of dopamine (DA) in the nucleus accumbens septi (NAS) are reviewed. A role of NAS DA in reward has long been suggested. However, some investigators have questioned the role of NAS DA in rewarding effects because of its role in aversive contexts. As findings supporting the role of NAS DA in mediating aversively motivated behaviors accumulate, it is necessary to accommodate such data for understanding the role of NAS DA in behavior. The aim of the present paper is to provide a unifying interpretation that can account for the functions of NAS DA in a variety of behavioral contexts: (1) its role in appetitive behavioral arousal, (2) its role as a facilitator as well as an inducer of reward processes, and (3) its presently undefined role in aversive contexts. The present analysis suggests that NAS DA plays an important role in sensorimotor integrations that facilitate flexible approach responses. Flexible approach responses are contrasted with fixed instrumental approach responses (habits), which may involve the nigro-striatal DA system more than the meso-accumbens DA system. Functional properties of NAS DA transmission are considered in two stages: unconditioned behavioral invigoration effects and incentive learning effects. (1) When organisms are presented with salient stimuli (e.g., novel stimuli and incentive stimuli), NAS DA is released and invigorates flexible approach responses (invigoration effects). (2) When proximal exteroceptive receptors are stimulated by unconditioned stimuli, NAS DA is released and enables stimulus representations to acquire incentive properties within specific environmental context. It is important to make a distinction that NAS DA is a critical component for the conditional formation of incentive representations but not the retrieval of incentive stimuli or behavioral expressions based on over-learned incentive responses (i.e., habits). Nor is NAS DA essential for the cognitive perception of environmental stimuli. Therefore, even without normal NAS DA transmission, the habit response system still allows animals to perform instrumental responses given that the tasks take place in fixed environment. Such a role of NAS DA as an incentive-property constructor is not limited to appetitive contexts but also aversive contexts. This dual action of NAS DA in invigoration and incentive learning may explain the rewarding effects of NAS DA as well as other effects of NAS DA in a variety of contexts including avoidance and unconditioned/conditioned increases in open-field locomotor activity. Particularly, the present hypothesis offers the following interpretation for the finding that both conditioned and unconditioned aversive stimuli stimulate DA release in the NAS: NAS DA invigorates approach responses toward 'safety'. Moreover, NAS DA modulates incentive properties of the environment so that organisms emit approach responses toward 'safety' (i.e., avoidance responses) when animals later encounter similar environmental contexts. There may be no obligatory relationship between NAS DA release and positive subjective effects, even though these systems probably interact with other brain systems which can mediate such effects. The present conceptual framework may be valuable in understanding the dynamic interplay of NAS DA neurochemistry and behavior, both normal and pathophysiological.

Cortically driven Fos induction in the striatum is amplified by local dopamine D2-class receptor blockade

Dopamine D2-class receptors have been shown to control the excitability of striatal neurons in response to cortical activation. It has been unclear, however, whether such receptors could regulate the number of striatal neurons activated by cortical stimulation, and thus affect the population response of the striatum to its cortical inputs. We used Fos induction as a readout to measure the ensemble response of striatal neurons to localized stimulation of the frontal cortex and tested for the effects of D2-class dopamine receptor blockade on this response. In freely moving rats, we stimulated the frontal cortex by local epidural application of a dose of a GABAA receptor antagonist (picrotoxin) just threshold for inducing Fos in the striatum. We combined this treatment with D2-class dopamine receptor antagonist treatments at dose levels also just threshold for inducing Fos, using either (i) systemic haloperidol or (ii) intrastriatal (-)sulpiride. Both systemic and intrastriatal blockade of D2-class receptors sharply increased the numbers of striatal neurons exhibiting cortically evoked Fos induction. These findings suggest that local activation of intrastriatal D2-class dopamine receptors can regulate the number of striatal neurons responsive to cortical inputs, thus dynamically shaping the flow of information through the striatum.

Afferent modulation of dopamine neuron firing patterns

In recent studies examining the modulation of dopamine (DA) cell firing patterns, particular emphasis has been placed on excitatory afferents from the prefrontal cortex and the subthalamic nucleus. A number of inconsistencies in recently published reports, however, do not support the contention that tonic activation of NMDA receptors is the sole determinate of DA neuronal firing patterns. The results of work on the basic mechanism of DA firing and the action of apamin suggest that excitatory projections to DA neurons from cholinergic and glutamatergic neurons in the tegmental pedunculopontine nucleus, and/or inhibitory GABAergic projections, are also involved in modulating DA neuron firing behavior.

Building neural representations of habits

Memories for habits and skills ("implicit or procedural memory") and memories for facts ("explicit or episodic memory") are built up in different brain systems and are vulnerable to different neurodegenerative disorders in humans. So that the striatum-based mechanisms underlying habit formation could be studied, chronic recordings from ensembles of striatal neurons were made with multiple tetrodes as rats learned a T-maze procedural task. Large and widely distributed changes in the neuronal activity patterns occurred in the sensorimotor striatum during behavioral acquisition, culminating in task-related activity emphasizing the beginning and end of the automatized procedure. The new ensemble patterns remained stable during weeks of subsequent performance of the same task. These results suggest that the encoding of action in the sensorimotor striatum undergoes dynamic reorganization as habit learning proceeds.

The dopamine hypothesis of reward: past and current status

Mesolimbic dopaminergic neurons are thought to serve as a final common neural pathway for mediating reinforcement processes. However, several recent findings have challenged the view that mesolimbic dopamine has a crucial role in the maintenance of reinforcement processes, or the subjective rewarding actions of natural rewards and drugs of abuse. Instead, there is growing evidence that dopamine is involved in the formation of associations between salient contextual stimuli and internal rewarding or aversive events. This evidence suggests that dopaminergic-neuron activation aids the organism in learning to recognize stimuli associated with such events. Thus, mesolimbic dopaminergic neurons have an important function in the acquisition of behavior reinforced by natural reward and drug stimuli. Furthermore, long-lasting neuroadaptive changes in mesolimbic dopamine-mediated transmission that develop during chronic drug use might contribute to compulsive drug-seeking behavior and relapse.

What are the computations of the cerebellum, the basal ganglia and the cerebral cortex?

The classical notion that the cerebellum and the basal ganglia are dedicated to motor control is under dispute given increasing evidence of their involvement in non-motor functions. Is it then impossible to characterize the functions of the cerebellum, the basal ganglia and the cerebral cortex in a simplistic manner? This paper presents a novel view that their computational roles can be characterized not by asking what are the "goals" of their computation, such as motor or sensory, but by asking what are the "methods" of their computation, specifically, their learning algorithms. There is currently enough anatomical, physiological, and theoretical evidence to support the hypotheses that the cerebellum is a specialized organism for supervised learning, the basal ganglia are for reinforcement learning, and the cerebral cortex is for unsupervised learning.This paper investigates how the learning modules specialized for these three kinds of learning can be assembled into goal-oriented behaving systems. In general, supervised learning modules in the cerebellum can be utilized as "internal models" of the environment. Reinforcement learning modules in the basal ganglia enable action selection by an "evaluation" of environmental states. Unsupervised learning modules in the cerebral cortex can provide statistically efficient representation of the states of the environment and the behaving system. Two basic action selection architectures are shown, namely, reactive action selection and predictive action selection. They can be implemented within the anatomical constraint of the network linking these structures. Furthermore, the use of the cerebellar supervised learning modules for state estimation, behavioral simulation, and encapsulation of learned skill is considered. Finally, the usefulness of such theoretical frameworks in interpreting brain imaging data is demonstrated in the paradigm of procedural learning.

Effect of expected reward magnitude on the response of neurons in the dorsolateral prefrontal cortex of the macaque

The dorsolateral prefrontal cortex plays a critical role in guiding actions that ensue seconds after an instruction. We recorded from neurons in area 46 and the frontal eye field (FEF) while monkeys performed a memory-guided eye movement task. A visual cue signaled whether a small or large liquid reward would accompany a correct response. Many neurons in area 46 responded more when the monkey expected a larger reward. Reward-related enhancement was evident throughout the memory period and was most pronounced when the remembered target appeared in the neuron's response field. Enhancement was not present in the FEF. The mixture of neural signals representing spatial working memory and reward expectation appears to be a distinct feature of area 46.

Single unit activity during a Go/NoGo task in the "prefrontal cortex" of pigeons

Single unit activity was recorded during a delayed auditory/visual Go/NoGo task from the neostriatum caudolaterale (NCL) of pigeons, a multimodal associative avian forebrain structure comparable to the prefrontal cortex (PFC). The animals were trained to mandibulate (to open their beak) during the Go period after which they received a drop of water as reward. Neuronal activity changes were observed during the delay period (DELAY) between auditory and visual stimulation, to the onset of the visual stimulus or to the delivery of the reward. In some neurons, responses were related to the behavioral significance of the stimulus such that the neuronal activity was statistically different between Go and NoGo trials. Moreover, some units anticipated the upcoming reward or changed their firing frequency in a correlated manner prior to beak movements. These neuronal activity patterns suggest that the NCL provides a neural network that participates in the integration and processing of external stimuli in order to generate goal directed behavior.

The role of mesoprefrontal dopamine neurons in the acquisition and expression of conditioned fear in the rat

The mesoprefrontal dopamine neurons are sensitive to physical, pharmacological and psychological stressors. In this report, the role of these neurons in the response to classical fear conditioning was investigated. 6-Hydroxydopamine lesions to the medial prefrontal cortex reduced dopamine levels to about 13% of controls but did not alter behavior during the acquisition of fear conditioning. As expected, conditioned fear increased dopamine metabolism (3,4-dihydroxyphenylacetic acid/dopamine ratio) in the nucleus accumbens in sham-lesion rats. The medial prefrontal 6-hydroxydopamine lesions did not alter this effect. During the expression, however, lesioned rats demonstrated a delayed extinction of the conditioned response without an overall increase in the initial conditioned response. This effect was consistent in rats receiving 6-hydroxydopamine lesions before or after the acquisition period. The calculated rates of extinction showed that the 6-hydroxydopamine lesioned rats had a reduced rate of extinction, but not acquisition, of fear conditioning. The results presented in this manuscript indicate that the mesoprefrontal dopamine neurons are involved in co-ordinating the normal extinction of a fear response but do not alter the acquisition of fearful behaviors. These data are consistent with the conclusion that the mesoprefrontal dopamine neurons are involved in maintaining the animal's response adaptability with regards to stress-related changes in the external environment.

A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task

This study investigated how the simulated response of dopamine neurons to reward-related stimuli could be used as reinforcement signal for learning a spatial delayed response task. Spatial delayed response tasks assess the functions of frontal cortex and basal ganglia in short-term memory, movement preparation and expectation of environmental events. In these tasks, a stimulus appears for a short period at a particular location, and after a delay the subject moves to the location indicated. Dopamine neurons are activated by unpredicted rewards and reward-predicting stimuli, are not influenced by fully predicted rewards, and are depressed by omitted rewards. Thus, they appear to report an error in the prediction of reward, which is the crucial reinforcement term in formal learning theories. Theoretical studies on reinforcement learning have shown that signals similar to dopamine responses can be used as effective teaching signals for learning. A neural network model implementing the temporal difference algorithm was trained to perform a simulated spatial delayed response task. The reinforcement signal was modeled according to the basic characteristics of dopamine responses to novel stimuli, primary rewards and reward-predicting stimuli. A Critic component analogous to dopamine neurons computed a temporal error in the prediction of reinforcement and emitted this signal to an Actor component which mediated the behavioral output. The spatial delayed response task was learned via two subtasks introducing spatial choices and temporal delays, in the same manner as monkeys in the laboratory. In all three tasks, the reinforcement signal of the Critic developed in a similar manner to the responses of natural dopamine neurons in comparable learning situations, and the learning curves of the Actor replicated the progress of learning observed in the animals. Several manipulations demonstrated further the efficacy of the particular characteristics of the dopamine-like reinforcement signal. Omission of reward induced a phasic reduction of the reinforcement signal at the time of the reward and led to extinction of learned actions. A reinforcement signal without prediction error resulted in impaired learning because of perseverative errors. Loss of learned behavior was seen with sustained reductions of the reinforcement signal, a situation in general comparable to the loss of dopamine innervation in Parkinsonian patients and experimentally lesioned animals. The striking similarities in teaching signals and learning behavior between the computational and biological results suggest that dopamine-like reward responses may serve as effective teaching signals for learning behavioral tasks that are typical for primate cognitive behavior, such as spatial delayed responding.

The regulation of forebrain dopamine transmission: relevance to the pathophysiology and psychopathology of schizophrenia

Since the discovery that the therapeutic efficacy of antipsychotic drugs was significantly correlated to their ability to block dopamine D2 receptors, abnormal dopamine transmission in the forebrain has been postulated to underlie psychosis in schizophrenia. In the past 15 years, an impressive amount of clinical and basic research aimed at the study of schizophrenia has indicated that prefrontal and temporal cortical abnormalities may be more important in the etiology of many of the symptoms of schizophrenia, including psychosis. However, the cortical systems that appear to have structural and/or metabolic abnormalities in schizophrenia patients potently regulate forebrain dopamine transmission through a number of mechanisms. In turn, dopamine modulates excitatory transmission mediated by frontal and temporal cortical projections to the basal ganglia and other regions. The present review summarizes the multiple interactions between forebrain DA systems and frontal and temporal corticostriatal transmission. It then examines the role of these interactions in normal behaviors and the psychopathology of schizophrenia.

No evidence for olfactory blocking in honeybee classical conditioning

We use binary odorant compounds to investigate ‘blocking’ in honeybees which learn to associate an odorant (A-D) with a sucrose reward as the reinforcer (+). ‘Blocking’ means that learning about a stimulus B is reduced when trained in compound with a stimulus A that has previously been trained alone. Thus, reinforcement of B in these circumstances is not sufficient to induce learning. Such blocking is a frequently observed phenomenon in vertebrate learning and has also recently been reported in honeybee olfactory learning. To explain blocking, current models of conditioning include cognition-like concepts of attention or expectation which, consequently, seem also to apply to honeybees. Here, we first reproduce a blocking-like effect in an experimental design taken from the literature. We identify two confounding variables in that design and experimentally demonstrate their potential to support a blocking-like effect. After eliminating these confounding variables using a series of different training procedures, the blocking-like effect disappeared. Thus, convincing evidence for blocking in honeybee classical conditioning is at present lacking. This casts doubt on the applicability of cognition-like concepts to honeybees.

Electrophysiological and morphological evidence for a GABAergic nigrostriatal pathway

The electrophysiological and neurochemical characteristics of the nondopaminergic nigrostriatal (NO-DA) cells and their functional response to the degeneration of dopaminergic nigrostriatal (DA) cells were studied. Three different criteria were used to identify NO-DA cells: (1) antidromic response to striatal stimulation with an electrophysiological behavior (firing rate, interspike interval variability, and conduction velocity) different from that of DA cells; (2) retrograde labeling after striatal injection of HRP but showing immunonegativity for DA cell markers (tyrosine hydroxylase, calretinin, calbindin-D28k, and cholecystokinin); and (3) resistance to neurotoxic effect of 6-hydroxydomine (6-OHDA). Our results showed that under normal conditions, 5-8% of nigrostriatal neurons are immunoreactive for GABA, glutamic acid decarboxylase, and parvalbumin, markers of GABAergic neurons, a percentage that reached 81-84% after 6-OHDA injection. Electrophysiologically, NO-DA cells showed a behavior similar to that found in other nigral GABAergic (nigrothalamic) cells. In addition, the 6-OHDA degeneration of DA cells induced a modification of their electrophysiological pattern similar to that found in GABAergic nigrothalamic neurons. Taken together, the present data indicate the existence of a small GABAergic nigrostriatal pathway and suggest their involvement in the pathophysiology of Parkinson's disease.

Effects of haloperidol or SCH-23390 on ethanol-induced conditioned taste aversion

Dopaminergic systems are thought to play an important role in the motivational effects of ethanol. The present experiments examined the effects of haloperidol (a D2 antagonist) and SCH-23390 (a D1 antagonist) on the acquisition of ethanol-induced conditioned taste aversion. In four separate experiments, adult male Swiss-Webster mice were acclimated to a 2-h/day water restriction regimen. Subsequently they received four conditioning trials consisting of 1-h access to either 0.2 M NaCl (experiments 1-3) or 0.15 % w/v saccharin (experiment 4). After flavor access on trials 1-3, subjects received either haloperidol (0.1, 0.15, or 0.3 mg/kg), SCH-23390 (0.05 mg/kg), or saline followed 30 min later by 0, 2, or 4 g/kg ethanol. Ethanol-flavor pairings reduced subsequent flavor intakes, indicating the development of conditioned taste aversion. Neither haloperidol of SCH-23390 reduced flavor intakes in the absence of ethanol. However, both haloperidol and SCH-23390 reduced ethanol-conditioned aversion depending on ethanol dose and conditioned flavor. These results are consistent with the notion that dopaminergic processes are important for the development of ethanol-induced conditioned taste aversion, and the notion that dopaminergic receptor systems influence both positive and negative motivational effects of ethanol.

Properties and plasticity of excitatory synapses on dopaminergic and GABAergic cells in the ventral tegmental area

Excitatory inputs to the ventral tegmental area (VTA) influence the activity of both dopaminergic (DA) and GABAergic (GABA) cells, yet little is known about the basic properties of excitatory synapses on these two cell types. Using a midbrain slice preparation and whole-cell recording techniques, we found that excitatory synapses on DA and GABA cells display several differences. Synapses on DA cells exhibit a depression in response to repetitive activation, are minimally affected by the GABAB receptor agonist baclofen, and express NMDA receptor-dependent long-term potentiation (LTP). In contrast, synapses on GABA cells exhibit a facilitation in response to repetitive activation, are depressed significantly by baclofen, and do not express LTP. The relative contribution of NMDA and non-NMDA receptors to the synaptic currents recorded from the two cell types is the same as is the depression of synaptic transmission elicited by the application of adenosine, serotonin, or methionine enkephalin (met-enkephalin). The significant differences in the manner in which excitatory synaptic inputs to DA and GABA cells in the VTA can be modulated have potentially important implications for understanding the behavior of VTA neurons during normal behavior and during pathological states such as addiction.

Is the short-latency dopamine response too short to signal reward error?

Unexpected stimuli that are behaviourally significant have the capacity to elicit a short-latency, short-duration burst of firing in mesencephalic dopaminergic neurones. An influential interpretation of the experimental data that characterize this response proposes that dopaminergic neurones have a crucial role in reinforcement learning because they signal error in the prediction of future reward. In this article we propose a different functional role for this 'short-latency dopamine response' in the mechanisms that underlie associative learning. We suggest that the initial burst of dopaminergic-neurone firing could represent an essential component in the process of switching attentional and behavioural selections to unexpected, behaviourally important stimuli. This switching response could be a crucial prerequisite for associative learning and might be part of a general short-latency response that is mediated by catecholamines and prepares the organism for an appropriate reaction to biologically significant events. Any act which in a given situation produces satisfaction becomes associated with that situation so that when the situation recurs the act is more likely than before to recur also. E.L. Thorndike (1911) 1.

Dopaminergic synapses mediate neuronal changes in an analogue of operant conditioning

Feeding behavior in Aplysia can be modified by operant conditioning in which contingent reinforcement is conveyed by the esophageal nerve (E n.). A neuronal analogue of this conditioning in the isolated buccal ganglia was developed by using stimulation of E n. as an analogue of contingent reinforcement. Previous studies indicated that E n. may release dopamine. We used a dopamine antagonist (methylergonovine) to investigate whether dopamine mediated the enhancement of motor patterns in the analogue of operant conditioning. Methylergonovine blocked synaptic connections from the reinforcement pathway and the contingent-dependent enhancement of the reinforced pattern. These results suggest that dopamine mediates at least part of the neuronal modifications induced by contingent reinforcement.

Dopamine and acetylcholine elevation on lever-press acquisition in rat prefrontal cortex

To determine whether the rat medial prefrontal cortex (PFC) is involved in acquiring operant learning, we observed changes in extracellular concentration of dopamine (DA) and acetylcholine (ACh) in the rat medial PFC during lever-press acquisition (acquisition group) or retrieval (retention group) using in vivo microdialysis. We found that DA or ACh elevation related to acquisition occurred. DA elevation was observed in the acquisition group only. These results indicate that the medical PFC is related to acquisition, and suggest that interaction between DA and ACh may be involved in learning acquisition.