Plastic Corticostriatal Circuits for Action Learning: What's Dopamine Got to Do with It?

Modeling borderline personality disorder based on the neurobehavioral foundation of major personality traits

Borderline personality disorder (BPD) is an exceedingly complex behavioral phenomenon that is in need of conceptual clarification within a larger model of personality disorders (PDs). The association of personality traits to BPD is discussed initially as a means of introducing a dimensional personality approach to understanding BPD. While this model suggests that PDs emerge at the extremes of personality dimensions, attempts to demonstrate such an association have been empirically disappointing and conceptually unilluminating. Therefore, in this article, we attempt to extend such models by outlining the neurobehavioral systems that underlie major personality traits, and highlight the evidence that they are subject to experience-dependent modification that can be enduring through effects on genetic expression, mainly through processes known as epigenetics. It is through such processes that risk for personality disorder may be modified by experience at any point in development, but perhaps especially during early critical periods of development. We conclude by presenting a multidimensional model of PDs, in general, and BPD, in particular, that relies on the concepts developed earlier in the article. Our goal is to provide a guide for novel clinical conceptualization and assessment of PDs, as well as research on their psychobiological nature and pharmacological treatment.

Striatum processes reward differently in adolescents versus adults

Adolescents often respond differently than adults to the same salient motivating contexts, such as peer interactions and pleasurable stimuli. Delineating the neural processing differences of adolescents is critical to understanding this phenomenon, as well as the bases of serious behavioral and psychiatric vulnerabilities, such as drug abuse, mood disorders, and schizophrenia. We believe that age-related changes in the ways salient stimuli are processed in key brain regions could underlie the unique predilections and vulnerabilities of adolescence. Because motivated behavior is the central issue, it is critical that age-related comparisons of brain activity be undertaken during motivational contexts. We compared single-unit activity and local field potentials in the nucleus accumbens (NAc) and dorsal striatum (DS) of adolescent and adult rats during a reward-motivated instrumental task. These regions are involved in motivated learning, reward processing, and action selection. We report adolescent neural processing differences in the DS, a region generally associated more with learning than reward processing in adults. Specifically, adolescents, but not adults, had a large proportion of neurons in the DS that activated in anticipation of reward. More similar response patterns were observed in NAc of the two age groups. DS single-unit activity differences were found despite similar local field potential oscillations. This study demonstrates that in adolescents, a region critically involved in learning and habit formation is highly responsive to reward. It thus suggests a mechanism for how rewards might shape adolescent behavior differently, and for their increased vulnerabilities to affective disorders.

Partial dopamine depletion in MPTP-treated mice differentially altered motor skill learning and action control

Recent findings suggest that the neurotransmitter dopamine (DA) system plays a role in motor control and the acquisition of habits and skills. However, isolating DA-mediated motor learning from motor performance remains challenging as most studies include often severely DA-depleted mice. Using the neurotoxin 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP), we investigated the effect of various degrees of DA-depletion in mice on three tests of motor behaviors: the accelerating rotarod, wire suspension and pole tests. Three protocols were performed to decrease DA synthesis to various extents: 4 injections (i.p.) of 9 mg/kg in 1 day; 4 injections (i.p.) of 15 mg/kg in 1 day; or 5 injections (s.c.) of 30 mg/kg in 5 days. Severity of DA-depletion was assessed by the evaluation of tyrosine hydroxylase (TH) and dopamine transporter levels in the striatum using the Western blot technique. Mice were gathered into four different groups according their TH levels: mild, moderate, marked and severe. In these mice, the general motor abilities such as coordination, motion speed and muscular strength were relatively intact whereas impaired acquisition of skilled behavior occurred in mice with marked and severe reduction in TH levels. Marked and severely DA-depleted mice exhibited lower scores within the first trials of the first training day as well as a much slower progression in the following days on the accelerating rotarod. Based on these results, we conclude that the learning of a skilled behavior is more vulnerable to DA depletion than the DA-mediated control of motor activity.

Preservation of function in Parkinson's disease: what's learning got to do with it?

Dopamine denervation gives rise to abnormal corticostriatal plasticity; however, its role in the symptoms and progression of Parkinson's disease (PD) has not been articulated or incorporated into current clinical models. The 'integrative selective gain' framework proposed here integrates dopaminergic mechanisms known to modulate basal ganglia throughput into a single conceptual framework: (1) synaptic weights, the neural instantiation of accumulated experience and skill modulated by dopamine-dependent plasticity and (2) system gain, the operating parameters of the basal ganglia, modulated by dopamine's on-line effects on cell excitability, glutamatergic transmission and the balance between facilitatory and inhibitory pathways. Within this framework and based on recent work, a hypothesis is presented that prior synaptic weights and established skills can facilitate motor performance and preserve function despite diminished dopamine; however, dopamine denervation induces aberrant corticostriatal plasticity that degrades established synaptic weights and replaces them with inappropriate, inhibitory learning that inverts the function of the basal ganglia resulting in 'anti-optimization' of motor performance. Consequently, mitigating aberrant corticostriatal plasticity represents an important therapeutic objective, as reflected in the long-duration response to levodopa, reinterpreted here as the correction of aberrant learning. It is proposed that viewing aberrant corticostriatal plasticity and learning as a provisional endophenotype of PD would facilitate investigation of this hypothesis.

Molecular substrates of action control in cortico-striatal circuits

The purpose of this review is to describe the molecular mechanisms in the striatum that mediate reward-based learning and action control during instrumental conditioning. Experiments assessing the neural bases of instrumental conditioning have uncovered functional circuits in the striatum, including dorsal and ventral striatal sub-regions, involved in action-outcome learning, stimulus-response learning, and the motivational control of action by reward-associated cues. Integration of dopamine (DA) and glutamate neurotransmission within these striatal sub-regions is hypothesized to enable learning and action control through its role in shaping synaptic plasticity and cellular excitability. The extracellular signal regulated kinase (ERK) appears to be particularly important for reward-based learning and action control due to its sensitivity to combined DA and glutamate receptor activation and its involvement in a range of cellular functions. ERK activation in striatal neurons is proposed to have a dual role in both the learning and performance factors that contribute to instrumental conditioning through its regulation of plasticity-related transcription factors and its modulation of intrinsic cellular excitability. Furthermore, perturbation of ERK activation by drugs of abuse may give rise to behavioral disorders such as addiction.

Identifying neurobiological phenotypes associated with alcohol use disorder severity

Although numerous studies provide general support for the importance of genetic factors in the risk for alcohol use disorders (AUDs), candidate gene and genome-wide studies have yet to identify a set of genetic variations that explain a significant portion of the variance in AUDs. One reason is that alcohol-related phenotypes used in genetic studies are typically based on highly heterogeneous diagnostic categories. Therefore, identifying neurobiological phenotypes related to neuroadaptations that drive the development of AUDs is critical for the future success of genetic and epigenetic studies. One such neurobiological phenotype is the degree to which exposure to alcohol taste cues recruits the basal ganglia, prefrontal cortex, and motor areas, all of which have been shown to have a critical role in addictive behaviors in animal studies. To that end, this study was designed to examine whether cue-elicited responses of these structures are associated with AUD severity in a large sample (n=326) using voxelwise and functional connectivity measures. Results suggested that alcohol cues significantly activated dorsal striatum, insula/orbitofrontal cortex, anterior cingulate cortex, and ventral tegmental area. AUD severity was moderately correlated with regions involved in incentive salience such as the nucleus accumbens and amygdala, and stronger relationships with precuneus, insula, and dorsal striatum. The findings indicate that AUDs are related to neuroadaptations in these regions and that these measures may represent important neurobiological phenotypes for subsequent genetic studies.

Parallel associative processing in the dorsal striatum: segregation of stimulus-response and cognitive control subregions

Although evidence suggests that the dorsal striatum contributes to multiple learning and memory functions, there nevertheless remains considerable disagreement on the specific associative roles of different neuroanatomical subregions. We review evidence indicating that the dorsolateral striatum (DLS) is a substrate for stimulus-response habit formation - incremental strengthening of simple S-R bonds - via input from sensorimotor neocortex while the dorsomedial striatum (DMS) contributes to behavioral flexibility - the cognitive control of behavior - via prefrontal and limbic circuits engaged in relational and spatial information processing. The parallel circuits through dorsal striatum interact with incentive/affective motivational processing in the ventral striatum and portions of the prefrontal cortex leading to overt responding under specific testing conditions. Converging evidence obtained through a detailed task analysis and neurobehavioral assessment is beginning to illuminate striatal subregional interactions and relations to the rest of the mammalian brain.

Neurogenetic and experiential processes underlying major personality traits: implications for modelling personality disorders

Abstract The association of personality traits to personality disorders (PDs) is assumed by many to fit a dimensional model, where PDs emerge at the extremes of personality dimensions. Nevertheless, attempts to demonstrate such an association have been empirically disappointing and conceptually unilluminating. In this article we attempt to extend such models by outlining the neurobehavioural systems that underlie major personality traits, and highlight the evidence that they are subject to experience-dependent modification that can be enduring through effects on genetic expression, mainly through processes known as epigenetics. It is through such processes that risk for personality disorder may be modified by experience at any point in development, but perhaps especially during early critical periods of development. We conclude by presenting a novel multidimensional model of PDs that relies on the concepts developed earlier in the article. Our goal is to provide a guide for research on the psychobiological nature and pharmacological treatment of PDs.

Cortical and striatal contributions to automaticity in information-integration categorization

In information-integration categorization, accuracy is maximized only if information from two or more stimulus components is integrated at some pre-decisional stage. In many cases the optimal strategy is difficult or impossible to describe verbally. Evidence suggests that success in information-integration tasks depends on procedural learning that is mediated largely within the striatum. Although many studies have examined initial information-integration learning, little is known about how automaticity develops in information-integration tasks. To address this issue, each of ten human participants received feedback training on the same information-integration categories for more than 11,000 trials spread over 20 different training sessions. Sessions 2, 4, 10, and 20 were performed inside an MRI scanner. The following results stood out. 1) Automaticity developed between sessions 10 and 20. 2) Pre-automatic performance depended on the putamen, but not on the body and tail of the caudate nucleus. 3) Automatic performance depended only on cortical regions, particularly the supplementary and pre-supplementary motor areas. 4) Feedback processing was mainly associated with deactivations in motor and premotor regions of cortex, and in the ventral lateral prefrontal cortex. 5) The overall effects of practice were consistent with the existing literature on the development of automaticity.

Spike-timing dependent plasticity in striatal interneurons

Basal ganglia, an ensemble of interconnected subcortical nuclei, are involved in adaptive motor planning and procedural learning. Striatum, the primary input nucleus of basal ganglia, extracts the pertinent cortical and thalamic information from background noise in relation with the environmental stimuli and motivation. The striatum comprises different neuronal populations: the GABAergic striatal output neurons, three classes of GABAergic interneurons and the cholinergic cells. Striatal interneurons exert a powerful control of striatal output neuron excitability and therefore shape the cortico-basal ganglia information processing. Besides output neurons, striatal interneurons also receive directly cortical information and are able to adapt their behavior depending on the level of cortical and striatal activation. In this review, we focus on the corticostriatal long-term synaptic efficacy changes occurring in interneurons, and especially the spike-timing dependent plasticity (STDP), as a Hebbian synaptic learning rule. Combined with the striatal local interactions between interneurons and output neurons, we will consider the functional consequences of the interneuron plasticity on the striatal output. This article is part of a Special Issue entitled 'Synaptic Plasticity & Interneurons'.

Human category learning 2.0

During the 1990s and early 2000s, cognitive neuroscience investigations of human category learning focused on the primary goal of showing that humans have multiple category-learning systems and on the secondary goals of identifying key qualitative properties of each system and of roughly mapping out the neural networks that mediate each system. Many researchers now accept the strength of the evidence supporting multiple systems, and as a result, during the past few years, work has begun on the second generation of research questions-that is, on questions that begin with the assumption that humans have multiple category-learning systems. This article reviews much of this second generation of research. Topics covered include (1) How do the various systems interact? (2) Are there different neural systems for categorization and category representation? (3) How does automaticity develop in each system? and (4) Exactly how does each system learn?

The dopaminergic system in the aging brain of Drosophila

Drosophila models of Parkinson's disease are characterized by two principal phenotypes: the specific loss of dopaminergic (DA) neurons in the aging brain and defects in motor behavior. However, an age-related analysis of these baseline parameters in wildtype Drosophila is lacking. Here we analyzed the DA system and motor behavior in aging Drosophila. DA neurons in the adult brain can be grouped into bilateral symmetric clusters, each comprising a stereotypical number of cells. Analysis of TH > mCD8::GFP and cell type-specific MARCM clones revealed that DA neurons show cluster-specific, stereotypical projection patterns with terminal arborization in target regions that represent distinct functional areas of the adult brain. Target areas include the mushroom bodies, involved in memory formation and motivation, and the central complex, involved in the control of motor behavior, indicating that similar to the mammalian brain, DA neurons in the fly brain are involved in the regulation of specific behaviors. Behavioral analysis revealed that Drosophila show an age-related decline in startle-induced locomotion and negative geotaxis. Motion tracking however, revealed that walking activity, and exploration behavior, but not centrophobism increase at late stages of life. Analysis of TH > Dcr2, mCD8::GFP revealed a specific effect of Dcr2 expression on walking activity but not on exploratory or centrophobic behavior, indicating that the siRNA pathway may modulate distinct DA behaviors in Drosophila. Moreover, DA neurons were maintained between early- and late life, as quantified by TH > mCD8::GFP and anti-TH labeling, indicating that adult onset, age-related degeneration of DA neurons does not occur in the aging brain of Drosophila. Taken together, our data establish baseline parameters in Drosophila for the study of Parkinson's disease as well as other disorders affecting DA neurons and movement control.

Movement chunking during sequence learning is a dopamine-dependant process: a study conducted in Parkinson's disease

Chunking of single movements into integrated sequences has been described during motor learning, and we have recently demonstrated that this process involves a dopamine-dependant mechanism in animal (Levesque et al. in Exp Brain Res 182:499-508, 2007; Tremblay et al. in Behav Brain Res 198:231-239, 2009). However, there is no such evidence in human. The aim of the present study was to assess this question in Parkinson's disease (PD), a neurological condition known for its dopamine depletion in the striatum. Eleven PD patients were tested under their usual levodopa medication (ON state), and following a 12-h levodopa withdrawal (OFF state). Patients were compared with 12 healthy participants on a motor learning sequencing task, requiring pressing fourteen buttons in the correct order, which was determined by visual stimuli presented on a computer screen. Learning was assessed from three blocks of 20 trials administered successively. Chunks of movements were intrinsically created by each participant during this learning period. Then, the sequence was shuffled according to the participant's own chunks, generating two new sequences, with either preserved or broken chunks. Those new motor sequences had to be performed separately in a fourth and fifth blocks of 20 trials. Results showed that execution time improved in every group during the learning period (from blocks 1 to 3). However, while motor chunking occurred in healthy controls and ON-PD patients, it did not in OFF-PD patients. In the shuffling conditions, a significant difference was seen between the preserved and the broken chunks conditions for both healthy participants and ON-PD patients, but not for OFF-PD patients. These results suggest that movement chunking during motor sequence learning is a dopamine-dependent process in human.

Spike-timing dependent plasticity in the striatum

The striatum is the major input nucleus of basal ganglia, an ensemble of interconnected sub-cortical nuclei associated with fundamental processes of action-selection and procedural learning and memory. The striatum receives afferents from the cerebral cortex and the thalamus. In turn, it relays the integrated information towards the basal ganglia output nuclei through which it operates a selected activation of behavioral effectors. The striatal output neurons, the GABAergic medium-sized spiny neurons (MSNs), are in charge of the detection and integration of behaviorally relevant information. This property confers to the striatum the ability to extract relevant information from the background noise and select cognitive-motor sequences adapted to environmental stimuli. As long-term synaptic efficacy changes are believed to underlie learning and memory, the corticostriatal long-term plasticity provides a fundamental mechanism for the function of the basal ganglia in procedural learning. Here, we reviewed the different forms of spike-timing dependent plasticity (STDP) occurring at corticostriatal synapses. Most of the studies have focused on MSNs and their ability to develop long-term plasticity. Nevertheless, the striatal interneurons (the fast-spiking GABAergic, NO-synthase and cholinergic interneurons) also receive monosynaptic afferents from the cortex and tightly regulated corticostriatal information processing. Therefore, it is important to take into account the variety of striatal neurons to fully understand the ability of striatum to develop long-term plasticity. Corticostriatal STDP with various spike-timing dependence have been observed depending on the neuronal sub-populations and experimental conditions. This complexity highlights the extraordinary potentiality in term of plasticity of the corticostriatal pathway.

Striatal dopamine modulates basal ganglia output and regulates social context-dependent behavioral variability through D1 receptors

Cortico-basal ganglia (BG) circuits are thought to promote the acquisition of motor skills through reinforcement learning. In songbirds, a specialized portion of the BG is responsible for song learning and plasticity. This circuit generates song variability that underlies vocal experimentation in young birds and modulates song variability depending on the social context in adult birds. When male birds sing in the presence of a female, a social context associated with decreased BG-induced song variability, the extracellular dopamine (DA) level is increased in the avian BG nucleus Area X. These results suggest that DA could trigger song variability changes through its action in Area X. Consistent with this hypothesis, we report that DA delivered to Area X weakens the output signal of the avian cortico-BG circuit. Acting through D(1) receptors, DA reduced responses in Area X to song playback and to electrical stimulation of its afferent cortical nucleus HVC (used as a proper name). Specifically, DA reduced the response to direct excitatory input and decreased firing variability in Area X pallidal neurons, which provide the output to the thalamus. As a consequence, DA delivery in Area X also decreased responses to song playback in the cortical output nucleus of the BG loop, the lateral magnocellular nucleus of the anterior nidopallium. Further, interfering with D(1) receptor transmission in Area X abolished social context-related changes in song variability. In conclusion, we propose that DA acts on D(1) receptors in Area X to modulate the BG output signal and trigger changes in song variability.

Cortical and basal ganglia contributions to habit learning and automaticity

In the 20th century it was thought that novel behaviors are mediated primarily in cortex and that the development of automaticity is a process of transferring control to subcortical structures. However, evidence supports the view that subcortical structures, such as the striatum, make significant contributions to initial learning. More recently, there has been increasing evidence that neurons in the associative striatum are selectively activated during early learning, whereas those in the sensorimotor striatum are more active after automaticity has developed. At the same time, other recent reports indicate that automatic behaviors are striatum- and dopamine-independent, and might be mediated entirely within cortex. Resolving this apparent conflict should be a major goal of future research.

Activation of calcium/calmodulin-dependent protein kinase IIalpha in the striatum by the heteromeric D1-D2 dopamine receptor complex

Synaptic plasticity in the striatum is a key mechanism that underlies processes such as reward related incentive learning and behavioral habit formation resulting from drugs of abuse. Key aspects of these functions are dependent on dopamine transmission as well as activation of calcium/calmodulin-dependent protein kinase IIalpha (CaMKIIalpha). In this study, we examined the ability of a recently identified heteromeric complex composed of D1 and D2 dopamine receptors coupled to Gq/11 to activate striatal CaMKIIalpha. Using the dopaminergic agonist SKF83959, which selectively activates the D1-D2 complex, we demonstrated phosphorylation of CaMKIIalpha at threonine 286, both in heterologous cells and in the murine striatum in vivo. Phosphorylation of CaMKIIalpha by activation of the receptor complex required concurrent agonism of both D1 and D2 receptors and was independent of receptor pathways that modulated adenylyl cyclase. The identification of this novel mechanism by which dopamine may modulate synaptic plasticity has implications for our understanding of striatal-mediated reward and motor function, as well as neuronal disorders in which striatal dopaminergic neurotransmission is involved.

Motor skill learning depends on protein synthesis in the dorsal striatum after training

Functional imaging studies in humans and electrophysiological data in animals suggest that corticostriatal circuits undergo plastic modifications during motor skill learning. In motor cortex and hippocampus circuit plasticity can be prevented by protein synthesis inhibition (PSI) which can interfere with certain forms learning. Here, the hypothesis was tested that inducing PSI in the dorsal striatum by bilateral intrastriatal injection of anisomycin (ANI) in rats interferes with learning a precision forelimb reaching task. Injecting ANI shortly after training on days 1 and 2 during 4 days of daily practice (n = 14) led to a significant impairment of motor skill learning as compared with vehicle-injected controls (n = 15, P = 0.033). ANI did not affect the animals' motivation as measured by intertrial latencies. Also, ANI did not affect reaching performance once learning was completed and performance reached a plateau. These findings demonstrate that PSI in the dorsal striatum after training impairs the acquisition of a novel motor skill. The results support the notion that plasticity in basal ganglia circuits, mediated by protein synthesis, contributes to motor skill learning.

A role for medial prefrontal dopaminergic innervation in instrumental conditioning

To investigate the involvement of dopaminergic projections to the prelimbic and infralimbic cortex in the control of goal-directed responses, a first experiment examined the effect of pretraining 6-OHDA lesions of these cortices. We used outcome devaluation and contingency degradation procedures to separately assess the representation of the outcome as a goal or the encoding of the contingency between the action and its outcome. All groups acquired the instrumental response at a normal rate, indicating that dopaminergic activity in the medial prefrontal cortex is not necessary for the acquisition of instrumental learning. Sham-operated animals showed sensitivity to both outcome devaluation and contingency degradation. Animals with dopaminergic lesions of the prelimbic cortex, but not the infralimbic cortex, failed to adapt their instrumental response to changes in contingency, whereas their response remained sensitive to outcome devaluation. In a second experiment, aimed at determining whether dopamine was specifically needed during contingency changes, we performed microinfusions of the dopamine D(1)/D(2) receptor antagonist flupenthixol in the prelimbic cortex only before contingency degradation sessions. Animals with infusions of flupenthixol failed to adapt their response to changes in contingency, thus replicating the deficit of animals with dopaminergic lesions in Experiment 1. These results demonstrate that dissociable neurobiological mechanisms support action-outcome relationships and goal representation, dopamine signaling in the prelimbic cortex being necessary for the former but not the latter.

Decision theory, reinforcement learning, and the brain

Decision making is a core competence for animals and humans acting and surviving in environments they only partially comprehend, gaining rewards and punishments for their troubles. Decision-theoretic concepts permeate experiments and computational models in ethology, psychology, and neuroscience. Here, we review a well-known, coherent Bayesian approach to decision making, showing how it unifies issues in Markovian decision problems, signal detection psychophysics, sequential sampling, and optimal exploration and discuss paradigmatic psychological and neural examples of each problem. We discuss computational issues concerning what subjects know about their task and how ambitious they are in seeking optimal solutions; we address algorithmic topics concerning model-based and model-free methods for making choices; and we highlight key aspects of the neural implementation of decision making.

Neurons in a forebrain nucleus required for vocal plasticity rapidly switch between precise firing and variable bursting depending on social context

Song is a learned vocal behavior influenced by social interactions. Prior work has suggested that the anterior forebrain pathway (AFP), a specialized pallial-basal ganglia circuit critical for vocal plasticity, mediates the influence of social signals on song. Here, we investigate the signals the AFP sends to song motor areas and their dependence on social context by characterizing singing-related activity of single neurons in the AFP output nucleus LMAN (lateral magnocellular nucleus of the anterior nidopallium). We show that interaction with females causes marked, real-time changes in firing properties of individual LMAN neurons. When males sing to females ("directed"), LMAN neurons exhibit reliable firing of single spikes precisely locked to song. In contrast, when males sing alone ("undirected"), the same LMAN neurons exhibit prominent burst firing and trial-by-trial variability. Burst structure and timing vary substantially across repeated undirected trials. Despite context-dependent differences in firing statistics, the average pattern of song-locked firing for an individual neuron is similar across behavioral contexts, suggesting a common underlying signal. Different LMAN neurons in the same bird, however, exhibit distinct firing patterns, suggesting that subsets of neurons jointly encode song features. Together, our findings demonstrate that behavioral interactions reversibly transform the signaling mode of LMAN neurons. Such changes may contribute to rapid switching of motor activity between variable and precise states. More generally, our results suggest that pallial-basal ganglia circuits contribute to motor learning and production through multiple mechanisms: patterned signals could guide changes in motor output while state-dependent variability could subserve motor exploration.

A dopaminergic basis for working memory, learning and attentional shifting in Parkinsonism

Parkinson's disease (PD) patients exhibit cognitive deficits, including reinforcement learning, working memory (WM) and set shifting. Computational models of the basal ganglia-frontal system posit similar mechanisms for these deficits in terms of reduced dynamic range of striatal dopamine (DA) signals in both medicated and non-medicated states. Here, we report results from the first study that tests PD patients on and off dopaminergic medications in a modified version of the AX continuous performance (AX-CPT) working memory task, consisting of three performance phases and one phase requiring WM associations to be learned via reinforcement feedback. Patients generally showed impairments relative to controls. Medicated patients showed deficits specifically when having to ignore distracting stimuli during the delay. Our models suggest that this impairment is due to medication causing excessive WM updating by enhancing striatal "Go" signals that facilitate such updating, while concurrently suppressing "NoGo" signals. In contrast, patients off medication showed deficits consistent with an overall reduction in striatal DA and associated Go updating signals. Supporting this dichotomy, patients on and off medication both showed attentional shifting deficits, but for different reasons. Deficits in non-medicated patients were consistent with an inability to update the new attentional set, whereas those in medicated patients were evident when having to ignore distractors that had previously been task relevant. Finally, in the feedback-based WM phase, medicated patients were better than unmedicated patients, suggesting a key role of striatal DA in using feedback to update information into WM. These results lend further insight into the role of basal ganglia dopamine in WM and broadly support predictions from neurocomputational models.

Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity

Single action potentials (APs) backpropagate into the higher-order dendrites of striatal spiny projection neurons during cortically driven "up" states. The timing of these backpropagating APs relative to the arriving corticostriatal excitatory inputs determines changes in dendritic calcium concentration. The question arises to whether this spike-timing relative to cortical excitatory inputs can also induce synaptic plasticity at corticostriatal synapses. Here we show that timing of single postsynaptic APs relative to the cortically evoked EPSP determines both the direction and the strength of synaptic plasticity in spiny projection neurons. Single APs occurring 30 ms before the cortically evoked EPSP induced long-term depression (LTD), whereas APs occurring 10 ms after the EPSP induced long-term potentiation (LTP). The amount of plasticity decreased as the time between the APs and EPSPs was increased, with the resulting spike-timing window being broader for LTD than for LTP. In addition, we show that dopamine receptor activation is required for this spike-timing-dependent plasticity (STDP). Blocking dopamine D(1)/D(5) receptors prevented both LTD and LTP induction. In contrast, blocking dopamine D(2) receptors delayed, but did not prevent, LTD and sped induction of LTP. We conclude (1) that, in combination with cortical inputs, single APs evoked in spiny projection neurons can induce both LTP and LTD of the corticostriatal pathway; (2) that the strength and direction of these synaptic changes depend deterministically on the AP timing relative to the arriving cortical inputs; (3) that, whereas dopamine D(2) receptor activation modulates the initial phase of striatal STDP, dopamine D(1)/D(5) receptor activation is critically required for striatal STDP. Thus, the timing of APs relative to cortical inputs alone is not enough to induce corticostriatal plasticity, implying that ongoing activity does not affect synaptic strength unless dopamine receptors are activated.

Endocannabinoid signaling is critical for habit formation

Extended training can induce a shift in behavioral control from goal-directed actions, which are governed by action-outcome contingencies and sensitive to changes in the expected value of the outcome, to habits which are less dependent on action-outcome relations and insensitive to changes in outcome value. Previous studies in rats have shown that interval schedules of reinforcement favor habit formation while ratio schedules favor goal-directed behavior. However, the molecular mechanisms underlying habit formation are not well understood. Endocannabinoids, which can function as retrograde messengers acting through presynaptic CB1 receptors, are highly expressed in the dorsolateral striatum, a key region involved in habit formation. Using a reversible devaluation paradigm, we confirmed that in mice random interval schedules also favor habit formation compared with random ratio schedules. We also found that training with interval schedules resulted in a preference for exploration of a novel lever, whereas training with ratio schedules resulted in less generalization and more exploitation of the reinforced lever. Furthermore, mice carrying either a heterozygous or a homozygous null mutation of the cannabinoid receptor type I (CB1) showed reduced habit formation and enhanced exploitation. The impaired habit formation in CB1 mutant mice cannot be attributed to chronic developmental or behavioral abnormalities because pharmacological blockade of CB1 receptors specifically during training also impairs habit formation. Taken together our data suggest that endocannabinoid signaling is critical for habit formation.

What is reinforced by phasic dopamine signals?

The basal ganglia have been associated with processes of reinforcement learning. A strong line of supporting evidence comes from the recording of dopamine (DA) neurones in behaving monkeys. Unpredicted, biologically salient events, including rewards cause a stereotypic short-latency (70-100 ms), short-duration (100-200 ms) burst of DA activity - the phasic response. This response is widely considered to represent reward prediction errors used as teaching signals in appetitive learning to promote actions that will maximise future reward acquisition. For DA signalling to perform this function, sensory processing afferent to DA neurones should discriminate unpredicted reward-related events. However, the comparative response latencies of DA neurones and orienting gaze-shifts indicate that phasic DA responses are triggered by pre-attentive sensory processing. Consequently, in circumstances where biologically salient events are both spatially and temporally unpredictable, it is unlikely their identity will be known at the time of DA signalling. The limited quality of afferent sensory processing and the precise timing of phasic DA signals, suggests that they may play a less direct role in 'Law of Effect' appetitive learning. Rather, the 'time-stamp' nature of the phasic response, in conjunction with the other signals likely to be present in the basal ganglia at the time of phasic DA input, suggests it may reinforce the discovery of unpredicted sensory events for which the organism is responsible. Furthermore, DA-promoted repetition of preceding actions/movements should enable the system to converge on those aspects of context and behavioural output that lead to the discovery of novel actions.

Dopaminergic mechanisms in actions and habits

Recent studies suggest new ways to interpret dopaminergic actions in goal-directed performance and habitual responding. In the early stages of learning dopamine plays an essential role, but with extended training dopamine appears to play a decreasing role in response expression. Experimental manipulation of dopamine levels alters the correlation of cortical and striatal neural activity in behaving animals, and these dopamine-dependent changes in corticostriatal correlations may be reflected in changes in action selection in the basal ganglia. Consistent with this hypothesis, changes in dopamine signaling brought about by sensitization with amphetamine mimic the transition from goal-directed to habit-based instrumental performance. At the cellular level, dopamine-dependent synaptic plasticity may be important initially, and subsequently lead to more persistent changes that no longer require dopamine. The locus of these actions within the cortical and corticostriatal circuitry is a focus on ongoing research.