Neuronal correlates of instrumental learning in the dorsal striatum

Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies

Behavior in spatial navigation is often organized into map-based (place-driven) vs. map-free (cue-driven) strategies; behavior in operant conditioning research is often organized into goal-directed vs. habitual strategies. Here we attempt to unify the two. We review one powerful theory for distinct forms of learning during instrumental conditioning, namely model-based (maintaining a representation of the world) and model-free (reacting to immediate stimuli) learning algorithms. We extend these lines of argument to propose an alternative taxonomy for spatial navigation, showing how various previously identified strategies can be distinguished as “model-based” or “model-free” depending on the usage of information and not on the type of information (e.g., cue vs. place). We argue that identifying “model-free” learning with dorsolateral striatum and “model-based” learning with dorsomedial striatum could reconcile numerous conflicting results in the spatial navigation literature. From this perspective, we further propose that the ventral striatum plays key roles in the model-building process. We propose that the core of the ventral striatum is positioned to learn the probability of action selection for every transition between states of the world. We further review suggestions that the ventral striatal core and shell are positioned to act as “critics” contributing to the computation of a reward prediction error for model-free and model-based systems, respectively.

Response inhibition signals and miscoding of direction in dorsomedial striatum

The ability to inhibit action is critical for everyday behavior and is affected by a variety of disorders. Behavioral control and response inhibition is thought to depend on a neural circuit that includes the dorsal striatum, yet the neural signals that lead to response inhibition and its failure are unclear. To address this issue, we recorded from neurons in rat dorsomedial striatum (mDS) in a novel task in which rats responded to a spatial cue that signaled that reward would be delivered either to the left or to the right. On 80% of trials rats were instructed to respond in the direction cued by the light (GO). On 20% of trials a second light illuminated instructing the rat to refrain from making the cued movement and move in the opposite direction (STOP). Many neurons in mDS encoded direction, firing more or less strongly for GO movements made ipsilateral or contralateral to the recording electrode. Neurons that fired more strongly for contralateral GO responses were more active when rats were faster, showed reduced activity on STOP trials, and miscoded direction on errors, suggesting that when these neurons were overly active, response inhibition failed. Neurons that decreased firing for contralateral movement were excited during trials in which the rat was required to stop the ipsilateral movement. For these neurons activity was reduced when errors were made and was negatively correlated with movement time suggesting that when these neurons were less active on STOP trials, response inhibition failed. Finally, the activity of a significant number of neurons represented a global inhibitory signal, firing more strongly during response inhibition regardless of response direction. Breakdown by cell type suggests that putative medium spiny neurons (MSNs) tended to fire more strongly under STOP trials, whereas putative interneurons exhibited both activity patterns.

Striatal mechanisms underlying movement, reinforcement, and punishment

Direct and indirect pathway striatal neurons are known to exert opposing control over motor output. In this review, we discuss a hypothetical extension of this framework, in which direct pathway striatal neurons also mediate reinforcement and reward, and indirect pathway neurons mediate punishment and aversion.

Lost in transition: aging-related changes in executive control by the medial prefrontal cortex

Neural correlates of aging in the medial prefrontal cortex (mPFC) were studied using an operant delayed response task. The task used blocks of trials with memory-guided (delayed alternation) and visually-guided (stimulus-response) responding. Older rats (24 months) performed at a slow pace compared with younger rats (6 months). They wasted time engaged in nonessential behaviors (e.g., licking on spouts beyond the period of reward delivery) and were slow to respond at the end of the delay period. Aged mPFC neurons showed normal spatial processing. They differed from neurons in younger rats by having reduced modulations by imperative stimuli indicating reward availability and reduced activity associated with response latencies for reward collection. Older rats showed reduced sensitivity to imperative stimuli at three levels of neural activity: reduced fractions of neurons with changes in firing rate around the stimulus, reduced correlation over neurons at the time of the stimulus as measured with analysis of population activity, and reduced amplitudes of event-related fluctuations in intracortical field potentials at the time of the imperative stimulus. Our findings suggest that aging alters the encoding of time-sensitive information and impairs the ability of prefrontal networks to keep subjects "on task."

Acquisition of "Start" and "Stop" response thresholds in peak-interval timing is differentially sensitive to protein synthesis inhibition in the dorsal and ventral striatum

Time-based decision-making in peak-interval timing procedures involves the setting of response thresholds for the initiation (“Start”) and termination (“Stop”) of a response sequence that is centered on a target duration. Using intracerebral infusions of the protein synthesis inhibitor anisomycin, we report that the acquisition of the “Start” response depends on normal functioning (including protein synthesis) in the dorsal striatum (DS), but not the ventral striatum (VS). Conversely, disruption of the VS, but not the DS, impairs the acquisition of the “Stop” response. We hypothesize that the dorsal and ventral regions of the striatum function as a competitive neural network that encodes the temporal boundaries marking the beginning and end of a timed response sequence.

Encoding of both positive and negative reward prediction errors by neurons of the primate lateral prefrontal cortex and caudate nucleus

Learning can be motivated by unanticipated success or unexpected failure. The former encourages us to repeat an action or activity, whereas the latter leads us to find an alternative strategy. Understanding the neural representation of these unexpected events is therefore critical to elucidate learning-related circuits. We examined the activity of neurons in the lateral prefrontal cortex (PFC) and caudate nucleus of monkeys as they performed a trial-and-error learning task. Unexpected outcomes were widely represented in both structures, and neurons driven by unexpectedly negative outcomes were as frequent as those activated by unexpectedly positive outcomes. Moreover, both positive and negative reward prediction errors (RPEs) were represented primarily by increases in firing rate, unlike the manner in which dopamine neurons have been observed to reflect these values. Interestingly, positive RPEs tended to appear with shorter latency than negative RPEs, perhaps reflecting the mechanism of their generation. Last, in the PFC but not the caudate, trial-by-trial variations in outcome-related activity were linked to the animals' subsequent behavioral decisions. More broadly, the robustness of RPE signaling by these neurons suggests that actor-critic models of reinforcement learning in which the PFC and particularly the caudate are considered primarily to be "actors" rather than "critics," should be reconsidered to include a prominent evaluative role for these structures.

Striatum processes reward differently in adolescents versus adults

Adolescents often respond differently than adults to the same salient motivating contexts, such as peer interactions and pleasurable stimuli. Delineating the neural processing differences of adolescents is critical to understanding this phenomenon, as well as the bases of serious behavioral and psychiatric vulnerabilities, such as drug abuse, mood disorders, and schizophrenia. We believe that age-related changes in the ways salient stimuli are processed in key brain regions could underlie the unique predilections and vulnerabilities of adolescence. Because motivated behavior is the central issue, it is critical that age-related comparisons of brain activity be undertaken during motivational contexts. We compared single-unit activity and local field potentials in the nucleus accumbens (NAc) and dorsal striatum (DS) of adolescent and adult rats during a reward-motivated instrumental task. These regions are involved in motivated learning, reward processing, and action selection. We report adolescent neural processing differences in the DS, a region generally associated more with learning than reward processing in adults. Specifically, adolescents, but not adults, had a large proportion of neurons in the DS that activated in anticipation of reward. More similar response patterns were observed in NAc of the two age groups. DS single-unit activity differences were found despite similar local field potential oscillations. This study demonstrates that in adolescents, a region critically involved in learning and habit formation is highly responsive to reward. It thus suggests a mechanism for how rewards might shape adolescent behavior differently, and for their increased vulnerabilities to affective disorders.

Preservation of function in Parkinson's disease: what's learning got to do with it?

Dopamine denervation gives rise to abnormal corticostriatal plasticity; however, its role in the symptoms and progression of Parkinson's disease (PD) has not been articulated or incorporated into current clinical models. The 'integrative selective gain' framework proposed here integrates dopaminergic mechanisms known to modulate basal ganglia throughput into a single conceptual framework: (1) synaptic weights, the neural instantiation of accumulated experience and skill modulated by dopamine-dependent plasticity and (2) system gain, the operating parameters of the basal ganglia, modulated by dopamine's on-line effects on cell excitability, glutamatergic transmission and the balance between facilitatory and inhibitory pathways. Within this framework and based on recent work, a hypothesis is presented that prior synaptic weights and established skills can facilitate motor performance and preserve function despite diminished dopamine; however, dopamine denervation induces aberrant corticostriatal plasticity that degrades established synaptic weights and replaces them with inappropriate, inhibitory learning that inverts the function of the basal ganglia resulting in 'anti-optimization' of motor performance. Consequently, mitigating aberrant corticostriatal plasticity represents an important therapeutic objective, as reflected in the long-duration response to levodopa, reinterpreted here as the correction of aberrant learning. It is proposed that viewing aberrant corticostriatal plasticity and learning as a provisional endophenotype of PD would facilitate investigation of this hypothesis.

Motoring ahead with rodents

How neural circuits underlie the acquisition and control of learned motor behaviors has traditionally been explored in monkeys and, more recently, songbirds. The development of genetic tools for functional circuit analysis in rodents, the availability of transgenic animals with well characterized phenotypes, and the relative ease with which rats and mice can be trained to perform various motor tasks, make rodents attractive models for exploring the neural circuit mechanisms underlying the acquisition and production of learned motor skills. Here we discuss the advantages and drawbacks of this approach, review recent trends and results, and outline possible strategies for wider adoption of rodents as a model system for complex motor learning.

Differential roles of caudate nucleus and putamen during instrumental learning

The dorsal striatum is crucial for the acquisition and consolidation of instrumental behaviour, but the underlying computations and internal dynamics remain elusive. To address this issue, we combined a model of key computations supporting decision-making during instrumental learning with human behavioural and functional magnetic resonance imaging (fMRI) data. The results showed that the associative and sensorimotor dorsal striatum host complementary computations that, we suggest, may differentially support goal-directed and habitual processes. The anterior caudate nucleus integrates information about performance and cognitive control demands, whereas the putamen tracks how likely the conditioning stimuli lead to correct response. Contrary to current models, the putamen is recruited during initial acquisition. As the exploratory phase proceeds, the relative contribution of the caudate nucleus becomes dominant over the putamen. During early consolidation, caudate nucleus and putamen settle to asymptotic values and share control. We then investigated how dorsal striatal computations may affect decision-making. We found that portion of reaction times' variance parallels the combined cost associated with the dorsal striatal computations. Overall, our findings provide a deeper insight into the functional heterogeneity within the dorsal striatum and suggest that the dynamic interplay between caudate nucleus and putamen, rather than their serial recruitment, underlies the acquisition and early consolidation of instrumental behaviours.

Selective theta-synchronization of choice-relevant information subserves goal-directed behavior

Theta activity reflects a state of rhythmic modulation of excitability at the level of single neuron membranes, within local neuronal groups and between distant nodes of a neuronal network. A wealth of evidence has shown that during theta states distant neuronal groups synchronize, forming networks of spatially confined neuronal clusters at specific time periods during task performance. Here, we show that a functional commonality of networks engaging in theta rhythmic states is that they emerge around decision points, reflecting rhythmic synchronization of choice-relevant information. Decision points characterize a point in time shortly before a subject chooses to select one action over another, i.e., when automatic behavior is terminated and the organism reactivates multiple sources of information to evaluate the evidence for available choices. As such, decision processes require the coordinated retrieval of choice-relevant information including (i) the retrieval of stimulus evaluations (stimulus–reward associations) and reward expectancies about future outcomes, (ii) the retrieval of past and prospective memories (e.g., stimulus–stimulus associations), (iii) the reactivation of contextual task rule representations (e.g., stimulus–response mappings), along with (iv) an ongoing assessment of sensory evidence. An increasing number of studies reveal that retrieval of these multiple types of information proceeds within few theta cycles through synchronized spiking activity across limbic, striatal, and cortical processing nodes. The outlined evidence suggests that evolving spatially and temporally specific theta synchronization could serve as the critical correlate underlying the selection of a choice during goal-directed behavior.

Differential influence of levodopa on reward-based learning in Parkinson's disease

The mesocorticolimbic dopamine (DA) system linking the dopaminergic midbrain to the prefrontal cortex and subcortical striatum has been shown to be sensitive to reinforcement in animals and humans. Within this system, coexistent segregated striato-frontal circuits have been linked to different functions. In the present study, we tested patients with Parkinson's disease (PD), a neurodegenerative disorder characterized by dopaminergic cell loss, on two reward-based learning tasks assumed to differentially involve dorsal and ventral striato-frontal circuits. 15 non-depressed and non-demented PD patients on levodopa monotherapy were tested both on and off medication. Levodopa had beneficial effects on the performance on an instrumental learning task with constant stimulus-reward associations, hypothesized to rely on dorsal striato-frontal circuits. In contrast, performance on a reversal learning task with changing reward contingencies, relying on ventral striato-frontal structures, was better in the unmedicated state. These results are in line with the "overdose hypothesis" which assumes detrimental effects of dopaminergic medication on functions relying upon less affected regions in PD. This study demonstrates, in a within-subject design, a double dissociation of dopaminergic medication and performance on two reward-based learning tasks differing in regard to whether reward contingencies are constant or dynamic. There was no evidence for a dose effect of levodopa on reward-based behavior with the patients' actual levodopa dose being uncorrelated to their performance on the reward-based learning tasks.

Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning

The basal ganglia are implicated in a remarkable range of functions influencing emotion and cognition as well as motor behavior. Current models of basal ganglia function hypothesize that parallel limbic, associative, and motor cortico-basal ganglia loops contribute to this diverse set of functions, but little is yet known about how these loops operate and how their activities evolve during learning. To address these issues, we recorded simultaneously in sensorimotor and associative regions of the striatum as rats learned different versions of a conditional T-maze task. We found highly contrasting patterns of activity in these regions during task performance and found that these different patterns of structured activity developed concurrently, but with sharply different dynamics. Based on the region-specific dynamics of these patterns across learning, we suggest a working model whereby dorsomedial associative loops can modulate the access of dorsolateral sensorimotor loops to the control of action.

Neural correlates of stimulus-response and response-outcome associations in dorsolateral versus dorsomedial striatum

Considerable evidence suggests that there is functional heterogeneity in the control of behavior by the dorsal striatum. Dorsomedial striatum may support goal-directed behavior by representing associations between responses and outcomes (R-O associations). The dorsolateral striatum, in contrast, may support motor habits by encoding associations between stimuli and responses (S-R associations). To test whether neural correlates in striatum in fact conform to this pattern, we recorded single-units in dorsomedial and dorsolateral striatum of rats performing a task in which R-O contingencies were manipulated independently of S-R contingencies. Among response-selective neurons in both regions, activity was significantly modulated by the initial stimulus, providing evidence of S-R encoding. Similarly, response selectivity was significantly modulated by the associated outcome in both regions, providing evidence of R-O encoding. In both regions, this outcome-modulation did not seem to reflect the relative value of the expected outcome, but rather its specific identity. Finally, in both regions we found correlates of the available action-outcome contingencies reflected in the baseline activity of many neurons. These results suggest that differences in information content in these two regions may not determine the differential roles they play in controlling behavior, demonstrated in previous studies.

Absence of cue-evoked firing in rat dorsolateral striatum neurons

The rat dorsolateral striatum (DLS) has been implicated in habit formation. Previous studies in our laboratory found that as animals acquired a motor habit or remained goal-directed, tested by reward devaluation, the vast majority of DLS neurons decreased firing rates during the same responses over training days. However, mixed results have been reported in the literature regarding whether DLS neurons exhibit cue reactivity. In the present study, we reanalyzed a sample of DLS neurons in a task in which habitual behavior was acquired (dataset of Tang et al., 2007 [45]) and found that somatic sensorimotor as well as nonsomatomotor neurons of the DLS exhibited no cue-evoked firing. A second sample of DLS neurons related to licking in a task in which goal-directed behavior occurred (dataset of Tang et al., 2009 [46]) was also reanalyzed for cue-evoked correlates. Although behavior was cue guided, lick neurons did not exhibit cue-evoked firing. Given the complete absence of cue-related firing during habitual or goal-directed behavior, adaptations in DLS firing patterns may be regulated by movement-related learning rather than nonsomatosensory cues, consistent with convergent S1 and M1 afferents to the region. Striatal cue reactivity in the rat, is likely mediated within the dorsomedial and ventromedial striatum, in line with associative and limbic afferents to these regions, respectively.

Singing-related neural activity distinguishes four classes of putative striatal neurons in the songbird basal ganglia

The striatum-the primary input nucleus of the basal ganglia-plays a major role in motor control and learning. Four main classes of striatal neuron are thought to be essential for normal striatal function: medium spiny neurons, fast-spiking interneurons, cholinergic tonically active neurons, and low-threshold spiking interneurons. However, the nature of the interaction of these neurons during behavior is poorly understood. The songbird area X is a specialized striato-pallidal basal ganglia nucleus that contains two pallidal cell types as well as the same four cell types found in the mammalian striatum. We recorded 185 single units in Area X of singing juvenile birds and, based on singing-related firing patterns and spike waveforms, find six distinct cell classes--two classes of putative pallidal neuron that exhibited a high spontaneous firing rate (> 60 Hz), and four cell classes that exhibited low spontaneous firing rates characteristic of striatal neurons. In this study, we examine in detail the four putative striatal cell classes. Type-1 neurons were the most frequently encountered and exhibited sparse temporally precise singing-related activity. Type-2 neurons were distinguished by their narrow spike waveforms and exhibited brief, high-frequency bursts during singing. Type-3 neurons were tonically active and did not burst, whereas type-4 neurons were inactive outside of singing and during singing generated long high-frequency bursts that could reach firing rates over 1 kHz. Based on comparison to the mammalian literature, we suggest that these four putative striatal cell classes correspond, respectively, to the medium spiny neurons, fast-spiking interneurons, tonically active neurons, and low-threshold spiking interneurons that are known to reside in area X.

The dorsomedial striatum reflects response bias during learning

Previous studies have established that neurons in the dorsomedial striatum track the behavioral significance of external stimuli, are sensitive to contingencies between actions and outcomes, and show rapid flexibility in representing task-related information. Here, we describe how neural activity in the dorsomedial striatum changes during the initial acquisition of a Go/NoGo task and during an initial reversal of stimulus-response contingencies. Rats made nosepoke responses over delay periods and then received one of two acoustic stimuli. Liquid rewards were delivered after one stimulus (S+) if the rats made a Go response (entering a reward port on the opposite wall of the chamber). If a Go response was made to other stimulus (S-), rats experienced a timeout. On 10% of trials, no stimulus was presented. These trials were used to assess response bias, the animals' tendency to collect reward independent of the stimulus. Response bias increased during the reversal, corresponding to the animals' uncertainty about the stimulus-response contingencies. Most task-modulated neurons fired during the response at the end of the delay period. The fraction of response-modulated neurons was correlated with response bias and neural activity was sensitive to the behavioral response made on the previous trial. During initial task acquisition and initial reversal learning, there was a remarkable change in the percentages of neurons that fired in relation to the task events, especially during withdrawal from the nosepoke aperture. These results suggest that changes in task-related activity in the dorsomedial striatum during learning are driven by the animal's bias to collect rewards.

Stable encoding of task structure coexists with flexible coding of task events in sensorimotor striatum

The sensorimotor striatum, as part of the brain's habit circuitry, has been suggested to store fixed action values as a result of stimulus-response learning and has been contrasted with a more flexible system that conditionally assigns values to behaviors. The stability of neural activity in the sensorimotor striatum is thought to underlie not only normal habits but also addiction and clinical syndromes characterized by behavioral fixity. By recording in the sensorimotor striatum of mice, we asked whether neuronal activity acquired during procedural learning would be stable even if the sensory stimuli triggering the habitual behavior were altered. Contrary to expectation, both fixed and flexible activity patterns appeared. One, representing the global structure of the acquired behavior, was stable across changes in task cuing. The second, a fine-grain representation of task events, adjusted rapidly. Such dual forms of representation may be critical to allow motor and cognitive flexibility despite habitual performance.