Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences

Assessing the influence of dopamine and mindfulness on the formation of routines in visual search

Given experience in cluttered but stable visual environments, our eye-movements form stereotyped routines that sample task-relevant locations, while not mixing-up routines between similar task-settings. Both dopamine signaling and mindfulness have been posited as factors that influence the formation of such routines, yet quantification of their impact remains to be tested in healthy humans. Over two sessions, participants searched through grids of doors to find hidden targets, using a gaze-contingent display. Within each session, door scenes appeared in either one of two colors, with each color signaling a differing set of likely target locations. We derived measures for how well target locations were learned (target-accuracy), how routine were sets of eye-movements (stereotypy), and the extent of interference between the two scenes (setting-accuracy). Participants completed two sessions, where they were administered either levodopa (dopamine precursor) or placebo (vitamin C), under double-blind counterbalanced conditions. Dopamine and trait mindfulness (assessed by questionnaire) interacted to influence both target-accuracy and stereotypy. Increasing dopamine improved accuracy and reduced stereotypy for high mindfulness scorers, but induced the opposite pattern for low mindfulness scorers. Dopamine also disrupted setting-accuracy invariant to mindfulness. Our findings show that mindfulness modulates the impact of dopamine on the target-accuracy and stereotypy of eye-movement routines, whereas increasing dopamine promotes interference between task-settings, regardless of mindfulness. These findings provide a link between non-human and human models regarding the influence of dopamine on the formation of task-relevant eye-movement routines and provide novel insights into behavior-trait factors that modulate the use of experience when building adaptive repertoires.

Synchronous Measurements of Extracellular Action Potentials and Neurochemical Activity with Carbon Fiber Electrodes in Nonhuman Primates

Measuring the dynamic relationship between neuromodulators, such as dopamine, and neuronal action potentials is imperative to understand how these fundamental modes of neural signaling interact to mediate behavior. We developed methods to measure concurrently dopamine and extracellular action potentials (i.e., spikes) in monkeys. Standard fast-scan cyclic voltammetric (FSCV) electrochemical (EChem) and electrophysiological (EPhys) recording systems are combined and used to collect spike and dopamine signals, respectively, from an array of carbon fiber (CF) sensors implanted in the monkey striatum. FSCV requires the application of small voltages at the implanted sensors to measure redox currents generated from target molecules, such as dopamine. These applied voltages create artifacts at neighboring EPhys measurement sensors which may lead to misclassification of these signals as physiological spikes. Therefore, simple automated temporal interpolation algorithms were designed to remove these artifacts and enable accurate spike extraction. We validated these methods using simulated artifacts and demonstrated an average spike recovery rate of 84.5%. We identified and discriminated cell type-specific units in the monkey striatum that were shown to correlate to specific behavioral task parameters related to reward size and eye movement direction. Synchronously recorded spike and dopamine signals displayed contrasting relations to the task variables, suggesting a complex relationship between these two modes of neural signaling. Future application of our methods will help advance our understanding of the interactions between neuromodulator signaling and neuronal activity, to elucidate more detailed mechanisms of neural circuitry and plasticity mediating behaviors in health and in disease.

Striosomes Target Nigral Dopamine-Containing Neurons via Direct-D1 and Indirect-D2 Pathways Paralleling Classic Direct-Indirect Basal Ganglia Systems

The classic output pathways of the basal ganglia are known as the direct-D1 and indirect-D2, or Go/No-Go, pathways. Balance of the activity in these canonical direct-indirect pathways is considered a core requirement for normal movement control, and their imbalance is a major etiologic factor in movement disorders including Parkinsons disease. We present evidence for a conceptually equivalent parallel system of direct-D1 and indirect-D2 pathways that arise from striatal projection neurons (SPNs) of the striosome compartment rather than from the matrix. These striosomal direct (S-D1) and indirect (S-D2) pathways, as a pair, target dopamine-containing neurons of the substantia nigra (SNpc) instead of the motor output nuclei of the basal ganglia. The novel anatomically and functionally distinct indirect-D2 striosomal pathway targets dopaminergic SNpc cells indirectly via a core region of the external pallidum (GPe). We demonstrate that these S-D1 and S-D2 pathways oppositely modulate striatal dopamine release in freely behaving mice under open-field conditions and oppositely modulate locomotor and other movements. These S-D1 and S-D2 pathways further exhibit different, time-dependent responses during performance of a probabilistic decision-making maze task and respond differently to rewarding and aversive stimuli. These contrasts depend on mediolateral and anteroposterior striatal locations of the SPNs as are the classic direct and indirect pathways. The effects of S-D1 and S-D2 stimulation on striatal dopamine release and voluntary locomotion are nearly opposite. The parallelism of the direct-indirect circuit design motifs of the striosomal S-D and S-D2 circuits and canonical matrix M-D1 and M-D2, and their contrasting behavioral effects, call for a major reformulation of the classic direct-indirect pathway model of basal ganglia function. Given that some striosomes receive limbic and association cortical inputs, the S-D1 and S-D2 circuits likely influence motivation for action and behavioral learning, complementing and possibly reorienting the motoric activities of the canonical matrix pathways. At a fundamental level, these findings suggest a unifying framework for aligning two sets of circuits that share the organizational motif of opponent D1 and D2 regulation, but that have different outputs and can even have opposite polarities in their targets and effects, albeit conditioned by striatal topography. Our findings further delineate a potentially therapeutically important set of pathways influencing dopamine, including a D2 receptor-linked S-D2 pathway likely unknowingly targeted by administration of many therapeutic drugs including those for Parkinsons disease. The novel parallel pathway model that we propose here could help to account for the normally integrated modulatory influence of the basal ganglia on motivation for actions as well as the actions themselves.

Synchronous Measurements of Extracellular Action Potentials and Neurochemical Activity with Carbon Fiber Electrodes in Nonhuman Primates

Measuring the dynamic relationship between neuromodulators, such as dopamine, and neuronal action potentials is imperative to understand how these fundamental modes of neural signaling interact to mediate behavior. Here, we developed methods to measure concurrently dopamine and extracellular action potentials (i.e., spikes) and applied these in a monkey performing a behavioral task. Standard fast-scan cyclic voltammetric (FSCV) electrochemical (EChem) and electrophysiological (EPhys) recording systems are combined and used to collect spike and dopamine signals, respectively, from an array of carbon fiber (CF) sensors implanted in the monkey striatum. FSCV requires the application of small voltages at the implanted sensors to measure redox currents generated from target molecules, such as dopamine. These applied voltages create artifacts at neighboring EPhys-measurement sensors, producing signals that may falsely be classified as physiological spikes. Therefore, simple automated temporal interpolation algorithms were designed to remove these artifacts and enable accurate spike extraction. We validated these methods using simulated artifacts and demonstrated an average spike recovery rate of 84.5%. This spike extraction was performed on data collected from concurrent EChem and EPhys recordings made in a task-performing monkey to discriminate cell-type specific striatal units. These identified units were shown to correlate to specific behavioral task parameters related to reward size and eye-movement direction. Synchronous measures of spike and dopamine signals displayed contrasting relations to the behavioral task parameters, as taken from our small set of representative data, suggesting a complex relationship between these two modes of neural signaling. Future application of our methods will help advance our understanding of the interactions between neuromodulator signaling and neuronal activity, to elucidate more detailed mechanisms of neural circuitry and plasticity mediating behaviors in health and in disease.

Dopaminergic D2 receptor modulation of striatal cholinergic interneurons contributes to sequence learning

Learning action sequences is necessary for normal daily activities. Medium spiny neurons (MSNs) in the dorsal striatum (dStr) encode action sequences through changes in firing at the start and/or stop of action sequences or sustained changes in firing throughout the sequence. Acetylcholine (ACh), released from cholinergic interneurons (ChIs), regulates striatal function by modulating MSN and interneuron excitability, dopamine and glutamate release, and synaptic plasticity. Cholinergic neurons in dStr pause their tonic firing during the performance of learned action sequences. Activation of dopamine type-2 receptors (D2Rs) on ChIs is one mechanism of ChI pausing. In this study we show that deleting D2Rs from ChIs by crossing D2-floxed with ChAT-Cre mice (D2Flox-ChATCre), which inhibits dopamine-mediated ChI pausing and leads to deficits in an operant action sequence task and lower breakpoints in a progressive ratio task. These data suggest that D2Flox-ChATCre mice have reduced motivation to work for sucrose reward, but show no generalized motor skill deficits. D2Flox-ChATCre mice perform similarly to controls in a simple reversal learning task, indicating normal behavioral flexibility, a cognitive function associated with ChIs. In vivo electrophysiological recordings show that D2Flox-ChatCre mice have deficits in sequence encoding, with fewer dStr MSNs encoding entire action sequences compared to controls. Thus, ChI D2R deletion appears to impair a neural substrate of action chunking. Virally replacing D2Rs in dStr ChIs in adult mice improves action sequence learning, but not the lower breakpoints, further suggesting that D2Rs on ChIs in the dStr are critical for sequence learning, but not for driving the motivational aspects of the task.

Knowledge generalization and the costs of multitasking

Humans are able to rapidly perform novel tasks, but show pervasive performance costs when attempting to do two things at once. Traditionally, empirical and theoretical investigations into the sources of such multitasking interference have largely focused on multitasking in isolation to other cognitive functions, characterizing the conditions that give rise to performance decrements. Here we instead ask whether multitasking costs are linked to the system's capacity for knowledge generalization, as is required to perform novel tasks. We show how interrogation of the neurophysiological circuitry underlying these two facets of cognition yields further insights for both. Specifically, we demonstrate how a system that rapidly generalizes knowledge may induce multitasking costs owing to sharing of task contingencies between contexts in neural representations encoded in frontoparietal and striatal brain regions. We discuss neurophysiological insights suggesting that prolonged learning segregates such representations by refining the brain's model of task-relevant contingencies, thereby reducing information sharing between contexts and improving multitasking performance while reducing flexibility and generalization. These proposed neural mechanisms explain why the brain shows rapid task understanding, multitasking limitations and practice effects. In short, multitasking limits are the price we pay for behavioural flexibility.

"Give me a break!" A systematic review and meta-analysis on the efficacy of micro-breaks for increasing well-being and performance

Recovery activities during short breaks taken between work tasks are solutions for preventing the impairing effects of accumulated strain. No wonder then that a growing body of scientific literature from various perspectives emerged on this topic. The present meta-analysis is aimed at estimating the efficacy of micro-breaks in enhancing well-being (vigor and fatigue) and performance, as well as in which conditions and for whom are the micro-breaks most effective. We searched the existent literature on this topic and aggregated the existing data from experimental and quasi-experimental studies. The systematic search revealed 19 records, which resulted in 22 independent study samples (N = 2335). Random-effects meta-analyses shown statistically significant but small effects of micro-breaks in boosting vigor (d = .36, p < .001; k = 9, n = 913), reducing fatigue (d = .35, p < .001; k = 9, n = 803), and a non-significant effect on increasing overall performance (d = .16, p = .116; k = 15, n = 1132). Sub-groups analyses on performance types revealed significant effects only for tasks with less cognitive demands. A meta-regression showed that the longer the break, the greater the boost was on performance. Overall, the data support the role of micro-breaks for well-being, while for performance, recovering from highly depleting tasks may need more than 10-minute breaks. Therefore, future studies should focus on this issue.

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical-subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

An insula hierarchical network architecture for active interoceptive inference

In the brain, the insular cortex receives a vast amount of interoceptive information, ascending through deep brain structures, from multiple visceral organs. The unique hierarchical and modular architecture of the insula suggests specialization for processing interoceptive afferents. Yet, the biological significance of the insula's neuroanatomical architecture, in relation to deep brain structures, remains obscure. In this opinion piece, we propose the Insula Hierarchical Modular Adaptive Interoception Control (IMAC) model to suggest that insula modules (granular, dysgranular and agranular), forming parallel networks with the prefrontal cortex and striatum, are specialized to form higher order interoceptive representations. These interoceptive representations are recruited in a context-dependent manner to support habitual, model-based and exploratory control of visceral organs and physiological processes. We discuss how insula interoceptive representations may give rise to conscious feelings that best explain lower order deep brain interoceptive representations, and how the insula may serve to defend the body and mind against pathological depression.

Caught in the ACTS: Defining Abstract Cognitive Task Sequences as an Independent Process

Cognitive neuroscience currently conflates the study of serial responses (e.g., delay match to sample/nonsample, n-back) with the study of sequential operations. In this essay, our goal is to define and disentangle the latter, termed abstract cognitive task sequences (ACTS). Existing literatures address tasks requiring serial events, including procedural learning of implicit motor responses, statistical learning of predictive relationships, and judgments of attributes. These findings do not describe the behavior and underlying mechanism required to succeed at remembering to evaluate color, then shape; or to multiply, then add. A new literature is needed to characterize these sorts of second-order cognitive demands of studying a sequence of operations. Our second goal is to characterize gaps in knowledge related to ACTS that merit further investigation. In the following sections, we define more precisely what we mean by ACTS and suggest research questions' further investigation would be positioned to address.

Unpacking self-ordered sequences

In this issue of Neuron, Chiang et al. examine population coding of self-ordered sequences in prefrontal cortex. They find better decoding, more distributed information, and less variability when order is consistent. Consistent ordering produces reliable population response patterns that may aid planning and memory.

Cognitive strategies shift information from single neurons to populations in prefrontal cortex

Neurons in primate lateral prefrontal cortex (LPFC) play a critical role in working memory (WM) and cognitive strategies. Consistent with adaptive coding models, responses of these neurons are not fixed but flexibly adjust on the basis of cognitive demands. However, little is known about how these adjustments affect population codes. Here, we investigated ensemble coding in LPFC while monkeys implemented different strategies in a WM task. Although single neurons were less tuned when monkeys used more stereotyped strategies, task information could still be accurately decoded from neural populations. This was due to changes in population codes that distributed information among a greater number of neurons, each contributing less to the overall population. Moreover, this shift occurred for task-relevant, but not irrelevant, information. These results demonstrate that cognitive strategies that impose structure on information held in mind rearrange population codes in LPFC, such that information becomes more distributed among neurons in an ensemble.

Computational validity: using computation to translate behaviours across species

We propose a new conceptual framework (computational validity) for translation across species and populations based on the computational similarity between the information processing underlying parallel tasks. Translating between species depends not on the superficial similarity of the tasks presented, but rather on the computational similarity of the strategies and mechanisms that underlie those behaviours. Computational validity goes beyond construct validity by directly addressing questions of information processing. Computational validity interacts with circuit validity as computation depends on circuits, but similar computations could be accomplished by different circuits. Because different individuals may use different computations to accomplish a given task, computational validity suggests that behaviour should be understood through the subject's point of view; thus, behaviour should be characterized on an individual level rather than a task level. Tasks can constrain the computational algorithms available to a subject and the observed subtleties of that behaviour can provide information about the computations used by each individual. Computational validity has especially high relevance for the study of psychiatric disorders, given the new views of psychiatry as identifying and mediating information processing dysfunctions that may show high inter-individual variability, as well as for animal models investigating aspects of human psychiatric disorders. This article is part of the theme issue 'Systems neuroscience through the lens of evolutionary theory'.

A Novel Self-Organizing Fuzzy Neural Network to Learn and Mimic Habitual Sequential Tasks

In this article, a new self-organizing fuzzy neural network (FNN) model is presented which is able to simultaneously and accurately learn and reproduce different sequences. Multiple sequence learning is important in performing habitual and skillful tasks, such as writing, signing signatures, and playing piano. Generally, it is indispensable for pattern generation applications. Since multiple sequences have similar parts, local information such as some previous samples is not sufficient to efficiently reproduce them. Instead, it is necessary to consider global and discriminative information, maybe in the very initial samples of each sequence, to first recognize them, and then predict their next sample based on the current local information. Therefore, the structure of the proposed network consists of two parts: 1) sequence identifier, which computes a novel sequence identity value based on initial samples of a sequence, and detects the sequence identity based on proper fuzzy rules and 2) sequence locator, which locates the input sample in the sequence. Therefore, by integrating outputs of these two parts in fuzzy rules, the network is able to produce the proper output based on the current state of each sequence. To learn the proposed structure, a gradual learning procedure is proposed. First, learning is performed by adding new fuzzy rules, based on coverage measure, using available correct data. Next, the initialized parameters are fine-tuned, by the gradient descent algorithm, based on fed back approximated network output as the next input. The proposed method has a dynamic structure able to learn new sequences online. Finally, to investigate the effectiveness of the presented approach, it is used to simultaneously learn and reproduce multiple sequences in different applications, including sequences with similar parts, different patterns, and writing different letters. The performance of the proposed method is evaluated and compared with other existing methods, including the adaptive network-based fuzzy inference system, GDFNN, CFNN, and long short-term memory (LSTM). According to these experiments, the proposed method outperforms traditional FNNs and LSTM in learning multiple sequences.

Parallel and hierarchical neural mechanisms for adaptive and predictive behavioral control

Our brain can be recognized as a network of largely hierarchically organized neural circuits that operate to control specific functions, but when acting in parallel, enable the performance of complex and simultaneous behaviors. Indeed, many of our daily actions require concurrent information processing in sensorimotor, associative, and limbic circuits that are dynamically and hierarchically modulated by sensory information and previous learning. This organization of information processing in biological organisms has served as a major inspiration for artificial intelligence and has helped to create in silico systems capable of matching or even outperforming humans in several specific tasks, including visual recognition and strategy-based games. However, the development of human-like robots that are able to move as quickly as humans and respond flexibly in various situations remains a major challenge and indicates an area where further use of parallel and hierarchical architectures may hold promise. In this article we review several important neural and behavioral mechanisms organizing hierarchical and predictive processing for the acquisition and realization of flexible behavioral control. Then, inspired by the organizational features of brain circuits, we introduce a multi-timescale parallel and hierarchical learning framework for the realization of versatile and agile movement in humanoid robots.

Striatal transcriptome changes linked to drug-induced repetitive behaviors

Disruptive or excessive repetitive motor patterns (stereotypies) are cardinal symptoms in numerous neuropsychiatric disorders. Stereotypies are also evoked by psychomotor stimulants such as amphetamine. The acquisition of motor sequences is paralleled by changes in activity patterns in the striatum, and stereotypies have been linked to abnormal plasticity in these reinforcement-related circuits. Here, we designed experiments in mice to identify transcriptomic changes that underlie striatal plasticity occurring alongside the development of drug-induced stereotypic behavior. We identified three schedules of amphetamine treatment inducing different degrees of stereotypy and used bulk RNAseq to compare striatal gene expression changes among groups of mice treated with the different drug-dose schedules and vehicle-treated, cage-mate controls. Mice were identified as naïve, sensitized, or tolerant to drug-induced stereotypy. All drug-treated groups exhibited expression changes in genes that encode members of the extracellular signal-regulated kinase (ERK) cascades known to regulate psychomotor stimulant responses. In the sensitized group with the most prolonged stereotypy, we found dysregulation of 20 genes that were not changed in other groups. Gene set enrichment analysis indicated highly significant overlap with genes regulated by neuregulin 1 (Nrg1). Nrg1 is known to be a schizophrenia and autism susceptibility gene that encodes a ligand for Erb-B receptors, which are involved in neuronal migration, myelination, and cell survival, including that of dopamine-containing neurons. Stimulant abuse is a risk factor for schizophrenia onset, and these two disorders share behavioral stereotypy phenotypes. Our results raise the possibility that drug-induced sensitization of the Nrg1 signaling pathway might underlie these links.

From the Gut to the Brain and Back: Therapeutic Approaches for the Treatment of Network Dysfunction in Parkinson's Disease

Parkinson's disease (PD) is a complex, multisystem, progressive, degenerative disorder characterized by severe, debilitating motor dysfunction, cognitive impairments, and mood disorders. Although preclinical research has traditionally focused on the motor deficits resulting from the loss of nigrostriatal dopaminergic neurons, up to two thirds of PD patients present separate and distinct behavioral changes. Loss of basal forebrain cholinergic neurons occurs as early as the loss of dopaminergic cells and contributes to the cognitive decline in PD. In addition, attentional deficits can limit posture control and movement efficacy caused by dopaminergic cell loss. Complicating the picture further is intracellular α-synuclein accumulation beginning in the enteric nervous system and diffusing to the substantia nigra through the dorsal motor neurons of the vagus nerve. It seems that α-synuclein's role is that of mediating dopamine synthesis, storage, and release, and its function has not been completely understood. Treating a complex, multistage network disorder, such as PD, likely requires a multipronged approach. Here, we describe a few approaches that could be used alone or perhaps in combination to achieve a greater mosaic of behavioral benefit. These include (1) using encapsulated, genetically modified cells as delivery vehicles for administering neuroprotective trophic factors, such as GDNF, in a direct and sustained means to the brain; (2) immunotherapeutic interventions, such as vaccination or the use of monoclonal antibodies against aggregated, pathological α-synuclein; (3) the continuous infusion of levodopa-carbidopa through an intestinal gel pad to attenuate the loss of dopaminergic function and manage the motor and non-motor complications in PD patients; and (4) specific rehabilitation treatment programs for drug-refractory motor complications.

Complementary Control over Habits and Behavioral Vigor by Phasic Activity in the Dorsolateral Striatum

Despite clear evidence linking the basal ganglia to the control of outcome insensitivity (i.e., habit) and behavioral vigor (i.e., its behavioral speed/fluidity), it remains unclear whether or how these functions relate to one another. Here, using male Long-Evans rats in response-based and cue-based maze-running tasks, we demonstrate that phasic dorsolateral striatum (DLS) activity occurring at the onset of a learned behavior regulates how vigorous and habitual it is. In a response-based task, brief optogenetic excitation at the onset of runs decreased run duration and the occurrence of deliberative behaviors, whereas midrun stimulation carried little effect. Outcome devaluation showed these runs to be habitual. DLS inhibition at run start did not produce robust effects on behavior until after outcome devaluation. At that time, when the DLS was plausibly most critically required for performance (i.e., habitual), inhibition reduced performance vigor measures and caused a dramatic loss of habitual responding (i.e., animals quit the task). In a second cue-based "beacon" task requiring behavior initiation at the start of the run and again in the middle of the run, DLS excitation at both time points could improve the vigor of runs. Postdevaluation testing showed behavior on the beacon task to be habitual as well. This pattern of results suggests that one role for phasic DLS activity at behavior initiation is to promote the execution of the behavior in a vigorous and habitual fashion by a diverse set of measures.SIGNIFICANCE STATEMENT Our research expands the literature twofold. First, we find that features of a habitual behavior that are typically studied separately (i.e., maze response performance, deliberation movements, running vigor, and outcome insensitivity) are quite closely linked together. Second, efforts have been made to understand "what" the dorsolateral striatum (DLS) does for habitual behavior, and our research provides a key set of results showing "when" it is important (i.e., at behavior initiation). By showing such dramatic control over habits by DLS activity in a phasic time window, plausible real-world applications could involve more informed DLS perturbations to curb intractably problematic habits.

Contributions of the basal ganglia to action sequence learning and performance

Animals engage in intricately woven and choreographed action sequences that are constructed from trial-and-error learning. The mechanisms by which the brain links together individual actions which are later recalled as fluid chains of behavior are not fully understood, but there is broad consensus that the basal ganglia play a crucial role in this process. This paper presents a comprehensive review of the role of the basal ganglia in action sequencing, with a focus on whether the computational framework of reinforcement learning can capture key behavioral features of sequencing and the neural mechanisms that underlie them. While a simple neurocomputational model of reinforcement learning can capture key features of action sequence learning, this model is not sufficient to capture goal-directed control of sequences or their hierarchical representation. The hierarchical structure of action sequences, in particular, poses a challenge for building better models of action sequencing, and it is in this regard that further investigations into basal ganglia information processing may be informative.

Exploring the relationship between actions, habits, and automaticity in an action sequence task

It is tempting to equate the automatization of an action sequence with the formation of a habit. However, the term “habit” specifically implies a failure to evaluate future consequences to guide behavior. To test if automatized sequences become habitual, we trained rats on an action sequence task for either 20 or 60 d and then conducted reward devaluation tests. While both groups showed equivalent goal-directed performance of the trained action sequence on a global measure of behavior, sequence initiation and completion times were differentially sensitive to outcome devaluation in moderately and extensively trained rats.

Thalamocortical Axonal Activity in Motor Cortex Exhibits Layer-Specific Dynamics during Motor Learning

The thalamus is the hub through which neural signals are transmitted from the basal ganglia and cerebellum to the neocortex. However, thalamocortical axonal activity during motor learning remains largely undescribed. We conducted two-photon calcium imaging of thalamocortical axonal activity in the motor cortex of mice learning a self-initiated lever-pull task. Layer 1 (L1) axons came to exhibit activity at lever-pull initiation and termination, while layer 3 (L3) axons did so at lever-pull initiation. L1 population activity had a sequence structure related to both lever-pull duration and reproducibility. Stimulation of the substantia nigra pars reticulata activated more L1 than L3 axons, whereas deep cerebellar nuclei (DCN) stimulation did the opposite. Lesions to either the dorsal striatum or the DCN impaired motor learning and disrupted temporal dynamics in both layers. Thus, layer-specific thalamocortical signals evolve with the progression of learning, which requires both the basal ganglia and cerebellar activities.

Monitoring and Updating of Action Selection for Goal-Directed Behavior through the Striatal Direct and Indirect Pathways

The basal ganglia play key roles in adaptive behaviors guided by reward and punishment. However, despite accumulating knowledge, few studies have tested how heterogeneous signals in the basal ganglia are organized and coordinated for goal-directed behavior. In this study, we investigated neuronal signals of the direct and indirect pathways of the basal ganglia as rats performed a lever push/pull task for a probabilistic reward. In the dorsomedial striatum, we found that optogenetically and electrophysiologically identified direct pathway neurons encoded reward outcomes, whereas indirect pathway neurons encoded no-reward outcome and next-action selection. Outcome coding occurred in association with the chosen action. In support of pathway-specific neuronal coding, light activation induced a bias on repeat selection of the same action in the direct pathway, but on switch selection in the indirect pathway. Our data reveal the mechanisms underlying monitoring and updating of action selection for goal-directed behavior through basal ganglia circuits.

The Basal Ganglia: More than just a switching device

The basal ganglia consist of a variety of subcortical nuclei engaged in motor control and executive functions, such as motor learning, behavioral control, and emotion. The striatum, a major basal ganglia component, is particularly useful for cognitive planning of purposive motor acts owing to its structural features and the neuronal circuitry established with the cerebral cortex. Recent data indicate emergent functions played by the striatum. Indeed, cortico-striatal circuits carrying motor information are paralleled by circuits originating from associative and limbic territories, which are functionally integrated in the striatum. Functional integration between brain areas is achieved through patterns of coherent activity. Coherence belonging to cortico-basal ganglia circuits is also present in Parkinson's disease patients. Excessive synchronization occurring in this pathology is reduced by dopaminergic therapies. The mechanisms through which the dopaminergic effects may be addressed are the object of several ongoing investigations. Overall, the bulk of data reported in recent years has provided new vistas concerning basal ganglia role in the organization and control of movement and behavior, both in physiological and pathological conditions. In this review, basal ganglia functions involved in the organization of main movement categories and behaviors are critically discussed. Comparatively, the multiplicity of Parkinson's disease symptomatology is also revised.

Kainate Receptors Inhibit Glutamate Release Via Mobilization of Endocannabinoids in Striatal Direct Pathway Spiny Projection Neurons

Kainate receptors are members of the glutamate receptor family that function by both generating ionotropic currents through an integral ion channel pore and coupling to downstream metabotropic signaling pathways. They are highly expressed in the striatum, yet their roles in regulating striatal synapses are not known. Using mice of both sexes, we demonstrate that GluK2-containing kainate receptors expressed in direct pathway spiny projection neurons (dSPNs) inhibit glutamate release at corticostriatal synapses in the dorsolateral striatum. This inhibition requires postsynaptic kainate-receptor-mediated mobilization of a retrograde endocannabinoid (eCB) signal and activation of presynaptic CB1 receptors. This pathway can be activated during repetitive 25 Hz trains of synaptic stimulation, causing short-term depression of corticostriatal synapses. This is the first study to demonstrate a role for kainate receptors in regulating eCB-mediated plasticity at the corticostriatal synapse and demonstrates an important role for these receptors in regulating basal ganglia circuits.SIGNIFICANCE STATEMENT The GRIK2 gene, encoding the GluK2 subunit of the kainate receptor, has been linked to several neuropsychiatric and neurodevelopmental disorders including obsessive compulsive disorder (OCD). Perseverative behaviors associated with OCD are known to result from pathophysiological changes in the striatum and kainate receptor knock-out mice have striatal-dependent phenotypes. However, the role of kainate receptors in striatal synapses is not known. We demonstrate that GluK2-containing kainate receptors regulate corticostriatal synapses by mobilizing endocannabinoids from direct pathway spiny projection neurons. Synaptic activation of GluK2 receptors during trains of synaptic input causes short-term synaptic depression, demonstrating a novel role for these receptors in regulating striatal circuits.

A Basal Ganglia Circuit Sufficient to Guide Birdsong Learning

Learning vocal behaviors, like speech and birdsong, is thought to rely on continued performance evaluation. Whether candidate performance evaluation circuits in the brain are sufficient to guide vocal learning is not known. Here, we test the sufficiency of VTA projections to the vocal basal ganglia in singing zebra finches, a songbird species that learns to produce a complex and stereotyped multi-syllabic courtship song during development. We optogenetically manipulate VTA axon terminals in singing birds contingent on how the pitch of an individual song syllable is naturally performed. We find that optical inhibition and excitation of VTA terminals are each sufficient to reliably guide learned changes in song. Inhibition and excitation have opponent effects on future performances of targeted song syllables, consistent with positive and negative reinforcement of performance outcomes. These findings define a central role for reinforcement mechanisms in learning vocalizations and demonstrate minimal circuit elements for learning vocal behaviors. VIDEO ABSTRACT.

Basal ganglia mechanisms in action selection, plasticity, and dystonia

Basal ganglia circuits are organized to selected desired actions and to inhibit potentially competing unwanted actions. This is accomplished through a complex circuitry that is modified through development and learning. Mechanisms of neural plasticity underlying these modifications are increasingly understood, but new mechanisms continue to be discovered. Dystonia, a movement disorder characterized by involuntary muscle contractions that cause abnormal postures and movements. Emerging evidence points to important links between mechanisms of plasticity and the manifestations of dystonia. Investigation of these mechanisms has improved understanding of the action of currently used medication and is informing the development of new treatments.

Learning new sequential stepping patterns requires striatal plasticity during the earliest phase of acquisition

Animals including humans execute motor behavior to reach their goals. For this purpose, they must choose correct strategies according to environmental conditions and shape many parameters of their movements, including their serial order and timing. To investigate the neurobiology underlying such skills, we used a multi-sensor equipped, motor-driven running wheel with adjustable sequences of foothold pegs on which mice ran to obtain water reward. When the peg patterns changed from a familiar pattern to a new pattern, the mice had to learn and implement new locomotor strategies in order to receive reward. We found that the accuracy of stepping and the achievement of water reward improved with the new learning after changes in the peg-pattern, and c-Fos expression levels assayed after the first post-switch session were high in both dorsolateral striatum and motor cortex, relative to post-switch plateau levels. Combined in situ hybridization and immunohistochemistry of striatal sections demonstrated that both enkephalin-positive (indirect pathway) neurons and substance P-positive (direct pathway) neurons were recruited specifically after the pattern switches, as were interneurons expressing neuronal nitric oxide synthase. When we blocked N-methyl-D-aspartate (NMDA) receptors in the dorsolateral striatum by injecting the NMDA receptor antagonist, D-2-amino-5-phosphonopentanoic acid (AP5), we found delays in early post-switch improvement in performance. These findings suggest that the dorsolateral striatum is activated on detecting shifts in environment to adapt motor behavior to the new context via NMDA-dependent plasticity, and that this plasticity may underlie forming and breaking skills and habits as well as to behavioral difficulties in clinical disorders.

Habit formation

Habits, both good ones and bad ones, are pervasive in animal behavior. Important frameworks have been developed to understand habits through psychological and neurobiological studies. This work has given us a rich understanding of brain networks that promote habits, and has also helped us to understand what constitutes a habitual behavior as opposed to a behavior that is more flexible and prospective. Mounting evidence from studies using neural recording methods suggests that habit formation is not a simple process. We review this evidence and take the position that habits could be sculpted from multiple dissociable changes in neural activity. These changes occur across multiple brain regions and even within single brain regions. This strategy of classifying components of a habit based on different brain signals provides a potentially useful new way to conceive of disorders that involve overly fixed behaviors as arising from different potential dysfunctions within the brain's habit network.

Pallidal spiking activity reflects learning dynamics and predicts performance

The basal ganglia (BG) network has been divided into interacting actor and critic components, modulating the probabilities of different state-action combinations through learning. Most models of learning and decision making in the BG focus on the roles of the striatum and its dopaminergic inputs, commonly overlooking the complexities and interactions of BG downstream nuclei. In this study, we aimed to reveal the learning-related activity of the external segment of the globus pallidus (GPe), a downstream structure whose computational role has remained relatively unexplored. Recording from monkeys engaged in a deterministic three-choice reversal learning task, we found that changes in GPe discharge rates predicted subsequent behavioral shifts on a trial-by-trial basis. Furthermore, the activity following the shift encoded whether it resulted in reward or not. The frequent changes in stimulus-outcome contingencies (i.e., reversals) allowed us to examine the learning-related neural activity and show that GPe discharge rates closely matched across-trial learning dynamics. Additionally, firing rates exhibited a linear decrease in sequences of correct responses, possibly reflecting a gradual shift from goal-directed execution to automaticity. Thus, modulations in GPe spiking activity are highest for attention-demanding aspects of behavior (i.e., switching choices) and decrease as attentional demands decline (i.e., as performance becomes automatic). These findings are contrasted with results from striatal tonically active neurons, which show none of these task-related modulations. Our results demonstrate that GPe, commonly studied in motor contexts, takes part in cognitive functions, in which movement plays a marginal role.

Input- and Output-Specific Regulation of Serial Order Performance by Corticostriatal Circuits

The serial ordering of individual movements into sequential patterns is thought to require synaptic plasticity within corticostriatal circuits that route information through the basal ganglia. We used genetically and anatomically targeted manipulations of specific circuit elements in mice to isolate the source and target of a corticostriatal synapse that regulates the performance of a serial order task. This excitatory synapse originates in secondary motor cortex, terminates on direct pathway medium spiny neurons in the dorsolateral striatum, and is strengthened by serial order learning. This experience-dependent and synapse-specific form of plasticity may sculpt the balance of activity in basal ganglia circuits during sequential movements, driving a disparity in striatal output that favors the direct pathway. This disparity is necessary for execution of responses in serial order, even though both direct and indirect pathways are active during movement initiation, suggesting dynamic modulation of corticostriatal circuitry contributes to the choreography of behavioral routines.

Translational Research on Habit and Alcohol

Habitual actions enable efficient daily living, but they can also contribute to pathological behaviors that resistant change, such as alcoholism. Habitual behaviors are learned actions that appear goal-directed but are in fact no longer under the control of the action's outcome. Instead, these actions are triggered by stimuli, which may be exogenous or interoceptive, discrete or contextual. A major hallmark characteristic of alcoholism is continued alcohol use despite serious negative consequences. In essence, although the outcome of alcohol seeking and drinking is dramatically devalued, these actions persist, often triggered by environmental cues associated with alcohol use. Thus, alcoholism meets the definition of an initially goal-directed behavior that converts to a habit-based process. Habit and alcohol have been well investigated in rodent models, with comparatively less research in non-human primates and people. This review focuses on translational research on habit and alcohol with an emphasis on cross-species methodology and neural circuitry.

The Monitoring and Control of Task Sequences in Human and Non-Human Primates

Our ability to plan and execute a series of tasks leading to a desired goal requires remarkable coordination between sensory, motor, and decision-related systems. Prefrontal cortex (PFC) is thought to play a central role in this coordination, especially when actions must be assembled extemporaneously and cannot be programmed as a rote series of movements. A central component of this flexible behavior is the moment-by-moment allocation of working memory and attention. The ubiquity of sequence planning in our everyday lives belies the neural complexity that supports this capacity, and little is known about how frontal cortical regions orchestrate the monitoring and control of sequential behaviors. For example, it remains unclear if and how sensory cortical areas, which provide essential driving inputs for behavior, are modulated by the frontal cortex during these tasks. Here, we review what is known about moment-to-moment monitoring as it relates to visually guided, rule-driven behaviors that change over time. We highlight recent human work that shows how the rostrolateral prefrontal cortex (RLPFC) participates in monitoring during task sequences. Neurophysiological data from monkeys suggests that monitoring may be accomplished by neurons that respond to items within the sequence and may in turn influence the tuning properties of neurons in posterior sensory areas. Understanding the interplay between proceduralized or habitual acts and supervised control of sequences is key to our understanding of sequential task execution. A crucial bridge will be the use of experimental protocols that allow for the examination of the functional homology between monkeys and humans. We illustrate how task sequences may be parceled into components and examined experimentally, thereby opening future avenues of investigation into the neural basis of sequential monitoring and control.

Pathway-Specific Striatal Substrates for Habitual Behavior

The dorsolateral striatum (DLS) is implicated in habit formation. However, the DLS circuit mechanisms underlying habit remain unclear. A key role for DLS is to transform sensorimotor cortical input into firing of output neurons that project to the mutually antagonistic direct and indirect basal ganglia pathways. Here we examine whether habit alters this input-output function. By imaging cortically evoked firing in large populations of pathway-defined striatal projection neurons (SPNs), we identify features that strongly correlate with habitual behavior on a subject-by-subject basis. Habitual behavior correlated with strengthened DLS output to both pathways as well as a tendency for action-promoting direct pathway SPNs to fire before indirect pathway SPNs. In contrast, habit suppression correlated solely with a weakened direct pathway output. Surprisingly, all effects were broadly distributed in space. Together, these findings indicate that the striatum imposes broad, pathway-specific modulations of incoming activity to render learned motor behaviors habitual.

Habit formation coincides with shifts in reinforcement representations in the sensorimotor striatum

Evaluating outcomes of behavior is a central function of the striatum. In circuits engaging the dorsomedial striatum, sensitivity to goal value is accentuated during learning, whereas outcome sensitivity is thought to be minimal in the dorsolateral striatum and its habit-related corticostriatal circuits. However, a distinct population of projection neurons in the dorsolateral striatum exhibits selective sensitivity to rewards. Here, we evaluated the outcome-related signaling in such neurons as rats performed an instructional T-maze task for two rewards. As the rats formed maze-running habits and then changed behavior after reward devaluation, we detected outcome-related spike activity in 116 units out of 1,479 recorded units. During initial training, nearly equal numbers of these units fired preferentially either after rewarded runs or after unrewarded runs, and the majority were responsive at only one of two reward locations. With overtraining, as habits formed, firing in nonrewarded trials almost disappeared, and reward-specific firing declined. Thus error-related signaling was lost, and reward signaling became generalized. Following reward devaluation, in an extinction test, postgoal activity was nearly undetectable, despite accurate running. Strikingly, when rewards were then returned, postgoal activity reappeared and recapitulated the original early response pattern, with nearly equal numbers responding to rewarded and unrewarded runs and to single rewards. These findings demonstrate that outcome evaluation in the dorsolateral striatum is highly plastic and tracks stages of behavioral exploration and exploitation. These signals could be a new target for understanding compulsive behaviors that involve changes to dorsal striatum function.

Bursts of beta oscillation differentiate postperformance activity in the striatum and motor cortex of monkeys performing movement tasks

Studies of neural oscillations in the beta band (13-30 Hz) have demonstrated modulations in beta-band power associated with sensory and motor events on time scales of 1 s or more, and have shown that these are exaggerated in Parkinson's disease. However, even early reports of beta activity noted extremely fleeting episodes of beta-band oscillation lasting <150 ms. Because the interpretation of possible functions for beta-band oscillations depends strongly on the time scale over which they occur, and because of these oscillations' potential importance in Parkinson's disease and related disorders, we analyzed in detail the distributions of duration and power for beta-band activity in a large dataset recorded in the striatum and motor-premotor cortex of macaque monkeys performing reaching tasks. Both regions exhibited typical beta-band suppression during movement and postmovement rebounds of up to 3 s as viewed in data averaged across trials, but single-trial analysis showed that most beta oscillations occurred in brief bursts, commonly 90-115 ms long. In the motor cortex, the burst probabilities peaked following the last movement, but in the striatum, the burst probabilities peaked at task end, after reward, and continued through the postperformance period. Thus, what appear to be extended periods of postperformance beta-band synchronization reflect primarily the modulated densities of short bursts of synchrony occurring in region-specific and task-time-specific patterns. We suggest that these short-time-scale events likely underlie the functions of most beta-band activity, so that prolongation of these beta episodes, as observed in Parkinson's disease, could produce deleterious network-level signaling.