Optogenetic Editing Reveals the Hierarchical Organization of Learned Action Sequences

Striatal projection neurons coexpressing dopamine D1 and D2 receptors modulate the motor function of D1- and D2-SPNs

The role of the striatum in motor control is commonly assumed to be mediated by the two striatal efferent pathways characterized by striatal projection neurons (SPNs) expressing dopamine (DA) D1 receptors or D2 receptors (D1-SPNs and D2-SPNs, respectively), without regard to SPNs coexpressing both receptors (D1/D2-SPNs). Here we developed an approach to target these hybrid SPNs in mice and demonstrate that, although these SPNs are less abundant, they have a major role in guiding the motor function of the other two populations. D1/D2-SPNs project exclusively to the external globus pallidus and have specific electrophysiological features with distinctive integration of DA signals. Gain- and loss-of-function experiments indicate that D1/D2-SPNs potentiate the prokinetic and antikinetic functions of D1-SPNs and D2-SPNs, respectively, and restrain the integrated motor response to psychostimulants. Overall, our findings demonstrate the essential role of this population of D1/D2-coexpressing neurons in orchestrating the fine-tuning of DA regulation in thalamo-cortico-striatal loops.

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Striosomes Target Nigral Dopamine-Containing Neurons via Direct-D1 and Indirect-D2 Pathways Paralleling Classic Direct-Indirect Basal Ganglia Systems

The classic output pathways of the basal ganglia are known as the direct-D1 and indirect-D2, or "Go/No-Go", pathways. Balance of the activity in these canonical direct-indirect pathways is considered a core requirement for normal movement control, and their imbalance is a major etiologic factor in movement disorders including Parkinson's disease. We present evidence for a conceptually equivalent parallel system of direct-D1 and indirect-D2 pathways that arise from striatal projection neurons (SPNs) of the striosome compartment rather than from the matrix. These striosomal direct (S-D1) and indirect (S-D2) pathways, as a pair, target dopamine-containing neurons of the substantia nigra (SNpc) instead of the motor output nuclei of the basal ganglia. The novel anatomically and functionally distinct indirect-D2 striosomal pathway targets dopaminergic SNpc cells indirectly via a core region of the external pallidum (GPe). We demonstrate that these S-D1 and S-D2 pathways oppositely modulate striatal dopamine release in freely behaving mice under open-field conditions and oppositely modulate locomotor and other movements. These S-D1 and S-D2 pathways further exhibit different, time-dependent responses during performance of a probabilistic decision-making maze task and respond differently to rewarding and aversive stimuli. These contrasts depend on mediolateral and anteroposterior striatal locations of the SPNs as are the classic direct and indirect pathways. The effects of S-D1 and S-D2 stimulation on striatal dopamine release and voluntary locomotion are nearly opposite. The parallelism of the direct-indirect circuit design motifs of the striosomal S-D and S-D2 circuits and canonical matrix M-D1 and M-D2, and their contrasting behavioral effects, call for a major reformulation of the classic direct-indirect pathway model of basal ganglia function. Given that some striosomes receive limbic and association cortical inputs, the S-D1 and S-D2 circuits likely influence motivation for action and behavioral learning, complementing and possibly reorienting the motoric activities of the canonical matrix pathways. At a fundamental level, these findings suggest a unifying framework for aligning two sets of circuits that share the organizational motif of opponent D1 and D2 regulation, but that have different outputs and can even have opposite polarities in their targets and effects, albeit conditioned by striatal topography. Our findings further delineate a potentially therapeutically important set of pathways influencing dopamine, including a D2 receptor-linked S-D2 pathway likely unknowingly targeted by administration of many therapeutic drugs including those for Parkinson's disease. The novel parallel pathway model that we propose here could help to account for the normally integrated modulatory influence of the basal ganglia on motivation for actions as well as the actions themselves.

How cortico-basal ganglia-thalamic subnetworks can shift decision policies to maximize reward rate

All mammals exhibit flexible decision policies that depend, at least in part, on the cortico-basal ganglia-thalamic (CBGT) pathways. Yet understanding how the complex connectivity, dynamics, and plasticity of CBGT circuits translates into experience-dependent shifts of decision policies represents a longstanding challenge in neuroscience. Here we used a computational approach to address this problem. Specifically, we simulated decisions driven by CBGT circuits under baseline, unrewarded conditions using a spiking neural network, and fit the resulting behavior to an evidence accumulation model. Using canonical correlation analysis, we then replicated the existence of three recently identified control ensembles (responsiveness, pliancy and choice) within CBGT circuits, with each ensemble mapping to a specific configuration of the evidence accumulation process. We subsequently simulated learning in a simple two-choice task with one optimal (i.e., rewarded) target. We find that value-based learning, via dopaminergic signals acting on cortico-striatal synapses, effectively manages the speed-accuracy tradeoff so as to increase reward rate over time. Within this process, learning-related changes in decision policy can be decomposed in terms of the contributions of each control ensemble, and these changes are driven by sequential reward prediction errors on individual trials. Our results provide a clear and simple mechanism for how dopaminergic plasticity shifts specific subnetworks within CBGT circuits so as to strategically modulate decision policies in order to maximize effective reward rate.

Striatal insights: a cellular and molecular perspective on repetitive behaviors in pathology

Animals often behave repetitively and predictably. These repetitive behaviors can have a component that is learned and ingrained as habits, which can be evolutionarily advantageous as they reduce cognitive load and the expenditure of attentional resources. Repetitive behaviors can also be conscious and deliberate, and may occur in the absence of habit formation, typically when they are a feature of normal development in children, or neuropsychiatric disorders. They can be considered pathological when they interfere with social relationships and daily activities. For instance, people affected by obsessive-compulsive disorder, autism spectrum disorder, Huntington's disease and Gilles de la Tourette syndrome can display a wide range of symptoms like compulsive, stereotyped and ritualistic behaviors. The striatum nucleus of the basal ganglia is proposed to act as a master regulator of these repetitive behaviors through its circuit connections with sensorimotor, associative, and limbic areas of the cortex. However, the precise mechanisms within the striatum, detailing its compartmental organization, cellular specificity, and the intricacies of its downstream connections, remain an area of active research. In this review, we summarize evidence across multiple scales, including circuit-level, cellular, and molecular dimensions, to elucidate the striatal mechanisms underpinning repetitive behaviors and offer perspectives on the implicated disorders. We consider the close relationship between behavioral output and transcriptional changes, and thereby structural and circuit alterations, including those occurring through epigenetic processes.

Opposing Motor Memories in the Direct and Indirect Pathways of the Basal Ganglia

Loss of dopamine neurons causes motor deterioration in Parkinson's disease patients. We have previously reported that in addition to acute motor impairment, the impaired motor behavior is encoded into long-term memory in an experience-dependent and task-specific manner, a phenomenon we refer to as aberrant inhibitory motor learning. Although normal motor learning and aberrant inhibitory learning oppose each other and this is manifested in apparent motor performance, in the present study, we found that normal motor memory acquired prior to aberrant inhibitory learning remains preserved in the brain, suggesting the existence of independent storage. To investigate the neuronal circuits underlying these two opposing memories, we took advantage of the RNA-binding protein YTHDF1, an m 6 A RNA methylation reader involved in the regulation of protein synthesis and learning/memory. Conditional deletion of Ythdf1 in either D1 or D2 receptor-expressing neurons revealed that normal motor memory is stored in the D1 (direct) pathway of the basal ganglia, while inhibitory memory is stored in the D2 (indirect) pathway. Furthermore, fiber photometry recordings of GCaMP signals from striatal D1 (dSPN) and D2 (iSPN) receptor-expressing neurons support the preservation of normal memory in the direct pathway after aberrant inhibitory learning, with activities of dSPN predictive of motor performance. Finally, a computational model based on activities of motor cortical neurons, dSPN and iSPN neurons, and their interactions through the basal ganglia loops supports the above observations. These findings have important implications for novel approaches in treating Parkinson's disease by reactivating preserved normal memory, and in treating hyperkinetic movement disorders such as chorea or tics by erasing aberrant motor memories.

Role of the basal ganglia in innate and learned behavioural sequences

Integrating individual actions into coherent, organised behavioural units, a process called chunking, is a fundamental, evolutionarily conserved process that renders actions automatic. In vertebrates, evidence points to the basal ganglia - a complex network believed to be involved in action selection - as a key component of action sequence encoding, although the underlying mechanisms are only just beginning to be understood. Central pattern generators control many innate automatic behavioural sequences that form some of the most basic behaviours in an animal's repertoire, and in vertebrates, brainstem and spinal pattern generators are under the control of higher order structures such as the basal ganglia. Evidence suggests that the basal ganglia play a crucial role in the concatenation of simpler behaviours into more complex chunks, in the context of innate behavioural sequences such as chain grooming in rats, as well as sequences in which innate capabilities and learning interact such as birdsong, and sequences that are learned from scratch, such as lever press sequences in operant behaviour. It has been proposed that the role of the striatum, the largest input structure of the basal ganglia, might lie in selecting and allowing the relevant central pattern generators to gain access to the motor system in the correct order, while inhibiting other behaviours. As behaviours become more complex and flexible, the pattern generators seem to become more dependent on descending signals. Indeed, during learning, the striatum itself may adopt the functional characteristics of a higher order pattern generator, facilitated at the microcircuit level by striatal neuropeptides.

Updating the striatal-pallidal wiring diagram

The striatal and pallidal complexes are basal ganglia structures that orchestrate learning and execution of flexible behavior. Models of how the basal ganglia subserve these functions have evolved considerably, and the advent of optogenetic and molecular tools has shed light on the heterogeneity of subcircuits within these pathways. However, a synthesis of how molecularly diverse neurons integrate into existing models of basal ganglia function is lacking. Here, we provide an overview of the neurochemical and molecular diversity of striatal and pallidal neurons and synthesize recent circuit connectivity studies in rodents that takes this diversity into account. We also highlight anatomical organizational principles that distinguish the dorsal and ventral basal ganglia pathways in rodents. Future work integrating the molecular and anatomical properties of striatal and pallidal subpopulations may resolve controversies regarding basal ganglia network function.

Nucleus accumbens and dorsal medial striatal dopamine and neural activity are essential for action sequence performance

Separable striatal circuits have unique functions in Pavlovian and instrumental behaviors but how these roles relate to performance of sequences of actions with and without associated cues are less clear. Here, we tested whether dopamine transmission and neural activity more generally in three striatal subdomains are necessary for performance of an action chain leading to reward delivery. Male and female Long-Evans rats were trained to press a series of three spatially distinct levers to receive reward. We assessed the contribution of neural activity or dopamine transmission within each striatal subdomain when progression through the action sequence was explicitly cued and in the absence of cues. Behavior in both task variations was substantially impacted following microinfusion of the dopamine antagonist, flupenthixol, into nucleus accumbens core (NAc) or dorsomedial striatum (DMS), with impairments in sequence timing and numbers of rewards earned after NAc flupenthixol. In contrast, after pharmacological inactivation to suppress overall activity, there was minimal impact on total rewards earned. Instead, inactivation of both NAc and DMS impaired sequence timing and led to sequence errors in the uncued, but not cued task. There was no impact of dopamine antagonism or reversible inactivation of dorsolateral striatum on either cued or uncued action sequence completion. These results highlight an essential contribution of NAc and DMS dopamine systems in motivational and performance aspects of chains of actions, whether cued or internally generated, as well as the impact of intact NAc and DMS function for correct sequence performance.

Asymmetric cortical projections to striatal direct and indirect pathways distinctly control actions

The striatal direct and indirect pathways constitute the core for basal ganglia function in action control. Although both striatal D1- and D2-spiny projection neurons (SPNs) receive excitatory inputs from the cerebral cortex, whether or not they share inputs from the same cortical neurons, and how pathway-specific corticostriatal projections control behavior remain largely unknown. Here using a new G-deleted rabies system in mice, we found that more than two-thirds of excitatory inputs to D2-SPNs also target D1-SPNs, while only one-third do so vice versa. Optogenetic stimulation of striatal D1- vs. D2-SPN-projecting cortical neurons differently regulate locomotion, reinforcement learning and sequence behavior, implying the functional dichotomy of pathway-specific corticostriatal subcircuits. These results reveal the partially segregated yet asymmetrically overlapping cortical projections on striatal D1- vs. D2-SPNs, and that the pathway-specific corticostriatal subcircuits distinctly control behavior. It has important implications in a wide range of neurological and psychiatric diseases affecting cortico-basal ganglia circuitry.

Timed Sequence Task: A New Paradigm to Study Motor Learning and Flexibility in Mice

Motor learning and flexibility allow animals to perform routine actions efficiently while keeping them flexible. A number of paradigms are used to test cognitive flexibility, but not many of them focus specifically on the learning of complex motor sequences and their flexibility. While many tests use operant or touchscreen boxes that offer high throughput and reproducibility, the motor actions themselves are mostly simple presses of a designated lever. To focus more on motor actions during the operant task and to probe the flexibility of these well trained actions, we developed a new operant paradigm for mice, the "timed sequence task." The task requires mice to learn a sequence of lever presses that have to be emitted in precisely defined time limits. After training, the required pressing sequence and/or timing of individual presses is modified to test the ability of mice to alter their previously trained motor actions. We provide a code for the new protocol that can be used and adapted to common types of operant boxes. In addition, we provide a set of scripts that allow automatic extraction and analysis of numerous parameters recorded during each session. We demonstrate that the analysis of multiple performance parameters is necessary for detailed insight into the behavior of animals during the task. We validate our paradigm in an experiment using the valproate model of autism as a model of cognitive inflexibility. We show that the valproate mice show superior performance at specific stages of the task, paradoxically because of their propensity to more stereotypic behavior.

Dissociating the contributions of sensorimotor striatum to automatic and visually guided motor sequences

The ability to sequence movements in response to new task demands enables rich and adaptive behavior. However, such flexibility is computationally costly and can result in halting performances. Practicing the same motor sequence repeatedly can render its execution precise, fast and effortless, that is, 'automatic'. The basal ganglia are thought to underlie both types of sequence execution, yet whether and how their contributions differ is unclear. We parse this in rats trained to perform the same motor sequence instructed by cues and in a self-initiated overtrained, or 'automatic,' condition. Neural recordings in the sensorimotor striatum revealed a kinematic code independent of the execution mode. Although lesions reduced the movement speed and affected detailed kinematics similarly, they disrupted high-level sequence structure for automatic, but not visually guided, behaviors. These results suggest that the basal ganglia are essential for 'automatic' motor skills that are defined in terms of continuous kinematics, but can be dispensable for discrete motor sequences guided by sensory cues.

Multiple dynamic interactions from basal ganglia direct and indirect pathways mediate action selection

The basal ganglia are known to be essential for action selection. However, the functional role of basal ganglia direct and indirect pathways in action selection remains unresolved. Here, by employing cell-type-specific neuronal recording and manipulation in mice trained in a choice task, we demonstrate that multiple dynamic interactions from the direct and indirect pathways control the action selection. While the direct pathway regulates the behavioral choice in a linear manner, the indirect pathway exerts a nonlinear inverted-U-shaped control over action selection, depending on the inputs and the network state. We propose a new center (direct)-surround (indirect)-context (indirect) 'Triple-control' functional model of basal ganglia, which can replicate the physiological and behavioral experimental observations that cannot be simply explained by either the traditional 'Go/No-go' or more recent 'Co-activation' model. These findings have important implications on understanding the basal ganglia circuitry and action selection in health and disease.

Inhibition of the dorsomedial striatal direct pathway is essential for the execution of action sequences

Contrary to the previous notion that the dorsomedial striatum (DMS) is crucial for acquiring new learning, accumulated evidence has suggested that the DMS also plays a role in the execution of already learned action sequences. Here, we examined how the direct and indirect pathways in the DMS regulate action sequences using a task that requires animals to press a lever consecutively. Cell-type-specific bulk Ca2+ recording revealed that the direct pathway was inhibited at the time of sequence execution. The sequence-related response was blunted in trials where the sequential behaviors were disrupted. Optogenetic activation at the sequence start caused distraction of action sequences without affecting motor function or memory of the task structure. By contrast with the direct pathway, the indirect pathway was slightly activated at the start of the sequence, but the optogenetic suppression of such sequence-related signaling did not impact the behaviors. These results suggest that the inhibition of the DMS direct pathway promotes sequence execution potentially by suppressing the formation of a new association.

Dopamine-mediated striatal activity and function is enhanced in GlyRα2 knockout animals

The glycine receptor alpha 2 (GlyRα2) is a ligand-gated ion channel which upon activation induces a chloride conductance. Here, we investigated the role of GlyRα2 in dopamine-stimulated striatal cell activity and behavior. We show that depletion of GlyRα2 enhances dopamine-induced increases in the activity of putative dopamine D1 receptor-expressing striatal projection neurons, but does not alter midbrain dopamine neuron activity. We next show that the locomotor response to d-amphetamine is enhanced in GlyRα2 knockout animals, and that this increase correlates with c-fos expression in the dorsal striatum. 3-D modeling revealed an increase in the neuronal ensemble size in the striatum in response to D-amphetamine in GlyRα2 KO mice. Finally, we show enhanced appetitive conditioning in GlyRα2 KO animals that is likely due to increased motivation, but not changes in associative learning or hedonic response. Taken together, we show that GlyRα2 is an important regulator of dopamine-stimulated striatal activity and function.

Enhancing reinforcement learning models by including direct and indirect pathways improves performance on striatal dependent tasks

A major advance in understanding learning behavior stems from experiments showing that reward learning requires dopamine inputs to striatal neurons and arises from synaptic plasticity of cortico-striatal synapses. Numerous reinforcement learning models mimic this dopamine-dependent synaptic plasticity by using the reward prediction error, which resembles dopamine neuron firing, to learn the best action in response to a set of cues. Though these models can explain many facets of behavior, reproducing some types of goal-directed behavior, such as renewal and reversal, require additional model components. Here we present a reinforcement learning model, TD2Q, which better corresponds to the basal ganglia with two Q matrices, one representing direct pathway neurons (G) and another representing indirect pathway neurons (N). Unlike previous two-Q architectures, a novel and critical aspect of TD2Q is to update the G and N matrices utilizing the temporal difference reward prediction error. A best action is selected for N and G using a softmax with a reward-dependent adaptive exploration parameter, and then differences are resolved using a second selection step applied to the two action probabilities. The model is tested on a range of multi-step tasks including extinction, renewal, discrimination; switching reward probability learning; and sequence learning. Simulations show that TD2Q produces behaviors similar to rodents in choice and sequence learning tasks, and that use of the temporal difference reward prediction error is required to learn multi-step tasks. Blocking the update rule on the N matrix blocks discrimination learning, as observed experimentally. Performance in the sequence learning task is dramatically improved with two matrices. These results suggest that including additional aspects of basal ganglia physiology can improve the performance of reinforcement learning models, better reproduce animal behaviors, and provide insight as to the role of direct- and indirect-pathway striatal neurons.

Multiple dynamic interactions from basal ganglia direct and indirect pathways mediate action selection

The basal ganglia are known to be essential for action selection. However, the functional role of basal ganglia direct and indirect pathways in action selection remains unresolved. Here by employing cell-type-specific neuronal recording and manipulation in mice trained in a choice task, we demonstrate that multiple dynamic interactions from the direct and indirect pathways control the action selection. While the direct pathway regulates the behavioral choice in a linear manner, the indirect pathway exerts a nonlinear inverted-U-shaped control over action selection, depending on the inputs and the network state. We propose a new center (direct) - surround (indirect) - context (indirect) "Triple-control" functional model of basal ganglia, which can replicate the physiological and behavioral experimental observations that cannot be simply explained by either the traditional "Go/No-go" or more recent "Co-activation" model. These findings have important implications on understanding the basal ganglia circuitry and action selection in health and disease.

Striatal ensemble activity in an innate naturalistic behavior

Self-grooming is an innate, naturalistic behavior found in a wide variety of organisms. The control of rodent grooming has been shown to be mediated by the dorsolateral striatum through lesion studies and in-vivo extracellular recordings. Yet, it is unclear how populations of neurons in the striatum encode grooming. We recorded single-unit extracellular activity from populations of neurons in freely moving mice and developed a semi-automated approach to detect self-grooming events from 117 hours of simultaneous multi-camera video recordings of mouse behavior. We first characterized the grooming transition-aligned response profiles of striatal projection neuron and fast spiking interneuron single units. We identified striatal ensembles whose units were more strongly correlated during grooming than during the entire session. These ensembles display varied grooming responses, including transient changes around grooming transitions or sustained changes in activity throughout the duration of grooming. Neural trajectories computed from the identified ensembles retain the grooming related dynamics present in trajectories computed from all units in the session. These results elaborate striatal function in rodent self-grooming and demonstrate that striatal grooming-related activity is organized within functional ensembles, improving our understanding of how the striatum guides action selection in a naturalistic behavior.

The deep cerebellar nuclei to striatum disynaptic connection contributes to skilled forelimb movement

Cerebellar-thalamo-striatal synaptic communication has been implicated in a wide range of behaviors, including goal-directed actions, and is altered in cerebellar dystonia. However, its detailed connectivity through the thalamus and its contribution to the execution of forelimb movements is unclear. Here, we use trans-synaptic and retrograde tracing, ex vivo slice recordings, and optogenetic inhibitions during the execution of unidirectional or sequential joystick displacements to demonstrate that the deep cerebellar nuclei (DCN) influence the dorsal striatum with a very high probability. We show that this mainly occurs through the centrolateral (CL), parafascicular (PF), and ventrolateral (VL) nuclei of the thalamus, observing that the DCN→VL and DCN→CL pathways contribute to the execution of unidirectional forelimb displacements while the DCN→PF and DCN→thalamo→striatal pathways contribute to the appropriate execution of forelimb reaching and sequential displacements. These findings highlight specific contributions of the different cerebellar-thalamo-striatal paths to the control of skilled forelimb movement.

Segregation of D1 and D2 dopamine receptors in the striatal direct and indirect pathways: An historical perspective

The direct and indirect striatal pathways form a cornerstone of the circuits of the basal ganglia. Dopamine has opponent affects on the function of these pathways due to the segregation of the D1- and D2-dopamine receptors in the spiny projection neurons giving rise to the direct and indirect pathways. An historical perspective is provided on the discovery of dopamine receptor segregation leading to models of how the direct and indirect affect motor behavior.

Arithmetic value representation for hierarchical behavior composition

The ability to compose new skills from a preacquired behavior repertoire is a hallmark of biological intelligence. Although artificial agents extract reusable skills from past experience and recombine them in a hierarchical manner, whether the brain similarly composes a novel behavior is largely unknown. In the present study, I show that deep reinforcement learning agents learn to solve a novel composite task by additively combining representations of prelearned action values of constituent subtasks. Learning efficacy in the composite task was further augmented by the introduction of stochasticity in behavior during pretraining. These theoretical predictions were empirically tested in mice, where subtask pretraining enhanced learning of the composite task. Cortex-wide, two-photon calcium imaging revealed analogous neural representations of combined action values, with improved learning when the behavior variability was amplified. Together, these results suggest that the brain composes a novel behavior with a simple arithmetic operation of preacquired action-value representations with stochastic policies.

Cilia in the Striatum Mediate Timing-Dependent Functions

Almost all brain cells contain cilia, antennae-like microtubule-based organelles. Yet, the significance of cilia, once considered vestigial organelles, in the higher-order brain functions is unknown. Cilia act as a hub that senses and transduces environmental sensory stimuli to generate an appropriate cellular response. Similarly, the striatum, a brain structure enriched in cilia, functions as a hub that receives and integrates various types of environmental information to drive appropriate motor response. To understand cilia's role in the striatum functions, we used loxP/Cre technology to ablate cilia from the dorsal striatum of male mice and monitored the behavioral consequences. Our results revealed an essential role for striatal cilia in the acquisition and brief storage of information, including learning new motor skills, but not in long-term consolidation of information or maintaining habitual/learned motor skills. A fundamental aspect of all disrupted functions was the "time perception/judgment deficit." Furthermore, the observed behavioral deficits form a cluster pertaining to clinical manifestations overlapping across psychiatric disorders that involve the striatum functions and are known to exhibit timing deficits. Thus, striatal cilia may act as a calibrator of the timing functions of the basal ganglia-cortical circuit by maintaining proper timing perception. Our findings suggest that dysfunctional cilia may contribute to the pathophysiology of neuro-psychiatric disorders, as related to deficits in timing perception.

Differential Alterations in Striatal Direct and Indirect Pathways Mediate Two Autism-like Behaviors in Valproate-Exposed Mice

Autism is characterized by two key diagnostic criteria including social deficits and repetitive behaviors. Although recent studies implicated ventral striatum in social deficits and dorsal striatum in repetitive behaviors, here we revealed coexisting and opposite morphologic and functional alterations in the dorsostriatal direct and indirect pathways, and such alterations in these two pathways were found to be responsible, respectively, for the two abovementioned different autism-like behaviors exhibited by male mice prenatally exposed to valproate. The alteration in direct pathway was characterized by a potentiated state of basal activity, with impairment in transient responsiveness of D1-MSNs during social exploration. Concurrent alteration in indirect pathway was a depressed state of basal activity, with enhancement in transient responsiveness of D2-MSNs during repetitive behaviors. A causal relationship linking such differential alterations in these two pathways to the coexistence of these two autism-like behaviors was demonstrated by the cell type-specific correction of abnormal basal activity in the D1-MSNs and D2-MSNs of valproate-exposed mice. The findings support those differential alterations in two striatal pathways mediate the two coexisting autism-like behavioral abnormalities, respectively. This result will help in developing therapeutic options targeting these circuit alterations.SIGNIFICANCE STATEMENT Autism is characterized by two key diagnostic criteria including social deficits and repetitive behaviors. Although a number of recent studies have implicated ventral striatum in social deficits and dorsal striatum in repetitive behaviors, but social behaviors need to be processed by a series of actions, and repetitive behaviors, especially the high-order repetitive behaviors such as restrictive interests, have its scope to cognitive and emotional domains. The current study, for the first time, revealed that prenatal valproate exposure induced coexisting and differential alterations in the dorsomedial striatal direct and indirect pathways, and that these alterations mediate the two coexisting autism-like behavioral abnormalities, respectively. This result will help in developing therapeutic options targeting these circuit alterations to address the behavioral abnormalities.

Learning-induced changes in the neural circuits underlying motor sequence execution

As the old adage goes: practice makes perfect. Yet, the neural mechanisms by which rote repetition transforms a halting behavior into a fluid, effortless, and "automatic" action are not well understood. Here we consider the possibility that well-practiced motor sequences, which initially rely on higher-level decision-making circuits, become wholly specified in lower-level control circuits. We review studies informing this idea, discuss the constraints on such shift in control, and suggest approaches to pinpoint circuit-level changes associated with motor sequence learning.

Activation, but not inhibition, of the indirect pathway disrupts choice rejection in a freely moving, multiple-choice foraging task

The dorsomedial striatum (DMS) plays a key role in action selection, but less is known about how direct and indirect pathway spiny projection neurons (dSPNs and iSPNs, respectively) contribute to choice rejection in freely moving animals. Here, we use pathway-specific chemogenetic manipulation during a serial choice foraging task to test the role of dSPNs and iSPNs in learned choice rejection. We find that chemogenetic activation, but not inhibition, of iSPNs disrupts rejection of nonrewarded choices, contrary to predictions of a simple "select/suppress" heuristic. Our findings suggest that iSPNs' role in stopping and freezing does not extend in a simple fashion to choice rejection in an ethological, freely moving context. These data may provide insights critical for the successful design of interventions for addiction or other conditions in which it is desirable to strengthen choice rejection.

Neural mechanisms underlying the temporal organization of naturalistic animal behavior

Naturalistic animal behavior exhibits a strikingly complex organization in the temporal domain, with variability arising from at least three sources: hierarchical, contextual, and stochastic. What neural mechanisms and computational principles underlie such intricate temporal features? In this review, we provide a critical assessment of the existing behavioral and neurophysiological evidence for these sources of temporal variability in naturalistic behavior. Recent research converges on an emergent mechanistic theory of temporal variability based on attractor neural networks and metastable dynamics, arising via coordinated interactions between mesoscopic neural circuits. We highlight the crucial role played by structural heterogeneities as well as noise from mesoscopic feedback loops in regulating flexible behavior. We assess the shortcomings and missing links in the current theoretical and experimental literature and propose new directions of investigation to fill these gaps.

Hierarchical Reinforcement Learning, Sequential Behavior, and the Dorsal Frontostriatal System

To effectively behave within ever-changing environments, biological agents must learn and act at varying hierarchical levels such that a complex task may be broken down into more tractable subtasks. Hierarchical reinforcement learning (HRL) is a computational framework that provides an understanding of this process by combining sequential actions into one temporally extended unit called an option. However, there are still open questions within the HRL framework, including how options are formed and how HRL mechanisms might be realized within the brain. In this review, we propose that the existing human motor sequence literature can aid in understanding both of these questions. We give specific emphasis to visuomotor sequence learning tasks such as the discrete sequence production task and the M × N (M steps × N sets) task to understand how hierarchical learning and behavior manifest across sequential action tasks as well as how the dorsal cortical-subcortical circuitry could support this kind of behavior. This review highlights how motor chunks within a motor sequence can function as HRL options. Furthermore, we aim to merge findings from motor sequence literature with reinforcement learning perspectives to inform experimental design in each respective subfield.

Habitual or hyper-controlled behavior: OCD symptoms and explicit sequence learning

BACKGROUND AND OBJECTIVES

This study examined whether ritualistic behaviors characteristic of obsessive-compulsive disorder (OCD) are a product of dysfunctional goal-directed behavior leading to habitual behavior (Gillan & Robbins, 2014). We used an explicit motor sequence learning task to investigate the repetition of chunked action sequences across the OC spectrum. As sequential motor behavior is practiced, action movements appear to get bundled together, and the initial movement of the sequence activates the entire sequence, leaving it relatively insensitive to change. Therefore, compulsive behavior in OCD may be a result of failing to inhibit the full activation of an extensively learned action sequence.

METHODS

Fifty-seven participants across the range of OCD symptoms practiced one sequence and then tested on a novel sequence in which one of the middle movements was omitted. Optimal performance for the new sequence required goal-directed inhibition of the original sequence and goal-directed execution of the new sequence instead. To manipulate activation of goal-directed behavior, we added a dual-task condition with a competing auditory tonal N-Back task. Data were analyzed using mixed-effects models.

RESULTS

Although we did observe expected learning patterns during learning of the original sequence, slower reaction times for the new sequences, and higher errors in the dual-task condition, performance was not significantly related to either obsessive-compulsive symptoms or distress symptoms.

LIMITATIONS

The current study used an analog sample; replication in a treatment seeking sample is warranted.

CONCLUSIONS

These findings challenge the goal-directed dysfunction model of OCD.

Striatal direct pathway neurons play leading roles in accelerating rotarod motor skill learning

Dorsal striatum is important for movement control and motor skill learning. However, it remains unclear how the spatially and temporally distributed striatal medium spiny neuron (MSN) activity in the direct and indirect pathways (D1 and D2 MSNs, respectively) encodes motor skill learning. Combining miniature fluorescence microscopy with an accelerating rotarod procedure, we identified two distinct MSN subpopulations involved in accelerating rotarod learning. In both D1 and D2 MSNs, we observed neurons that displayed activity tuned to acceleration during early stages of trials, as well as movement speed during late stages of trials. We found a distinct evolution trajectory for early-stage neurons during motor skill learning, with the evolution of D1 MSNs correlating strongly with performance improvement. Importantly, optogenetic inhibition of the early-stage neural activity in D1 MSNs, but not D2 MSNs, impaired accelerating rotarod learning. Together, this study provides insight into striatal D1 and D2 MSNs encoding motor skill learning.

Secondary auditory cortex mediates a sensorimotor mechanism for action timing

The ability to accurately determine when to perform an action is a fundamental brain function and vital to adaptive behavior. The behavioral mechanism and neural circuit for action timing, however, remain largely unknown. Using a new, self-paced action timing task in mice, we found that deprivation of auditory, but not somatosensory or visual input, disrupts learned action timing. The hearing effect was dependent on the auditory feedback derived from the animal's own actions, rather than passive environmental cues. Neuronal activity in the secondary auditory cortex was found to be both correlated with and necessary for the proper execution of learned action timing. Closed-loop, action-dependent optogenetic stimulation of the specific task-related neuronal population within the secondary auditory cortex rescued the key features of learned action timing under auditory deprivation. These results unveil a previously underappreciated sensorimotor mechanism in which the secondary auditory cortex transduces self-generated audiomotor feedback to control action timing.

Opponent control of behavior by dorsomedial striatal pathways depends on task demands and internal state

A classic view of the striatum holds that activity in direct and indirect pathways oppositely modulates motor output. Whether this involves direct control of movement, or reflects a cognitive process underlying movement, remains unresolved. Here we find that strong, opponent control of behavior by the two pathways of the dorsomedial striatum depends on the cognitive requirements of a task. Furthermore, a latent state model (a hidden Markov model with generalized linear model observations) reveals that-even within a single task-the contribution of the two pathways to behavior is state dependent. Specifically, the two pathways have large contributions in one of two states associated with a strategy of evidence accumulation, compared to a state associated with a strategy of repeating previous choices. Thus, both the demands imposed by a task, as well as the internal state of mice when performing a task, determine whether dorsomedial striatum pathways provide strong and opponent control of behavior.

Thunderstruck: The ACDC model of flexible sequences and rhythms in recurrent neural circuits

Adaptive sequential behavior is a hallmark of human cognition. In particular, humans can learn to produce precise spatiotemporal sequences given a certain context. For instance, musicians can not only reproduce learned action sequences in a context-dependent manner, they can also quickly and flexibly reapply them in any desired tempo or rhythm without overwriting previous learning. Existing neural network models fail to account for these properties. We argue that this limitation emerges from the fact that sequence information (i.e., the position of the action) and timing (i.e., the moment of response execution) are typically stored in the same neural network weights. Here, we augment a biologically plausible recurrent neural network of cortical dynamics to include a basal ganglia-thalamic module which uses reinforcement learning to dynamically modulate action. This “associative cluster-dependent chain” (ACDC) model modularly stores sequence and timing information in distinct loci of the network. This feature increases computational power and allows ACDC to display a wide range of temporal properties (e.g., multiple sequences, temporal shifting, rescaling, and compositionality), while still accounting for several behavioral and neurophysiological empirical observations. Finally, we apply this ACDC network to show how it can learn the famous “Thunderstruck” song intro and then flexibly play it in a “bossa nova” rhythm without further training.

How do humans flexibly adapt action sequences? For instance, musicians can learn a song and quickly speed up or slow down the tempo, or even play the song following a completely different rhythm (e.g., a rock song using a bossa nova rhythm). In this work, we build a biologically plausible network of cortico-basal ganglia interactions that explains how this temporal flexibility may emerge in the brain. Crucially, our model factorizes sequence order and action timing, respectively represented in cortical and basal ganglia dynamics. This factorization allows full temporal flexibility, i.e. the timing of a learned action sequence can be recomposed without interfering with the order of the sequence. As such, our model is capable of learning asynchronous action sequences, and flexibly shift, rescale, and recompose them, while accounting for biological data.

Descending neurons coordinate anterior grooming behavior in Drosophila

The brain coordinates the movements that constitute behavior, but how descending neurons convey the myriad of commands required to activate the motor neurons of the limbs in the right order and combinations to produce those movements is not well understood. For anterior grooming behavior in the fly, we show that its component head sweeps and leg rubs can be initiated separately, or as a set, by different descending neurons. Head sweeps and leg rubs are mutually exclusive movements of the front legs that normally alternate, and we show that circuits in the ventral nerve cord as well as in the brain can resolve competing commands. Finally, the left and right legs must work together to remove debris. The coordination for leg rubs can be achieved by unilateral activation of a single descending neuron, while a similar manipulation of a different descending neuron decouples the legs to produce single-sided head sweeps. Taken together, these results demonstrate that distinct descending neurons orchestrate the complex alternation between the movements that make up anterior grooming.

Dorsal striatum coding for the timely execution of action sequences

The automatic initiation of actions can be highly functional. But occasionally these actions cannot be withheld and are released at inappropriate times, impulsively. Striatal activity has been shown to participate in the timing of action sequence initiation and it has been linked to impulsivity. Using a self-initiated task, we trained adult male rats to withhold a rewarded action sequence until a waiting time interval has elapsed. By analyzing neuronal activity we show that the striatal response preceding the initiation of the learned sequence is strongly modulated by the time subjects wait before eliciting the sequence. Interestingly, the modulation is steeper in adolescent rats, which show a strong prevalence of impulsive responses compared to adults. We hypothesize this anticipatory striatal activity reflects the animals’ subjective reward expectation, based on the elapsed waiting time, while the steeper waiting modulation in adolescence reflects age-related differences in temporal discounting, internal urgency states, or explore–exploit balance.

Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors

Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning, signaling discrepancies between actual versus expected reward predicted by the cues.^1-5 While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. The vast majority of previous studies examining dopamine responses primarily have used discrete reward-predictive stimuli,^1-15 whether Pavlovian conditioned stimuli for which no action is required to earn reward or explicit discriminative stimuli that essentially instruct an animal how and when to respond for reward. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single and sequential instrumental actions, in behavioral contexts with minimal overt changes in the animal's external environment. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal's own action, relative to non-contingent passive stimulation. This dopamine suppression generalized to food rewards was specific to the reinforced action, was temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. These findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.

Neural mechanisms for learning hierarchical structures of information

Spatial and temporal information from the environment is often hierarchically organized, so is our knowledge formed about the environment. Identifying the meaningful segments embedded in hierarchically structured information is crucial for cognitive functions, including visual, auditory, motor, memory, and language processing. Segmentation enables the grasping of the links between isolated entities, offering the basis for reasoning and thinking. Importantly, the brain learns such segmentation without external instructions. Here, we review the underlying computational mechanisms implemented at the single-cell and network levels. The network-level mechanism has an interesting similarity to machine-learning methods for graph segmentation. The brain possibly implements methods for the analysis of the hierarchical structures of the environment at multiple levels of its processing hierarchy.

Striatal indirect pathway mediates exploration via collicular competition

The ability to suppress actions that lead to a negative outcome and explore alternative actions is necessary for optimal decision making. Although the basal ganglia have been implicated in these processes^1-5, the circuit mechanisms underlying action selection and exploration remain unclear. Here, using a simple lateralized licking task, we show that indirect striatal projection neurons (iSPN) in the basal ganglia contribute to these processes through modulation of the superior colliculus (SC). Optogenetic activation of iSPNs suppresses contraversive licking and promotes ipsiversive licking. Activity in lateral superior colliculus (lSC), a region downstream of the basal ganglia, is necessary for task performance and predicts lick direction. Furthermore, iSPN activation suppresses ipsilateral lSC, but surprisingly excites contralateral lSC, explaining the emergence of ipsiversive licking. Optogenetic inactivation reveals inter-collicular competition whereby each hemisphere of the superior colliculus inhibits the other, thus allowing the indirect pathway to disinhibit the contralateral lSC and trigger licking. Finally, inactivating iSPNs impairs suppression of devalued but previously rewarded licking and reduces exploratory licking. Our results reveal that iSPNs engage the competitive interaction between lSC hemispheres to trigger a motor action and suggest a general circuit mechanism for exploration during action selection.

Direct and indirect pathway neurons in ventrolateral striatum differentially regulate licking movement and nigral responses

Drinking behavior in rodents is characterized by stereotyped, rhythmic licking movement, which is regulated by the basal ganglia. It is unclear how direct and indirect pathways control the lick bout and individual spout contact. We find that inactivating D1 and D2 receptor-expressing medium spiny neurons (MSNs) in the ventrolateral striatum (VLS) oppositely alters the number of licks in a bout. D1- and D2-MSNs exhibit different patterns of lick-sequence-related activity and different phases of oscillation time-locked to the lick cycle. On the timescale of a lick cycle, transient inactivation of D1-MSNs during tongue protrusion reduces spout contact probability, whereas transiently inactivating D2-MSNs has no effect. On the timescale of a lick bout, inactivation of D1-MSNs (D2-MSNs) causes rate increase (decrease) in a subset of basal ganglia output neurons that decrease firing during licking. Our results reveal the distinct roles of D1- and D2-MSNs in regulating licking at both coarse and fine timescales.

Anterior thalamic NMDA-induced damage impairs extrapolation relying on serial stimulus patterns, in rats

Extrapolation of serial stimulus patterns seems to depend upon identification and application of patterns relating sequences of stimuli stored in memory, thus allowing prediction of pending events never experienced before. There have been proposals that such a "generator of predictions system" would include the subiculum, mammillary bodies, anteroventral thalamus and cingulate cortex (e.g., Gray, 1982). The anteroventral thalamus (AVT) seems to be in a strategic position, both hodologically and experimentally, to allow testing of this hypothesis. This study investigated the effect of NMDA-induced damage to the anteroventral thalamus [part of the anterodorsal (AD) thalamus was also damaged in some animals], following stereotaxic minute topic microinjections, on the ability of male Wistar rats to extrapolate relying on serial stimulus patterns. Corresponding sham-operated controls received phosphate-saline buffer microinjections at the same stereotaxic coordinates. The subjects were trained to run through a straight alleyway along 31 sessions, one session per day, to get rewarded. Each session included four successive trials. Subjects exposed to the monotonic serial pattern received 14, 7, 3, 1 sunflower seeds along trials. Subjects exposed to the non-monotonic serial pattern received 14, 3, 7, 1 sunflower seeds. On the 32nd testing session, a fifth trial, never experienced before, was included immediately after the fourth trial. Sham-operated control subjects exposed to the monotonic serial pattern were expected to exhibit longer running times, since the content of their prediction in the fifth trial should be "less than 1 sunflower seeds". In contrast, control subjects exposed to the non-monotonic serial pattern were expected to exhibit shorter running times, since the content of their prediction would be "more than 1 sunflower seeds". Confirming these predictions, control subjects exposed to the monotonic serial pattern exhibited longer running times as compared to both, their own running times in previous trials within the same session and control subjects exposed to the non-monotonic schedule, thus indicating the occurrence of extrapolation. In contrast, AVT/AD lesioned subjects exposed to the monotonic schedule did not exhibit this increase in running times on the fifth trial, indicating lack of extrapolation. These results indicate that extrapolation relying on serial stimulus patterns is disrupted following extensive NMDA-induced damage to AVT and part of the AD. This represents the first consistent demonstration that the anterior thalamic nuclei are required for extrapolation of serial stimulus patterns and generation of predictions.

Optogenetic stimulation of striatal patches modifies habit formation and inhibits dopamine release

Habits are inflexible behaviors that develop after extensive repetition, and overreliance on habits is a hallmark of many pathological states. The striatum is involved in the transition from flexible to inflexible responding, and interspersed throughout the striatum are patches, or striosomes, which make up ~15% of the volume of the striatum relative to the surrounding matrix compartment. Previous studies have suggested that patches are necessary for normal habit formation, but it remains unknown exactly how patches contribute to habit formation and expression. Here, using optogenetics, we stimulated striatal patches in Sepw1-NP67 mice during variable interval training (VI60), which is used to establish habitual responding. We found that activation of patches at reward retrieval resulted in elevated responding during VI60 training by modifying the pattern of head entry and pressing. Further, this optogenetic manipulation reduced subsequent responding following reinforcer devaluation, suggesting modified habit formation. However, patch stimulation did not generally increase extinction rates during a subsequent extinction probe, but did result in a small 'extinction burst', further suggesting goal-directed behavior. On the other hand, this manipulation had no effect in omission trials, where mice had to withhold responses to obtain rewards. Finally, we utilized fast-scan cyclic voltammetry to investigate how patch activation modifies evoked striatal dopamine release and found that optogenetic activation of patch projections to the substantia nigra pars compacta (SNc) is sufficient to suppress dopamine release in the dorsal striatum. Overall, this work provides novel insight into the role of the patch compartment in habit formation, and provides a potential mechanism for how patches modify habitual behavior by exerting control over dopamine signaling.

Dynamic encoding of saccade sequences in primate frontal eye field

The frontal eye field (FEF) is a key part of the oculomotor system, with dominant responses to the direction of single saccades. However, whether and how FEF contributes to sequential saccades remain largely unknown. By training rhesus monkeys to perform saccade sequences, we found sequence-related activities in FEF neurons, whose selectivity to saccade direction undergoes dynamic changes during sequential vs. single saccades. These sequence-related activities are context-dependent, exhibiting different firing activities during memory- vs. visually guided sequences. When the monkey was performing the sequential saccade task, the thresholds of microstimulation to evoke saccades in FEF were increased and the percentage of the successfully induced saccades was significantly reduced compared with the fixation condition. Pharmacological inactivation of FEF impaired the monkey's performance of previously learned sequential saccades, with different effects on the same actions depending on its position within the sequence. These results reveal the context-dependent, sequence-specific dynamic encoding of saccades in FEF, and underscore the crucial role of FEF in the planning and execution of sequential saccades. KEY POINTS: FEF neurons respond differently during sequential vs. single saccades Sequence-related FEF activity is context-dependent The microstimulation threshold in FEF was increased during the sequential task but the evoked saccade did not alter the sequence structure FEF inactivation severely impaired the performance of sequential saccades.

The Contribution of Premotor Cortico-Striatal Projections to the Execution of Serial Order Sequences

Striatal activity is necessary to initiate and execute sequences of actions. The main excitatory input to the striatum comes from the cortex. While it is hypothesized that motor and premotor cortico-striatal projections are important to guide striatal activity during the execution of sequences of actions, technical limitations have made this challenging to address. Here, we implemented a task in mice that allows for the study of different moments to execute a serial order sequence consisting of two subsequences of actions. Using this task, we performed electrophysiological recordings in the premotor (M2) and primary motor (M1) cortices, and state-dependent optogenetic inhibitions of their cortico-striatal projections. We show that while both M2 and M1 contain activity modulations related to the execution of self-paced sequences, mainly, the premotor cortico-striatal projections contribute to the proper execution/structuring of these sequences.

The basal ganglia control the detailed kinematics of learned motor skills

The basal ganglia are known to influence action selection and modulation of movement vigor, but whether and how they contribute to specifying the kinematics of learned motor skills is not understood. Here, we probe this question by recording and manipulating basal ganglia activity in rats trained to generate complex task-specific movement patterns with rich kinematic structure. We find that the sensorimotor arm of the basal ganglia circuit is crucial for generating the detailed movement patterns underlying the acquired motor skills. Furthermore, the neural representations in the striatum, and the control function they subserve, do not depend on inputs from the motor cortex. Taken together, these results extend our understanding of the basal ganglia by showing that they can specify and control the fine-grained details of learned motor skills through their interactions with lower-level motor circuits.

Interhemispheric Cortico-Cortical Pathway for Sequential Bimanual Movements in Mice

Animals precisely coordinate their left and right limbs for various adaptive purposes. While the left and right limbs are clearly controlled by different cortical hemispheres, the neural mechanisms that determine the action sequence between them remains elusive. Here, we have established a novel head-fixed bimanual-press (biPress) sequence task in which mice sequentially press left and right pedals with their forelimbs in a predetermined order. Using this motor task, we found that the motor cortical neurons responsible for the first press (1P) also generate independent motor signals for the second press (2P) by the opposite forelimb during the movement transitions between forelimbs. Projection-specific calcium imaging and optogenetic manipulation revealed these motor signals are transferred from one motor cortical hemisphere to the other via corticocortical projections. Together, our results suggest the motor cortices coordinate sequential bimanual movements through corticocortical pathways.

Involvement of Midbrain Dopamine Neuron Activity in Negative Reinforcement Learning in Mice

The activity of the midbrain dopamine system reflects the valence of environmental events and modulates various brain structures to modify an organism's behavior. A series of recent studies reported that the direct and indirect pathways in the striatum are critical for instrumental learning, but the dynamic changes in dopamine neuron activity that occur during negative reinforcement learning are still largely unclear. In the present study, by using a negative reinforcement learning paradigm employing foot shocks as aversive stimuli, bidirectional changes in substantia nigra pars compacta (SNc) dopamine neuron activity in the learning and habituation phases were observed. The results showed that in the learning phase, before mice had mastered the skill of escaping foot shocks, the presence of foot shocks induced a transient reduction in the activity of SNc dopamine neurons; however, in the habituation phase, in which the learned skill was automated, it induced a transient increase. Microinjection of a dopamine D1 receptor (D1R) or D2 receptor (D2R) antagonist into the dorsomedial striatum (DMS) significantly impaired learning behavior, suggesting that the modulatory effects of dopamine on both the direct and indirect pathways are required. Moreover, during the learning phase, excitatory synaptic transmission to DMS D2R-expressing medium spiny neurons (D2-MSNs) was potentiated. However, upon completion of the learning and habituation phases, the synapses onto D1R-expressing medium spiny neurons (D1-MSNs) were potentiated, and those onto D2-MSNs were restored to normal levels. The bidirectional changes in both SNc dopamine neuron activity and DMS synaptic plasticity might be the critical neural correlates for negative reinforcement learning.

α-Synuclein Selectively Impairs Motor Sequence Learning and Value Sensitivity: Reversal by the Adenosine A2A Receptor Antagonists

Parkinson's disease (PD) is characterized pathologically by alpha-synuclein (α-Syn) aggregates and clinically by the motor as well as cognitive deficits, including impairments in sequence learning and habit learning. Using intracerebral injection of WT and A53T mutant α-Syn fibrils, we investigate the behavioral mechanism of α-Syn for procedure-learning deficit in PD by critically determining the α-Syn-induced effects on model-based goal-directed behavior, model-free (probability-based) habit learning, and hierarchically organized sequence learning. 1) Contrary to the widely held view of habit-learning deficit in early PD, α-Syn aggregates in the dorsomedial striatum (DMS) and dorsolateral striatum (DLS) did not affect acquisition of habit learning, but selectively impaired goal-directed behavior with reduced value sensitivity. 2) α-Syn in the DLS (but not DMS) and SNc selectively impaired the sequence learning by affecting sequence initiation with the reduced first-step accuracy. 3) Adenosine A2A receptor (A2AR) antagonist KW6002 selectively improved sequence learning by preferentially improving sequence initiation and shift of sequence learning as well as behavioral reactivity. These findings established a casual role of α-Syn in the SN-DLS pathway in sequence-learning deficit and DMS α-Syn in goal-directed behavior deficit and suggest a novel therapeutic strategy to improve sequence-learning deficit in PD with enhanced sequence initiation by A2AR antagonists.

How the Mind Creates Structure: Hierarchical Learning of Action Sequences

Humans have the astonishing capacity to quickly adapt to varying environmental demands and reach complex goals in the absence of extrinsic rewards. Part of what underlies this capacity is the ability to flexibly reuse and recombine previous experiences, and to plan future courses of action in a psychological space that is shaped by these experiences. Decades of research have suggested that humans use hierarchical representations for efficient planning and flexibility, but the origin of these representations has remained elusive. This study investigates how 73 participants learned hierarchical representations through experience, in a task in which they had to perform complex action sequences to obtain rewards. Complex action sequences were composed of simpler action sequences, which were not rewarded, but whose completion was signaled to participants. We investigated the process with which participants learned to perform simpler action sequences and combined them into complex action sequences. After learning action sequences, participants completed a transfer phase in which either simple sequences or complex sequences were manipulated without notice. Relearning progressed slower when simple than complex sequences were changed, in accordance with a hierarchical representations in which lower levels are quickly consolidated, potentially stabilizing exploration, while higher levels remain malleable, with benefits for flexible recombination.

Complete representation of action space and value in all dorsal striatal pathways

The dorsal striatum plays a central role in the selection, execution, and evaluation of actions. An emerging model attributes action selection to the matrix and evaluation to the striosome compartment. Here, we use large-scale cell-type-specific calcium imaging to determine the activity of striatal projection neurons (SPNs) during motor and decision behaviors in the three major outputs of the dorsomedial striatum: Oprm1⁺ striosome versus D1⁺ direct and A2A⁺ indirect pathway SPNs. We find that Oprm1⁺ SPNs show complex tunings to simple movements and value-guided actions, which are conserved across many sessions in a single task but remap between contexts. During decision making, the SPN tuning profiles form a complete representation in which sequential SPN activity jointly encodes task progress and value. We propose that the three major output pathways in the dorsomedial striatum share a similarly complete representation of the entire action space, including task- and phase-specific signals of action value and choice.

Thalamic control of cortical dynamics in a model of flexible motor sequencing

The neural mechanisms that generate an extensible library of motor motifs and flexibly string them into arbitrary sequences are unclear. We developed a model in which inhibitory basal ganglia output neurons project to thalamic units that are themselves bidirectionally connected to a recurrent cortical network. We model the basal ganglia inhibitory patterns as silencing some thalamic neurons while leaving others disinhibited and free to interact with cortex during specific motifs. We show that a small number of disinhibited thalamic neurons can control cortical dynamics to generate specific motor output in a noise-robust way. Additionally, a single "preparatory" thalamocortical network can produce fast cortical dynamics that support rapid transitions between any pair of learned motifs. If the thalamic units associated with each sequence component are segregated, many motor outputs can be learned without interference and then combined in arbitrary orders for the flexible production of long and complex motor sequences.

Interpreting the role of the striatum during multiple phases of motor learning

The synaptic pathways in the striatum are central to basal ganglia functions including motor control, learning and organization, action selection, acquisition of motor skills, cognitive function, and emotion. Here, we review the role of the striatum and its connections in motor learning and performance. The development of new techniques to record neuronal activity and animal models of motor disorders using neurotoxin, pharmacological, and genetic manipulations are revealing pathways that underlie motor performance and motor learning, as well as how they are altered by pathophysiological mechanisms. We discuss approaches that can be used to analyze complex motor skills, particularly in rodents, and identify specific questions central to understanding how striatal circuits mediate motor learning.