Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training

Striatal arbitration between choice strategies guides few-shot adaptation

Animals often exhibit rapid action changes in context-switching environments. This study hypothesized that, compared to the expected outcome, an unexpected outcome leads to distinctly different action-selection strategies to guide rapid adaptation. We designed behavioral measures differentiating between trial-by-trial dynamics after expected and unexpected events. In various reversal learning data with different rodent species and task complexities, conventional learning models failed to replicate the choice behavior following an unexpected outcome. This discrepancy was resolved by the proposed model with two different decision variables contingent on outcome expectation: the support-stay and conflict-shift bias. Electrophysiological data analyses revealed that striatal neurons encode our model's key variables. Furthermore, the inactivation of striatal direct and indirect pathways neutralizes the effect of past expected and unexpected outcomes, respectively, on the action-selection strategy following an unexpected outcome. Our study suggests unique roles of the striatum in arbitrating between different action selection strategies for few-shot adaptation.

Cortico-striatal action control inherent of opponent cognitive-motivational styles

Turning on cue or stopping at a red light requires attending to such cues to select action sequences, or suppress action, in accordance with learned cue-associated action rules. Cortico-striatal projections are an essential part of the brain's attention-motor interface. Glutamate-sensing microelectrode arrays were used to measure glutamate transients in the dorsomedial striatum (DMS) of male and female rats walking a treadmill and executing cued turns and stops. Prelimbic-DMS projections were chemogenetically inhibited to determine their behavioral necessity and the cortico-striatal origin of cue-evoked glutamate transients. Furthermore, we investigated rats exhibiting preferably goal-directed (goal trackers, GTs) versus cue-driven attention (sign-trackers, STs), to determine the impact of such cognitive-motivational biases on cortico-striatal control. GTs executed more cued turns and initiated such turns more slowly than STs. During turns, but not missed turns or cued stops, cue-evoked glutamate concentrations were higher in GTs than in STs. In STs, turn cue-locked glutamate concentrations frequently peaked twice or three times, contrasting with predominately single peaks in GTs. In GTs, but not STs, inhibition of prelimbic-DMS projections attenuated turn rates and turn cue-evoked glutamate concentrations and increased the number of turn cue-locked glutamate peaks. These findings indicate that turn cue-evoked glutamate release in GTs is tightly controlled by cortico-striatal neuronal activity. In contrast, in STs, glutamate release from DMS glutamatergic terminals may be regulated by other striatal circuitry, preferably mediating cued suppression of action and reward tracking. As cortico-striatal dysfunction has been hypothesized to contribute to a wide range of disorders, including complex movement control deficits in Parkinson's disease and compulsive drug taking, the demonstration of phenotypic contrasts in cortico-striatal control implies the presence of individual vulnerabilities for such disorders.

Striatal cell-type specific stability and reorganization underlying agency and habit

Adaptive decision making requires agency, knowledge that actions produce particular outcomes. For well-practiced routines, agency is relinquished in favor of habit. Here, we asked how dorsomedial striatum D1+ and D2/A2A+ neurons contribute to agency and habit. We imaged calcium activity of these neurons as mice learned to lever press with agency and formed habits with overtraining. Whereas many D1+ neurons stably encoded actions throughout learning and developed encoding of reward outcomes, A2A+ neurons reorganized their encoding of actions from initial action-outcome learning to habit formation. Chemogenetic manipulations indicated that both D1+ and A2A+ neurons support action-outcome learning, but only D1+ neurons enable the use of such agency for adaptive, goal-directed decision making. These data reveal coordinated dorsomedial striatum D1+ and A2A+ function for the development of agency, cell-type specific stability and reorganization underlying agency and habit, and important insights into the neuronal circuits of how we learn and decide.

SENP3 knockdown improves motor and cognitive impairments in the intranasal MPTP rodent model of Parkinson's disease

Several mechanisms underlying Parkinson's disease (PD) remain unclear, and effective treatments are still lacking. The conjugation of the small ubiquitin-like modifier (SUMO), known as SUMOylation, to key proteins in PD has shown potential beneficial effects. Considering that this process is reversed by SUMO-specific proteases (SENPs), this study addressed the effects of increased SUMO-2/3 conjugation, mediated by SENP3 knockdown, in the intranasal 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP) rodent model of PD. Two weeks after infusion of the shRNA-containing lentiviral vector into the dorsolateral striatum and one week following intranasal MPTP administration, male Wistar rats were evaluated using cognitive and motor behavioural tests. Infection efficiency was confirmed by detecting GFP expression in the dorsolateral striatum. SENP3 knockdown, verified by Western blotting, resulted in increased SUMO-2/3 conjugation. MPTP-administered rats displayed impairments in both recognition and spatial memories, while SENP3 knockdown prevented these deficits. Rats exposed to MPTP also exhibited motor dysfunction, which was ameliorated by SENP3 knockdown. These findings underscore the involvement of SUMO-2/3 conjugation in PD and its potential as a novel therapeutic target to counteract cognitive and motor impairments induced by neurodegeneration.

Dorsomedial Striatum (DMS) CB1R Signaling Promotes Pavlovian Devaluation Sensitivity in Male Long Evans Rats and Reduces DMS Inhibitory Synaptic Transmission in Both Sexes

Cannabinoid receptor-1 (CB1R) signaling in the dorsal striatum regulates the shift from flexible to habitual behavior in instrumental outcome devaluation. Based on prior work establishing individual-, sex-, and experience-dependent differences in pavlovian behaviors, we predicted a role for dorsomedial striatum (DMS) CB1R signaling in driving rigid responding in pavlovian autoshaping and outcome devaluation. We trained male and female Long Evans rats in pavlovian lever autoshaping (PLA). We gave intra-DMS infusions of the CB1R inverse agonist, rimonabant, before satiety-induced outcome devaluation test sessions, where we sated rats on training pellets or home cage chow and tested them in brief nonreinforced PLA sessions. Overall, inhibition of DMS CB1R signaling prevented pavlovian outcome devaluation but did not affect behavior in reinforced PLA sessions. Males were sensitive to devaluation while females were not, and DMS CB1R blockade impaired devaluation sensitivity in males. Because these results suggest DMS CB1R signaling supports flexible responding, we investigated how DMS CB1R signaling impacts local inhibitory synaptic transmission in male and female Long Evans rats. We recorded spontaneous inhibitory postsynaptic currents (sIPSC) from DMS neurons at baseline and after application of a CB1R agonist, WIN 55,212-2. We found that male rats showed decreased sIPSC frequency compared with females and that CB1R activation reduced DMS inhibitory transmission independent of sex. Altogether our results demonstrate that DMS CB1Rs regulate pavlovian devaluation sensitivity and DMS inhibitory synaptic transmission and suggest that basal sex differences in inhibitory synaptic transmission may underly sex differences in DMS function and behavioral flexibility.

Neuronal encoding of behaviors and instrumental learning in the dorsal striatum

The dorsal striatum is instrumental in regulating motor control and goal-directed behaviors. The classical description of the two output pathways of the dorsal striatum highlights their antagonistic control over actions. However, recent experimental evidence implicates both pathways and their coordinated activities during actions. In this review, we examine the different models proposed for striatal encoding of actions during self-paced behaviors and how they can account for evidence harvested during goal-directed behaviors. We also discuss how the activation of striatal ensembles can be reshaped and reorganized to support the formation of instrumental learning and behavioral flexibility. Future work integrating these considerations may resolve controversies regarding the control of actions by striatal networks.

Chronic Ethanol Exposure Produces Sex-Dependent Impairments in Value Computations in the Striatum

Value-based decision-making relies on the striatum, where neural plasticity can be altered by chronic ethanol (EtOH) exposure, but the effects of such plasticity on striatal neural dynamics during decision-making remain unclear. This study investigated the long-term impacts of EtOH on reward-driven decision-making and striatal neurocomputations in male and female rats using a dynamic probabilistic reversal learning task. Following a prolonged withdrawal period, EtOH-exposed male rats exhibited deficits in adaptability and exploratory behavior, with aberrant outcome-driven value updating that heightened preference for chosen action. These behavioral changes were linked to altered neural activity in the dorsomedial striatum (DMS), where EtOH increased outcome-related encoding and decreased choice-related encoding. In contrast, female rats showed minimal behavioral changes with distinct EtOH-evoked alterations of neural activity, revealing significant sex differences in the impact of chronic EtOH. Our findings underscore the impact of chronic EtOH exposure on adaptive decision-making, revealing enduring changes in neurocomputational processes in the striatum underlying cognitive deficits that differ by sex.

Cortico-striatal action control inherent of opponent cognitive-motivational styles

Turning on cue or stopping at a red light requires attending to such cues to select action sequences, or suppress action, in accordance with learned cue-associated action rules. Cortico-striatal projections are an essential part of the brain's attention-motor interface. Glutamate-sensing microelectrode arrays were used to measure glutamate transients in the dorsomedial striatum (DMS) of male and female rats walking a treadmill and executing cued turns and stops. Prelimbic-DMS projections were chemogenetically inhibited to determine their behavioral necessity and the cortico-striatal origin of cue-evoked glutamate transients. Furthermore, we investigated rats exhibiting preferably goal-directed (goal trackers, GTs) versus cue-driven attention (sign trackers, STs), to determine the impact of such cognitive-motivational biases on cortico-striatal control. GTs executed more cued turns and initiated such turns more slowly than STs. During turns, but not missed turns or cued stops, cue-evoked glutamate concentrations were higher in GTs than in STs. In STs, turn cue-locked glutamate concentrations frequently peaked twice or three times, contrasting with predominately single peaks in GTs. In GTs, but not STs, inhibition of prelimbic-DMS projections attenuated turn rates and turn cue-evoked glutamate concentrations and increased the number of turn cue-locked glutamate peaks. These findings indicate that turn cue-evoked glutamate release in GTs is tightly controlled by cortico-striatal neuronal activity. In contrast, in STs, glutamate release from DMS glutamatergic terminals may be regulated by other striatal circuitry, preferably mediating cued suppression of action and reward tracking. As cortico-striatal dysfunction has been hypothesized to contribute to a wide range of disorders, including complex movement control deficits in Parkinson's disease and compulsive drug taking, the demonstration of phenotypic contrasts in cortico-striatal control implies the presence of individual vulnerabilities for such disorders.

Significance Statement

Adaptive behavior involves the selection of behaviorally significant cues and the capacity of selected cues to control behavioral action. Neuronal projections from cortex to striatum are essential for such an integration of attentional with motor functions. Here we demonstrated that glutamate release from cortico-striatal projections primarily influences cued turns but not cued suppression of actions (cued stops). Cortico-striatal control of cued turning was especially powerful in rats which, as a psychological trait, preferably deploy goal-directed attention. Together, our findings demonstrate the role of cortico-striatal input in cued action selection, and they emphasize the experimental and biopsychological significance of investigating the brain's attentional-motor interface in the context of broader individual differences in cognitive-motivational styles.

Coding Dynamics of the Striatal Networks During Learning

The rat dorsomedial (DMS) and dorsolateral striatum (DLS), equivalent to caudate nucleus and putamen in primates, are required for goal-directed and habit behaviour, respectively. However, it is still unclear whether and how this functional dichotomy emerges in the course of learning. In this study, we investigated this issue by recording DMS and DLS single neuron activity in rats performing a continuous spatial alternation task, from the acquisition to optimized performance. We first applied a classical analytical approach to identify task-related activity based on the modifications of single neuron firing rate in relation to specific task events or maze trajectories. We then used an innovative approach based on Hawkes process to reconstruct a directed connectivity graph of simultaneously recorded neurons, that was used to decode animal behavior. This approach enabled us to better unravel the role of DMS and DLS neural networks across learning stages. We showed that DMS and DLS display different task-related activity throughout learning stages, and the proportion of coding neurons over time decreases in the DMS and increases in the DLS. Despite these major differences, the decoding power of both networks increases during learning. These results suggest that DMS and DLS neural networks gradually reorganize in different ways in order to progressively increase their control over the behavioral performance.

A large-scale c-Fos brain mapping study on extinction of cocaine-primed reinstatement

Individuals with cocaine addiction can experience many craving episodes and subsequent relapses, which represents the main obstacle to recovery. Craving is often favored when abstinent individuals ingest a small dose of cocaine, encounter cues associated with drug use or are exposed to stressors. Using a cocaine-primed reinstatement model in rat, we recently showed that cocaine-conditioned interoceptive cues can be extinguished with repeated cocaine priming in the absence of drug reinforcement, a phenomenon we called extinction of cocaine priming. Here, we applied a large-scale c-Fos brain mapping approach following extinction of cocaine priming in male rats to identify brain regions implicated in processing the conditioned interoceptive stimuli of cocaine priming. We found that cocaine-primed reinstatement is associated with increased c-Fos expression in key brain regions (e.g., dorsal and ventral striatum, several prefrontal areas and insular cortex), while its extinction mostly disengages them. Moreover, while reinstatement behavior was correlated with insular and accumbal activation, extinction of cocaine priming implicated parts of the ventral pallidum, the mediodorsal thalamus and the median raphe. These brain patterns of activation and inhibition suggest that after repeated priming, interoceptive signals lose their conditioned discriminative properties and that action-outcome associations systems are mobilized in search for new contingencies, a brain state that may predispose to rapid relapse.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward rewards. Dopamine (DA) neurons in the ventral tegmental area (VTA) and substantia nigra (SNc) are crucial for this process, via engagement of a reciprocally connected network with their striatal targets. Critically, it remains unknown how dopamine neuron activity itself engages dopamine signals throughout the striatum, across learning. Here, we investigated how optogenetic Pavlovian cue conditioning of VTA or SNc dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. Although VTA stimulation-evoked robust dopamine release in NAc core, shell, and DMS, predictive cues preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS. Despite the development of vigorous movement late in training, conditioned dopamine signals did not emerge in the DLS, even during Pavlovian conditioning with SNc dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show a broad dissociation in the fundamental prediction and reward-related information generated by VTA and SNc dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale adaptations across the striatal network emerge during learning to coordinate behavior.

Functional states of prelimbic and related circuits during the acquisition of a GO/noGO task in rats

GO/noGO tasks enable assessing decision-making processes and the ability to suppress a specific action according to the context. Here, rats had to discriminate between 2 visual stimuli (GO or noGO) shown on an iPad screen. The execution (for GO) or nonexecution (for noGO) of the selected action (to touch or not the visual display) were reinforced with food. The main goal was to record and to analyze local field potentials collected from cortical and subcortical structures when the visual stimuli were shown on the touch screen and during the subsequent activities. Rats were implanted with recording electrodes in the prelimbic cortex, primary motor cortex, nucleus accumbens septi, basolateral amygdala, dorsolateral and dorsomedial striatum, hippocampal CA1, and mediodorsal thalamic nucleus. Spectral analyses of the collected data demonstrate that the prelimbic cortex was selectively involved in the cognitive and motivational processing of the learning task but not in the execution of reward-directed behaviors. In addition, the other recorded structures presented specific tendencies to be involved in these 2 types of brain activity in response to the presentation of GO or noGO stimuli. Spectral analyses, spectrograms, and coherence between the recorded brain areas indicate their specific involvement in GO vs. noGO tasks.

A transdiagnostic and translational framework for delineating the neuronal mechanisms of compulsive exercise in anorexia nervosa

OBJECTIVE

The development of novel treatments for anorexia nervosa (AN) requires a detailed understanding of the biological underpinnings of specific, commonly occurring symptoms, including compulsive exercise. There is considerable bio-behavioral overlap between AN and obsessive-compulsive disorder (OCD), therefore it is plausible that similar mechanisms underlie compulsive behavior in both populations. While the association between these conditions is widely acknowledged, defining the shared mechanisms for compulsive behavior in AN and OCD requires a novel approach.

METHODS

We present an argument that a better understanding of the neurobiological mechanisms that underpin compulsive exercise in AN can be achieved in two critical ways. First, by applying a framework of the neuronal control of OCD to exercise behavior in AN, and second, by taking better advantage of the activity-based anorexia (ABA) rodent model to directly test this framework in the context of feeding pathology.

RESULTS

A cross-disciplinary approach that spans preclinical, neuroimaging, and clinical research as well as compulsive neurocircuitry and behavior can advance our understanding of when, why, and how compulsive exercise develops in the context of AN and provide targets for novel treatment strategies.

DISCUSSION

In this article, we (i) link the expression of compulsive behavior in AN and OCD via a transition between goal-directed and habitual behavior, (ii) present disrupted cortico-striatal circuitry as a key substrate for the development of compulsive behavior in both conditions, and (iii) highlight the utility of the ABA rodent model to better understand the mechanisms of compulsive behavior relevant to AN.

PUBLIC SIGNIFICANCE

Individuals with AN who exercise compulsively are at risk of worse health outcomes and have poorer responses to standard treatments. However, when, why, and how compulsive exercise develops in AN remains inadequately understood. Identifying whether the neural circuitry underlying compulsive behavior in OCD also controls hyperactivity in the activity-based anorexia model will aid in the development of novel eating disorder treatment strategies for this high-risk population.

Rethinking dopamine-guided action sequence learning

As opposed to those requiring a single action for reward acquisition, tasks necessitating action sequences demand that animals learn action elements and their sequential order and sustain the behaviour until the sequence is completed. With repeated learning, animals not only exhibit precise execution of these sequences but also demonstrate enhanced smoothness and efficiency. Previous research has demonstrated that midbrain dopamine and its major projection target, the striatum, play crucial roles in these processes. Recent studies have shown that dopamine from the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA) serve distinct functions in action sequence learning. The distinct contributions of dopamine also depend on the striatal subregions, namely the ventral, dorsomedial and dorsolateral striatum. Here, we have reviewed recent findings on the role of striatal dopamine in action sequence learning, with a focus on recent rodent studies.

Changes in dorsomedial striatum activity during expression of goal-directed vs. habit-like cue-induced cocaine seeking

A preclinical model of cue exposure therapy, cue extinction, reduces cue-induced cocaine seeking that is goal-directed but not habit-like. Goal-directed and habitual behaviors differentially rely on the dorsomedial striatum (DMS) and dorsolateral striatum (DLS), but the effects of cue extinction on dorsal striatal responses to cue-induced drug seeking are unknown. We used fiber photometry in rats trained to self-administer cocaine paired with an audiovisual cue to examine how dorsal striatal intracellular calcium and extracellular dopamine activity differs between goal-directed and habit-like cue-induced cocaine seeking and how it is impacted by cue extinction. After minimal fixed-ratio training, rats showed enhanced DMS and DLS calcium responses to cue-reinforced compared to unreinforced lever presses. After rats were trained on goal-promoting fixed ratio schedules or habit-promoting second-order schedules of reinforcement, different patterns of dorsal striatal calcium and dopamine responses to cue-reinforced lever presses emerged. Rats trained on habit-promoting second-order schedules showed reduced DMS calcium responses and enhanced DLS dopamine responses to cue-reinforced lever presses. Cue extinction reduced calcium responses during subsequent drug seeking in the DMS, but not in the DLS. Therefore, cue extinction may reduce goal-directed behavior through its effects on the DMS, whereas habit-like behavior and the DLS are unaffected.

Dopamine neurons drive spatiotemporally heterogeneous striatal dopamine signals during learning

Environmental cues, through Pavlovian learning, become conditioned stimuli that invigorate and guide animals toward acquisition of rewards. Dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SNC) are crucial for this process. Dopamine neurons are embedded in a reciprocally connected network with their striatal targets, the functional organization of which remains poorly understood. Here, we investigated how learning during optogenetic Pavlovian cue conditioning of VTA or SNC dopamine neurons directs cue-evoked behavior and shapes subregion-specific striatal dopamine dynamics. We used a fluorescent dopamine biosensor to monitor dopamine in the nucleus accumbens (NAc) core and shell, dorsomedial striatum (DMS), and dorsolateral striatum (DLS). We demonstrate spatially heterogeneous, learning-dependent dopamine changes across striatal regions. While VTA stimulation evoked robust dopamine release in NAc core, shell, and DMS, cues predictive of this activation preferentially recruited dopamine release in NAc core, starting early in training, and DMS, late in training. Corresponding negative prediction error signals, reflecting a violation in the expectation of dopamine neuron activation, only emerged in the NAc core and DMS, and not the shell. Despite development of vigorous movement late in training, conditioned dopamine signals did not similarly emerge in the DLS, even during Pavlovian conditioning with SNC dopamine neuron activation, which elicited robust DLS dopamine release. Together, our studies show broad dissociation in the fundamental prediction and reward-related information generated by different dopamine neuron populations and signaled by dopamine across the striatum. Further, they offer new insight into how larger-scale plasticity across the striatal network emerges during Pavlovian learning to coordinate behavior.

CBD treatment following early life seizures alters orbitofrontal-striatal signaling during adulthood

Obsessive compulsive disorder (OCD) is a comorbid condition of epilepsy and often adds to the burden of epilepsy. Both OCD and epilepsy are disorders of hyperexcitable circuits. Fronto-striatal circuit dysfunction is implicated in OCD. Prior work in our laboratory has shown that in rat pups following a series of flurothyl-induced early life seizures (ELS) exhibit frontal-lobe dysfunction along with alterations in electrographic temporal coordination between the orbitofrontal cortex (OFC) and dorsomedial striatum (DMS), circuits implicated in OCD. Here, we studied the effects of ELS in male and female rat pups on OCD-like behaviors as adults using the marble burying test (MBT). Because cannabidiol (CBD) is an effective antiseizure medication and has shown efficacy in the treatment of individuals with OCD, we also randomized rats to CBD or vehicle treatment following ELS to determine if CBD had any effect on OCD-like behaviors. While the flurothyl model of ELS did not induce OCD-like behaviors, as measured in the MBT, ELS did alter neural signaling in structures implicated in OCD and CBD had sex-dependent effects of temporal coordination in a way which suggests it may have a beneficial effect on epilepsy-related OCD.

Disuse-driven plasticity in the human thalamus and putamen

Motor adaptation in cortico-striato-thalamo-cortical loops has been studied mainly in animals using invasive electrophysiology. Here, we leverage functional neuroimaging in humans to study motor circuit plasticity in the human subcortex. We employed an experimental paradigm that combined two weeks of upper-extremity immobilization with daily resting-state and motor task fMRI before, during, and after the casting period. We previously showed that limb disuse leads to decreased functional connectivity (FC) of the contralateral somatomotor cortex (SM1) with the ipsilateral somatomotor cortex, increased FC with the cingulo-opercular network (CON) as well as the emergence of high amplitude, fMRI signal pulses localized in the contralateral SM1, supplementary motor area and the cerebellum. From our prior observations, it remains unclear whether the disuse plasticity affects the thalamus and striatum. We extended our analysis to include these subcortical regions and found that both exhibit strengthened cortical FC and spontaneous fMRI signal pulses induced by limb disuse. The dorsal posterior putamen and the central thalamus, mainly CM, VLP and VIM nuclei, showed disuse pulses and FC changes that lined up with fmri task activations from the Human connectome project motor system localizer, acquired before casting for each participant. Our findings provide a novel understanding of the role of the cortico-striato-thalamo-cortical loops in human motor plasticity and a potential link with the physiology of sleep regulation. Additionally, similarities with FC observation from Parkinson Disease (PD) questions a pathophysiological link with limb disuse.

Making and breaking habits: Revisiting the definitions and behavioral factors that influence habits in animals

Habits have garnered significant interest in studies of associative learning and maladaptive behavior. However, habit research has faced scrutiny and challenges related to the definitions and methods. Differences in the conceptualizations of habits between animal and human studies create difficulties for translational research. Here, we review the definitions and commonly used methods for studying habits in animals and humans and discuss potential alternative ways to assess habits, such as automaticity. To better understand habits, we then focus on the behavioral factors that have been shown to make or break habits in animals, as well as potential mechanisms underlying the influence of these factors. We discuss the evidence that habitual and goal-directed systems learn in parallel and that they seem to interact in competitive and cooperative manners. Finally, we draw parallels between habitual responding and compulsive drug seeking in animals to delineate the similarities and differences in these behaviors.

Nucleus accumbens and dorsal medial striatal dopamine and neural activity are essential for action sequence performance

Separable striatal circuits have unique functions in Pavlovian and instrumental behaviors but how these roles relate to performance of sequences of actions with and without associated cues are less clear. Here, we tested whether dopamine transmission and neural activity more generally in three striatal subdomains are necessary for performance of an action chain leading to reward delivery. Male and female Long-Evans rats were trained to press a series of three spatially distinct levers to receive reward. We assessed the contribution of neural activity or dopamine transmission within each striatal subdomain when progression through the action sequence was explicitly cued and in the absence of cues. Behavior in both task variations was substantially impacted following microinfusion of the dopamine antagonist, flupenthixol, into nucleus accumbens core (NAc) or dorsomedial striatum (DMS), with impairments in sequence timing and numbers of rewards earned after NAc flupenthixol. In contrast, after pharmacological inactivation to suppress overall activity, there was minimal impact on total rewards earned. Instead, inactivation of both NAc and DMS impaired sequence timing and led to sequence errors in the uncued, but not cued task. There was no impact of dopamine antagonism or reversible inactivation of dorsolateral striatum on either cued or uncued action sequence completion. These results highlight an essential contribution of NAc and DMS dopamine systems in motivational and performance aspects of chains of actions, whether cued or internally generated, as well as the impact of intact NAc and DMS function for correct sequence performance.

Changes in dorsomedial striatum activity mediate expression of goal-directed vs. habit-like cue-induced cocaine seeking

A preclinical model of cue exposure therapy, cue extinction, reduces cue-induced cocaine seeking when drug seeking is goal-directed but not habitual. Goal-directed and habitual behaviors differentially rely on the dorsomedial striatum (DMS) and dorsolateral striatum (DLS), but the effects of cue extinction on dorsal striatal responses to cue-induced drug seeking are unknown. We used fiber photometry to examine how dorsal striatal intracellular calcium and extracellular dopamine activity differs between goal-directed and habitual cue-induced cocaine seeking and how it is impacted by cue extinction. Rats trained to self-administer cocaine paired with an audiovisual cue on schedules of reinforcement that promote goal-directed or habitual cocaine seeking had different patterns of dorsal striatal calcium and dopamine responses to cue-reinforced lever presses. Cue extinction reduced calcium and dopamine responses during subsequent drug seeking in the DMS, but not in the DLS. Therefore, cue extinction may reduce goal-directed behavior through its effects on the DMS, whereas habitual behavior and the DLS are unaffected.

Revealing abrupt transitions from goal-directed to habitual behavior

A fundamental tenet of animal behavior is that decision-making involves multiple 'controllers.' Initially, behavior is goal-directed, driven by desired outcomes, shifting later to habitual control, where cues trigger actions independent of motivational state. Clark Hull's question from 1943 still resonates today: "Is this transition abrupt, or is it gradual and progressive?"1 Despite a century-long belief in gradual transitions, this question remains unanswered2,3 as current methods cannot disambiguate goal-directed versus habitual control in real-time. Here, we introduce a novel 'volitional engagement' approach, motivating animals by palatability rather than biological need. Offering less palatable water in the home cage4,5 reduced motivation to 'work' for plain water in an auditory discrimination task when compared to water-restricted animals. Using quantitative behavior and computational modeling6, we found that palatability-driven animals learned to discriminate as quickly as water-restricted animals but exhibited state-like fluctuations when responding to the reward-predicting cue-reflecting goal-directed behavior. These fluctuations spontaneously and abruptly ceased after thousands of trials, with animals now always responding to the reward-predicting cue. In line with habitual control, post-transition behavior displayed motor automaticity, decreased error sensitivity (assessed via pupillary responses), and insensitivity to outcome devaluation. Bilateral lesions of the habit-related dorsolateral striatum7 blocked transitions to habitual behavior. Thus, 'volitional engagement' reveals spontaneous and abrupt transitions from goal-directed to habitual behavior, suggesting the involvement of a higher-level process that arbitrates between the two.

Identification of Novel BDNF-Specific Corticostriatal Circuitries

Brain-derived neurotrophic factor (BDNF) is released from axon terminals originating in the cerebral cortex onto striatal neurons. Here, we characterized BDNF neurons in the corticostriatal circuitry. First, we used BDNF-Cre and Ribotag transgenic mouse lines to label BDNF-positive neurons in the cortex and detected BDNF expression in all the subregions of the prefrontal cortex (PFC). Next, we used a retrograde viral tracing strategy, in combination with BDNF-Cre knock-in mice, to map the cortical outputs of BDNF neurons in the dorsomedial and dorsolateral striatum (DMS and DLS, respectively). We found that BDNF-expressing neurons located in the medial prefrontal cortex (mPFC) project mainly to the DMS, and those located in the primary and secondary motor cortices (M1 and M2, respectively) and agranular insular cortex (AI) project mainly to the DLS. In contrast, BDNF-expressing orbitofrontal cortical (OFC) neurons differentially target the dorsal striatum (DS) depending on their mediolateral and rostrocaudal location. Specifically, the DMS is mainly innervated by the medial and ventral part of the orbitofrontal cortex (MO and VO, respectively), whereas the DLS receives projections specifically from the lateral part of the OFC (LO). Together, our study uncovers previously unknown BDNF corticostriatal circuitries. These findings could have important implications for the role of BDNF signaling in corticostriatal pathways.

Abstinence-Induced Nicotine Seeking Relays on a Persistent Hypoglutamatergic State within the Amygdalo-Striatal Neurocircuitry

Nicotine robustly sustains smoking behavior by acting as a primary reinforcer and by enhancing the incentive salience of the nicotine-associated stimuli. The motivational effects produced by environmental cues associated with nicotine delivery can progressively manifest during abstinence resulting in reinstatement of nicotine seeking. However, how the activity in reward neuronal circuits is transformed during abstinence-induced nicotine seeking is not yet fully understood. In here we used a contingent nicotine and saline control self-administration model to disentangle the contribution of cue-elicited seeking responding for nicotine after drug abstinence in male Wistar rats. Using ex vivo electrophysiological recordings and a network analysis approach, we defined temporal and brain-region specific amygdalo-striatal glutamatergic alterations that occur during nicotine abstinence. The results from this study provide critical evidence indicating a persistent hypoglutamatergic state within the amygdalo-striatal neurocircuitry over protracted nicotine abstinence. During abstinence-induced nicotine seeking, electrophysiological recordings showed progressive neuroadaptations in dorsal and ventral striatum already at 14-d abstinence while neuroadaptations in subregions of the amygdala emerged only after 28-d abstinence. The observed neuroadaptations pointed to a brain network involving the amygdala and the dorsolateral striatum (DLS) to be implied in cue-induced reinstatement of nicotine seeking. Together these data suggest long-lasting neuroadaptations that might reflect neuroplastic changes responsible to abstinence-induced nicotine craving. Neurophysiological transformations were detected within a time window that allows therapeutic intervention advancing clinical development of preventive strategies in nicotine addiction.

Lack of action monitoring as a prerequisite for habitual and chunked behavior: Behavioral and neural correlates

We previously reported the rapid development of habitual behavior in a discrete-trials instrumental task in which lever insertion and retraction act as reward-predictive cues delineating sequence execution. Here we asked whether lever cues or performance variables reflective of skill and automaticity might account for habitual behavior in male rats. Behavior in the discrete-trials habit-promoting task was compared with two task variants lacking the sequence-delineating cues of lever extension and retraction. We find that behavior is under goal-directed control in absence of sequence-delineating cues but not in their presence, and that skilled performance does not predict goal-directed vs. habitual behavior. Neural activity recordings revealed an engagement of dorsolateral striatum and a disengagement of dorsomedial striatum during the sequence execution of the habit-promoting task, specifically. Together, these results indicate that sequence delineation cues promote habit and differential engagement of striatal subregions during instrumental responding, a pattern that may reflect cue-elicited behavioral chunking.

Male DAT Val559 Mice Exhibit Compulsive Behavior under Devalued Reward Conditions Accompanied by Cellular and Pharmacological Changes

Identified across multiple psychiatric disorders, the dopamine (DA) transporter (DAT) Ala559Val substitution triggers non-vesicular, anomalous DA efflux (ADE), perturbing DA neurotransmission and behavior. We have shown that DAT Val559 mice display a waiting impulsivity and changes in cognitive performance associated with enhanced reward motivation. Here, utilizing a within-subject, lever-pressing paradigm designed to bias the formation of goal-directed or habitual behavior, we demonstrate that DAT Val559 mice modulate their nose poke behavior appropriately to match context, but demonstrate a perseverative checking behavior. Although DAT Val559 mice display no issues with the cognitive flexibility required to acquire and re-learn a visual pairwise discrimination task, devaluation of reward evoked habitual reward seeking in DAT Val559 mutants in operant tasks regardless of reinforcement schedule. The direct DA agonist apomorphine also elicits locomotor stereotypies in DAT Val559, but not WT mice. Our observation that dendritic spine density is increased in the dorsal medial striatum (DMS) of DAT Val559 mice speaks to an imbalance in striatal circuitry that might underlie the propensity of DAT Val559 mutants to exhibit compulsive behaviors when reward is devalued. Thus, DAT Val559 mice represent a model for dissection of how altered DA signaling perturbs circuits that normally balance habitual and goal-directed behaviors.

Decreased Dorsomedial Striatum Direct Pathway Neuronal Activity Is Required for Learned Motor Coordination

It has been suggested that the dorsomedial striatum (DMS) is engaged in the early stages of motor learning for goal-directed actions, whereas at later stages, control is transferred to the dorsolateral striatum (DLS), a process that enables learned motor actions to become a skill or habit. It is not known whether these striatal regions are simultaneously active while the expertise is acquired. To address this question, we developed a mouse "Treadmill Training Task" that tracks changes in mouse locomotor coordination during running practice and simultaneously provides a means to measure local neuronal activity using photometry. To measure change in motor coordination over treadmill practice sessions, we used DeepLabCut (DLC) and custom-built code to analyze body position and paw movements. By evaluating improvements in motor coordination during training with simultaneous neuronal calcium activity in the striatum, we found that DMS direct pathway neurons exhibited decreased activity as the mouse gained proficiency at running. In contrast, direct pathway activity in the DLS was similar throughout training. Pharmacological blockade of D1 dopamine receptors in these subregions during task performance demonstrated that dopamine neurotransmission in the direct pathway activity is necessary for efficient motor coordination learning. These results provide new tools to measure changes in fine motor skills with simultaneous recordings of brain activity and reveal fundamental features of the neuronal substrates of motor learning.

Presynaptic supervision of cortical spine dynamics in motor learning

In mammalian neocortex, learning triggers the formation and turnover of new postsynaptic spines on pyramidal cell dendrites. However, the biological principles of spine reorganization during learning remain elusive because the identity of their presynaptic neuronal partners is unknown. Here, we show that two presynaptic neural circuits supervise distinct programs of spine dynamics to execute learning. We imaged spine dynamics in motor cortex during learning and performed post hoc identification of their afferent presynaptic neurons. New spines that appeared during learning formed small transient contacts with corticocortical neurons that were eliminated on skill acquisition. In contrast, persistent spines with axons from thalamic neurons were formed and enlarged. These results suggest that pyramidal cell dendrites in motor cortex use a neural circuit division of labor during skill learning, with dynamic teaching contacts from top-down intracortical axons followed by synaptic memory formation driven by thalamic axons. Dual spine supervision may govern diverse skill learning in the neocortex.

Spatiotemporal reorganization of corticostriatal networks encodes motor skill learning

Motor skill learning requires the activity of the dorsal striatum, with a differential global implication of the dorsomedial and dorsolateral territories. We investigate here whether and how specific striatal neurons encode the acquisition and consolidation of a motor skill. Using ex vivo two-photon calcium imaging after rotarod training, we report that highly active (HA) striatal populations arise from distinct spatiotemporal reorganization in the dorsomedial (DMS) and dorsolateral (DLS) striatum networks and are correlated with learning performance. The DMS overall activity decreases in early training, with few and sparsely distributed HA cells, while the DLS shows a progressive and long-lasting formation of HA cell clusters. These reorganizations result from reinforcement of synaptic connections to the DMS and anatomical rearrangements to the DLS. Targeted silencing of DMS or DLS HA cells with the cFos-TRAP strategy strongly impairs individual performance. Our data reveal that discrete domains of striatal populations encode acquisition and long-lasting retention of a motor skill.

Portable Neuroimaging-Guided Noninvasive Brain Stimulation of the Cortico-Cerebello-Thalamo-Cortical Loop—Hypothesis and Theory in Cannabis Use Disorder

Background: Maladaptive neuroplasticity-related learned response in substance use disorder (SUD) can be ameliorated using noninvasive brain stimulation (NIBS); however, inter-individual variability needs to be addressed for clinical translation. Objective: Our first objective was to develop a hypothesis for NIBS for learned response in SUD based on a competing neurobehavioral decision systems model. The next objective was to develop the theory by conducting a computational simulation of NIBS of the cortico-cerebello-thalamo-cortical (CCTC) loop in cannabis use disorder (CUD)-related dysfunctional “cue-reactivity”—a construct closely related to “craving”—that is a core symptom. Our third objective was to test the feasibility of a neuroimaging-guided rational NIBS approach in healthy humans. Methods: “Cue-reactivity” can be measured using behavioral paradigms and portable neuroimaging, including functional near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) metrics of sensorimotor gating. Therefore, we conducted a computational simulation of NIBS, including transcranial direct current stimulation (tDCS) and transcranial alternating current stimulation (tACS) of the cerebellar cortex and deep cerebellar nuclei (DCN) of the CCTC loop for its postulated effects on fNIRS and EEG metrics. We also developed a rational neuroimaging-guided NIBS approach for the cerebellar lobule (VII) and prefrontal cortex based on a healthy human study. Results: Simulation of cerebellar tDCS induced gamma oscillations in the cerebral cortex, while transcranial temporal interference stimulation induced a gamma-to-beta frequency shift. A preliminary healthy human study (N = 10) found that 2 mA cerebellar tDCS evoked similar oxyhemoglobin (HbO) response in the range of 5 × 10⁻⁶ M across the cerebellum and PFC brain regions (α = 0.01); however, infra-slow (0.01–0.10 Hz) prefrontal cortex HbO-driven phase–amplitude-coupled (PAC; 4 Hz, ±2 mA (max)) cerebellar tACS evoked HbO levels in the range of 10⁻⁷ M that were statistically different (α = 0.01) across these brain regions. Conclusion: Our healthy human study showed the feasibility of fNIRS of cerebellum and PFC and closed-loop fNIRS-driven ctACS at 4 Hz, which may facilitate cerebellar cognitive function via the frontoparietal network. Future work needs to combine fNIRS with EEG for multi-modal imaging for closed-loop NIBS during operant conditioning.

Striatum expresses region-specific plasticity consistent with distinct memory abilities

The striatum mediates two learning modalities: goal-directed behavior in dorsomedial (DMS) and habits in dorsolateral (DLS) striata. The synaptic bases of these learnings are still elusive. Indeed, while ample research has described DLS plasticity, little remains known about DMS plasticity and its involvement in procedural learning. Here, we find symmetric and asymmetric anti-Hebbian spike-timing-dependent plasticity (STDP) in DMS and DLS, respectively, with opposite plasticity dominance upon increasing corticostriatal activity. During motor-skill learning, plasticity is engaged in DMS and striatonigral DLS neurons only during early learning stages, whereas striatopallidal DLS neurons are mobilized only during late phases. With a mathematical modeling approach, we find that symmetric anti-Hebbian STDP favors memory flexibility, while asymmetric anti-Hebbian STDP favors memory maintenance, consistent with memory processes at play in procedural learning.

Opposing Roles of the Dorsolateral and Dorsomedial Striatum in the Acquisition of Skilled Action Sequencing in Rats

The shift in control from dorsomedial to dorsolateral striatum during skill and habit formation has been well established, but whether striatal subregions orchestrate this shift cooperatively or competitively remains unclear. Cortical inputs have also been implicated in the shift toward automaticity, but it is unknown whether they mirror their downstream striatal targets across this transition. We addressed these questions using a five step heterogeneous action sequencing task in male rats that is optimally performed by automated chains of actions. By optimizing automatic habitual responding, we discovered that loss of function in the dorsomedial striatum accelerated sequence acquisition. In contrast, loss of function in the dorsolateral striatum impeded acquisition of sequencing, demonstrating functional opposition within the striatum. Unexpectedly, the mPFC was not involved; however, the lateral orbitofrontal cortex was critical. These results shift current theories about striatal control of behavior to a model of competitive opposition, where the dorsomedial striatum interferes with the development of dorsolateral-striatum dependent behavior.SIGNIFICANCE STATEMENT We provide the most direct evidence to date that the dorsomedial and dorsolateral striatum compete for control in the acquisition of habitual action sequences. The dorsolateral striatum was critical for sequencing behavior, but loss of dorsomedial striatum function enhanced acquisition. In addition, we found that the mPFC was not required for the formation of automated actions. Using a task that optimizes habitual responding, we demonstrate that the arbitration of dorsomedial and dorsolateral control is not modulated by medial prefrontal cortical activity. However, we find evidence for the role of the lateral orbitofrontal cortex in action sequencing. These results have implications for our understanding of how habits and skills form.

Distinct roles for motor cortical and thalamic inputs to striatum during motor skill learning and execution

The acquisition and execution of motor skills are mediated by a distributed motor network, spanning cortical and subcortical brain areas. The sensorimotor striatum is an important cog in this network, yet the roles of its two main inputs, from motor cortex and thalamus, remain largely unknown. To address this, we silenced the inputs in rats trained on a task that results in highly stereotyped and idiosyncratic movement patterns. While striatal-projecting motor cortex neurons were critical for learning these skills, silencing this pathway after learning had no effect on performance. In contrast, silencing striatal-projecting thalamus neurons disrupted the execution of the learned skills, causing rats to revert to species-typical pressing behaviors and preventing them from relearning the task. These results show distinct roles for motor cortex and thalamus in the learning and execution of motor skills and suggest that their interaction in the striatum underlies experience-dependent changes in subcortical motor circuits.

Dorsal striatum coding for the timely execution of action sequences

The automatic initiation of actions can be highly functional. But occasionally these actions cannot be withheld and are released at inappropriate times, impulsively. Striatal activity has been shown to participate in the timing of action sequence initiation and it has been linked to impulsivity. Using a self-initiated task, we trained adult male rats to withhold a rewarded action sequence until a waiting time interval has elapsed. By analyzing neuronal activity we show that the striatal response preceding the initiation of the learned sequence is strongly modulated by the time subjects wait before eliciting the sequence. Interestingly, the modulation is steeper in adolescent rats, which show a strong prevalence of impulsive responses compared to adults. We hypothesize this anticipatory striatal activity reflects the animals’ subjective reward expectation, based on the elapsed waiting time, while the steeper waiting modulation in adolescence reflects age-related differences in temporal discounting, internal urgency states, or explore–exploit balance.

Deconstructing and reconstructing behaviour relevant to mental health disorders: The benefits of a psychological approach, with a focus on addiction

RUTHERFORD, L.G. and Milton, A.L. Deconstructing and reconstructing behaviour relevant to mental health disorders: what can psychology offer? NEUROSCI BIOBEHAV REV XX(X)XXX-XXX, 2021. - Current treatments for mental health disorders are successful only for some patients, and there is an unmet clinical need for new treatment development. One challenge for treatment development has been how best to model complex human conditions in animals, where mechanism can be more readily studied with a range of neuroscientific techniques. We suggest that an approach to modelling based on associative animal learning theory provides a good framework for deconstructing complex mental health disorders such that they can be studied in animals. These individual simple models can subsequently be used in combination to 'reconstruct' a more complex model of the mental health disorder of interest. Using examples primarily from the field of drug addiction, we explore the 'psychological approach' and suggest that in addition to facilitating translation and backtranslation of tasks between animal models and patients, it is also highly concordant with the concept of triangulation.

Overexpressing Histone Deacetylase 5 in Rat Dorsal Striatum Alters Reward-Guided Decision-Making and Associated Neural Encoding

Accumulating evidence in the past decade implicates histone-modifying enzymes, such as class I histone deacetylases (HDACs), in learning and memory and, recently, habit formation. However, it is unclear whether HDACs play roles in complex cognitive function. To address this issue, we examined the role of dorsal striatal HDAC5, a class II HDAC, in reward-guided decision-making and associated neural encoding in rats. We first injected adeno-associated virus to overexpress a nuclear-localized HDAC5 in dorsal striatum (DS). We then recorded neural correlates from dorsolateral striatum (DLS) as rats performed two reward-guided choice tasks, in which we manipulated either the size of or delay to reward. During these tasks, rats first learned which of two options led to the better reward and then reversed those contingencies in a second block of trials. We found that rats with HDAC5 overexpression in DS responded faster and chose higher value reward more often during the first block of trials but were less able to reverse those contingencies in the second block of trials. At the neural level, HDAC5 overexpression in DS elevated and reduced the number of cells in DLS that increased firing to stimuli and reward, respectively, and shifted encoding toward cues that predicted more immediate reward. These results suggest that the HDAC5 overexpression in DS contributes to inflexible decision-making, demonstrating a role of histone-modifying enzymes in complex cognitive function.SIGNIFICANCE STATEMENT HDACs are important for learning and habit formation. Here, we expanded on these functions and found that overexpression of HDAC5 produced faster and more automatic behavior, and related changes in dorsolateral striatal neural firing in rats performing a value-based decision-making task. These results implicate HDAC5 as a potential therapeutic target for psychiatric conditions that impair decision-making and executive function.

The learning of prospective and retrospective cognitive maps within neural circuits

Brain circuits are thought to form a "cognitive map" to process and store statistical relationships in the environment. A cognitive map is commonly defined as a mental representation that describes environmental states (i.e., variables or events) and the relationship between these states. This process is commonly conceptualized as a prospective process, as it is based on the relationships between states in chronological order (e.g., does reward follow a given state?). In this perspective, we expand this concept on the basis of recent findings to postulate that in addition to a prospective map, the brain forms and uses a retrospective cognitive map (e.g., does a given state precede reward?). In doing so, we demonstrate that many neural signals and behaviors (e.g., habits) that seem inflexible and non-cognitive can result from retrospective cognitive maps. Together, we present a significant conceptual reframing of the neurobiological study of associative learning, memory, and decision making.

Dorsomedial Striatal Activity Tracks Completion of Behavioral Sequences in Rats

For proper execution of goal-directed behaviors, individuals require both a general representation of the goal and an ability to monitor their own progress toward that goal. Here, we examine how dorsomedial striatum (DMS), a region pivotal for forming associations among stimuli, actions, and outcomes, encodes the execution of goal-directed action sequences that require self-monitoring of behavior. We trained rats to complete a sequence of at least five consecutive lever presses (without visiting the reward port) to obtain a reward and recorded the activity of individual cells in DMS while rats performed the task. We found that the pattern of DMS activity gradually changed during the execution of the sequence, permitting accurate decoding of sequence progress from neural activity at a population level. Moreover, this sequence-related activity was blunted on trials where rats did not complete a sufficient number of presses. Overall, these data suggest a link between DMS activity and the execution of behavioral sequences that require monitoring of ongoing behavior.

Direct and indirect pathway neurons in ventrolateral striatum differentially regulate licking movement and nigral responses

Drinking behavior in rodents is characterized by stereotyped, rhythmic licking movement, which is regulated by the basal ganglia. It is unclear how direct and indirect pathways control the lick bout and individual spout contact. We find that inactivating D1 and D2 receptor-expressing medium spiny neurons (MSNs) in the ventrolateral striatum (VLS) oppositely alters the number of licks in a bout. D1- and D2-MSNs exhibit different patterns of lick-sequence-related activity and different phases of oscillation time-locked to the lick cycle. On the timescale of a lick cycle, transient inactivation of D1-MSNs during tongue protrusion reduces spout contact probability, whereas transiently inactivating D2-MSNs has no effect. On the timescale of a lick bout, inactivation of D1-MSNs (D2-MSNs) causes rate increase (decrease) in a subset of basal ganglia output neurons that decrease firing during licking. Our results reveal the distinct roles of D1- and D2-MSNs in regulating licking at both coarse and fine timescales.

Parallel and hierarchical neural mechanisms for adaptive and predictive behavioral control

Our brain can be recognized as a network of largely hierarchically organized neural circuits that operate to control specific functions, but when acting in parallel, enable the performance of complex and simultaneous behaviors. Indeed, many of our daily actions require concurrent information processing in sensorimotor, associative, and limbic circuits that are dynamically and hierarchically modulated by sensory information and previous learning. This organization of information processing in biological organisms has served as a major inspiration for artificial intelligence and has helped to create in silico systems capable of matching or even outperforming humans in several specific tasks, including visual recognition and strategy-based games. However, the development of human-like robots that are able to move as quickly as humans and respond flexibly in various situations remains a major challenge and indicates an area where further use of parallel and hierarchical architectures may hold promise. In this article we review several important neural and behavioral mechanisms organizing hierarchical and predictive processing for the acquisition and realization of flexible behavioral control. Then, inspired by the organizational features of brain circuits, we introduce a multi-timescale parallel and hierarchical learning framework for the realization of versatile and agile movement in humanoid robots.

Dorsolateral Striatal Task-initiation Bursts Represent Past Experiences More than Future Action Plans

The dorsolateral striatum (DLS) is involved in learning and executing procedural actions. Cell ensembles in the DLS, but not the dorsomedial striatum (DMS), exhibit a burst of firing at the start of a well-learned action sequence ("task-bracketing"). However, it is currently unclear what information is contained in these bursts. Some theories suggest that these bursts should represent the procedural action sequence itself (that they should be about future action chains), whereas others suggest that they should contain representations of the current state of the world, taking into account primarily past information. In addition, the DLS local field potential shows transient bursts of power in the 50 Hz range (γ50) around the time a learned action sequence is initiated. However, it is currently unknown how bursts of activity in DLS cell ensembles and bursts of γ50 power in the DLS local field potential are related to each other. We found that DLS bursts at lap initiation in rats represented recently experienced reward locations more than future procedural actions, indicating that task-initiation DLS bursts contain primarily retrospective, rather than prospective, information to guide procedural actions. Furthermore, representations of past reward locations increased during periods of increased γ50 power in the DLS. There was no evidence of task-initiation bursts, increased γ50 power, or retrospective reward location information in the neighboring dorsomedial striatum. These data support a role for the DLS in model-free theories of procedural decision-making over planned action-chain theories, suggesting that procedural actions derive from representations of the current and recent past.SIGNIFICANCE STATEMENT While it is well-established that the dorsolateral striatum (DLS) plays a critical role in procedural decision-making, open questions remain about the kinds of representations contained in DLS ensemble activity that guide procedural actions. We found that DLS, but not DMS, cell ensembles contained nonlocal representations of past reward locations that appear moments before task-initiation DLS bursts. These retrospective representations were temporally linked to a rise in γ50 power that also preceded the characteristic DLS burst at task-initiation. These results support models of procedural decision-making based on associations between available actions and the current state of the world over models based on planning over action-chains.

The basal ganglia control the detailed kinematics of learned motor skills

The basal ganglia are known to influence action selection and modulation of movement vigor, but whether and how they contribute to specifying the kinematics of learned motor skills is not understood. Here, we probe this question by recording and manipulating basal ganglia activity in rats trained to generate complex task-specific movement patterns with rich kinematic structure. We find that the sensorimotor arm of the basal ganglia circuit is crucial for generating the detailed movement patterns underlying the acquired motor skills. Furthermore, the neural representations in the striatum, and the control function they subserve, do not depend on inputs from the motor cortex. Taken together, these results extend our understanding of the basal ganglia by showing that they can specify and control the fine-grained details of learned motor skills through their interactions with lower-level motor circuits.

Interpreting the role of the striatum during multiple phases of motor learning

The synaptic pathways in the striatum are central to basal ganglia functions including motor control, learning and organization, action selection, acquisition of motor skills, cognitive function, and emotion. Here, we review the role of the striatum and its connections in motor learning and performance. The development of new techniques to record neuronal activity and animal models of motor disorders using neurotoxin, pharmacological, and genetic manipulations are revealing pathways that underlie motor performance and motor learning, as well as how they are altered by pathophysiological mechanisms. We discuss approaches that can be used to analyze complex motor skills, particularly in rodents, and identify specific questions central to understanding how striatal circuits mediate motor learning.

Probing the decision-making mechanisms underlying choice between drug and nondrug rewards in rats

Delineating the decision-making mechanisms underlying choice between drug and nondrug rewards remains a challenge. This study adopts an original approach to probe these mechanisms by comparing response latencies during sampling versus choice trials. While lengthening of latencies during choice is predicted in a deliberative choice model (DCM), the race-like response competition mechanism postulated by the Sequential choice model (SCM) predicts a shortening of latencies during choice compared to sampling. Here, we tested these predictions by conducting a retrospective analysis of cocaine-versus-saccharin choice experiments conducted in our laboratory. We found that rats engage deliberative decision-making mechanisms after limited training, but adopt a SCM-like response selection mechanism after more extended training, while their behavior is presumably habitual. Thus, the DCM and SCM may not be general models of choice, as initially formulated, but could be dynamically engaged to control choice behavior across early and extended training.

Feasibility of combining functional near-infrared spectroscopy with electroencephalography to identify chronic stroke responders to cerebellar transcranial direct current stimulation-a computational modeling and portable neuroimaging methodological study

Feasibility of portable neuroimaging of cerebellar transcranial direct current stimulation (ctDCS) effects on the cerebral cortex has not been investigated vis-à-vis cerebellar lobular electric field strength. We studied functional near-infrared spectroscopy (fNIRS) in conjunction with electroencephalography (EEG) to measure changes in the brain activation at the prefrontal cortex (PFC) and the sensorimotor cortex (SMC) following ctDCS as well as virtual reality-based balance training (VBaT) before and after ctDCS treatment in 12 hemiparetic chronic stroke survivors. We performed general linear modeling (GLM) that putatively associated the lobular electric field strength with the changes in the fNIRS-EEG measures at the ipsilesional and contra-lesional PFC and SMC. Here, fNIRS-EEG measures were found in the latent space from canonical correlation analysis (CCA) between the changes in total hemoglobin (tHb) concentrations (0.01-0.07Hz and 0.07-0.13Hz bands) and log10-transformed EEG bandpower within 1-45 Hz where significant (Wilks' lambda>0.95) canonical correlations were found only for the 0.07-0.13-Hz band. Also, the first principal component (97.5% variance accounted for) of the mean lobular electric field strength was a good predictor of the latent variables of oxy-hemoglobin (O2Hb) concentrations and log10-transformed EEG bandpower. GLM also provided insights into non-responders to ctDCS who also performed poorly in the VBaT due to ideomotor apraxia. Future studies should investigate fNIRS-EEG joint-imaging in a larger cohort to identify non-responders based on GLM fitting to the fNIRS-EEG data.

Habit, choice, and addiction

Addiction was suggested to emerge from the progressive dominance of habits over goal-directed behaviors. However, it is generally assumed that habits do not persist in choice settings. Therefore, it is unclear how drug habits may persist in real-world scenarios where this factor predominates. Here, we discuss the poor translational validity of the habit construct, which impedes our ability to determine its role in addiction. New evidence of habitual behavior in a drug choice setting are then described and discussed. Interestingly, habitual preference did not promote drug choice but instead favored abstinence. Here, we propose several clues to reconcile these unexpected results with the habit theory of addiction, and we highlight the need in experimental research to face the complexity of drug addicts' decision-making environments by investigating drug habits in the context of choice and in the presence of cues. On a theoretical level, we need to consider more complex frameworks, taking into account continuous interactions between goal-directed and habitual systems, and alternative decision-making models more representative of real-world conditions.

Concurrent LI-rTMS induces changes in c-Fos expression but not behavior during a progressive ratio task with adult ephrin-A2A5-/- mice

Changes within the dopaminergic system induced by repetitive transcranial magnetic stimulation (rTMS) may contribute to its therapeutic effects; however, dopamine-related behavioral effects of rTMS have not been widely investigated. We recently showed that ephrin-A2A5^-/- mice completed significantly fewer trials in a visual task than wildtype mice, and that concurrent low-intensity (LI-) rTMS during the task could partially rescue the abnormal behavior [Poh et al. 2018, eNeuro, vol. 5]. Here, we investigated whether the behavioral differences in ephrin-A2A5^-/- mice are due to abnormal motivation, primarily a dopamine-modulated behavior, and whether LI-rTMS would increase motivation. Ephrin-A2A5^-/- and wildtype mice underwent 14 daily sessions of progressive ratio (PR) tasks and received either sham or LI-rTMS during the first 10 min. Ephrin-A2A5^-/- mice responded more than wildtype comparisons, and LI-rTMS did not influence task performance for either strain. Therefore concurrent stimulation does not influence motivation in a PR task. However, ephrin-A2A5^-/- mice did have abnormal performance in the PR tasks after a change in the PR schedule which suggests perseverative behavior. We stained for c-Fos in the prelimbic area (PrL), ventral tegmental area and nucleus accumbens (NAc) core and shell to examine neuronal activity from the final PR session. Sham ephrin-A2A5^-/- mice had lower c-Fos expression in the PrL and NAc vs. wildtype mice. Ephrin-A2A5^-/- mice that received LI-rTMS showed c-Fos expression closer to wildtype levels in the NAc. Combined with high PR performance, ephrin-A2A5^-/- mice show an abnormal shift to habitual responding and LI-rTMS may attenuate this shift.

The Dorsal Striatum Energizes Motor Routines

The dorsal striatum (dS) has been implicated in storing procedural memories and controlling movement kinematics. Since procedural memories are expressed through movements, the exact nature of the dS function has proven difficult to delineate. Here, we challenged rats in complementary locomotion-based tasks designed to alleviate this confound. Surprisingly, dS lesions did not impair the rats' ability to remember the procedure for the successful completion of motor routines. However, the speed and initiation of the reward-oriented phase of the routines were irreversibly altered by the dS lesion. Further behavioral analyses, combined with modeling in the optimal control framework, indicated that these kinematic alterations were well explained by an increased sensitivity to effort. Our work provides evidence supporting a primary role of the dS in modulating the kinematics of reward-oriented actions, a function that may be related to the optimization of the energetic costs of moving.

Dorsolateral striatum engagement during reversal learning

Most experimental preparations demonstrate a role for dorsolateral striatum (DLS) in stimulus-response, but not outcome-based, learning. Here, we assessed DLS involvement in a touchscreen-based reversal task requiring mice to update choice following a change in stimulus-reward contingencies. In vivo single-unit recordings in the DLS showed reversal produced a population-level shift from excited to inhibited neuronal activity prior to choices being made. The larger the shift, the faster mice reversed. Furthermore, optogenetic photosilencing DLS neurons during choice increased early reversal errors. These findings suggest dynamic DLS engagement may facilitate reversal, possibly by signaling a change in contingencies to other striatal and cortical regions.