Habit learning in hierarchical cortex-basal ganglia loops

Distinct dopaminergic spike-timing-dependent plasticity rules are suited to different functional roles

Various mathematical models have been formulated to describe the changes in synaptic strengths resulting from spike-timing-dependent plasticity (STDP). A subset of these models include a third factor, dopamine, which interacts with the timing of pre- and postsynaptic spiking to contribute to plasticity at specific synapses, notably those from cortex to striatum at the input layer of the basal ganglia. Theoretical work to analyze these plasticity models has largely focused on abstract issues, such as the conditions under which they may promote synchronization and the weight distributions induced by inputs with simple correlation structures, rather than on scenarios associated with specific tasks, and has generally not considered dopamine-dependent forms of STDP. In this paper, we analyze, mathematically and with simulations, three forms of dopamine-modulated STDP in three scenarios that are relevant to corticostriatal synapses. Two of the models considered comprise previously proposed STDP rules with modifications to incorporate dopamine, while the third is a corticostriatal dopamine-dependent STDP rule adapted from a similar one already in the literature. We test the ability of each of the three models to maintain its weights in the face of noise and to complete simple reward prediction and action selection tasks, studying the learned weight distributions and corresponding task performance in each setting. Interestingly, we find that each of the three plasticity rules is well suited to a subset of the scenarios studied but falls short in others. These results show that different tasks may require different forms of synaptic plasticity, yielding the prediction that the precise form of the STDP mechanism may vary across regions of the striatum, and other brain areas impacted by dopamine, that are involved in distinct computational functions.

The contribution of the basal ganglia and cerebellum to motor learning: A neuro-computational approach

Motor learning involves a widespread brain network including the basal ganglia, cerebellum, motor cortex, and brainstem. Despite its importance, little is known about how this network learns motor tasks and which role different parts of this network take. We designed a systems-level computational model of motor learning, including a cortex-basal ganglia motor loop and the cerebellum that both determine the response of central pattern generators in the brainstem. First, we demonstrate its ability to learn arm movements toward different motor goals. Second, we test the model in a motor adaptation task with cognitive control, where the model replicates human data. We conclude that the cortex-basal ganglia loop learns via a novelty-based motor prediction error to determine concrete actions given a desired outcome, and that the cerebellum minimizes the remaining aiming error.

Distinct cortico-striatal compartments drive competition between adaptive and automatized behavior

Cortical and basal ganglia circuits play a crucial role in the formation of goal-directed and habitual behaviors. In this study, we investigate the cortico-striatal circuitry involved in learning and the role of this circuitry in the emergence of inflexible behaviors such as those observed in addiction. Specifically, we develop a computational model of cortico-striatal interactions that performs concurrent goal-directed and habit learning. The model accomplishes this by distinguishing learning processes in the dorsomedial striatum (DMS) that rely on reward prediction error signals as distinct from the dorsolateral striatum (DLS) where learning is supported by salience signals. These striatal subregions each operate on unique cortical input: the DMS receives input from the prefrontal cortex (PFC) which represents outcomes, and the DLS receives input from the premotor cortex which determines action selection. Following an initial learning of a two-alternative forced choice task, we subjected the model to reversal learning, reward devaluation, and learning a punished outcome. Behavior driven by stimulus-response associations in the DLS resisted goal-directed learning of new reward feedback rules despite devaluation or punishment, indicating the expression of habit. We repeated these simulations after the impairment of executive control, which was implemented as poor outcome representation in the PFC. The degraded executive control reduced the efficacy of goal-directed learning, and stimulus-response associations in the DLS were even more resistant to the learning of new reward feedback rules. In summary, this model describes how circuits of the dorsal striatum are dynamically engaged to control behavior and how the impairment of executive control by the PFC enhances inflexible behavior.

Enhanced habit formation in Tourette patients explained by shortcut modulation in a hierarchical cortico-basal ganglia model

Devaluation protocols reveal that Tourette patients show an increased propensity to habitual behaviors as they continue to respond to devalued outcomes in a cognitive stimulus-response-outcome association task. We use a neuro-computational model of hierarchically organized cortico-basal ganglia-thalamo-cortical loops to shed more light on habit formation and its alteration in Tourette patients. In our model, habitual behavior emerges from cortico-thalamic shortcut connections, where enhanced habit formation can be linked to faster plasticity in the shortcut or to a stronger feedback from the shortcut to the basal ganglia. We explore two major hypotheses of Tourette pathophysiology-local striatal disinhibition and increased dopaminergic modulation of striatal medium spiny neurons-as causes for altered shortcut activation. Both model changes altered shortcut functioning and resulted in higher rates of responses towards devalued outcomes, similar to what is observed in Tourette patients. We recommend future experimental neuroscientific studies to locate shortcuts between cortico-basal ganglia-thalamo-cortical loops in the human brain and study their potential role in health and disease.

Astrocyte-neuron interaction in the dorsal striatum-pallidal circuits and alcohol-seeking behaviors

In the striatum, two main types of GABAergic medium spiny neurons (MSNs), denoted striatonigral (or direct-pathway MSNs, dMSNs) and striatopallidal neurons (indirect-pathway MSNs, iMSNs), form circuits with distinct pallidal nuclei, which sends "GO" or "NO-GO" signals through the thalamus. These striatopallidal circuits evaluate and execute reward-seeking and taking behaviors. Especially, the dorsal striatum can be further divided into the dorsomedial striatum (DMS, equivalent to caudate in primates and humans) and dorsolateral striatum (DLS, equivalent to putamen), which orchestrates goal-directed and habitual reward-seeking and taking behaviors, respectively. Using optogenetics, chemogenetics and in vivo calcium imaging technologies combined with electrophysiology and digitalized behavior phenotyping, recent studies have revealed cell-, circuit- and context-specific functions of these microcircuits in addictive behaviors. Also, region-specific astrocytes regulate the homeostatic activities of the dMSNs and iMSNs as well as the downstream circuits, which determine the net balance of cortico-striato-pallidal activities to the thalamic neurons. This review will summarize the recent progress of striatopallidal circuits focusing on astrocyte-neuron interaction and, reward- and alcohol-seeking behaviors. Our review will also discuss the translational and clinical implications of these microcircuit studies. This article is part of the special Issue on "Neurocircuitry Modulating Drug and Alcohol Abuse".

Optimal attention tuning in a neuro-computational model of the visual cortex-basal ganglia-prefrontal cortex loop

Visual attention is widely considered a vital factor in the perception and analysis of a visual scene. Several studies explored the effects and mechanisms of top-down attention, but the mechanisms that determine the attentional signal are less explored. By developing a neuro-computational model of visual attention including the visual cortex-basal ganglia loop, we demonstrate how attentional alignment can evolve based on dopaminergic reward during a visual search task. Unlike most previous modeling studies of feature-based attention, we do not implement a manually predefined attention template. Dopamine-modulated covariance learning enable the basal ganglia to learn rewarded associations between the visual input and the attentional gain represented in the PFC of the model. Hence, the model shows human-like performance on a visual search task by optimally tuning the attention signal. In particular, similar as in humans, this reward-based tuning in the model leads to an attentional template that is not centered on the target feature, but a relevant feature deviating away from the target due to the presence of highly similar distractors. Further analyses of the model shows, attention is mainly guided by the signal-to-noise ratio between target and distractors.

Rigid reduced successor representation as a potential mechanism for addiction

Difficulty in cessation of drinking, smoking, or gambling has been widely recognized. Conventional theories proposed relative dominance of habitual over goal-directed control, but human studies have not convincingly supported them. Referring to the recently suggested "successor representation (SR)" of states that enables partially goal-directed control, we propose a dopamine-related mechanism that makes resistance to habitual reward-obtaining particularly difficult. We considered that long-standing behavior towards a certain reward without resisting temptation can (but not always) lead to a formation of rigid dimension-reduced SR based on the goal state, which cannot be updated. Then, in our model assuming such rigid reduced SR, whereas no reward prediction error (RPE) is generated at the goal while no resistance is made, a sustained large positive RPE is generated upon goal reaching once the person starts resisting temptation. Such sustained RPE is somewhat similar to the hypothesized sustained fictitious RPE caused by drug-induced dopamine. In contrast, if rigid reduced SR is not formed and states are represented individually as in simple reinforcement learning models, no sustained RPE is generated at the goal. Formation of rigid reduced SR also attenuates the resistance-dependent decrease in the value of the cue for behavior, makes subsequent introduction of punishment after the goal ineffective, and potentially enhances the propensity of nonresistance through the influence of RPEs via the spiral striatum-midbrain circuit. These results suggest that formation of rigid reduced SR makes cessation of habitual reward-obtaining particularly difficult and can thus be a mechanism for addiction, common to substance and nonsubstance reward.

Interpreting the role of the striatum during multiple phases of motor learning

The synaptic pathways in the striatum are central to basal ganglia functions including motor control, learning and organization, action selection, acquisition of motor skills, cognitive function, and emotion. Here, we review the role of the striatum and its connections in motor learning and performance. The development of new techniques to record neuronal activity and animal models of motor disorders using neurotoxin, pharmacological, and genetic manipulations are revealing pathways that underlie motor performance and motor learning, as well as how they are altered by pathophysiological mechanisms. We discuss approaches that can be used to analyze complex motor skills, particularly in rodents, and identify specific questions central to understanding how striatal circuits mediate motor learning.

Neurotransmitters, Cell Types, and Circuit Mechanisms of Motor Skill Learning and Clinical Applications

Animals acquire motor skills to better survive and adapt to a changing environment. The ability to learn novel motor actions without disturbing learned ones is essential to maintaining a broad motor repertoire. During motor learning, the brain makes a series of adjustments to build novel sensory–motor relationships that are stored within specific circuits for long-term retention. The neural mechanism of learning novel motor actions and transforming them into long-term memory still remains unclear. Here we review the latest findings with regard to the contributions of various brain subregions, cell types, and neurotransmitters to motor learning. Aiming to seek therapeutic strategies to restore the motor memory in relative neurodegenerative disorders, we also briefly describe the common experimental tests and manipulations for motor memory in rodents.

The basal ganglia corticostriatal loops and conditional learning

Brief maneuvering of the literature as to the various roles attributed to the basal ganglia corticostriatal circuits in a variety of cognitive processes such as working memory, selective attention, and category learning has inspired us to investigate the interplay of the two major basal ganglia open-recurrent loops, namely, visual and executive loops specifically the possible involvement of their overlap in conditional learning. We propose that the interaction of the visual and executive loops reflected through their cortical overlap in the dorsolateral prefrontal cortex (DL-PFC), lateral orbitofrontal cortex (LO-PFC), and presupplementary motor area (SMA) plays an instrumental role preliminary first in forming associations between a series of correct responses following similar stimuli and then in shifting, abstracting, and generalizing conditioned responses. The premotor and supplementary motor areas have been shown essential to producing a sequence of movements while the SMA is engaged in monitoring complex movements. In light of the recent studies, we will suggest that the interaction of visual and executive loops could strengthen or weaken learned associations following different reward values. Furthermore, we speculate that the overlap of the visual and executive loops can account for the switching between the associative vs. rule-based category learning systems.