A computational model of how the basal ganglia produce sequences

A Novel Self-Organizing Fuzzy Neural Network to Learn and Mimic Habitual Sequential Tasks

In this article, a new self-organizing fuzzy neural network (FNN) model is presented which is able to simultaneously and accurately learn and reproduce different sequences. Multiple sequence learning is important in performing habitual and skillful tasks, such as writing, signing signatures, and playing piano. Generally, it is indispensable for pattern generation applications. Since multiple sequences have similar parts, local information such as some previous samples is not sufficient to efficiently reproduce them. Instead, it is necessary to consider global and discriminative information, maybe in the very initial samples of each sequence, to first recognize them, and then predict their next sample based on the current local information. Therefore, the structure of the proposed network consists of two parts: 1) sequence identifier, which computes a novel sequence identity value based on initial samples of a sequence, and detects the sequence identity based on proper fuzzy rules and 2) sequence locator, which locates the input sample in the sequence. Therefore, by integrating outputs of these two parts in fuzzy rules, the network is able to produce the proper output based on the current state of each sequence. To learn the proposed structure, a gradual learning procedure is proposed. First, learning is performed by adding new fuzzy rules, based on coverage measure, using available correct data. Next, the initialized parameters are fine-tuned, by the gradient descent algorithm, based on fed back approximated network output as the next input. The proposed method has a dynamic structure able to learn new sequences online. Finally, to investigate the effectiveness of the presented approach, it is used to simultaneously learn and reproduce multiple sequences in different applications, including sequences with similar parts, different patterns, and writing different letters. The performance of the proposed method is evaluated and compared with other existing methods, including the adaptive network-based fuzzy inference system, GDFNN, CFNN, and long short-term memory (LSTM). According to these experiments, the proposed method outperforms traditional FNNs and LSTM in learning multiple sequences.

Basal impulses: findings from the last twenty years on impulsivity and reward pathways using deep brain stimulation

INTRODUCTION

Deep brain stimulation (DBS) is an important treatment modality for movement disorders. Its role in tasks and processes of higher cortical function continues to increase in importance and relevance. This systematic review investigates the impact of DBS on measures of impulsivity.

EVIDENCE ACQUISITION

A total of 45 studies were collated from PubMed (30 prospective, 8 animal, 4 questionnaire-based, and 3 computational models), excluding case reports and review articles. Two areas extensively studied are the subthalamic nucleus (STN) and nucleus accumbens (NAc).

EVIDENCE SYNTHESIS

While both are part of the basal ganglia, the STN and NAc have extensive connections to the prefrontal cortex, cingulate cortex, and limbic system. Therefore, understanding cause and treatment of impulsivity requires understanding motor pathways, learning, memory, and emotional processing. DBS of the STN and NAc shell can increase objective measures of impulsivity, as measured by reaction times or reward-based learning, independent from patient insight. The ability for DBS to treat impulse control disorders, and also cause and/or worsen impulsivity in Parkinson's disease, may be explained by the affected closely-related neuroanatomical areas with discrete and sometimes opposing functions.

CONCLUSIONS

As newer, more refined DBS technology emerges, large-scale prospective studies specifically aimed at treatment of impulsivity disorders are needed.

Turning the body into a clock: Accurate timing is facilitated by simple stereotyped interactions with the environment

How animals adapt their behavior according to regular time intervals between events is not well understood, especially when intervals last several seconds. One possibility is that animals use disembodied internal neuronal representations of time to decide when to initiate a given action at the end of an interval. However, animals rarely remain immobile during time intervals but tend to perform stereotyped behaviors, raising the possibility that motor routines improve timing accuracy. To test this possibility, we used a task in which rats, freely moving on a motorized treadmill, could obtain a reward if they approached it after a fixed interval. Most animals took advantage of the treadmill length and its moving direction to develop, by trial-and-error, the same motor routine whose execution resulted in the precise timing of their reward approaches. Noticeably, when proficient animals did not follow this routine, their temporal accuracy decreased. Then, naïve animals were trained in modified versions of the task designed to prevent the development of this routine. Compared to rats trained in the first protocol, these animals didn't reach a comparable level of timing accuracy. Altogether, our results indicate that timing accuracy in rats is improved when the environment affords cues that animals can incorporate into motor routines.

Impaired Motor Recycling during Action Selection in Parkinson's Disease

Behavioral studies have shown that the human motor system recycles motor parameters of previous actions, such as movement amplitude, when programming new actions. Shifting motor plans toward a new action forms a particularly severe problem for patients with Parkinson’s disease (PD), a disorder that, in its early stage, is dominated by basal ganglia dysfunction. Here, we test whether this action selection deficit in Parkinson’s patients arises from an impaired ability to recycle motor parameters shared across subsequent actions. Parkinson’s patients off dopaminergic medication (n = 16) and matched healthy controls (n = 16) performed a task that involved moving a handheld dowel over an obstacle in the context of a sequence of aiming movements. Consistent with previous research, healthy participants continued making unnecessarily large hand movements after clearing the obstacle (defined as “hand path priming effect”), even after switching movements between hands. In contrast, Parkinson’s patients showed a reduced hand path priming effect, i.e., they performed biomechanically more efficient movements than controls, but only when switching movements between hands. This effect correlated with disease severity, such that patients with more severe motor symptoms had a smaller hand path priming effect. We propose that the basal ganglia mediate recycling of movement parameters across subsequent actions.

Spontaneous eyeblinks are sensitive to sequential learning

Although sequential learning and spontaneous eyeblink rate (EBR) have both been shown to be tightly related to cerebral dopaminergic activity, they have never been investigated at the same time. In the present study, EBR, taken as an indirect marker of dopaminergic activity, was investigated in two resting state conditions, both before and after visuomotor sequence learning in a serial reaction time task (SRT) and during task practice. Participants' abilities to produce and manipulate their knowledge about the sequential material were probed in a generation task. We hypothesized that the time course of spontaneous EBR might follow the progressive decrease of RTs during the SRT session. Additionally, we manipulated the structure of the transfer blocks as well as their respective order, assuming that (1) fully random trials might generate a larger psychophysiological response than an unlearned but structured material, and (2) a second (final) block of transfer might give rise to larger effects given that the sequential material was better consolidated after further practice. Finally, we tentatively hypothesized that, in addition to their online version, spontaneous EBR recorded during the pre- and post-learning resting sessions might be predictive of (1) the SRT learning curve, (2) the magnitude of the transfer effects, and (3) performance in the generation task. Results showed successful sequence learning with decreased accuracy and increased reaction times (RTs) in transfer blocks featuring a different material (random trials or a structured, novel sequence). In line with our hypothesis that EBR reflects dopaminergic activity associated with sequential learning, we observed increased EBR in random trials as well as when the second transfer block occurred at the end of the learning session. There was a positive relationship between the learning curve (RTs) and the slope of EBR during the SRT session. Additionally, inter-individual differences in resting and real-time EBR predicted the magnitude of accuracy and RTs transfer effects, respectively, but they were not related to participants' performances during the generation task. Notwithstanding, our results suggest that the degree of explicit sequential knowledge modulates the association between the magnitude of the transfer effect in EBR and SRT performance. Overall, the present study provides evidence that EBR may represent a valid indirect psychophysiological correlate of dopaminergic activity coupled to sequential learning.

Non-motor Characterization of the Basal Ganglia: Evidence From Human and Non-human Primate Electrophysiology

Although the basal ganglia have been implicated in a growing list of human behaviors, they include some of the least understood nuclei in the brain. For several decades studies have employed numerous methodologies to uncover evidence pointing to the basal ganglia as a hub of both motor and non-motor function. Recently, new electrophysiological characterization of the basal ganglia in humans has become possible through direct access to these deep structures as part of routine neurosurgery. Electrophysiological approaches for identifying non-motor function have the potential to unlock a deeper understanding of pathways that may inform clinical interventions and particularly neuromodulation. Various electrophysiological modalities can also be combined to reveal functional connections between the basal ganglia and traditional structures throughout the neocortex that have been linked to non-motor behavior. Several reviews have previously summarized evidence for non-motor function in the basal ganglia stemming from behavioral, clinical, computational, imaging, and non-primate animal studies; in this review, instead we turn to electrophysiological studies of non-human primates and humans. We begin by introducing common electrophysiological methodologies for basal ganglia investigation, and then we discuss studies across numerous non-motor domains–emotion, response inhibition, conflict, decision-making, error-detection and surprise, reward processing, language, and time processing. We discuss the limitations of current approaches and highlight the current state of the information.

A loop-based neural architecture for structured behavior encoding and decoding

We present a new type of artificial neural network that generalizes on anatomical and dynamical aspects of the mammal brain. Its main novelty lies in its topological structure which is built as an array of interacting elementary motifs shaped like loops. These loops come in various types and can implement functions such as gating, inhibitory or executive control, or encoding of task elements to name a few. Each loop features two sets of neurons and a control region, linked together by non-recurrent projections. The two neural sets do the bulk of the loop's computations while the control unit specifies the timing and the conditions under which the computations implemented by the loop are to be performed. By functionally linking many such loops together, a neural network is obtained that may perform complex cognitive computations. To demonstrate the potential offered by such a system, we present two neural network simulations. The first illustrates the structure and dynamics of a single loop implementing a simple gating mechanism. The second simulation shows how connecting four loops in series can produce neural activity patterns that are sufficient to pass a simplified delayed-response task. We also show that this network reproduces electrophysiological measurements gathered in various regions of the brain of monkeys performing similar tasks. We also demonstrate connections between this type of neural network and recurrent or long short-term memory network models, and suggest ways to generalize them for future artificial intelligence research.

A system-level mathematical model of Basal Ganglia motor-circuit for kinematic planning of arm movements

In this paper, a novel system-level mathematical model of the Basal Ganglia (BG) for kinematic planning, is proposed. An arm composed of several segments presents a geometric redundancy. Thus, selecting one trajectory among an infinite number of possible ones requires overcoming redundancy, according to some kinds of optimization. Solving this optimization is assumed to be the function of BG in planning. In the proposed model, first, a mathematical solution of kinematic planning is proposed for movements of a redundant arm in a plane, based on minimizing energy consumption. Next, the function of each part in the model is interpreted as a possible role of a nucleus of BG. Since the kinematic variables are considered as vectors, the proposed model is presented based on the vector calculus. This vector model predicts different neuronal populations in BG which is in accordance with some recent experimental studies. According to the proposed model, the function of the direct pathway is to calculate the necessary rotation of each joint, and the function of the indirect pathway is to control each joint rotation considering the movement of the other joints. In the proposed model, the local feedback loop between Subthalamic Nucleus and Globus Pallidus externus is interpreted as a local memory to store the previous amounts of movements of the other joints, which are utilized by the indirect pathway. In this model, activities of dopaminergic neurons would encode, at short-term, the error between the desired and actual positions of the end-effector. The short-term modulating effect of dopamine on Striatum is also modeled as cross product. The model is simulated to generate the commands of a redundant manipulator. The performance of the model is studied for different reaching movements between 8 points in a plane. Finally, some symptoms of Parkinson's disease such as bradykinesia and akinesia are simulated by modifying the model parameters, inspired by the dopamine depletion.

Dysfunctions of the basal ganglia-cerebellar-thalamo-cortical system produce motor tics in Tourette syndrome

Motor tics are a cardinal feature of Tourette syndrome and are traditionally associated with an excess of striatal dopamine in the basal ganglia. Recent evidence increasingly supports a more articulated view where cerebellum and cortex, working closely in concert with basal ganglia, are also involved in tic production. Building on such evidence, this article proposes a computational model of the basal ganglia-cerebellar-thalamo-cortical system to study how motor tics are generated in Tourette syndrome. In particular, the model: (i) reproduces the main results of recent experiments about the involvement of the basal ganglia-cerebellar-thalamo-cortical system in tic generation; (ii) suggests an explanation of the system-level mechanisms underlying motor tic production: in this respect, the model predicts that the interplay between dopaminergic signal and cortical activity contributes to triggering the tic event and that the recently discovered basal ganglia-cerebellar anatomical pathway may support the involvement of the cerebellum in tic production; (iii) furnishes predictions on the amount of tics generated when striatal dopamine increases and when the cortex is externally stimulated. These predictions could be important in identifying new brain target areas for future therapies. Finally, the model represents the first computational attempt to study the role of the recently discovered basal ganglia-cerebellar anatomical links. Studying this non-cortex-mediated basal ganglia-cerebellar interaction could radically change our perspective about how these areas interact with each other and with the cortex. Overall, the model also shows the utility of casting Tourette syndrome within a system-level perspective rather than viewing it as related to the dysfunction of a single brain area.

Tourette syndrome is a neuropsychiatric disorder characterized by vocal and motor tics. Tics represent a cardinal symptom traditionally associated with a dysfunction of the basal ganglia leading to an excess of the dopamine neurotransmitter. This view gives a restricted clinical picture and limits therapeutic approaches because it ignores the influence of altered interactions between the basal ganglia and other brain areas. In this respect, recent evidence supports a more articulated framework where cerebellum and cortex are also involved in tic production. Building on these data, we propose a computational model of the basal ganglia-cerebellar-thalamo-cortical network to investigate the specific mechanisms underlying motor tic production. The model reproduces the results of recent experiments and suggests an explanation of the system-level processes underlying tic production. Moreover, it furnishes predictions related to the amount of tics generated when there are dysfunctions in the basal ganglia-cerebellar-thalamo-cortical circuits. These predictions could be important in identifying new brain target areas for future therapies based on a system-level view of Tourette syndrome.

Properties of Neurons in External Globus Pallidus Can Support Optimal Action Selection

The external globus pallidus (GPe) is a key nucleus within basal ganglia circuits that are thought to be involved in action selection. A class of computational models assumes that, during action selection, the basal ganglia compute for all actions available in a given context the probabilities that they should be selected. These models suggest that a network of GPe and subthalamic nucleus (STN) neurons computes the normalization term in Bayes’ equation. In order to perform such computation, the GPe needs to send feedback to the STN equal to a particular function of the activity of STN neurons. However, the complex form of this function makes it unlikely that individual GPe neurons, or even a single GPe cell type, could compute it. Here, we demonstrate how this function could be computed within a network containing two types of GABAergic GPe projection neuron, so-called ‘prototypic’ and ‘arkypallidal’ neurons, that have different response properties in vivo and distinct connections. We compare our model predictions with the experimentally-reported connectivity and input-output functions (f-I curves) of the two populations of GPe neurons. We show that, together, these dichotomous cell types fulfil the requirements necessary to compute the function needed for optimal action selection. We conclude that, by virtue of their distinct response properties and connectivities, a network of arkypallidal and prototypic GPe neurons comprises a neural substrate capable of supporting the computation of the posterior probabilities of actions.

Choosing an appropriate action as quickly and accurately as possible in a given situation is critical for the survival of animals and humans. One of the brain regions involved in action selection is a set of subcortical nuclei known as the basal ganglia. The importance of understanding information processing in the basal ganglia is further emphasised by the fact that their disturbed interactions in Parkinson’s disease results in profound difficulties in movement. Computational models have suggested how the basal ganglia could select actions in the fastest possible way for the required accuracy level. These models further predict that a part of basal ganglia, called the external globus pallidus (GPe), needs to calculate a particular function of its inputs. This paper proposes how this function could be computed in a mathematical model of a network within GPe. Furthermore, it shows that the experimentally observed connectivity and response properties of GPe neurons fulfil the requirements necessary to support optimal action selection. This suggests the GPe neurons have properties that allow them to contribute to optimal action selection in the whole basal ganglia.

New Models of Frontal-Subcortical Skeletomotor Circuit Pathology in Tardive Dyskinesia

Tardive dyskinesia (TD) is a hyperkinetic movement disorder that can occur as a side effect of treatment with antipsychotic medications. Because antipsychotics block the D2 family of dopamine receptors in the striatum, it has long been suspected this blockade contributes to the development of TD. Specifically, increased sensitivity of the dopamine receptors following chronic blockade has been thought to result in abnormal functioning of the frontal-subcortical (FSC) skeletomotor circuit and the symptoms of TD. However, this hypothesis remains unproven. In recent years, substantial research has focused on the basal ganglia and FSC circuits. This research has resulted in the development of the focused selection model of skeletomotor circuit function. This hypothesis provides a compelling model of neurocircuit abnormalities in TD. A greater understanding of the neuropathology of TD may lead to the development of better treatment and prevention strategies for this disorder. Furthermore, this information may contribute to a more complete understanding of normal skeletomotor circuit function and the role of circuit pathology in numerous neuropsychiatric conditions.

Selection of cortical dynamics for motor behaviour by the basal ganglia

The basal ganglia and cortex are strongly implicated in the control of motor preparation and execution. Re-entrant loops between these two brain areas are thought to determine the selection of motor repertoires for instrumental action. The nature of neural encoding and processing in the motor cortex as well as the way in which selection by the basal ganglia acts on them is currently debated. The classic view of the motor cortex implementing a direct mapping of information from perception to muscular responses is challenged by proposals viewing it as a set of dynamical systems controlling muscles. Consequently, the common idea that a competition between relatively segregated cortico-striato-nigro-thalamo-cortical channels selects patterns of activity in the motor cortex is no more sufficient to explain how action selection works. Here, we contribute to develop the dynamical view of the basal ganglia-cortical system by proposing a computational model in which a thalamo-cortical dynamical neural reservoir is modulated by disinhibitory selection of the basal ganglia guided by top-down information, so that it responds with different dynamics to the same bottom-up input. The model shows how different motor trajectories can so be produced by controlling the same set of joint actuators. Furthermore, the model shows how the basal ganglia might modulate cortical dynamics by preserving coarse-grained spatiotemporal information throughout cortico-cortical pathways.

Habit Learning by Naive Macaques Is Marked by Response Sharpening of Striatal Neurons Representing the Cost and Outcome of Acquired Action Sequences

Over a century of scientific work has focused on defining the factors motivating behavioral learning. Observations in animals and humans trained on a wide range of tasks support reinforcement learning (RL) algorithms as accounting for the learning. Still unknown, however, are the signals that drive learning in naive, untrained subjects. Here, we capitalized on a sequential saccade task in which macaque monkeys acquired repetitive scanning sequences without instruction. We found that spike activity in the caudate nucleus after each trial corresponded to an integrated cost-benefit signal that was highly correlated with the degree of naturalistic untutored learning by the monkeys. Across learning, neurons encoding both cost and outcome gradually acquired increasingly sharp phasic trial-end responses that paralleled the development of the habit-like, repetitive saccade sequences. Our findings demonstrate an integrated cost-benefit signal by which RL and its neural correlates could drive naturalistic behaviors in freely behaving primates.

Intact action segmentation in Parkinson's disease: Hypothesis testing using a novel computational approach

Action observation is known to trigger predictions of the ongoing course of action and thus considered a hallmark example for predictive perception. A related task, which explicitly taps into the ability to predict actions based on their internal representations, is action segmentation; the task requires participants to demarcate where one action step is completed and another one begins. It thus benefits from a temporally precise prediction of the current action. Formation and exploitation of these temporal predictions of external events is now closely associated with a network including the basal ganglia and prefrontal cortex. Because decline of dopaminergic innervation leads to impaired function of the basal ganglia and prefrontal cortex in Parkinson's disease (PD), we hypothesised that PD patients would show increased temporal variability in the action segmentation task, especially under medication withdrawal (hypothesis 1). Another crucial aspect of action segmentation is its reliance on a semantic representation of actions. There is no evidence to suggest that action representations are substantially altered, or cannot be accessed, in non-demented PD patients. We therefore expected action segmentation judgments to follow the same overall patterns in PD patients and healthy controls (hypothesis 2), resulting in comparable segmentation profiles. Both hypotheses were tested with a novel classification approach. We present evidence for both hypotheses in the present study: classifier performance was slightly decreased when it was tested for its ability to predict the identity of movies segmented by PD patients, and a measure of normativity of response behaviour was decreased when patients segmented movies under medication-withdrawal without access to an episodic memory of the sequence. This pattern of results is consistent with hypothesis 1. However, the classifier analysis also revealed that responses given by patients and controls create very similar action-specific patterns, thus delivering evidence in favour hypothesis 2. In terms of methodology, the use of classifiers in the present study allowed us to establish similarity of behaviour across groups (hypothesis 2). The approach opens up a new avenue that standard statistical methods often fail to provide and is discussed in terms of its merits to measure hypothesised similarities across study populations.

Morphological changes of glutamatergic synapses in animal models of Parkinson's disease

The striatum and the subthalamic nucleus (STN) are the main entry doors for extrinsic inputs to reach the basal ganglia (BG) circuitry. The cerebral cortex, thalamus and brainstem are the key sources of glutamatergic inputs to these nuclei. There is anatomical, functional and neurochemical evidence that glutamatergic neurotransmission is altered in the striatum and STN of animal models of Parkinson’s disease (PD) and that these changes may contribute to aberrant network neuronal activity in the BG-thalamocortical circuitry. Postmortem studies of animal models and PD patients have revealed significant pathology of glutamatergic synapses, dendritic spines and microcircuits in the striatum of parkinsonians. More recent findings have also demonstrated a significant breakdown of the glutamatergic corticosubthalamic system in parkinsonian monkeys. In this review, we will discuss evidence for synaptic glutamatergic dysfunction and pathology of cortical and thalamic inputs to the striatum and STN in models of PD. The potential functional implication of these alterations on synaptic integration, processing and transmission of extrinsic information through the BG circuits will be considered. Finally, the significance of these pathological changes in the pathophysiology of motor and non-motor symptoms in PD will be examined.

Abnormal neuronal activity in Tourette syndrome and its modulation using deep brain stimulation

Tourette syndrome (TS) is a common childhood-onset disorder characterized by motor and vocal tics that are typically accompanied by a multitude of comorbid symptoms. Pharmacological treatment options are limited, which has led to the exploration of deep brain stimulation (DBS) as a possible treatment for severe cases. Multiple lines of evidence have linked TS with abnormalities in the motor and limbic cortico-basal ganglia (CBG) pathways. Neurophysiological data have only recently started to slowly accumulate from multiple sources: noninvasive imaging and electrophysiological techniques, invasive electrophysiological recordings in TS patients undergoing DBS implantation surgery, and animal models of the disorder. These converging sources point to system-level physiological changes throughout the CBG pathway, including both general altered baseline neuronal activity patterns and specific tic-related activity. DBS has been applied to different regions along the motor and limbic pathways, primarily to the globus pallidus internus, thalamic nuclei, and nucleus accumbens. In line with the findings that also draw on the more abundant application of DBS to Parkinson's disease, this stimulation is assumed to result in changes in the neuronal firing patterns and the passage of information through the stimulated nuclei. We present an overview of recent experimental findings on abnormal neuronal activity associated with TS and the changes in this activity following DBS. These findings are then discussed in the context of current models of CBG function in the normal state, during TS, and finally in the wider context of DBS in CBG-related disorders.

The role of prediction and outcomes in adaptive cognitive control

Humans adaptively perform actions to achieve their goals. This flexible behaviour requires two core abilities: the ability to anticipate the outcomes of candidate actions and the ability to select and implement actions in a goal-directed manner. The ability to predict outcomes has been extensively researched in reinforcement learning paradigms, but this work has often focused on simple actions that are not embedded in hierarchical and sequential structures that are characteristic of goal-directed human behaviour. On the other hand, the ability to select actions in accordance with high-level task goals, particularly in the presence of alternative responses and salient distractors, has been widely researched in cognitive control paradigms. Cognitive control research, however, has often paid less attention to the role of action outcomes. The present review attempts to bridge these accounts by proposing an outcome-guided mechanism for selection of extended actions. Our proposal builds on constructs from the hierarchical reinforcement learning literature, which emphasises the concept of reaching and evaluating informative states, i.e., states that constitute subgoals in complex actions. We develop an account of the neural mechanisms that allow outcome-guided action selection to be achieved in a network that relies on projections from cortical areas to the basal ganglia and back-projections from the basal ganglia to the cortex. These cortico-basal ganglia-thalamo-cortical 'loops' allow convergence - and thus integration - of information from non-adjacent cortical areas (for example between sensory and motor representations). This integration is essential in action sequences, for which achieving an anticipated sensory state signals the successful completion of an action. We further describe how projection pathways within the basal ganglia allow selection between representations, which may pertain to movements, actions, or extended action plans. The model lastly envisages a role for hierarchical projections from the striatum to dopaminergic midbrain areas that enable more rostral frontal areas to bias the selection of inputs from more posterior frontal areas via their respective representations in the basal ganglia.

Hemispheric differences in the mesostriatal dopaminergic system

The mesostriatal dopaminergic system, which comprises the mesolimbic and the nigrostriatal pathways, plays a major role in neural processing underlying motor and limbic functions. Multiple reports suggest that these processes are influenced by hemispheric differences in striatal dopamine (DA) levels, DA turnover and its receptor activity. Here, we review studies which measured the concentration of DA and its metabolites to examine the relationship between DA imbalance and animal behavior under different conditions. Specifically, we assess evidence in support of endogenous, inter-hemispheric DA imbalance; determine whether the known anatomy provides a suitable substrate for this imbalance; examine the relationship between DA imbalance and animal behavior; and characterize the symmetry of the observed inter-hemispheric laterality in the nigrostriatal and the mesolimbic DA systems. We conclude that many studies provide supporting evidence for the occurrence of experience-dependent endogenous DA imbalance which is controlled by a dedicated regulatory/compensatory mechanism. Additionally, it seems that the link between DA imbalance and animal behavior is better characterized in the nigrostriatal than in the mesolimbic system. Nonetheless, a variety of brain and behavioral manipulations demonstrate that the nigrostriatal system displays symmetrical laterality whereas the mesolimbic system displays asymmetrical laterality which supports hemispheric specialization in rodents. The reciprocity of the relationship between DA imbalance and animal behavior (i.e., the capacity of animal training to alter DA imbalance for prolonged time periods) remains controversial, however, if confirmed, it may provide a valuable non-invasive therapeutic means for treating abnormal DA imbalance.

Closed-loop brain-machine-body interfaces for noninvasive rehabilitation of movement disorders

Traditional approaches for neurological rehabilitation of patients affected with movement disorders, such as Parkinson's disease (PD), dystonia, and essential tremor (ET) consist mainly of oral medication, physical therapy, and botulinum toxin injections. Recently, the more invasive method of deep brain stimulation (DBS) showed significant improvement of the physical symptoms associated with these disorders. In the past several years, the adoption of feedback control theory helped DBS protocols to take into account the progressive and dynamic nature of these neurological movement disorders that had largely been ignored so far. As a result, a more efficient and effective management of PD cardinal symptoms has emerged. In this paper, we review closed-loop systems for rehabilitation of movement disorders, focusing on PD, for which several invasive and noninvasive methods have been developed during the last decade, reducing the complications and side effects associated with traditional rehabilitation approaches and paving the way for tailored individual therapeutics. We then present a novel, transformative, noninvasive closed-loop framework based on force neurofeedback and discuss several future developments of closed-loop systems that might bring us closer to individualized solutions for neurological rehabilitation of movement disorders.

The role of efference copy in striatal learning

Reinforcement learning requires the convergence of signals representing context, action, and reward. While models of basal ganglia function have well-founded hypotheses about the neural origin of signals representing context and reward, the function and origin of signals representing action are less clear. Recent findings suggest that exploratory or variable behaviors are initiated by a wide array of 'action-generating' circuits in the midbrain, brainstem, and cortex. Thus, in order to learn, the striatum must incorporate an efference copy of action decisions made in these action-generating circuits. Here we review several recent neural models of reinforcement learning that emphasize the role of efference copy signals. Also described are ideas about how these signals might be integrated with inputs signaling context and reward.

Investigating habits: strategies, technologies and models

Understanding habits at a biological level requires a combination of behavioral observations and measures of ongoing neural activity. Theoretical frameworks as well as definitions of habitual behaviors emerging from classic behavioral research have been enriched by new approaches taking account of the identification of brain regions and circuits related to habitual behavior. Together, this combination of experimental and theoretical work has provided key insights into how brain circuits underlying action-learning and action-selection are organized, and how a balance between behavioral flexibility and fixity is achieved. New methods to monitor and manipulate neural activity in real time are allowing us to have a first look “under the hood” of a habit as it is formed and expressed. Here we discuss ideas emerging from such approaches. We pay special attention to the unexpected findings that have arisen from our own experiments suggesting that habitual behaviors likely require the simultaneous activity of multiple distinct components, or operators, seen as responsible for the contrasting dynamics of neural activity in both cortico-limbic and sensorimotor circuits recorded concurrently during different stages of habit learning. The neural dynamics identified thus far do not fully meet expectations derived from traditional models of the structure of habits, and the behavioral measures of habits that we have made also are not fully aligned with these models. We explore these new clues as opportunities to refine an understanding of habits.

Transient and steady-state selection in the striatal microcircuit

Although the basal ganglia have been widely studied and implicated in signal processing and action selection, little information is known about the active role the striatal microcircuit plays in action selection in the basal ganglia-thalamo-cortical loops. To address this knowledge gap we use a large scale three dimensional spiking model of the striatum, combined with a rate coded model of the basal ganglia-thalamo-cortical loop, to asses the computational role the striatum plays in action selection. We identify a robust transient phenomena generated by the striatal microcircuit, which temporarily enhances the difference between two competing cortical inputs. We show that this transient is sufficient to modulate decision making in the basal ganglia-thalamo-cortical circuit. We also find that the transient selection originates from a novel adaptation effect in single striatal projection neurons, which is amenable to experimental testing. Finally, we compared transient selection with models implementing classical steady-state selection. We challenged both forms of model to account for recent reports of paradoxically enhanced response selection in Huntington's disease patients. We found that steady-state selection was uniformly impaired under all simulated Huntington's conditions, but transient selection was enhanced given a sufficient Huntington's-like increase in NMDA receptor sensitivity. Thus our models provide an intriguing hypothesis for the mechanisms underlying the paradoxical cognitive improvements in manifest Huntington's patients.

Desynchronizing the abnormally synchronized neural activity in the subthalamic nucleus: a modeling study

A mathematical model of a target area for deep brain stimulation was used to investigate the effects of electrical stimulation on pathologically synchronized clusters of neurons. In total, three newly developed stimulation techniques based on multisite coordinated reset and delayed feedback were tested and compared with a high-frequency stimulation method that is currently used as a standard stimulation protocol for deep brain stimulation. By modeling both excitatory and inhibitory actions of the electrical stimulation, we revealed the desynchronization impacts of the novel stimulation techniques. This contrasts with standard high-frequency stimulation, which failed to desynchronize the target population and whose inhibitory effects blocked all neuronal activity. We also explored the demand-controlled character of the proposed methods, and demonstrated that the amount of stimulation current required was considerably smaller than that for high-frequency stimulation. These novel stimulation methods appear to be superior to standard high-frequency stimulation techniques, and we propose the methods now be used for deep brain stimulation.

Computational models of basal-ganglia pathway functions: focus on functional neuroanatomy

Over the past 15 years, computational models have had a considerable impact on basal-ganglia research. Most of these models implement multiple distinct basal-ganglia pathways and assume them to fulfill different functions. As there is now a multitude of different models, it has become complex to keep track of their various, sometimes just marginally different assumptions on pathway functions. Moreover, it has become a challenge to oversee to what extent individual assumptions are corroborated or challenged by empirical data. Focusing on computational, but also considering non-computational models, we review influential concepts of pathway functions and show to what extent they are compatible with or contradict each other. Moreover, we outline how empirical evidence favors or challenges specific model assumptions and propose experiments that allow testing assumptions against each other.

Basal ganglia output to the thalamus: still a paradox

The basal ganglia (BG)-recipient thalamus controls motor output but it remains unclear how its activity is regulated. Several studies report that thalamic activation occurs via disinhibition during pauses in the firing of inhibitory pallidal inputs from the BG. Other studies indicate that thalamic spiking is triggered by pallidal inputs via post-inhibitory 'rebound' calcium spikes. Finally excitatory cortical inputs can drive thalamic activity, which becomes entrained, or time-locked, to pallidal spikes. We present a unifying framework where these seemingly distinct results arise from a continuum of thalamic firing 'modes' controlled by excitatory inputs. We provide a mechanistic explanation for paradoxical pallidothalamic coactivations observed during behavior that raises new questions about what information is integrated in the thalamus to control behavior.

The implication of neuroactive steroids in Tourette's syndrome pathogenesis: A role for 5α-reductase?

Tourette's syndrome (TS) is a neurodevelopmental disorder characterised by recurring motor and phonic tics. The pathogenesis of TS is considered to reflect dysregulations in the signalling of dopamine (DA) and other neurotransmitters, which lead to excitation/inhibition imbalances in cortico-striato-thalamocortical circuits. The causes of these deficits may reflect complex gene × environment × sex (G × E × S) interactions; indeed, the disorder is markedly predominant in males, with a male-to-female prevalence ratio of approximately 4 : 1. Converging lines of evidence point to neuroactive steroids as being likely molecular candidates to account for G × E × S interactions in TS. Building on these premises, our group has begun examining the possibility that alterations in the steroid biosynthetic process may be directly implicated in TS pathophysiology; in particular, our research has focused on 5α-reductase (5αR), the enzyme catalysing the key rate-limiting step in the synthesis of pregnane and androstane neurosteroids. In clinical and preclinical studies, we found that 5αR inhibitors exerted marked anti-DAergic and tic-suppressing properties, suggesting a central role for this enzyme in TS pathogenesis. Based on these data, we hypothesise that enhancements in 5αR activity in early developmental stages may lead to an inappropriate activation of the 'backdoor' pathway for androgen synthesis from adrenarche until the end of puberty. We predict that the ensuing imbalances in steroid homeostasis may impair the signalling of DA and other neurotransmitters, ultimately resulting in the facilitation of tics and other behavioural abnormalities in TS.

Developing intuition: neural correlates of cognitive-skill learning in caudate nucleus

The superior capability of cognitive experts largely depends on automatic, quick information processing, which is often referred to as intuition. Intuition develops following extensive long-term training. There are many cognitive models on intuition development, but its neural basis is not known. Here we trained novices for 15 weeks to learn a simple board game and measured their brain activities in early and end phases of the training while they quickly generated the best next-move to a given board pattern. We found that the activation in the head of caudate nucleus developed over the course of training, in parallel to the development of the capability to quickly generate the best next-move, and the magnitude of the caudate activity was correlated with the subject's performance. In contrast, cortical activations, which already appeared in the early phase of training, did not further change. Thus, neural activation in the caudate head, but not those in cortical areas, tracked the development of capability to quickly generate the best next-move, indicating that circuitries including the caudate head may automate cognitive computations.

Habits, action sequences and reinforcement learning

It is now widely accepted that instrumental actions can be either goal-directed or habitual; whereas the former are rapidly acquired and regulated by their outcome, the latter are reflexive, elicited by antecedent stimuli rather than their consequences. Model-based reinforcement learning (RL) provides an elegant description of goal-directed action. Through exposure to states, actions and rewards, the agent rapidly constructs a model of the world and can choose an appropriate action based on quite abstract changes in environmental and evaluative demands. This model is powerful but has a problem explaining the development of habitual actions. To account for habits, theorists have argued that another action controller is required, called model-free RL, that does not form a model of the world but rather caches action values within states allowing a state to select an action based on its reward history rather than its consequences. Nevertheless, there are persistent problems with important predictions from the model; most notably the failure of model-free RL correctly to predict the insensitivity of habitual actions to changes in the action-reward contingency. Here, we suggest that introducing model-free RL in instrumental conditioning is unnecessary, and demonstrate that reconceptualizing habits as action sequences allows model-based RL to be applied to both goal-directed and habitual actions in a manner consistent with what real animals do. This approach has significant implications for the way habits are currently investigated and generates new experimental predictions.

200-300Hz movement modulated oscillations in the internal globus pallidus of patients with Parkinson's Disease

Symptoms in Parkinson's Disease (PD) have been linked to oscillatory activity within the basal ganglia. In humans, such activity has been detected mainly in the local field potentials (LFPs) recorded from electrode contacts used for deep brain stimulation. Although most studies have focused on activity within the subthalamic nucleus (STN), the internal part of the globus pallidus (GPi) is considered an equally efficacious site for therapeutic neuromodulation. Moreover, while most investigations have evaluated changes in oscillatory activity in the beta (12-35Hz) and gamma (35-100Hz) bands, our preliminary spectral analysis of LFP signals in the GPi suggested distinct activity at higher frequencies as well. We hypothesized there is a unique LFP signature in the GPi that consists of movement modulated spectral power increases above 100Hz. Using invasive recordings from the GPi of patients undergoing DBS, in addition to confirming increased beta band activity within the GPi of patients with PD, we have identified and characterized a previously undescribed peak between 200 and 300Hz centered at approximately 235Hz, whose height and width but not center frequency are movement modulated. An increase in peak height is not transient, but rather persists for the duration of movement. The 200-300Hz rhythms in the GPi could have a functional role in the basal ganglia reentrant circuits by encoding output information entering the thalamo-cortical network or by organizing downstream activity for the successful execution of tasks.

Synchronizing with auditory and visual rhythms: an fMRI assessment of modality differences and modality appropriateness

Synchronizing movements with auditory beats, compared to visual flashes, yields divergent activation in timing-related brain areas as well as more stable tapping synchronization. The differences in timing-related brain activation could reflect differences in tapping synchronization stability, rather than differences between modality (i.e., audio-motor vs. visuo-motor integration). In the current fMRI study, participants synchronized their finger taps with four types of visual and auditory pacing sequences: flashes and a moving bar, as well as beeps and a frequency-modulated 'siren'. Behavioral tapping results showed that visuo-motor synchronization improved with moving targets, whereas audio-motor synchronization degraded with frequency-modulated sirens. Consequently, a modality difference in synchronization occurred between the discrete beeps and flashes, but not between the novel continuous siren and moving bar. Imaging results showed that activation in the putamen, a key timing area, paralleled the behavioral results: putamen activation was highest for beeps, intermediate for the continuous siren and moving bar, and was lowest for the flashes. Putamen activation differed between modalities for beeps and flashes, but not for the novel moving bar and siren. By dissociating synchronization performance from modality, we show that activation in the basal ganglia is associated with sensorimotor synchronization stability rather than modality-specificity in this task. Synchronization stability is apparently contingent upon the modality's processing affinity: discrete auditory and moving visual signals are modality appropriate, and can be encoded reliably for integration with the motor system.

Oculomotor learning revisited: a model of reinforcement learning in the basal ganglia incorporating an efference copy of motor actions

In its simplest formulation, reinforcement learning is based on the idea that if an action taken in a particular context is followed by a favorable outcome, then, in the same context, the tendency to produce that action should be strengthened, or reinforced. While reinforcement learning forms the basis of many current theories of basal ganglia (BG) function, these models do not incorporate distinct computational roles for signals that convey context, and those that convey what action an animal takes. Recent experiments in the songbird suggest that vocal-related BG circuitry receives two functionally distinct excitatory inputs. One input is from a cortical region that carries context information about the current “time” in the motor sequence. The other is an efference copy of motor commands from a separate cortical brain region that generates vocal variability during learning. Based on these findings, I propose here a general model of vertebrate BG function that combines context information with a distinct motor efference copy signal. The signals are integrated by a learning rule in which efference copy inputs gate the potentiation of context inputs (but not efference copy inputs) onto medium spiny neurons in response to a rewarded action. The hypothesis is described in terms of a circuit that implements the learning of visually guided saccades. The model makes testable predictions about the anatomical and functional properties of hypothesized context and efference copy inputs to the striatum from both thalamic and cortical sources.

Action selection and action value in frontal-striatal circuits

The role that frontal-striatal circuits play in normal behavior remains unclear. Two of the leading hypotheses suggest that these circuits are important for action selection or reinforcement learning. To examine these hypotheses, we carried out an experiment in which monkeys had to select actions in two different task conditions. In the first (random) condition, actions were selected on the basis of perceptual inference. In the second (fixed) condition, the animals used reinforcement from previous trials to select actions. Examination of neural activity showed that the representation of the selected action was stronger in lateral prefrontal cortex (lPFC), and occurred earlier in the lPFC than it did in the dorsal striatum (dSTR). In contrast to this, the representation of action values, in both the random and fixed conditions, was stronger in the dSTR. Thus, the dSTR contains an enriched representation of action value, but it followed frontal cortex in action selection.

Systems biology in psychiatric research: from complex data sets over wiring diagrams to computer simulations

The classification of psychiatric disorders has always been a problem in clinical settings. The present debate about the major systems in clinical practice, DSM-IV and ICD-10, has resulted in attempts to improve and replace those schemes by some that include more endophenotypic and molecular features. However, these disorders not only require more precise diagnostic tools, but also have to be viewed more extensively in their dynamic behaviors, which require more precise data sets related to their origins and developments. This enormous challenge in brain research has to be approached on different levels of the biological system by new methods, including improvements in electroencephalography, brain imaging, and molecular biology. All these methods entail accumulations of large data sets that become more and more difficult to interpret. In particular, on the molecular level, there is an apparent need to use highly sophisticated computer programs to tackle these problems. Evidently, only interdisciplinary work among mathematicians, physicists, biologists, and clinicians can further improve our understanding of complex diseases of the brain.

Acetylcholine-based entropy in response selection: a model of how striatal interneurons modulate exploration, exploitation, and response variability in decision-making

The basal ganglia play a fundamental role in decision-making. Their contribution is typically modeled within a reinforcement learning framework, with the basal ganglia learning to select the options associated with highest value and their dopamine inputs conveying performance feedback. This basic framework, however, does not account for the role of cholinergic interneurons in the striatum, and does not easily explain certain dynamic aspects of decision-making and skill acquisition like the generation of exploratory actions. This paper describes basal ganglia acetylcholine-based entropy (BABE), a model of the acetylcholine system in the striatum that provides a unified explanation for these phenomena. According to this model, cholinergic interneurons in the striatum control the level of variability in behavior by modulating the number of possible responses that are considered by the basal ganglia, as well as the level of competition between them. This mechanism provides a natural way to account for the role of basal ganglia in generating behavioral variability during the acquisition of certain cognitive skills, as well as for modulating exploration and exploitation in decision-making. Compared to a typical reinforcement learning model, BABE showed a greater modulation of response variability in the face of changes in the reward contingences, allowing for faster learning (and re-learning) of option values. Finally, the paper discusses the possible applications of the model to other domains.

Expectancy, ambiguity, and behavioral flexibility: separable and complementary roles of the orbital frontal cortex and amygdala in processing reward expectancies

Appetitive goal-directed behavior can be associated with a cue-triggered expectancy that it will lead to a particular reward, a process thought to depend on the OFC and basolateral amygdala complex. We developed a biologically informed neural network model of this system to investigate the separable and complementary roles of these areas as the main components of a flexible expectancy system. These areas of interest are part of a neural network with additional subcortical areas, including the central nucleus of amygdala, ventral (limbic) and dorsomedial (associative) striatum. Our simulations are consistent with the view that the amygdala maintains Pavlovian associations through incremental updating of synaptic strength and that the OFC supports flexibility by maintaining an activation-based working memory of the recent reward history. Our model provides a mechanistic explanation for electrophysiological evidence that cue-related firing in OFC neurons is nonselectively early after a contingency change and why this nonselective firing is critical for promoting plasticity in the amygdala. This ambiguous activation results from the simultaneous maintenance of recent outcomes and obsolete Pavlovian contingencies in working memory. Furthermore, at the beginning of reversal, the OFC is critical for supporting responses that are no longer inappropriate. This result is inconsistent with an exclusive inhibitory account of OFC function.

A neural correlate of predicted and actual reward-value information in monkey pedunculopontine tegmental and dorsal raphe nucleus during saccade tasks

Dopamine, acetylcholine, and serotonin, the main modulators of the central nervous system, have been proposed to play important roles in the execution of movement, control of several forms of attentional behavior, and reinforcement learning. While the response pattern of midbrain dopaminergic neurons and its specific role in reinforcement learning have been revealed, the role of the other neuromodulators remains rather elusive. Here, we review our recent studies using extracellular recording from neurons in the pedunculopontine tegmental nucleus, where many cholinergic neurons exist, and the dorsal raphe nucleus, where many serotonergic neurons exist, while monkeys performed eye movement tasks to obtain different reward values. The firing patterns of these neurons are often tonic throughout the task period, while dopaminergic neurons exhibited a phasic activity pattern to the task event. The different modulation patterns, together with the activity of dopaminergic neurons, reveal dynamic information processing between these different neuromodulator systems.

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.

What are the physiological changes that take place in the brain when we solve a problem or learn a new skill? It is commonly assumed that behavior adaptations are realized on the microscopic level by changes in synaptic efficacies. However, this is hard to verify experimentally due to the difficulties of identifying the relevant synapses and monitoring them over long periods during a behavioral task. To address this question computationally, we develop a spiking neuronal network model of actor-critic temporal-difference learning, a variant of reinforcement learning for which neural correlates have already been partially established. The network learns a complex task by means of an internally generated reward signal constrained by recent findings on the dopaminergic system. Our model combines top-down and bottom-up modelling approaches to bridge the gap between synaptic plasticity and system-level learning. It paves the way for further investigations of the dopaminergic system in reward learning in the healthy brain and in pathological conditions such as Parkinson's disease, and can be used as a module in functional models based on brain-scale circuitry.

The neuropharmacology of implicit learning

Two decades of pharmacologic research on the human capacity to implicitly acquire knowledge as well as cognitive skills and procedures have yielded surprisingly few conclusive insights. We review the empirical literature of the neuropharmacology of implicit learning. We evaluate the findings in the context of relevant computational models related to neurotransmittors such as dopamine, serotonin, acetylcholine and noradrenalin. These include models for reinforcement learning, sequence production, and categorization. We conclude, based on the reviewed literature, that one can predict improved implicit acquisition by moderately elevated dopamine levels and impaired implicit acquisition by moderately decreased dopamine levels. These effects are most prominent in the dorsal striatum. This is supported by a range of behavioral tasks in the empirical literature. Similar predictions can be made for serotonin, although there is yet a lack of support in the literature for serotonin involvement in classical implicit learning tasks. There is currently a lack of evidence for a role of the noradrenergic and cholinergic systems in implicit and related forms of learning. GABA modulators, including benzodiazepines, seem to affect implicit learning in a complex manner and further research is needed. Finally, we identify allosteric AMPA receptors modulators as a potentially interesting target for future investigation of the neuropharmacology of procedural and implicit learning.

What do the basal ganglia do? A modeling perspective

Basal ganglia (BG) constitute a network of seven deep brain nuclei involved in a variety of crucial brain functions including: action selection, action gating, reward based learning, motor preparation, timing, etc. In spite of the immense amount of data available today, researchers continue to wonder how a single deep brain circuit performs such a bewildering range of functions. Computational models of BG have focused on individual functions and fail to give an integrative picture of BG function. A major breakthrough in our understanding of BG function is perhaps the insight that activities of mesencephalic dopaminergic cells represent some form of 'reward' to the organism. This insight enabled application of tools from 'reinforcement learning,' a branch of machine learning, in the study of BG function. Nevertheless, in spite of these bright spots, we are far from the goal of arriving at a comprehensive understanding of these 'mysterious nuclei.' A comprehensive knowledge of BG function has the potential to radically alter treatment and management of a variety of BG-related neurological disorders (Parkinson's disease, Huntington's chorea, etc.) and neuropsychiatric disorders (schizophrenia, obsessive compulsive disorder, etc.) also. In this article, we review the existing modeling literature on BG and hypothesize an integrative picture of the function of these nuclei.

Reinforcement learning on slow features of high-dimensional input streams

Humans and animals are able to learn complex behaviors based on a massive stream of sensory information from different modalities. Early animal studies have identified learning mechanisms that are based on reward and punishment such that animals tend to avoid actions that lead to punishment whereas rewarded actions are reinforced. However, most algorithms for reward-based learning are only applicable if the dimensionality of the state-space is sufficiently small or its structure is sufficiently simple. Therefore, the question arises how the problem of learning on high-dimensional data is solved in the brain. In this article, we propose a biologically plausible generic two-stage learning system that can directly be applied to raw high-dimensional input streams. The system is composed of a hierarchical slow feature analysis (SFA) network for preprocessing and a simple neural network on top that is trained based on rewards. We demonstrate by computer simulations that this generic architecture is able to learn quite demanding reinforcement learning tasks on high-dimensional visual input streams in a time that is comparable to the time needed when an explicit highly informative low-dimensional state-space representation is given instead of the high-dimensional visual input. The learning speed of the proposed architecture in a task similar to the Morris water maze task is comparable to that found in experimental studies with rats. This study thus supports the hypothesis that slowness learning is one important unsupervised learning principle utilized in the brain to form efficient state representations for behavioral learning.

Conditional routing of information to the cortex: a model of the basal ganglia's role in cognitive coordination

The basal ganglia play a central role in cognition and are involved in such general functions as action selection and reinforcement learning. Here, we present a model exploring the hypothesis that the basal ganglia implement a conditional information-routing system. The system directs the transmission of cortical signals between pairs of regions by manipulating separately the selection of sources and destinations of information transfers. We suggest that such a mechanism provides an account for several cognitive functions of the basal ganglia. The model also incorporates a possible mechanism by which subsequent transfers of information control the release of dopamine. This signal is used to produce novel stimulus-response associations by internalizing transferred cortical representations in the striatum. We discuss how the model is related to production systems and cognitive architectures. A series of simulations is presented to illustrate how the model can perform simple stimulus-response tasks, develop automatic behaviors, and provide an account of impairments in Parkinson's and Huntington's diseases.