A computational model of how the basal ganglia produce sequences

Conditional routing of information to the cortex: a model of the basal ganglia's role in cognitive coordination

The basal ganglia play a central role in cognition and are involved in such general functions as action selection and reinforcement learning. Here, we present a model exploring the hypothesis that the basal ganglia implement a conditional information-routing system. The system directs the transmission of cortical signals between pairs of regions by manipulating separately the selection of sources and destinations of information transfers. We suggest that such a mechanism provides an account for several cognitive functions of the basal ganglia. The model also incorporates a possible mechanism by which subsequent transfers of information control the release of dopamine. This signal is used to produce novel stimulus-response associations by internalizing transferred cortical representations in the striatum. We discuss how the model is related to production systems and cognitive architectures. A series of simulations is presented to illustrate how the model can perform simple stimulus-response tasks, develop automatic behaviors, and provide an account of impairments in Parkinson's and Huntington's diseases.

Convergent evidence for abnormal striatal synaptic plasticity in dystonia

Dystonia is a functionally disabling movement disorder characterized by abnormal movements and postures. Although substantial recent progress has been made in identifying genetic factors, the pathophysiology of the disease remains a mystery. A provocative suggestion gaining broader acceptance is that some aspect of neural plasticity may be abnormal. There is also evidence that, at least in some forms of dystonia, sensorimotor "use" may be a contributing factor. Most empirical evidence of abnormal plasticity in dystonia comes from measures of sensorimotor cortical organization and physiology. However, the basal ganglia also play a critical role in sensorimotor function. Furthermore, the basal ganglia are prominently implicated in traditional models of dystonia, are the primary targets of stereotactic neurosurgical interventions, and provide a neural substrate for sensorimotor learning influenced by neuromodulators. Our working hypothesis is that abnormal plasticity in the basal ganglia is a critical link between the etiology and pathophysiology of dystonia. In this review we set up the background for this hypothesis by integrating a large body of disparate indirect evidence that dystonia may involve abnormalities in synaptic plasticity in the striatum. After reviewing evidence implicating the striatum in dystonia, we focus on the influence of two neuromodulatory systems: dopamine and acetylcholine. For both of these neuromodulators, we first describe the evidence for abnormalities in dystonia and then the means by which it may influence striatal synaptic plasticity. Collectively, the evidence suggests that many different forms of dystonia may involve abnormal plasticity in the striatum. An improved understanding of these altered plastic processes would help inform our understanding of the pathophysiology of dystonia, and, given the role of the striatum in sensorimotor learning, provide a principled basis for designing therapies aimed at the dynamic processes linking etiology to pathophysiology of the disease.

The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward

The basal ganglia are often conceptualised as three parallel domains that include all the constituent nuclei. The 'ventral domain' appears to be critical for learning flexible behaviours for exploration and foraging, as it is the recipient of converging inputs from amygdala, hippocampal formation and prefrontal cortex, putatively centres for stimulus evaluation, spatial navigation, and planning/contingency, respectively. However, compared to work on the dorsal domains, the rich potential for quantitative theories and models of the ventral domain remains largely untapped, and the purpose of this review is to provide the stimulus for this work. We systematically review the ventral domain's structures and internal organisation, and propose a functional architecture as the basis for computational models. Using a full schematic of the structure of inputs to the ventral striatum (nucleus accumbens core and shell), we argue for the existence of many identifiable processing channels on the basis of unique combinations of afferent inputs. We then identify the potential information represented in these channels by reconciling a broad range of studies from the hippocampal, amygdala and prefrontal cortex literatures with known properties of the ventral striatum from lesion, pharmacological, and electrophysiological studies. Dopamine's key role in learning is reviewed within the three current major computational frameworks; we also show that the shell-based basal ganglia sub-circuits are well placed to generate the phasic burst and dip responses of dopaminergic neurons. We detail dopamine's modulation of ventral basal ganglia's inputs by its actions on pre-synaptic terminals and post-synaptic membranes in the striatum, arguing that the complexity of these effects hint at computational roles for dopamine beyond current ideas. The ventral basal ganglia are revealed as a constellation of multiple functional systems for the learning and selection of flexible behaviours and of behavioural strategies, sharing the common operations of selection-by-disinhibition and of dopaminergic modulation.

Recognizing sequences of sequences

The brain's decoding of fast sensory streams is currently impossible to emulate, even approximately, with artificial agents. For example, robust speech recognition is relatively easy for humans but exceptionally difficult for artificial speech recognition systems. In this paper, we propose that recognition can be simplified with an internal model of how sensory input is generated, when formulated in a Bayesian framework. We show that a plausible candidate for an internal or generative model is a hierarchy of ‘stable heteroclinic channels’. This model describes continuous dynamics in the environment as a hierarchy of sequences, where slower sequences cause faster sequences. Under this model, online recognition corresponds to the dynamic decoding of causal sequences, giving a representation of the environment with predictive power on several timescales. We illustrate the ensuing decoding or recognition scheme using synthetic sequences of syllables, where syllables are sequences of phonemes and phonemes are sequences of sound-wave modulations. By presenting anomalous stimuli, we find that the resulting recognition dynamics disclose inference at multiple time scales and are reminiscent of neuronal dynamics seen in the real brain.

Despite tremendous advances in neuroscience, we cannot yet build machines that recognize the world as effortlessly as we do. One reason might be that there are computational approaches to recognition that have not yet been exploited. Here, we demonstrate that the ability to recognize temporal sequences might play an important part. We show that an artificial decoding device can extract natural speech sounds from sound waves if speech is generated as dynamic and transient sequences of sequences. In principle, this means that artificial recognition can be implemented robustly and online using dynamic systems theory and Bayesian inference.

The first second: models of short-term memory traces in the brain

Many network models in computational neuroscience rise to the challenge of explaining behavioural phenomena ranging from microseconds to tens of seconds using components operating mostly on a time-scale of milliseconds. These models have in common that the underlying system has a memory, which implies that its output depends on its past input history. In this review we compare how such memory traces or delayed responses may be implemented in different brain areas supporting a diversity of functions.

Probabilistic reversal learning is impaired in Parkinson's disease

In many everyday settings, the relationship between our choices and their potentially rewarding outcomes is probabilistic and dynamic. In addition, the difficulty of the choices can vary widely. Although a large body of theoretical and empirical evidence suggests that dopamine mediates rewarded learning, the influence of dopamine in probabilistic and dynamic rewarded learning remains unclear. We adapted a probabilistic rewarded learning task originally used to study firing rates of dopamine cells in primate substantia nigra pars compacta [Morris G, Nevet A, Arkadir D, Vaadia E, Bergman H (2006) Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9:1057-1063] for use as a reversal learning task with humans. We sought to investigate how the dopamine depletion in Parkinson's disease (PD) affects probabilistic reward learning and adaptation to a reversal in reward contingencies. Over the course of 256 trials subjects learned to choose the more favorable from among pairs of images with small or large differences in reward probabilities. During a subsequent otherwise identical reversal phase, the reward probability contingencies for the stimuli were reversed. Seventeen PD patients of mild to moderate severity were studied off of their dopaminergic medications and compared to 15 age-matched controls. Compared to controls, PD patients had distinct pre- and post-reversal deficiencies depending upon the difficulty of the choices they had to learn. The patients also exhibited compromised adaptability to the reversal. A computational model of the subjects' trial-by-trial choices demonstrated that the adaptability was sensitive to the gain with which patients weighted pre-reversal feedback. Collectively, the results implicate the nigral dopaminergic system in learning to make choices in environments with probabilistic and dynamic reward contingencies.

Differential effect of reward and punishment on procedural learning

Reward and punishment are potent modulators of associative learning in instrumental and classical conditioning. However, the effect of reward and punishment on procedural learning is not known. The striatum is known to be an important locus of reward-related neural signals and part of the neural substrate of procedural learning. Here, using an implicit motor learning task, we show that reward leads to enhancement of learning in human subjects, whereas punishment is associated only with improvement in motor performance. Furthermore, these behavioral effects have distinct neural substrates with the learning effect of reward being mediated through the dorsal striatum and the performance effect of punishment through the insula. Our results suggest that reward and punishment engage separate motivational systems with distinctive behavioral effects and neural substrates.

Striatal versus hippocampal representations during win-stay maze performance

The striatum and hippocampus are widely held to be components of distinct memory systems that can guide competing behavioral strategies. However, some electrophysiological studies have suggested that neurons in both structures encode spatial information and may therefore make similar contributions to behavior. In rats well trained to perform a win-stay radial maze task, we recorded simultaneously from dorsal hippocampus and from multiple striatal subregions, including both lateral areas implicated in motor responses to cues and medial areas that work cooperatively with hippocampus in cognitive operations. In each brain region, movement through the maze was accompanied by the continuous sequential activation of sets of projection neurons. Hippocampal neurons overwhelmingly were active at a single spatial location (place cells). Striatal projection neurons were active at discrete points within the progression of every trial-especially during choices or following reward delivery-regardless of spatial position. Place-cell-type firing was not observed even for medial striatal cells entrained to the hippocampal theta rhythm. We also examined neural coding in earlier training sessions, when rats made use of spatial working memory to guide choices, and again found that striatal cells did not show place-cell-type firing. Prospective or retrospective encoding of trajectory was not observed in either hippocampus or striatum, at either training stage. Our results indicate that, at least in this task, dorsal hippocampus uses a spatial foundation for information processing that is not substantially modulated by spatial working memory demands. By contrast, striatal cells do not use such a spatial foundation, even in medial subregions that cooperate with hippocampus in the selection of spatial strategies. The progressive dominance of a striatum-dependent strategy does not appear to be accompanied by large changes in striatal or hippocampal single-cell representations, suggesting that the conflict between strategies may be resolved elsewhere.

Encoding by Response Duration in the Basal Ganglia

Several models have suggested that information transmission in the basal ganglia (BG) involves gating mechanisms, where neuronal activity modulates the extent of gate aperture and its duration. Here, we demonstrate that BG response duration is informative about a highly abstract stimulus feature and show that the duration of “gate opening” can indeed be used for information transmission through the BG. We analyzed recordings from three BG locations: the external part of the globus pallidus (GPe), the substantia nigra pars reticulata (SNr), and dopaminergic neurons from the substantia nigra pars compacta (SNc) during performance of a probabilistic visuomotor task. Most (>85%) of the neurons showed significant rate modulation following the appearance of cues predicting future reward. Trial-to-trial mutual information analysis revealed that response duration encoded reward prospects in many (42%) of the responsive SNr neurons, as well as in the SNc (26.9%), and the GPe (29.3%). Whereas the low-frequency discharge SNc neurons responded with only an increase in firing rate, SNr and GPe neurons with high-frequency tonic discharge responded with both increases and decreases. Conversely, many duration-informative neurons in SNr (68%) and GPe (50%) responded with a decreased rather than an increased rate. The response duration was more informative than the extreme (minimal or maximal) amplitude or spike count in responsive bins of duration-informative neurons. Thus response duration is not simply correlated with the discharge rate and can provide additional information to the target structures of the BG.

Path-finding in real and simulated rats: assessing the influence of path characteristics on navigation learning

A large body of experimental evidence suggests that the hippocampal place field system is involved in reward based navigation learning in rodents. Reinforcement learning (RL) mechanisms have been used to model this, associating the state space in an RL-algorithm to the place-field map in a rat. The convergence properties of RL-algorithms are affected by the exploration patterns of the learner. Therefore, we first analyzed the path characteristics of freely exploring rats in a test arena. We found that straight path segments with mean length 23 cm up to a maximal length of 80 cm take up a significant proportion of the total paths. Thus, rat paths are biased as compared to random exploration. Next we designed a RL system that reproduces these specific path characteristics. Our model arena is covered by overlapping, probabilistically firing place fields (PF) of realistic size and coverage. Because convergence of RL-algorithms is also influenced by the state space characteristics, different PF-sizes and densities, leading to a different degree of overlap, were also investigated. The model rat learns finding a reward opposite to its starting point. We observed that the combination of biased straight exploration, overlapping coverage and probabilistic firing will strongly impair the convergence of learning. When the degree of randomness in the exploration is increased, convergence improves, but the distribution of straight path segments becomes unrealistic and paths become 'wiggly'. To mend this situation without affecting the path characteristic two additional mechanisms are implemented: a gradual drop of the learned weights (weight decay) and path length limitation, which prevents learning if the reward is not found after some expected time. Both mechanisms limit the memory of the system and thereby counteract effects of getting trapped on a wrong path. When using these strategies individually divergent cases get substantially reduced and for some parameter settings no divergence was found anymore at all. Using weight decay and path length limitation at the same time, convergence is not much improved but instead time to convergence increases as the memory limiting effect is getting too strong. The degree of improvement relies also on the size and degree of overlap (coverage density) in the place field system. The used combination of these two parameters leads to a trade-off between convergence and speed to convergence. Thus, this study suggests that the role of the PF-system in navigation learning cannot be considered independently from the animals' exploration pattern.

Modeling parkinsonian circuitry and the DBS electrode. I. Biophysical background and software

Deep brain stimulation (DBS) of the subthalamic nucleus (STN) for Parkinson's disease (PD) has become routine over the past decade, utilizing microelectrode recordings to ensure accurate placement of the stimulating electrodes. The clinical benefits of STN DBS for PD are well documented, but the mechanisms by which DBS achieves these results remain elusive. We have created a closed-form mathematical function of the potential field generated by a typical 4-contact DBS electrode and inserted this function into a computational model designed to simulate individual neurons and neural circuitry of significant portions of the basal ganglia. We present the mathematical function representing the potential field itself and the basis for the neural circuitry modeling in this paper.

Modeling parkinsonian circuitry and the DBS electrode. II. Evaluation of a computer simulation model of the basal ganglia with and without subthalamic nucleus stimulation

Treatment with deep brain stimulation (DBS) for Parkinson's disease (PD) has become routine over the past decade, particularly using the subthalamic nucleus (STN) as a target and utilizing microelectrode recordings to ensure accurate placement of the stimulating electrodes. The clinical changes seen with DBS in the STN for PD are consistently beneficial, but there continues to be only marginal understanding of the mechanisms by which DBS achieves these results. Using an analytical model of the typical DBS 4-contact electrode and software developed to simulate individual neurons and neural circuitry of the basal ganglia we compare the results of the model to those of data obtained during DBS surgery of the STN. Firing rate, interspike intervals and regularity analyses were performed on the simulated data and compared to results in the literature.

Towards an executive without a homunculus: computational models of the prefrontal cortex/basal ganglia system

The prefrontal cortex (PFC) has long been thought to serve as an 'executive' that controls the selection of actions and cognitive functions more generally. However, the mechanistic basis of this executive function has not been clearly specified often amounting to a homunculus. This paper reviews recent attempts to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying the executive functions of the PFC. The overall approach builds upon existing mechanistic models of the basal ganglia (BG) and frontal systems known to play a critical role in motor control and action selection, where the BG provide a 'Go' versus 'NoGo' modulation of frontal action representations. In our model, the BG modulate working memory representations in prefrontal areas to support more abstract executive functions. We have developed a computational model of this system that is capable of developing human-like performance on working memory and executive control tasks through trial-and-error learning. This learning is based on reinforcement learning mechanisms associated with the midbrain dopaminergic system and its activation via the BG and amygdala. Finally, we briefly describe various empirical tests of this framework.

A scale-free systems theory of motivation and addiction

Scale-free organizations, characterized by uneven distributions of linkages between nodal elements, describe the structure and function of many life-based complex systems developing under evolutionary pressures. We explore motivated behavior as a scale-free map toward a comprehensive translational theory of addiction. Motivational and behavioral repertoires are reframed as link and nodal element sets, respectively, comprising a scale-free structure. These sets are generated by semi-independent information-processing streams within cortical-striatal circuits that cooperatively provide decision-making and sequential processing functions necessary for traversing maps of motivational links connecting behavioral nodes. Dopamine modulation of cortical-striatal plasticity serves a central-hierarchical mechanism for survival-adaptive sculpting and development of motivational-behavioral repertoires by guiding a scale-free design. Drug-induced dopamine activity promotes drug taking as a highly connected behavioral hub at the expense of natural-adaptive motivational links and behavioral nodes. Conceptualizing addiction as pathological alteration of scale-free motivational-behavioral repertoires unifies neurobiological, neurocomputational and behavioral research while addressing addiction vulnerability in adolescence and psychiatric illness. This model may inform integrative research in defining more effective prevention and treatment strategies for addiction.

Serotonin and the Evaluation of Future Rewards: Theory, Experiments, and Possible Neural Mechanisms

The ability to select an action by considering both delays and amount of reward outcome is critical for survival and well-being of animals and humans. Previous animal experiments suggest a role of serotonin in action choice by modulating the evaluation of delayed rewards. It remains unclear, however, through which neural circuits, and through what receptors and intracellular mechanisms, serotonin affects the evaluation of delayed rewards. Here, we review experimental studies and computational theory of decisions under delayed rewards, and propose that serotonin controls the timescale of reward prediction by regulating neural activity in the basal ganglia.

Reward prediction error computation in the pedunculopontine tegmental nucleus neurons

In this article, we address the role of neuronal activity in the pathways of the brainstem-midbrain circuit in reward and the basis for believing that this circuit provides advantages over previous reinforcement learning theory. Several lines of evidence support the reward-based learning theory proposing that midbrain dopamine (DA) neurons send a teaching signal (the reward prediction error signal) to control synaptic plasticity of the projection area. However, the underlying mechanism of where and how the reward prediction error signal is computed still remains unclear. Since the pedunculopontine tegmental nucleus (PPTN) in the brainstem is one of the strongest excitatory input sources to DA neurons, we hypothesized that the PPTN may play an important role in activating DA neurons and reinforcement learning by relaying necessary signals for reward prediction error computation to DA neurons. To investigate the involvement of the PPTN neurons in computation of reward prediction error, we used a visually guided saccade task (VGST) during recording of neuronal activity in monkeys. Here, we predict that PPTN neurons may relay the excitatory component of tonic reward prediction and phasic primary reward signals, and derive a new computational theory of the reward prediction error in DA neurons.

A physiologically plausible model of action selection and oscillatory activity in the basal ganglia

The basal ganglia (BG) have long been implicated in both motor function and dysfunction. It has been proposed that the BG form a centralized action selection circuit, resolving conflict between multiple neural systems competing for access to the final common motor pathway. We present a new spiking neuron model of the BG circuitry to test this proposal, incorporating all major features and many physiologically plausible details. We include the following: effects of dopamine in the subthalamic nucleus (STN) and globus pallidus (GP), transmission delays between neurons, and specific distributions of synaptic inputs over dendrites. All main parameters were derived from experimental studies. We find that the BG circuitry supports motor program selection and switching, which deteriorates under dopamine-depleted and dopamine-excessive conditions in a manner consistent with some pathologies associated with those dopamine states. We also validated the model against data describing oscillatory properties of BG. We find that the same model displayed detailed features of both gamma-band (30-80 Hz) and slow (approximately 1 Hz) oscillatory phenomena reported by Brown et al. (2002) and Magill et al. (2001), respectively. Only the parameters required to mimic experimental conditions (e.g., anesthetic) or manipulations (e.g., lesions) were changed. From the results, we derive the following novel predictions about the STN-GP feedback loop: (1) the loop is functionally decoupled by tonic dopamine under normal conditions and recoupled by dopamine depletion; (2) the loop does not show pacemaking activity under normal conditions in vivo (but does after combined dopamine depletion and cortical lesion); (3) the loop has a resonant frequency in the gamma-band.

Computational models simulating electrophysiological activity in the basal ganglia

Modeling of the basal ganglia has played a substantial role in gaining insight into the mechanisms involved in the computational processes performed by this elusive group of nuclei. Models of the basal ganglia have undergone revolutionary changes over the last twenty years due to the rapid accumulation of neuroscientific data. In this chapter, we present distinct modeling approaches that can be used to enhance our understanding of the functional dynamics of information processing within the basal ganglia, and their interactions with the rest of the brain. Specific examples of recently developed models dealing with the analysis of computational processing issues at different structural levels of the basal ganglia are discussed.

Transient calcium and dopamine increase PKA activity and DARPP-32 phosphorylation

Reinforcement learning theorizes that strengthening of synaptic connections in medium spiny neurons of the striatum occurs when glutamatergic input (from cortex) and dopaminergic input (from substantia nigra) are received simultaneously. Subsequent to learning, medium spiny neurons with strengthened synapses are more likely to fire in response to cortical input alone. This synaptic plasticity is produced by phosphorylation of AMPA receptors, caused by phosphorylation of various signalling molecules. A key signalling molecule is the phosphoprotein DARPP-32, highly expressed in striatal medium spiny neurons. DARPP-32 is regulated by several neurotransmitters through a complex network of intracellular signalling pathways involving cAMP (increased through dopamine stimulation) and calcium (increased through glutamate stimulation). Since DARPP-32 controls several kinases and phosphatases involved in striatal synaptic plasticity, understanding the interactions between cAMP and calcium, in particular the effect of transient stimuli on DARPP-32 phosphorylation, has major implications for understanding reinforcement learning. We developed a computer model of the biochemical reaction pathways involved in the phosphorylation of DARPP-32 on Thr34 and Thr75. Ordinary differential equations describing the biochemical reactions were implemented in a single compartment model using the software XPPAUT. Reaction rate constants were obtained from the biochemical literature. The first set of simulations using sustained elevations of dopamine and calcium produced phosphorylation levels of DARPP-32 similar to that measured experimentally, thereby validating the model. The second set of simulations, using the validated model, showed that transient dopamine elevations increased the phosphorylation of Thr34 as expected, but transient calcium elevations also increased the phosphorylation of Thr34, contrary to what is believed. When transient calcium and dopamine stimuli were paired, PKA activation and Thr34 phosphorylation increased compared with dopamine alone. This result, which is robust to variation in model parameters, supports reinforcement learning theories in which activity-dependent long-term synaptic plasticity requires paired glutamate and dopamine inputs.

Reinforcement learning, based on the association of a stimulus-triggered movement with a reward, involves changes in connection strength between neurons. Memory storage occurs in the striatum, the input stage of the basal ganglia, when a stimulus or movement signal originating from the cortex and a reward signal originating from the midbrain reach the target striatal cells together. Repetitive pairing of these two signals strengthens the connection between cortical and striatal cells. The strengthening of the connections is caused by activation of biochemical signalling pathways inside the striatal cells. These intracellular signalling pathways are explored in a quantitative computational model describing the biochemical pathways important for reinforcement learning. Lindskog et al.'s study shows that when brief reward and stimuli signals are paired, a stronger response in the intracellular signalling occurs compared with the situation when each signal is given alone. This result illustrates mechanisms whereby paired stimuli, but not unpaired stimuli, can cause learning. Furthermore, the model predicts that the biochemical responses are different after brief stimulation as compared with prolonged stimulation. This result highlights the difficulties in predicting the nonlinear interactions within signalling cascades based on prolonged stimulations, which often are used in biochemical experiments.

Neural mechanism for stochastic behaviour during a competitive game

Previous studies have shown that non-human primates can generate highly stochastic choice behaviour, especially when this is required during a competitive interaction with another agent. To understand the neural mechanism of such dynamic choice behaviour, we propose a biologically plausible model of decision making endowed with synaptic plasticity that follows a reward-dependent stochastic Hebbian learning rule. This model constitutes a biophysical implementation of reinforcement learning, and it reproduces salient features of behavioural data from an experiment with monkeys playing a matching pennies game. Due to interaction with an opponent and learning dynamics, the model generates quasi-random behaviour robustly in spite of intrinsic biases. Furthermore, non-random choice behaviour can also emerge when the model plays against a non-interactive opponent, as observed in the monkey experiment. Finally, when combined with a meta-learning algorithm, our model accounts for the slow drift in the animal's strategy based on a process of reward maximization.

Visuomotor learning in immersive 3D virtual reality in Parkinson's disease and in aging

Successful adaptation to novel sensorimotor contexts critically depends on efficient sensory processing and integration mechanisms, particularly those required to combine visual and proprioceptive inputs. If the basal ganglia are a critical part of specialized circuits that adapt motor behavior to new sensorimotor contexts, then patients who are suffering from basal ganglia dysfunction, as in Parkinson's disease should show sensorimotor learning impairments. However, this issue has been under-explored. We tested the ability of 8 patients with Parkinson's disease (PD), off medication, ten healthy elderly subjects and ten healthy young adults to reach to a remembered 3D location presented in an immersive virtual environment. A multi-phase learning paradigm was used having four conditions: baseline, initial learning, reversal learning and aftereffect. In initial learning, the computer altered the position of a simulated arm endpoint used for movement feedback by shifting its apparent location diagonally, requiring thereby both horizontal and vertical compensations. This visual distortion forced subjects to learn new coordinations between what they saw in the virtual environment and the actual position of their limbs, which they had to derive from proprioceptive information (or efference copy). In reversal learning, the sign of the distortion was reversed. Both elderly subjects and PD patients showed learning phase-dependent difficulties. First, elderly controls were slower than young subjects when learning both dimensions of the initial biaxial discordance. However, their performance improved during reversal learning and as a result elderly and young controls showed similar adaptation rates during reversal learning. Second, in striking contrast to healthy elderly subjects, PD patients were more profoundly impaired during the reversal phase of learning. PD patients were able to learn the initial biaxial discordance but were on average slower than age-matched controls in adapting to the horizontal component of the biaxial discordance. More importantly, when the biaxial discordance was reversed, PD patients were unable to make appropriate movement corrections. Therefore, they showed significantly degraded learning indices relative to age-matched controls for both dimensions of the biaxial discordance. Together, these results suggest that the ability to adapt to a sudden biaxial visuomotor discordance applied in three-dimensional space declines in normal aging and Parkinson disease. Furthermore, the presence of learning rate differences in the PD patients relative to age-matched controls supports an important contribution of basal ganglia-related circuits in learning novel visuomotor coordinations, particularly those in which subjects must learn to adapt to sensorimotor contingencies that were reversed from those just learned.

Dissociating local and global levels of perceptuo-motor control in masked priming

Masked prime stimuli presented near the threshold of conscious awareness affect responses to subsequent targets. The direction of these priming effects depends on the interval between masked prime and target. With short intervals, benefits for compatible trials (primes and targets mapped to the same response) and costs for incompatible trials are observed. This pattern reverses with longer intervals. We argue (a) that these effects reflect the initial activation and subsequent self-inhibition of the primed response, and the corresponding inhibition and subsequent disinhibition of the nonprimed response, and (b) that they are generated at dissociable local (within response channels) and global (between channels) levels of motor control. In two experiments, global-level priming effects were modulated by changing the number of response alternatives, whereas local-level effects remained unaffected. These experiments suggest that low-level motor control mechanisms can be successfully decomposed into separable subcomponents, operating at different levels within the motor system.

The role of the basal ganglia in exploration in a neural model based on reinforcement learning

We present a computational model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated water pool experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system's state space. We model the Subthalamic Nucleus-Globus Pallidus externa (STN-GPe) segment of the basal ganglia as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes such as chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore state-space, we suggest that the complex exploratory dynamics of STN-GPe system in conjunction with dopamine-based reward signaling from the Substantia Nigra pars compacta (SNc) present the two key ingredients of a reinforcement learning system.

Microstimulation of frontal cortex can reorder a remembered spatial sequence

Complex goal-directed behaviors extend over time and thus depend on the ability to serially order memories and assemble compound, temporally coordinated movements. Theories of sequential processing range from simple associative chaining to hierarchical models in which order is encoded explicitly and separately from sequence components. To examine how short-term memory and planning for sequences might be coded, we used microstimulation to perturb neural activity in the supplementary eye field (SEF) while animals held a sequence of two cued locations in memory over a short delay. We found that stimulation affected the order in which animals saccaded to the locations, but not the memory for which locations were cued. These results imply that memory for sequential order can be dissociated from that of its components. Furthermore, stimulation of the SEF appeared to bias sequence endpoints to converge toward a location in contralateral space, suggesting that this area encodes sequences in terms of their endpoints rather than their individual components.

Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia

The prefrontal cortex has long been thought to subserve both working memory (the holding of information online for processing) and executive functions (deciding how to manipulate working memory and perform processing). Although many computational models of working memory have been developed, the mechanistic basis of executive function remains elusive, often amounting to a homunculus. This article presents an attempt to deconstruct this homunculus through powerful learning mechanisms that allow a computational model of the prefrontal cortex to control both itself and other brain areas in a strategic, task-appropriate manner. These learning mechanisms are based on subcortical structures in the midbrain, basal ganglia, and amygdala, which together form an actor-critic architecture. The critic system learns which prefrontal representations are task relevant and trains the actor, which in turn provides a dynamic gating mechanism for controlling working memory updating. Computationally, the learning mechanism is designed to simultaneously solve the temporal and structural credit assignment problems. The model's performance compares favorably with standard backpropagation-based temporal learning mechanisms on the challenging 1-2-AX working memory task and other benchmark working memory tasks.

Banishing the homunculus: making working memory work

The prefrontal cortex has long been thought to subserve both working memory and "executive" function, but the mechanistic basis of their integrated function has remained poorly understood, often amounting to a homunculus. This paper reviews the progress in our laboratory and others pursuing a long-term research agenda to deconstruct this homunculus by elucidating the precise computational and neural mechanisms underlying these phenomena. We outline six key functional demands underlying working memory, and then describe the current state of our computational model of the prefrontal cortex and associated systems in the basal ganglia (BG). The model, called PBWM (prefrontal cortex, basal ganglia working memory model), relies on actively maintained representations in the prefrontal cortex, which are dynamically updated/gated by the basal ganglia. It is capable of developing human-like performance largely on its own by taking advantage of powerful reinforcement learning mechanisms, based on the midbrain dopaminergic system and its activation via the basal ganglia and amygdala. These learning mechanisms enable the model to learn to control both itself and other brain areas in a strategic, task-appropriate manner. The model can learn challenging working memory tasks, and has been corroborated by several important empirical studies.

A neural network model of Parkinson's disease bradykinesia

Parkinson's disease (PD) is caused by dopamine (DA) depletion consequent to cell degeneration in the substantia nigra pars compacta (SNc) and the ventral tegmental area (VTA). Although computational analyses of PD have focused on DA depletion in DA-recipient parts of the basal ganglia, there is also extensive DAergic innervation of the frontal and parietal cortex as well as the spinal cord. To understand PD bradykinesia, a comprehensive network model is needed to study how patterns of DA depletion at key cellular sites in the basal ganglia, cortex and spinal cord contribute to disordered neuronal and spinal cord activity and other PD symptoms. We extend a basal ganglia-cortico-spinal circuit for control of voluntary arm movements by incorporating DAergic innervation of cells in the cortical and spinal components of the circuit. The resultant model simulates successfully several of the main reported effects of DA depletion on neuronal, electromyographic (EMG), and movement parameters of PD bradykinesia.

Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms

In this review, we compare methods for temporal sequence learning (TSL) across the disciplines machine-control, classical conditioning, neuronal models for TSL as well as spike-timing-dependent plasticity (STDP). This review introduces the most influential models and focuses on two questions: To what degree are reward-based (e.g., TD learning) and correlation-based (Hebbian) learning related? and How do the different models correspond to possibly underlying biological mechanisms of synaptic plasticity? We first compare the different models in an open-loop condition, where behavioral feedback does not alter the learning. Here we observe that reward-based and correlation-based learning are indeed very similar. Machine control is then used to introduce the problem of closed-loop control (e.g., actor-critic architectures). Here the problem of evaluative (rewards) versus nonevaluative (correlations) feedback from the environment will be discussed, showing that both learning approaches are fundamentally different in the closed-loop condition. In trying to answer the second question, we compare neuronal versions of the different learning architectures to the anatomy of the involved brain structures (basal-ganglia, thalamus, and cortex) and the molecular biophysics of glutamatergic and dopaminergic synapses. Finally, we discuss the different algorithms used to model STDP and compare them to reward-based learning rules. Certain similarities are found in spite of the strongly different timescales. Here we focus on the biophysics of the different calcium-release mechanisms known to be involved in STDP.

Physiological studies of information processing in the normal and Parkinsonian basal ganglia: pallidal activity in Go/No-Go task and following MPTP treatment

Understanding the role of the basal ganglia in day to day behavior is critical for a better understanding of the role of these structures in pathological states--such as Parkinson's disease. To elucidate this connection, we studied pallidal activity in a monkey performing a delayed release Go/No-Go task and in monkeys treated with the dopaminergic neurotoxin--MPTP. We compared the results with the predictions of the action selection and reinforcement driven dimensionality reduction models of the basal ganglia. The fraction of responding pallidal neurons, as well as the ratio of positive to negative responses, were equal in the Go and the No-Go modes. The fraction of pallidal neurons with significant responses following the trigger signal (19/26) was higher than that following the visual cue (11/26); however, the fraction of negative responses was significantly higher following the cue signal (47%) than that following the trigger signal (22%). Most (80%) of the cue responses were sensitive to the laterality of the cue, whereas only 25% of the responses following the trigger signal were sensitive to the cue or movement direction. Finally, pallidal spiking activity was not correlated in the normal behaving monkey, and became highly synchronized following MPTP treatment. We conclude that pallidal activity in the normal monkey is consistent with the model of action selection, assuming that action is selected following the visual cue. However, the reinforcement driven dimensionality reduction model is consistent with both the Go/No-Go responses and the normal/MPTP correlation studies.

Sequential super-stereotypy of an instinctive fixed action pattern in hyper-dopaminergic mutant mice: a model of obsessive compulsive disorder and Tourette's

BACKGROUND

Excessive sequential stereotypy of behavioral patterns (sequential super-stereotypy) in Tourette's syndrome and obsessive compulsive disorder (OCD) is thought to involve dysfunction in nigrostriatal dopamine systems. In sequential super-stereotypy, patients become trapped in overly rigid sequential patterns of action, language, or thought. Some instinctive behavioral patterns of animals, such as the syntactic grooming chain pattern of rodents, have sufficiently complex and stereotyped serial structure to detect potential production of overly-rigid sequential patterns. A syntactic grooming chain is a fixed action pattern that serially links up to 25 grooming movements into 4 predictable phases that follow 1 syntactic rule. New mutant mouse models allow gene-based manipulation of brain function relevant to sequential patterns, but no current animal model of spontaneous OCD-like behaviors has so far been reported to exhibit sequential super-stereotypy in the sense of a whole complex serial pattern that becomes stronger and excessively rigid. Here we used a hyper-dopaminergic mutant mouse to examine whether an OCD-like behavioral sequence in animals shows sequential super-stereotypy. Knockdown mutation of the dopamine transporter gene (DAT) causes extracellular dopamine levels in the neostriatum of these adult mutant mice to rise to 170% of wild-type control levels.

RESULTS

We found that the serial pattern of this instinctive behavioral sequence becomes strengthened as an entire entity in hyper-dopaminergic mutants, and more resistant to interruption. Hyper-dopaminergic mutant mice have stronger and more rigid syntactic grooming chain patterns than wild-type control mice. Mutants showed sequential super-stereotypy in the sense of having more stereotyped and predictable syntactic grooming sequences, and were also more likely to resist disruption of the pattern en route, by returning after a disruption to complete the pattern from the appropriate point in the sequence. By contrast, wild-type mice exhibited weaker forms of the fixed action pattern, and often failed to complete the full sequence.

CONCLUSIONS

Sequential super-stereotypy occurs in the complex fixed action patterns of hyper-dopaminergic mutant mice. Elucidation of the basis for sequential super-stereotypy of instinctive behavior in DAT knockdown mutant mice may offer insights into neural mechanisms of overly-rigid sequences of action or thought in human patients with disorders such as Tourette's or OCD.

Computational models of the basal ganglia: from robots to membranes

With the rapid accumulation of neuroscientific data comes a pressing need to develop models that can explain the computational processes performed by the basal ganglia. Relevant biological information spans a range of structural levels, from the activity of neuronal membranes to the role of the basal ganglia in overt behavioural control. This viewpoint presents a framework for understanding the aims, limitations and methods for testing of computational models across all structural levels. We identify distinct modelling strategies that can deliver important and complementary insights into the nature of problems the basal ganglia have evolved to solve, and describe methods that are used to solve them.

Identification of prosodic attitudes by a temporal recurrent network

Human speakers modulate the fundamental frequency (F0) of their utterances in order to express different 'prosodic' attitudes such as surprise or curiosity. How are these prosodic attitudes then decoded? The current research addresses the issue of how the temporal structure of F0 can be used in order to discriminate between prosodic attitudes in natural language using a temporal recurrent neural network (TRN) that was initially developed to simulate the neurophysiology of the primate frontostriatal system. In the TRN, a recurrent network of leaky integrator neurons encodes a continuous trajectory of internal states that characterizes the input sequence. The input to the model is a population coding of the continuous, time-varying values of the fundamental frequency (F0) of natural language sentences. We expose the model to an experiment based on one in which human subjects were required to discriminate between different prosodic attitudes (surprise, exclamation, question, etc.). After training, the model discriminates between six prosodic attitudes in new sentences at 82.52% correct, compared to 72.8% correct for human subjects. These results reveal (1) that F0 provides relevant information for prosodic attitude discrimination, and (2) that the TRN demonstrates a categorical sensitivity to this information that can be used for classifying new sentences.

How laminar frontal cortex and basal ganglia circuits interact to control planned and reactive saccades

How does the brain learn to balance between reactive and planned behaviors? The basal ganglia (BG) and frontal cortex together allow animals to learn planned behaviors that acquire rewards when prepotent reactive behaviors are insufficient. This paper proposes a new model, called TELOS, to explain how laminar circuitry of the frontal cortex, exemplified by the frontal eye fields, interacts with the BG, thalamus, superior colliculus, and inferotemporal and parietal cortices to learn and perform reactive and planned eye movements. The model is formulated as fourteen computational hypotheses. These specify how strategy priming and action planning (in cortical layers III, Va and VI) are dissociated from movement execution (in layer Vb), how the BG help to choose among and gate competing plans, and how a visual stimulus may serve either as a movement target or as a discriminative cue to move elsewhere. The direct, indirect and hyperdirect pathways through the BG are shown to enable complex gating functions, including deferred execution of selected plans, and switching among alternative sensory-motor mappings. Notably, the model can learn and gate the use of a What-to-Where transformation that enables spatially invariant object representations to selectively excite spatially coded movement plans. Model simulations show how dopaminergic reward and non-reward signals guide monkeys to learn and perform saccadic eye movements in the fixation, single saccade, overlap, gap, and delay (memory-guided) saccade tasks. Model cell activation dynamics quantitatively simulate seventeen established types of dynamics exhibited by corresponding real cells during performance of these tasks.

Cerebral modeling and dynamic Bayesian networks

The understanding and the prediction of the clinical outcomes of focal or degenerative cerebral lesions, as well as the assessment of rehabilitation procedures, necessitate knowing the cerebral substratum of cognitive or sensorimotor functions. This is achieved by activation studies, where subjects are asked to perform a specific task while data of their brain functioning are obtained through functional neuroimaging techniques. Such studies, as well as animal experiments, have shown that sensorimotor or cognitive functions are the offspring of the activity of large-scale networks of anatomically connected cerebral regions. However, no one-to-one correspondence between activated networks adn functions can be found. Our research aims at understanding how the activation of large-scale networks derives from cerebral information processing mechanisms, which can only explain apparently conflicting activation data. Our work falls at the crossroads of neuroimaging interpretation techniques and computational neuroscience. Since knowledge in cognitive neuroscience is permanently evolving, our research aims more precisely at defining a new modeling formalism and at building a flexible simulator, allowing a quick implementation of the models, for a better interpretation of cerebral functional images. It also aims at providing plausible models, at eht level of large-scale networks, of cerebral information processing mechanisms in humans. In this paper, we propose a formalism, based on dynamic Bayesian networks (DBNs), that respects the following constraints: an oriented, networks architecture, whose nodes (the cerebral structures) can all be different, the implementation of causality--the activation of the structure is caused by upstream nodes' activation--the explicit representation of different time scales (from 1 ms for the cerebral activity to many seconds for a PET scan image acquisition), the representation of cerebral information at the integrated level of neuronal populations, the imprecision of functional neuroimaging data, the nonlinearity and the uncertainty in cerebral mechanisms, and brain's plasticity (learning, reorganization, modulation). One of the main problems, nonlinearity, has been tackled thanks to new extensions of the Kalman filter. The capabilities of the formalism's current version are illustrated by the modeling of a phoneme categorization process, explaining the different cerebral activations in normal and dyslexic subjects.

Dopamine Neurons Can Represent Context-Dependent Prediction Error

Midbrain dopamine (DA) neurons are thought to encode reward prediction error. Reward prediction can be improved if any relevant context is taken into account. We found that monkey DA neurons can encode a context-dependent prediction error. In the first noncontextual task, a light stimulus was randomly followed by reward, with a fixed equal probability. The response of DA neurons was positively correlated with the number of preceding unrewarded trials and could be simulated by a conventional temporal difference (TD) model. In the second contextual task, a reward-indicating light stimulus was presented with the probability that, while fixed overall, was incremented as a function of the number of preceding unrewarded trials. The DA neuronal response then was negatively correlated with this number. This history effect corresponded to the prediction error based on the conditional probability of reward and could be simulated only by implementing the relevant context into the TD model.

Doing Without Schema Hierarchies: A Recurrent Connectionist Approach to Normal and Impaired Routine Sequential Action

In everyday tasks, selecting actions in the proper sequence requires a continuously updated representation of temporal context. Previous models have addressed this problem by positing a hierarchy of processing units, mirroring the roughly hierarchical structure of naturalistic tasks themselves. The present study considers an alternative framework, in which the representation of context depends on recurrent connections within a network mapping from environmental inputs to actions. The ability of this approach to account for human performance was evaluated by applying it, through simulation, to a specific everyday task. The resulting model learned to deal flexibly with a complex set of sequencing constraints, encoding contextual information at multiple time scales within a single, distributed internal representation. Degrading this representation led to errors resembling those observed both in everyday behavior and in apraxia. Analysis of the model's function yielded numerous predictions relevant to both normal and apraxic performance.

Information processing, dimensionality reduction and reinforcement learning in the basal ganglia

Modeling of the basal ganglia has played a major role in our understanding of this elusive group of nuclei. Models of the basal ganglia have undergone evolutionary and revolutionary changes over the last 20 years, as new research in the fields of anatomy, physiology and biochemistry of these nuclei has yielded new information. Early models dealt with a single pathway through the nuclei and focused on the nature of the processing performed within it, convergence of information versus parallel processing of information. Later, the Albin-DeLong "box-and-arrow" model characterized the inter-nuclei interaction as multiple pathways while maintaining a simplistic scalar representation of the nuclei themselves. This model made a breakthrough by providing key insights into the behavior of these nuclei in hypo- and hyper-kinetic movement disorders. The next generation of models elaborated the intra-nuclei interactions and focused on the role of the basal ganglia in action selection and sequence generation which form the most current consensus regarding basal ganglia function in both normal and pathological conditions. However, new findings challenge these models and point to a different neural network approach to information processing in the basal ganglia. Here, we take an in-depth look at the reinforcement driven dimensionality reduction (RDDR) model which postulates that the basal ganglia compress cortical information according to a reinforcement signal using optimal extraction methods. The model provides new insights and experimental predictions on the computational capacity of the basal ganglia and their role in health and disease.

Modeling facilitation and inhibition of competing motor programs in basal ganglia subthalamic nucleus-pallidal circuits

The motor symptoms of Parkinson's disease (PD) implicate the basal ganglia (BG) in some aspect of motor control, although the role the BG play in regulation of motor behavior is not completely understood. The modeling study presented here takes advantage of available cellular, systems, and clinical data on BG and PD to begin to build a biophysically based network model of pallidosubthalamic circuits of BG, to integrate this information and better understand the physiology of the normal BG and PD pathophysiology. The model reflects the experimentally supported hypothesis that the BG are involved in facilitation of the desired motor program and inhibition of competing motor programs that interfere with the desired movement. Our model network consists of subthalamic and pallidal (both external and internal segments) neural assemblies, with inputs from cortex and striatum. Functional subsets within each of the BG nuclei correspond to the desired motor program and the unwanted motor programs. A single compartment conductance-based model represents each subset. This network can discriminate between competing signals for motor program initiation, thus facilitating a single motor program. This ability depends on metabotropic gamma-aminobutyric acid B projections from the external pallidum to subthalamic nucleus and rebound properties of subthalamic cells, as well as on the structure of projections between pallidum and subthalamus. The loss of this ability leads to hypokinesia, known PD motor deficits characterized by a slowness or inability to switch between motor programs.

Encoding of direction and combination of movements by primate putamen neurons

To perform multiple movements in a preprogrammed order, the brain needs to compute both the visuospatial and temporal organization of such multiple movements. Previous studies revealed participation of the parietal and premotor areas in visuospatial processing, and of the supplementary motor and presupplementary motor areas in temporal structuring of multiple movements. In the basal ganglia, on the other hand, relatively little has been known about how the neuronal processing of the visuospatial and temporal structuring of multiple movements occurs. In the present study, monkeys performed combinations of hand movements, either a lever turn-lever turn or lever turn-button press. Combinations of the two movements were performed under visually instructed condition first, then under remembered condition. We found that activity of 43% (30/69) and 60% (42/69) of putamen neurons was selective to the preprogrammed combination of movements and to the direction of the first movement. The neurons preferring remembered condition were mainly observed in the dorsomedial part of the putamen, where most of neurons were also selective to both direction and combination of movements, while those in the ventrolateral part of the putamen were not selective to the instructed and remembered conditions. The results supported a hypothesis that the movement direction-selective and the movement combination-selective neuron activities in the striatum may play an indispensable role in the visuospatial and temporal organization of movements through the cortico-basal ganglia loop system.

Functional correlations between neighboring neurons in the primate globus pallidus are weak or nonexistent

The anatomical structure of the basal ganglia displays topographical organization and massive funneling of neuronal projections toward the globus pallidus as well as an axonal collateral system within this nucleus. This structure suggests the formation of correlations between the spiking activities of pallidal cells. Nevertheless, previous studies of remote neurons in the pallidum have reported uncorrelated spiking activity. These correlation results may be challenged, because remote pallidal neurons may be located in different pallidal territories. To further test the independence of pallidal activity, we studied the spiking activity of neighboring pairs recorded by the same electrodes. A narrow peak dominated the correlations of all pairs of neurons recorded on the same electrode. This type of peak is classically interpreted as a sign of strong common input. However, recent mathematical analysis shows that such peaks may derive from a technical inability to detect overlapping spikes by spike-sorting techniques. A long-term shallow trough in the correlation of neighboring neurons may also result from the same effect, which we have termed the "shadowing effect." A comparison of the expected shadowing effect with the actual correlations suggests that no real correlations exist between 93.9% of neighboring pallidal pairs. The remaining 6.1% of the pairs display symmetric long-term positive correlations centered on time 0. Thus, functional interactions between neighboring pallidal neurons do not display any significant differences from the interactions between physically remote neurons in this brain area. Moreover, the combination of anatomical data and current physiological results suggests an active decorrelating process performed in the basal ganglia.

Evidence for lasting sequence segmentation in the discrete sequence-production task

It is well known that movement sequences are initiated and executed more slowly as they become longer. Those effects of sequence length, which have been found to lessen with practice, have been attributed to the development of a single motor chunk that represents the entire sequence. But an increasingly efficient distribution of programming can also explain the effects. To examine the mechanisms underlying skill in executing keying sequences, the authors examined the performance of participants (N = 18) who practiced a discrete sequence-production task involving fixed sequences of 3 and 6 key presses. Detailed examination of the effects of extensive practice, of regularities in key pressing order, and of a preceding choice RT task on the production of those sequences showed that most participants executed the 6-key sequence as 2 or more successive segments and continued to do so in the various conditions. The preceding choice RT task restored the sequence-length effect in latency that had disappeared with practice. The present results suggest that practice induces the development of motor chunks, each representing a short segment, and with longer sequences a control scheme for concatenating the motor chunks. Segmentation of longer sequences appeared to be concealed by individual segmentation differences unless there were regularities that imposed a common segmentation pattern.