Prediction of complex two-dimensional trajectories by a cerebellar model of smooth pursuit eye movement

Age-related decline of online visuomotor adaptation: a combined effect of deteriorations of motor anticipation and execution

The literature has established that the capability of visuomotor adaptation decreases with aging. However, the underlying mechanisms of this decline are yet to be fully understood. The current study addressed this issue by examining how aging affected visuomotor adaptation in a continuous manual tracking task with delayed visual feedback. To distinguish separate contributions of the declined capability of motor anticipation and deterioration of motor execution to this age-related decline, we recorded and analyzed participants' manual tracking performances and their eye movements during tracking. Twenty-nine older people and twenty-three young adults (control group) participated in this experiment. The results showed that the age-related decline of visuomotor adaptation was strongly linked to degraded performance in predictive pursuit eye movement, indicating that declined capability motor anticipation with aging had critical influences on the age-related decline of visuomotor adaptation. Additionally, deterioration of motor execution, measured by random error after controlling for the lag between target and cursor, was found to have an independent contribution to the decline of visuomotor adaptation. Taking these findings together, we see a picture that the age-related decline of visuomotor adaptation is a joint effect of the declined capability of motor anticipation and the deterioration of motor execution with aging.

Cortico-Cerebellar Hyper-Connections and Reduced Purkinje Cells Behind Abnormal Eyeblink Conditioning in a Computational Model of Autism Spectrum Disorder

Empirical evidence suggests that children with autism spectrum disorder (ASD) show abnormal behavior during delay eyeblink conditioning. They show a higher conditioned response learning rate and earlier peak latency of the conditioned response signal. The neuronal mechanisms underlying this autistic behavioral phenotype are still unclear. Here, we use a physiologically constrained spiking neuron model of the cerebellar-cortical system to investigate which features are critical to explaining atypical learning in ASD. Significantly, the computer simulations run with the model suggest that the higher conditioned responses learning rate mainly depends on the reduced number of Purkinje cells. In contrast, the earlier peak latency mainly depends on the hyper-connections of the cerebellum with sensory and motor cortex. Notably, the model has been validated by reproducing the behavioral data collected from studies with real children. Overall, this article is a starting point to understanding the link between the behavioral and neurobiological basis in ASD learning. At the end of the paper, we discuss how this knowledge could be critical for devising new treatments.

A cerebellar-based solution to the nondeterministic time delay problem in robotic control

[Figure: see text].

How the Cerebellum and Prefrontal Cortex Cooperate During Trace Eyeblinking Conditioning

Several data have demonstrated that during the widely used experimental paradigm for studying associative learning, trace eye blinking conditioning (TEBC), there is a strong interaction between cerebellum and medial prefrontal cortex (mPFC). Despite this evidence, the neural mechanisms underlying this interaction are still not clear. Here, we propose a neurophysiologically plausible computational model to address this issue. The model is constrained on the basis of two critical anatomo-physiological features: (i) the cerebello-cortical organization through two circuits, respectively, targeting M1 and mPFC; (ii) the different timing in the plasticity mechanisms of these parallel circuits produced by the granule cells time sensitivity according to which different subpopulations are active at different moments during conditioned stimuli. The computer simulations run with the model suggest that these features are critical to understand how the cooperation between cerebellum and mPFC supports motor areas during TEBC. In particular, a greater trace interval produces greater plasticity changes at the slow path synapses involving mPFC with respect to plasticity changes at the fast path involving M1. As a consequence, the greater is the trace interval, the stronger is the mPFC involvement. The model has been validated by reproducing data collected through recent real mice experiments.

World Statistics Drive Learning of Cerebellar Internal Models in Adaptive Feedback Control: A Case Study Using the Optokinetic Reflex

The cerebellum is widely implicated in having an important role in adaptive motor control. Many of the computational studies on cerebellar motor control to date have focused on the associated architecture and learning algorithms in an effort to further understand cerebellar function. In this paper we switch focus to the signals driving cerebellar adaptation that arise through different motor behavior. To do this, we investigate computationally the contribution of the cerebellum to the optokinetic reflex (OKR), a visual feedback control scheme for image stabilization. We develop a computational model of the adaptation of the cerebellar response to the world velocity signals that excite the OKR (where world velocity signals are used to emulate head velocity signals when studying the OKR in head-fixed experimental laboratory conditions). The results show that the filter learnt by the cerebellar model is highly dependent on the power spectrum of the colored noise world velocity excitation signal. Thus, the key finding here is that the cerebellar filter is determined by the statistics of the OKR excitation signal.

Implicit adaptation compensates for erratic explicit strategy in human motor learning

Sports are replete with strategies, yet coaching lore often emphasizes 'quieting the mind', 'trusting the body' and 'avoiding overthinking' in referring to the importance of relying less on high-level explicit strategies in favor of low-level implicit motor learning. We investigated the interactions between explicit strategy and implicit motor adaptation by designing a sensorimotor learning paradigm that drives adaptive changes in some dimensions but not others. We find that strategy and implicit adaptation synergize in driven dimensions, but effectively cancel each other in undriven dimensions. Independent analyses-based on time lags, the correlational structure in the data and computational modeling-demonstrate that this cancellation occurs because implicit adaptation effectively compensates for noise in explicit strategy rather than the converse, acting to clean up the motor noise resulting from low-fidelity explicit strategy during motor learning. These results provide new insight into why implicit learning increasingly takes over from explicit strategy as skill learning proceeds.

Spike burst-pause dynamics of Purkinje cells regulate sensorimotor adaptation

Cerebellar Purkinje cells mediate accurate eye movement coordination. However, it remains unclear how oculomotor adaptation depends on the interplay between the characteristic Purkinje cell response patterns, namely tonic, bursting, and spike pauses. Here, a spiking cerebellar model assesses the role of Purkinje cell firing patterns in vestibular ocular reflex (VOR) adaptation. The model captures the cerebellar microcircuit properties and it incorporates spike-based synaptic plasticity at multiple cerebellar sites. A detailed Purkinje cell model reproduces the three spike-firing patterns that are shown to regulate the cerebellar output. Our results suggest that pauses following Purkinje complex spikes (bursts) encode transient disinhibition of target medial vestibular nuclei, critically gating the vestibular signals conveyed by mossy fibres. This gating mechanism accounts for early and coarse VOR acquisition, prior to the late reflex consolidation. In addition, properly timed and sized Purkinje cell bursts allow the ratio between long-term depression and potentiation (LTD/LTP) to be finely shaped at mossy fibre-medial vestibular nuclei synapses, which optimises VOR consolidation. Tonic Purkinje cell firing maintains the consolidated VOR through time. Importantly, pauses are crucial to facilitate VOR phase-reversal learning, by reshaping previously learnt synaptic weight distributions. Altogether, these results predict that Purkinje spike burst-pause dynamics are instrumental to VOR learning and reversal adaptation.

Cerebellar Purkinje cells regulate accurate eye movement coordination. However, it remains unclear how cerebellar-dependent oculomotor adaptation depends on the interplay between Purkinje cell characteristic response patterns: tonic, high frequency bursting, and post-complex spike pauses. We explore the role of Purkinje spike burst-pause dynamics in VOR adaptation. A biophysical model of Purkinje cell is at the core of a spiking network model, which captures the cerebellar microcircuit properties and incorporates spike-based synaptic plasticity mechanisms at different cerebellar sites. We show that Purkinje spike burst-pause dynamics are critical for (1) gating the vestibular-motor response association during VOR acquisition; (2) mediating the LTD/LTP balance for VOR consolidation; (3) reshaping synaptic efficacy distributions for VOR phase-reversal adaptation; (4) explaining the reversal VOR gain discontinuities during sleeping.

Sustained sensorimotor control as intermittent decisions about prediction errors: computational framework and application to ground vehicle steering

A conceptual and computational framework is proposed for modelling of human sensorimotor control and is exemplified for the sensorimotor task of steering a car. The framework emphasises control intermittency and extends on existing models by suggesting that the nervous system implements intermittent control using a combination of (1) motor primitives, (2) prediction of sensory outcomes of motor actions, and (3) evidence accumulation of prediction errors. It is shown that approximate but useful sensory predictions in the intermittent control context can be constructed without detailed forward models, as a superposition of simple prediction primitives, resembling neurobiologically observed corollary discharges. The proposed mathematical framework allows straightforward extension to intermittent behaviour from existing one-dimensional continuous models in the linear control and ecological psychology traditions. Empirical data from a driving simulator are used in model-fitting analyses to test some of the framework's main theoretical predictions: it is shown that human steering control, in routine lane-keeping and in a demanding near-limit task, is better described as a sequence of discrete stepwise control adjustments, than as continuous control. Results on the possible roles of sensory prediction in control adjustment amplitudes, and of evidence accumulation mechanisms in control onset timing, show trends that match the theoretical predictions; these warrant further investigation. The results for the accumulation-based model align with other recent literature, in a possibly converging case against the type of threshold mechanisms that are often assumed in existing models of intermittent control.

Machine Learning Capabilities of a Simulated Cerebellum

This paper describes the learning and control capabilities of a biologically constrained bottom-up model of the mammalian cerebellum. Results are presented from six tasks: 1) eyelid conditioning; 2) pendulum balancing; 3) proportional-integral-derivative control; 4) robot balancing; 5) pattern recognition; and 6) MNIST handwritten digit recognition. These tasks span several paradigms of machine learning, including supervised learning, reinforcement learning, control, and pattern recognition. Results over these six domains indicate that the cerebellar simulation is capable of robustly identifying static input patterns even when randomized across the sensory apparatus. This capability allows the simulated cerebellum to perform several different supervised learning and control tasks. On the other hand, both reinforcement learning and temporal pattern recognition prove problematic due to the delayed nature of error signals and the simulator's inability to solve the credit assignment problem. These results are consistent with previous findings which hypothesize that in the human brain, the basal ganglia is responsible for reinforcement learning, while the cerebellum handles supervised learning.

Evaluation of Teaching Signals for Motor Control in the Cerebellum during Real-World Robot Application

Motor learning in the cerebellum is believed to entail plastic changes at synapses between parallel fibers and Purkinje cells, induced by the teaching signal conveyed in the climbing fiber (CF) input. Despite the abundant research on the cerebellum, the nature of this signal is still a matter of debate. Two types of movement error information have been proposed to be plausible teaching signals: sensory error (SE) and motor command error (ME); however, their plausibility has not been tested in the real world. Here, we conducted a comparison of different types of CF teaching signals in real-world engineering applications by using a realistic neuronal network model of the cerebellum. We employed a direct current motor (simple task) and a two-wheeled balancing robot (difficult task). We demonstrate that SE, ME or a linear combination of the two is sufficient to yield comparable performance in a simple task. When the task is more difficult, although SE slightly outperformed ME, these types of error information are all able to adequately control the robot. We categorize granular cells according to their inputs and the error signal revealing that different granule cells are preferably engaged for SE, ME or their combination. Thus, unlike previous theoretical and simulation studies that support either SE or ME, it is demonstrated for the first time in a real-world engineering application that both SE and ME are adequate as the CF teaching signal in a realistic computational cerebellar model, even when the control task is as difficult as stabilizing a two-wheeled balancing robot.

Distributed Cerebellar Motor Learning: A Spike-Timing-Dependent Plasticity Model

Deep cerebellar nuclei neurons receive both inhibitory (GABAergic) synaptic currents from Purkinje cells (within the cerebellar cortex) and excitatory (glutamatergic) synaptic currents from mossy fibers. Those two deep cerebellar nucleus inputs are thought to be also adaptive, embedding interesting properties in the framework of accurate movements. We show that distributed spike-timing-dependent plasticity mechanisms (STDP) located at different cerebellar sites (parallel fibers to Purkinje cells, mossy fibers to deep cerebellar nucleus cells, and Purkinje cells to deep cerebellar nucleus cells) in close-loop simulations provide an explanation for the complex learning properties of the cerebellum in motor learning. Concretely, we propose a new mechanistic cerebellar spiking model. In this new model, deep cerebellar nuclei embed a dual functionality: deep cerebellar nuclei acting as a gain adaptation mechanism and as a facilitator for the slow memory consolidation at mossy fibers to deep cerebellar nucleus synapses. Equipping the cerebellum with excitatory (e-STDP) and inhibitory (i-STDP) mechanisms at deep cerebellar nuclei afferents allows the accommodation of synaptic memories that were formed at parallel fibers to Purkinje cells synapses and then transferred to mossy fibers to deep cerebellar nucleus synapses. These adaptive mechanisms also contribute to modulate the deep-cerebellar-nucleus-output firing rate (output gain modulation toward optimizing its working range).

Neuronal Reward and Decision Signals: From Theories to Data

Rewards are crucial objects that induce learning, approach behavior, choices, and emotions. Whereas emotions are difficult to investigate in animals, the learning function is mediated by neuronal reward prediction error signals which implement basic constructs of reinforcement learning theory. These signals are found in dopamine neurons, which emit a global reward signal to striatum and frontal cortex, and in specific neurons in striatum, amygdala, and frontal cortex projecting to select neuronal populations. The approach and choice functions involve subjective value, which is objectively assessed by behavioral choices eliciting internal, subjective reward preferences. Utility is the formal mathematical characterization of subjective value and a prime decision variable in economic choice theory. It is coded as utility prediction error by phasic dopamine responses. Utility can incorporate various influences, including risk, delay, effort, and social interaction. Appropriate for formal decision mechanisms, rewards are coded as object value, action value, difference value, and chosen value by specific neurons. Although all reward, reinforcement, and decision variables are theoretical constructs, their neuronal signals constitute measurable physical implementations and as such confirm the validity of these concepts. The neuronal reward signals provide guidance for behavior while constraining the free will to act.

A realistic bi-hemispheric model of the cerebellum uncovers the purpose of the abundant granule cells during motor control

The cerebellar granule cells (GCs) have been proposed to perform lossless, adaptive spatio-temporal coding of incoming sensory/motor information required by downstream cerebellar circuits to support motor learning, motor coordination, and cognition. Here we use a physio-anatomically inspired bi-hemispheric cerebellar neuronal network (biCNN) to selectively enable/disable the output of GCs and evaluate the behavioral and neural consequences during three different control scenarios. The control scenarios are a simple direct current motor (1 degree of freedom: DOF), an unstable two-wheel balancing robot (2 DOFs), and a simulation model of a quadcopter (6 DOFs). Results showed that adequate control was maintained with a relatively small number of GCs (< 200) in all the control scenarios. However, the minimum number of GCs required to successfully govern each control plant increased with their complexity (i.e., DOFs). It was also shown that increasing the number of GCs resulted in higher robustness against changes in the initialization parameters of the biCNN model (i.e., synaptic connections and synaptic weights). Therefore, we suggest that the abundant GCs in the cerebellar cortex provide the computational power during the large repertoire of motor activities and motor plants the cerebellum is involved with, and bring robustness against changes in the cerebellar microcircuit (e.g., neuronal connections).

Fast convergence of learning requires plasticity between inferior olive and deep cerebellar nuclei in a manipulation task: a closed-loop robotic simulation

The cerebellum is known to play a critical role in learning relevant patterns of activity for adaptive motor control, but the underlying network mechanisms are only partly understood. The classical long-term synaptic plasticity between parallel fibers (PFs) and Purkinje cells (PCs), which is driven by the inferior olive (IO), can only account for limited aspects of learning. Recently, the role of additional forms of plasticity in the granular layer, molecular layer and deep cerebellar nuclei (DCN) has been considered. In particular, learning at DCN synapses allows for generalization, but convergence to a stable state requires hundreds of repetitions. In this paper we have explored the putative role of the IO-DCN connection by endowing it with adaptable weights and exploring its implications in a closed-loop robotic manipulation task. Our results show that IO-DCN plasticity accelerates convergence of learning by up to two orders of magnitude without conflicting with the generalization properties conferred by DCN plasticity. Thus, this model suggests that multiple distributed learning mechanisms provide a key for explaining the complex properties of procedural learning and open up new experimental questions for synaptic plasticity in the cerebellar network.

Excessive Sensitivity to Uncertain Visual Input in L-DOPA-Induced Dyskinesias in Parkinson's Disease: Further Implications for Cerebellar Involvement

When faced with visual uncertainty during motor performance, humans rely more on predictive forward models and proprioception and attribute lesser importance to the ambiguous visual feedback. Though disrupted predictive control is typical of patients with cerebellar disease, sensorimotor deficits associated with the involuntary and often unconscious nature of l-DOPA-induced dyskinesias in Parkinson’s disease (PD) suggests dyskinetic subjects may also demonstrate impaired predictive motor control.

Methods: We investigated the motor performance of 9 dyskinetic and 10 non-dyskinetic PD subjects on and off l-DOPA, and of 10 age-matched control subjects, during a large-amplitude, overlearned, visually guided tracking task. Ambiguous visual feedback was introduced by adding “jitter” to a moving target that followed a Lissajous pattern. Root mean square (RMS) tracking error was calculated, and ANOVA, robust multivariate linear regression, and linear dynamical system analyses were used to determine the contribution of speed and ambiguity to tracking performance.

Results: Increasing target ambiguity and speed contributed significantly more to the RMS error of dyskinetic subjects off medication. l-DOPA improved the RMS tracking performance of both PD groups. At higher speeds, controls and PDs without dyskinesia were able to effectively de-weight ambiguous visual information.

Conclusion: PDs’ visually guided motor performance degrades with visual jitter and speed of movement to a greater degree compared to age-matched controls. However, there are fundamental differences in PDs with and without dyskinesia: subjects without dyskinesia are generally slow, and less responsive to dynamic changes in motor task requirements, but in PDs with dyskinesia, there was a trade-off between overall performance and inappropriate reliance on ambiguous visual feedback. This is likely associated with functional changes in posterior parietal–ponto–cerebellar pathways.

Evidence for a retinal velocity memory underlying the direction of anticipatory smooth pursuit eye movements

To compute spatially correct smooth pursuit eye movements, the brain uses both retinal motion and extraretinal signals about the eyes and head in space (Blohm and Lefèvre 2010). However, when smooth eye movements rely solely on memorized target velocity, such as during anticipatory pursuit, it is unknown if this velocity memory also accounts for extraretinal information, such as head roll and ocular torsion. To answer this question, we used a novel behavioral updating paradigm in which participants pursued a repetitive, spatially constant fixation-gap-ramp stimulus in series of five trials. During the first four trials, participants' heads were rolled toward one shoulder, inducing ocular counterroll (OCR). With each repetition, participants increased their anticipatory pursuit gain, indicating a robust encoding of velocity memory. On the fifth trial, they rolled their heads to the opposite shoulder before pursuit, also inducing changes in ocular torsion. Consequently, for spatially accurate anticipatory pursuit, the velocity memory had to be updated across changes in head roll and ocular torsion. We tested how the velocity memory accounted for head roll and OCR by observing the effects of changes to these signals on anticipatory trajectories of the memory decoding (fifth) trials. We found that anticipatory pursuit was updated for changes in head roll; however, we observed no evidence of compensation for OCR, representing the absence of ocular torsion signals within the velocity memory. This indicated that the directional component of the memory must be coded retinally and updated to account for changes in head roll, but not OCR.

Physiology of clinical dysfunction of the cerebellum

Experimental and theoretical research into cerebellar function has begun to converge toward understanding the cerebellum as a "controller" in the engineering sense. The purpose of a controller is to convert high-level intent commands and information describing the current state of a system into low-level control signals suitable for maintaining or changing system behavior. The cerebellar subsystem appears to play this role for parts of the body and other parts of the brain. As with engineering controllers, fundamental functions include stabilization at a fixed posture or state, adjustment of movement or transition amplitude, facilitation of movement/transition speed and crispness of launch and braking, improvement of resistance to disturbances, coordination of control across multiple degrees of freedom, and assistance with estimation and/or prediction of current and future system states. As with adaptive engineering controllers, the cerebellar subsystem also readily tunes itself over time. At a more detailed level, many of the specific actions of cerebellar circuits can be understood in terms of proportional (P), integrator-like (I), and differentiator-like (D) signal processing which are fundamental components of many engineering control systems. This chapter presents an integrated, mechanistic view of ataxia, tremor, and several cerebellar oculomotor signs in terms of PID control and the neural centers that appear to subserve these functions. It also suggests the manner in which impairments in motor learning, perception, and cognition that are associated with cerebellar dysfunction may be viewed from a similar perspective.

ADAPTIVE CEREBELLAR SPIKING MODEL EMBEDDED IN THE CONTROL LOOP: CONTEXT SWITCHING AND ROBUSTNESS AGAINST NOISE

This work evaluates the capability of a spiking cerebellar model embedded in different loop architectures (recurrent, forward, and forward&recurrent) to control a robotic arm (three degrees of freedom) using a biologically-inspired approach. The implemented spiking network relies on synaptic plasticity (long-term potentiation and long-term depression) to adapt and cope with perturbations in the manipulation scenario: changes in dynamics and kinematics of the simulated robot. Furthermore, the effect of several degrees of noise in the cerebellar input pathway (mossy fibers) was assessed depending on the employed control architecture. The implemented cerebellar model managed to adapt in the three control architectures to different dynamics and kinematics providing corrective actions for more accurate movements. According to the obtained results, coupling both control architectures (forward&recurrent) provides benefits of the two of them and leads to a higher robustness against noise.

Eye movements: the past 25 years

This article reviews the past 25 years of research on eye movements (1986-2011). Emphasis is on three oculomotor behaviors: gaze control, smooth pursuit and saccades, and on their interactions with vision. Focus over the past 25 years has remained on the fundamental and classical questions: What are the mechanisms that keep gaze stable with either stationary or moving targets? How does the motion of the image on the retina affect vision? Where do we look - and why - when performing a complex task? How can the world appear clear and stable despite continual movements of the eyes? The past 25 years of investigation of these questions has seen progress and transformations at all levels due to new approaches (behavioral, neural and theoretical) aimed at studying how eye movements cope with real-world visual and cognitive demands. The work has led to a better understanding of how prediction, learning and attention work with sensory signals to contribute to the effective operation of eye movements in visually rich environments.

Cerebellar input configuration toward object model abstraction in manipulation tasks

It is widely assumed that the cerebellum is one of the main nervous centers involved in correcting and refining planned movement and accounting for disturbances occurring during movement, for instance, due to the manipulation of objects which affect the kinematics and dynamics of the robot-arm plant model. In this brief, we evaluate a way in which a cerebellar-like structure can store a model in the granular and molecular layers. Furthermore, we study how its microstructure and input representations (context labels and sensorimotor signals) can efficiently support model abstraction toward delivering accurate corrective torque values for increasing precision during different-object manipulation. We also describe how the explicit (object-related input labels) and implicit state input representations (sensorimotor signals) complement each other to better handle different models and allow interpolation between two already stored models. This facilitates accurate corrections during manipulations of new objects taking advantage of already stored models.

Cerebellarlike corrective model inference engine for manipulation tasks

This paper presents how a simple cerebellumlike architecture can infer corrective models in the framework of a control task when manipulating objects that significantly affect the dynamics model of the system. The main motivation of this paper is to evaluate a simplified bio-mimetic approach in the framework of a manipulation task. More concretely, the paper focuses on how the model inference process takes place within a feedforward control loop based on the cerebellar structure and on how these internal models are built up by means of biologically plausible synaptic adaptation mechanisms. This kind of investigation may provide clues on how biology achieves accurate control of non-stiff-joint robot with low-power actuators which involve controlling systems with high inertial components. This paper studies how a basic temporal-correlation kernel including long-term depression (LTD) and a constant long-term potentiation (LTP) at parallel fiber-Purkinje cell synapses can effectively infer corrective models. We evaluate how this spike-timing-dependent plasticity correlates sensorimotor activity arriving through the parallel fibers with teaching signals (dependent on error estimates) arriving through the climbing fibers from the inferior olive. This paper addresses the study of how these LTD and LTP components need to be well balanced with each other to achieve accurate learning. This is of interest to evaluate the relevant role of homeostatic mechanisms in biological systems where adaptation occurs in a distributed manner. Furthermore, we illustrate how the temporal-correlation kernel can also work in the presence of transmission delays in sensorimotor pathways. We use a cerebellumlike spiking neural network which stores the corrective models as well-structured weight patterns distributed among the parallel fibers to Purkinje cell connections.

The cerebellum: a neural system for the study of reinforcement learning

In its strictest application, the term “reinforcement learning” refers to a computational approach to learning in which an agent (often a machine) interacts with a mutable environment to maximize reward through trial and error. The approach borrows essentials from several fields, most notably Computer Science, Behavioral Neuroscience, and Psychology. At the most basic level, a neural system capable of mediating reinforcement learning must be able to acquire sensory information about the external environment and internal milieu (either directly or through connectivities with other brain regions), must be able to select a behavior to be executed, and must be capable of providing evaluative feedback about the success of that behavior. Given that Psychology informs us that reinforcers, both positive and negative, are stimuli or consequences that increase the probability that the immediately antecedent behavior will be repeated and that reinforcer strength or viability is modulated by the organism's past experience with the reinforcer, its affect, and even the state of its muscles (e.g., eyes open or closed); it is the case that any neural system that supports reinforcement learning must also be sensitive to these same considerations. Once learning is established, such a neural system must finally be able to maintain continued response expression and prevent response drift. In this report, we examine both historical and recent evidence that the cerebellum satisfies all of these requirements. While we report evidence from a variety of learning paradigms, the majority of our discussion will focus on classical conditioning of the rabbit eye blink response as an ideal model system for the study of reinforcement and reinforcement learning.

Neural mechanisms of observational learning

Individuals can learn by interacting with the environment and experiencing a difference between predicted and obtained outcomes (prediction error). However, many species also learn by observing the actions and outcomes of others. In contrast to individual learning, observational learning cannot be based on directly experienced outcome prediction errors. Accordingly, the behavioral and neural mechanisms of learning through observation remain elusive. Here we propose that human observational learning can be explained by two previously uncharacterized forms of prediction error, observational action prediction errors (the actual minus the predicted choice of others) and observational outcome prediction errors (the actual minus predicted outcome received by others). In a functional MRI experiment, we found that brain activity in the dorsolateral prefrontal cortex and the ventromedial prefrontal cortex respectively corresponded to these two distinct observational learning signals.

Using input minimization to train a cerebellar model to simulate regulation of smooth pursuit

Cerebellar learning appears to be driven by motor error, but whether or not error signals are provided by climbing fibers (CFs) remains a matter of controversy. Here we show that a model of the cerebellum can be trained to simulate the regulation of smooth pursuit eye movements by minimizing its inputs from parallel fibers (PFs), which carry various signals including error and efference copy. The CF spikes act as "learn now" signals. The model can be trained to simulate the regulation of smooth pursuit of visual objects following circular or complex trajectories and provides insight into how Purkinje cells might encode pursuit parameters. In minimizing both error and efference copy, the model demonstrates how cerebellar learning through PF input minimization (InMin) can make movements more accurate and more efficient. An experimental test is derived that would distinguish InMin from other models of cerebellar learning which assume that CFs carry error signals.

Dynamic correspondence between Purkinje cell discharge and forelimb muscle activity during reaching

There remain conflicting models of the cerebellar control of limb movement, ranging from the suggestion that the inhibitory output from Purkinje cells (PCs) is meant to suppress unwanted muscle activity, to the hypothesis that the cerebellar cortex embodies complex internal models of limb dynamics. To test these ideas, we undertook a quantitative comparison of PC simple spike dynamics to those of muscle activity. We recorded simultaneously from Purkinje cells in the paravermal anterior lobe and from muscles of the hand and arm in the behaving monkey during a simple, sequential button pressing task. The task-related discharge of each neuron was determined from peri-event histograms aligned to the onset of the behavior. Bursts of discharge were more than twice as common as pauses, but there was no difference in their timing relative to movement. From the same recordings, the similarity between discharge and muscle activity was evaluated by calculating the cross correlation between firing rate and rectified EMG. Surprisingly, given the inhibitory projection of PCs, most of the bursts of PC discharge were positively correlated with muscle activity. Although our results do not support a simple correspondence of pauses and bursts with limb acceleration and deceleration respectively, they are consistent with a more complex PC regulation of cerebellar nuclear activity from task-related, corticopontine drive.

Adaptive-filter models of the cerebellum: computational analysis

Many current models of the cerebellar cortical microcircuit are equivalent to an adaptive filter using the covariance learning rule. The adaptive filter is a development of the original Marr-Albus framework that deals naturally with continuous time-varying signals, thus addressing the issue of 'timing' in cerebellar function, and it can be connected in a variety of ways to other parts of the system, consistent with the microzonal organization of cerebellar cortex. However, its computational capacities are not well understood. Here we summarise the results of recent work that has focused on two of its intrinsic properties. First, an adaptive filter seeks to decorrelate its (mossy fibre) inputs from a (climbing fibre) teaching signal. This procedure can be used both for sensory processing, e.g. removal of interference from sensory signals, and for learning accurate motor commands, by decorrelating an efference copy of those commands from a sensory signal of inaccuracy. As a model of the cerebellum the adaptive filter thus forms a natural link between events at the cellular level, such as forms of synaptic plasticity and the learning rules they embody, and intelligent behaviour at the system level. Secondly, it has been shown that the covariance learning rule enables the filter to handle input and intrinsic noise optimally. Such optimality may underlie the recently described role of the cerebellum in producing accurate smooth pursuit eye movements in the face of sensory noise. Moreover, it has the consequence of driving most input weights to very small values, consistent with experimental data that many parallel-fibre synapses are normally silent. The effectiveness of silent synapses can only be altered by LTP, so learning tasks depending on a reduction of Purkinje cell firing require the synapses to be embedded in a second, inhibitory pathway from parallel fibre to Purkinje cell. This pathway and the appropriate climbing-fibre related plasticity have been described experimentally, and its presence has implications for asymmetries and hysteresis in behavioural learning rates that are also consistent with experimental observations. These computational properties of the adaptive filter suggest that it is both powerful and realistic enough to be a suitable candidate model of the cerebellar cortical microcircuit.

Cerebellar-inspired adaptive control of a robot eye actuated by pneumatic artificial muscles

In this paper, a model of cerebellar function is implemented and evaluated in the control of a robot eye actuated by pneumatic artificial muscles. The investigated control problem is stabilization of the visual image in response to disturbances. This is analogous to the vestibuloocular reflex (VOR) in humans. The cerebellar model is structurally based on the adaptive filter, and the learning rule is computationally analogous to least-mean squares, where parameter adaptation at the parallel fiber/Purkinje cell synapse is driven by the correlation of the sensory error signal (carried by the climbing fiber) and the motor command signal. Convergence of the algorithm is first analyzed in simulation on a model of the robot and then tested online in both one and two degrees of freedom. The results show that this model of neural function successfully works on a real-world problem, providing empirical evidence for validating: 1) the generic cerebellar learning algorithm; 2) the function of the cerebellum in the VOR; and 3) the signal transmission between functional neural components of the VOR.

A real-time spiking cerebellum model for learning robot control

We describe a neural network model of the cerebellum based on integrate-and-fire spiking neurons with conductance-based synapses. The neuron characteristics are derived from our earlier detailed models of the different cerebellar neurons. We tested the cerebellum model in a real-time control application with a robotic platform. Delays were introduced in the different sensorimotor pathways according to the biological system. The main plasticity in the cerebellar model is a spike-timing dependent plasticity (STDP) at the parallel fiber to Purkinje cell connections. This STDP is driven by the inferior olive (IO) activity, which encodes an error signal using a novel probabilistic low frequency model. We demonstrate the cerebellar model in a robot control system using a target-reaching task. We test whether the system learns to reach different target positions in a non-destructive way, therefore abstracting a general dynamics model. To test the system's ability to self-adapt to different dynamical situations, we present results obtained after changing the dynamics of the robotic platform significantly (its friction and load). The experimental results show that the cerebellar-based system is able to adapt dynamically to different contexts.

Silent synapses, LTP, and the indirect parallel-fibre pathway: computational consequences of optimal cerebellar noise-processing

Computational analysis of neural systems is at its most useful when it uncovers principles that provide a unified account of phenomena across multiple scales and levels of description. Here we analyse a widely used model of the cerebellar contribution to sensori-motor learning to demonstrate both that its response to intrinsic and sensor noise is optimal, and that the unexpected synaptic and behavioural consequences of this optimality can explain a wide range of experimental data. The response of the Marr-Albus adaptive-filter model of the cerebellar microcircuit to noise was examined in the context of vestibulo-ocular reflex calibration. We found that, when appropriately connected, an adaptive-filter model using the covariance learning rule to adjust the weights of synapses between parallel fibres and Purkinje cells learns weight values that are optimal given the relative amount of signal and noise carried by each parallel fibre. This optimality principle is consistent with data on the cerebellar role in smooth pursuit eye movements, and predicts that many synaptic weights must be very small, providing an explanation for the experimentally observed preponderance of silent synapses. Such a preponderance has in its turn two further consequences. First, an additional inhibitory pathway from parallel fibre to Purkinje cell is required if Purkinje cell activity is to be altered in either direction from a starting point of silent synapses. Second, cerebellar learning tasks must often proceed via LTP, rather than LTD as is widely assumed. Taken together, these considerations have profound behavioural consequences, including the optimal combination of sensori-motor information, and asymmetry and hysteresis of sensori-motor learning rates.

The cerebellum or “little brain” is a fist-sized structure located towards the rear of the brain, containing as many neurons as the rest of the brain combined, whose functions include learning to perform skilled motor tasks accurately and automatically. It is wired up into repeating microcircuits, sometimes referred to as cerebellar chips, in which learning alters the strength of the synapses between the parallel fibres, which carry input information, and the large Purkinje cell neurons, which produce outputs contributing to skilled movements. The cerebellar chip has a striking resemblance to a mathematical structure called an adaptive filter used by control engineers, and we have used this analogy to analyse its information-processing properties. We show that it learns synaptic strengths that minimise the errors in performance caused by unavoidable noise in sensors and cerebellar circuitry. Optimality principles of this kind have proved to be powerful tools for understanding complex systems. Here we show that optimality explains neuronal-level features of cerebellar learning such as the mysterious preponderance of “silent” synapses between parallel fibres and Purkinje cells and behavioural-level features such as the dependence of rate of learning of a motor skill on learning history.

Embodied models of delayed neural responses: spatiotemporal categorization and predictive motor control in brain based devices

In order to respond appropriately to environmental stimuli, organisms must integrate over time spatiotemporal signals that reflect object motion and self-movement. One possible mechanism to achieve this spatiotemporal transformation is to delay or lag neural responses. This paper reviews our recent modeling work testing the sufficiency of delayed responses in the nervous system in two different behavioral tasks: (1) Categorizing spatiotemporal tactile cues with thalamic "lag" cells and downstream coincidence detectors, and (2) Predictive motor control was achieved by the cerebellum through a delayed eligibility trace rule at cerebellar synapses. Since the timing of these neural signals must closely match real-world dynamics, we tested these ideas using the brain based device (BBD) approach in which a simulated nervous system is embodied in a robotic device. In both tasks, biologically inspired neural simulations with delayed neural responses were critical for successful behavior by the device.

Cerebellar motor learning: when is cortical plasticity not enough?

Classical Marr-Albus theories of cerebellar learning employ only cortical sites of plasticity. However, tests of these theories using adaptive calibration of the vestibulo-ocular reflex (VOR) have indicated plasticity in both cerebellar cortex and the brainstem. To resolve this long-standing conflict, we attempted to identify the computational role of the brainstem site, by using an adaptive filter version of the cerebellar microcircuit to model VOR calibration for changes in the oculomotor plant. With only cortical plasticity, introducing a realistic delay in the retinal-slip error signal of 100 ms prevented learning at frequencies higher than 2.5 Hz, although the VOR itself is accurate up to at least 25 Hz. However, the introduction of an additional brainstem site of plasticity, driven by the correlation between cerebellar and vestibular inputs, overcame the 2.5 Hz limitation and allowed learning of accurate high-frequency gains. This "cortex-first" learning mechanism is consistent with a wide variety of evidence concerning the role of the flocculus in VOR calibration, and complements rather than replaces the previously proposed "brainstem-first" mechanism that operates when ocular tracking mechanisms are effective. These results (i) describe a process whereby information originally learnt in one area of the brain (cerebellar cortex) can be transferred and expressed in another (brainstem), and (ii) indicate for the first time why a brainstem site of plasticity is actually required by Marr-Albus type models when high-frequency gains must be learned in the presence of error delay.

Synchronized firing among retinal ganglion cells signals motion reversal

We show that when a moving object suddenly reverses direction, there is a brief, synchronous burst of firing within a population of retinal ganglion cells. This burst can be driven by either the leading or trailing edge of the object. The latency is constant for movement at different speeds, objects of different size, and bright versus dark contrasts. The same ganglion cells that signal a motion reversal also respond to smooth motion. We show that the brain can build a pure reversal detector using only a linear filter that reads out synchrony from a group of ganglion cells. These results indicate that not only can the retina anticipate the location of a smoothly moving object, but that it can also signal violations in its own prediction. We show that the reversal response cannot be explained by models of the classical receptive field and suggest that nonlinear receptive field subunits may be responsible.

Evolution of motor memory during the seconds after observation of motor error

When a movement results in error, the nervous system amends the motor commands that generate the subsequent movement. Here we show that this adaptation depends not just on error, but also on passage of time between the two movements. We observed that subjects learned a reaching task faster, i.e., with fewer trials, when the intertrial time intervals (ITIs) were lengthened. We hypothesized two computational mechanisms that could have accounted for this. First, learning could have been driven by a Bayesian process where the learner assumed that errors are the result of perturbations that have multiple timescales. In theory, longer ITIs can produce faster learning because passage of time might increase uncertainty, which in turn increases sensitivity to error. Second, error in a trial may result in a trace that decays with time. If the learner continued to sample from the trace during the ITI, then adaptation would increase with increased ITIs. The two models made separate predictions: The Bayesian model predicted that when movements are separated by random ITIs, the learner would learn most from a trial that followed a long time interval. In contrast, the trace model predicted that the learner would learn most from a trial that preceded a long time interval. We performed two experiments to test for these predictions and in both experiments found evidence for the trace model. We suggest that motor error produces an error memory trace that decays with a time constant of about 4 s, continuously promoting adaptation until the next movement.

Semianalytical transient solution of a delayed differential equation and its application to the tracking motion in the sensory-motor system

We derived semianalytically the transient solution of a delayed differential equation that had been shown to be a simple but good model of the sensory-motor system. In the present Brief Report, we applied this transient solution for studying the global nature of the transient tracking motion when visual target information is changed suddenly. The results clarified that the dynamic error minimization principle in hand motion observed experimentally is robust over a wide range of the parameter space of the delay time, the time constant, and the feedforward parameter.

Computation of inverse functions in a model of cerebellar and reflex pathways allows to control a mobile mechanical segment

The command and control of limb movements by the cerebellar and reflex pathways are modeled by means of a circuit whose structure is deduced from functional constraints. One constraint is that fast limb movements must be accurate although they cannot be continuously controlled in closed loop by use of sensory signals. Thus, the pathways which process the motor orders must contain approximate inverse functions of the bio-mechanical functions of the limb and of the muscles. This can be achieved by means of parallel feedback loops, whose pattern turns out to be comparable to the anatomy of the cerebellar pathways. They contain neural networks able to anticipate the motor consequences of the motor orders, modeled by artificial neural networks whose connectivity is similar to that of the cerebellar cortex. These networks learn the direct biomechanical functions of the limbs and muscles by means of a supervised learning process. Teaching signals calculated from motor errors are sent to the learning sites, as, in the cerebellum, complex spikes issued from the inferior olive are conveyed to the Purkinje cells by climbing fibers. Learning rules are deduced by a differential calculation, as classical gradient rules, and they account for the long term depression which takes place in the dendritic arborizations of the Purkinje cells. Another constraint is that reflexes must not impede voluntary movements while remaining at any instant ready to oppose perturbations. Therefore, efferent copies of the motor orders are sent to the interneurones of the reflexes, where they cancel the sensory-motor consequences of the voluntary movements. After learning, the model is able to drive accurately, both in velocity and position, angular movements of a rod actuated by two pneumatic McKibben muscles. Reflexes comparable to the myotatic and tendinous reflexes, and stabilizing reactions comparable to the cerebellar sensory-motor reactions, reduce efficiently the effects of perturbing torques. These results allow to link the behavioral concepts of the equilibrium-point "lambda model" [J Motor Behav 18 (1986) 17] with anatomical and physiological features: gains of reflexes and sensori-motor reactions set the slope of the "invariant characteristic," and efferent copies set the "threshold of the stretch reflex." Thus, mathematical and physical laws account for the raison d'etre of the inhibitory nature of Purkinje cells and for the conspicuous anatomical pattern of the cerebellar pathways. These properties of these pathways allow to perform approximate inverse calculations after learning of direct functions, and insure also the coordination of voluntary and reflex motor orders.

Associative short-term synaptic plasticity mediated by endocannabinoids

Associative learning is important on rapid timescales, but no suitable form of short-term plasticity has been identified that is both associative and synapse specific. Here, we assess whether endocannabinoids can mediate such plasticity. In the cerebellum, bursts of parallel fiber (PF) activity evoke endocannabinoid release from Purkinje cell dendrites that results in retrograde synaptic inhibition lasting seconds. We find that the powerful climbing fiber (CF) to Purkinje cell synapse regulates this inhibition. Compared to PF stimulation alone, coactivation of PF and CF synapses greatly enhanced endocannabinoid-mediated inhibition of PF synapses. Retrograde inhibition was restricted to PFs activated within several hundred milliseconds of CF activation. This associative plasticity reflects two aspects of calcium-dependent endocannabinoid release. First, PF-mediated activation of metabotropic glutamate receptors locally reduced the dendritic calcium levels required for endocannabinoid release. Second, CF and PF coactivation evoked localized supralinear dendritic calcium signals. Thus, endocannabinoids mediate transient associative synaptic plasticity.

Temporal dynamics of cerebro-cerebellar network recruitment during a cognitive task

Previous investigations have demonstrated that two regions in the right cerebellum, one located superiorly in hemispheral lobule VI/Crus I and another located inferiorly in hemispheral lobule VIIB/VIIIA, are activated during verbal working memory performance. On the basis of functional neuroimaging patterns of activation, as well as known cortico-pontine and ponto-cerebellar projections, the superior region has been hypothesized to contribute to the articulatory control system of working memory whereas the inferior region has been linked to the phonological store. The present study used event-related fMRI and individual estimates of hemodynamic response for both the cerebellum and neocortex to test this model and characterize the task phase specific cerebro-cerebellar activations for a Sternberg verbal working memory task. Results demonstrated that the right superior cerebellum showed the strongest activation during the initial encoding phase of the task, and, consistent with predictions, a similar pattern was observed in left opercular inferior frontal and premotor regions. In contrast, the right inferior cerebellum exhibited the greatest activation during the maintenance phase of the task, and as predicted, corresponded with activation in the left inferior parietal lobule. The significance of the results with respect to cerebro-cerebellar models of verbal working memory and to theoretical accounts of cerebellar involvement in cognition is discussed.

Human hand moves proactively to the external stimulus: an evolutional strategy for minimizing transient error

We investigated particularly the proactive nature of the visual-motor system by steady and transient experiments of a hand-tracking task, and confirmed that the hand motion precedes on the average the target motion in steady runs within a finite frequency range of the sinusoidal target motion. The question why and how much the hand motion should precede was answered by frequency-jump experiments. The results implied that the positive phase shift of the hand motion represents the proactive nature of the visual-motor control system which is adaptationally developed for each person to minimize the transient error of the hand motion when the target motion changes unexpectedly.

A model of cerebellum stabilized and scheduled hybrid long-loop control of upright balance

A recurrent integrator proportional integral derivative (PID) model that has been used to account for cerebrocerebellar stabilization and scaling of transcortical proprioceptive feedback in the control of horizontal planar arm movements has been augmented with long-loop force feedback and gainscheduling to describe the control of human upright balance. The cerebellar component of the controller is represented by two sets of gains that each provide linear scaling of same-joint and interjoint long-loop stretch responses between ankle, knee, and hip. The cerebral component of the model includes a single set of same-joint linear force feedback gains. Responses to platform translations of a three-segment body model operating under this hybrid proprioception and force-based long-loop control were simulated. With low-velocity platform disturbances, "ankle-strategy"-type postural recovery kinematics and electromyogram (EMG) patterns were generated using the first set of cerebeller control gains. With faster disturbances, balance was maintained by including the second set of gains cerebellar control gains that yielded "mixed ankle-hip strategy"-type kinematics and EMG patterns. The addition of small amounts of simulated muscular coactivation improved the fit to certain human datasets. It is proposed that the cerebellum switches control gainsets as a function of sensed body kinematic state. Reduction of cerebellar gains with a compensatory increase in muscular stiffness yielded posture recovery with abnormal motions consistent with those found in cerebellar disease. The model demonstrates that stabilized hybrid long-loop feedback with scheduling of linear gains may afford realistic balance control in the absence of explicit internal dynamics models and suggests that the cerebellum and cerebral cortex may contribute to balance control by such a mechanism.

Recurrent cerebellar architecture solves the motor-error problem

Current views of cerebellar function have been heavily influenced by the models of Marr and Albus, who suggested that the climbing fibre input to the cerebellum acts as a teaching signal for motor learning. It is commonly assumed that this teaching signal must be motor error (the difference between actual and correct motor command), but this approach requires complex neural structures to estimate unobservable motor error from its observed sensory consequences. We have proposed elsewhere a recurrent decorrelation control architecture in which Marr-Albus models learn without requiring motor error. Here, we prove convergence for this architecture and demonstrate important advantages for the modular control of systems with multiple degrees of freedom. These results are illustrated by modelling adaptive plant compensation for the three-dimensional vestibular ocular reflex. This provides a functional role for recurrent cerebellar connectivity, which may be a generic anatomical feature of projections between regions of cerebral and cerebellar cortex.

Visual awareness and the cerebellum: possible role of decorrelation control

The two roles in awareness most often suggested for the cerebellum are (i) keeping the details of motor skills away from forebrain computation, and (ii) signaling to the forebrain when a sensory event is not predictable from prior motor commands. However, it is unclear how current models of the cerebellum could carry out these roles. Their architecture, based on the seminal ideas of Marr and Albus, appears to need 'motor error' to learn correct motor commands. However, since motor error is the difference between the actual motor command and what the command should have been, it is a signal unavailable to the organism in principle. We propose a possible solution to this problem, termed decorrelation control, in which the cerebellum learns to decorrelate the motor command sent to the muscles from the sensory consequences of motor error. This method was tested in a linear model of oculomotor plant compensation in the vestibulo-ocular reflex. A copy of the eye-movement command was sent as mossy-fiber input to the flocculus, represented as a simple adaptive filter version of the Marr-Albus architecture. The sensory consequences of motor error were retinal slip, delivered as climbing fiber input to the flocculus. A standard anti-Hebbian learning rule was used to decorrelate the two. Simulations of the linearized problem showed the method to be effective and robust for plant compensation. Decorrelation control is thus a candidate algorithm for the basic cerebellar microcircuit, indicating how it could achieve motor learning using only signals available to the system. Such learning might then enable the cerebellum to free up visual awareness, and also, by providing a sensory signal decorrelated from motor command, supply awareness with crucial information about the external world.

Evidence for object permanence in the smooth-pursuit eye movements of monkeys

We recorded the smooth-pursuit eye movements of monkeys in response to targets that were extinguished (blinked) for 200 ms in mid-trajectory. Eye velocity declined considerably during the target blinks, even when the blinks were completely predictable in time and space. Eye velocity declined whether blinks were presented during steady-state pursuit of a constant-velocity target, during initiation of pursuit before target velocity was reached, or during eye accelerations induced by a change in target velocity. When a physical occluder covered the trajectory of the target during blinks, creating the impression that the target moved behind it, the decline in eye velocity was reduced or abolished. If the target was occluded once the eye had reached target velocity, pursuit was only slightly poorer than normal, uninterrupted pursuit. In contrast, if the target was occluded during the initiation of pursuit, while the eye was accelerating toward target velocity, pursuit during occlusion was very different from normal pursuit. Eye velocity remained relatively stable during target occlusion, showing much less acceleration than normal pursuit and much less of a decline than was produced by a target blink. Anticipatory or predictive eye acceleration was typically observed just prior to the reappearance of the target. Computer simulations show that these results are best understood by assuming that a mechanism of eye-velocity memory remains engaged during target occlusion but is disengaged during target blinks.

What the cerebellum computes

The brain is an organ that processes information. Brain systems such as the cerebellum receive inputs from other systems and generate outputs according to their internal rules of information processing. Thus, our understanding of the cerebellum is ultimately best expressed in terms of the information processing it accomplishes and how cerebellar neurons and synapses produce this processing. We review evidence that indicates how Pavlovian eyelid conditioning reveals cerebellar processing to be an example of feedforward control. Eyelid conditioning demonstrates a capacity for learning in the cerebellum that is error driven, associative and temporally specific--as is required for feedforward control. This computation-centered view is consistent with a variety of proposed functions of the cerebellum, including sensory-motor integration, motor coordination, motor learning and timing. Moreover, feedforward processing could be the common link between motor and non-motor functions of the cerebellum.