From brainstem to cortex: computational models of saccade generation circuitry

Preparatory activity and the expansive null-space

The study of the cortical control of movement experienced a conceptual shift over recent decades, as the basic currency of understanding shifted from single-neuron tuning towards population-level factors and their dynamics. This transition was informed by a maturing understanding of recurrent networks, where mechanism is often characterized in terms of population-level factors. By estimating factors from data, experimenters could test network-inspired hypotheses. Central to such hypotheses are 'output-null' factors that do not directly drive motor outputs yet are essential to the overall computation. In this Review, we highlight how the hypothesis of output-null factors was motivated by the venerable observation that motor-cortex neurons are active during movement preparation, well before movement begins. We discuss how output-null factors then became similarly central to understanding neural activity during movement. We discuss how this conceptual framework provided key analysis tools, making it possible for experimenters to address long-standing questions regarding motor control. We highlight an intriguing trend: as experimental and theoretical discoveries accumulate, the range of computational roles hypothesized to be subserved by output-null factors continues to expand.

Differential eye movement features between Alzheimer's disease patients with and without depressive symptoms

BACKGROUND

Accurately diagnosing depressive symptoms in Alzheimer's disease (AD) patients is often challenging. Eye movement parameters have been demonstrated as biomarkers for assessing cognition and psychological conditions.

AIM

To investigate the differences in eye movement between AD patients with and without depressive symptoms.

METHODS

Eye movement data of 65 AD patients were compared between the depressed AD (D-AD) and non-depressed AD (nD-AD) groups. Logistic regression analysis was employed to identify diagnostic biomarkers and the ROC curve was plotted. The correlation between eye movement and HAMD-17 scores was assessed by partial correlation analysis.

RESULTS

The D-AD patients showed longer saccade latency and faster average/peak saccade velocities in the overlap prosaccade test, longer average reaction time and faster average saccade velocity in the gap prosaccade test, longer start-up durations, slower pursuit velocity, more offsets, and larger total offset degrees in the smooth pursuit test, and poorer fixation stability in both the central and lateral fixation tests compared to nD-AD patients. The start-up duration in the smooth pursuit test and the number of offsets in the central fixation test were identified as the diagnostic eye movement parameters for D-AD patients with the area under the ROC curves of 0.8011. Partial correlation analysis revealed that the start-up duration and pursuit velocity in the smooth pursuit test and the total offset degrees in the lateral fixation test were correlated with HAMD-17 scores in D-AD patients.

DISCUSSION AND CONCLUSIONS

Eye movement differences may help to differentiate D-AD patients from nD-AD patients in a non-invasive and cost-effective manner.

Regularities in vertical saccadic metrics: new insights, and future perspectives

Introduction

Asymmetries in processing by the healthy brain demonstrate regularities that facilitate the modeling of brain operations. The goal of the present study was to determine asymmetries in saccadic metrics during visual exploration, devoid of confounding clutter in the visual field.

Methods

Twenty healthy adults searched for a small, low-contrast gaze-contingent target on a blank computer screen. The target was visible, only if eye fixation was within a 5 deg. by 5 deg. area of the target's location.

Results

Replicating previously-reported asymmetries, repeated measures contrast analyses indicated that up-directed saccades were executed earlier, were smaller in amplitude, and had greater probability than down-directed saccades. Given that saccade velocities are confounded by saccade amplitudes, it was also useful to investigate saccade kinematics of visual exploration, as a function of vertical saccade direction. Saccade kinematics were modeled for each participant, as a square root relationship between average saccade velocity (i.e., average velocity between launching and landing of a saccade) and corresponding saccade amplitude (Velocity = S*[Saccade Amplitude]^0.5). A comparison of the vertical scaling parameter (S) for up- and down-directed saccades showed that up-directed saccades tended to be slower than down-directed ones.

Discussion

To motivate future research, an ecological theory of asymmetric pre-saccadic inhibition was presented to explain the collection of vertical saccadic regularities. For example, given that the theory proposes strong inhibition for the releasing of reflexive down-directed prosaccades (cued by an attracting peripheral target below eye fixation), and weak inhibition for the releasing of up-directed prosaccades (cued by an attracting peripheral target above eye fixation), a prediction for future studies is longer reaction times for vertical anti-saccade cues above eye fixation. Finally, the present study with healthy individuals demonstrates a rationale for further study of vertical saccades in psychiatric disorders, as bio-markers for brain pathology.

Head Orientation Influences Saccade Directions during Free Viewing

When looking around a visual scene, humans make saccadic eye movements to fixate objects of interest. While the extraocular muscles can execute saccades in any direction, not all saccade directions are equally likely: saccades in horizontal and vertical directions are most prevalent. Here, we asked whether head orientation plays a role in determining saccade direction biases. Study participants (n = 14) viewed natural scenes and abstract fractals (radially symmetric patterns) through a virtual reality headset equipped with eye tracking. Participants' heads were stabilized and tilted at -30°, 0°, or 30° while viewing the images, which could also be tilted by -30°, 0°, and 30° relative to the head. To determine whether the biases in saccade direction changed with head tilt, we calculated polar histograms of saccade directions and cross-correlated pairs of histograms to find the angular displacement resulting in the maximum correlation. During free viewing of fractals, saccade biases largely followed the orientation of the head with an average displacement value of 24° when comparing head upright to head tilt in world-referenced coordinates (t ₍₁₃₎ = 17.63, p < 0.001). There was a systematic offset of 2.6° in saccade directions, likely reflecting ocular counter roll (OCR; t ₍₁₃₎ = 3.13, p = 0.008). When participants viewed an Earth upright natural scene during head tilt, we found that the orientation of the head still influenced saccade directions (t ₍₁₃₎ = 3.7, p = 0.001). These results suggest that nonvisual information about head orientation, such as that acquired by vestibular sensors, likely plays a role in saccade generation.

Developmental maturation of causal signaling hubs in voluntary control of saccades and their functional controllability

The ability to adaptively respond to behaviorally relevant cues in the environment, including voluntary control of automatic but inappropriate responses and deployment of a goal-relevant alternative response, undergoes significant maturation from childhood to adulthood. Importantly, the maturation of voluntary control processes influences the developmental trajectories of several key cognitive domains, including executive function and emotion regulation. Understanding the maturation of voluntary control is therefore of fundamental importance, but little is known about the underlying causal functional circuit mechanisms. Here, we use state-space and control-theoretic modeling to investigate the maturation of causal signaling mechanisms underlying voluntary control over saccades. We demonstrate that directed causal interactions in a canonical saccade network undergo significant maturation between childhood and adulthood. Crucially, we show that the frontal eye field (FEF) is an immature causal signaling hub in children during control over saccades. Using control-theoretic analysis, we then demonstrate that the saccade network is less controllable in children and that greater energy is required to drive FEF dynamics in children compared to adults. Our findings provide novel evidence that strengthening of causal signaling hubs and controllability of FEF are key mechanisms underlying age-related improvements in the ability to plan and execute voluntary control over saccades.

A Dynamic Neural Field Model of Multimodal Merging: Application to the Ventriloquist Effect

Multimodal merging encompasses the ability to localize stimuli based on imprecise information sampled through individual senses such as sight and hearing. Merging decisions are standardly described using Bayesian models that fit behaviors over many trials, encapsulated in a probability distribution. We introduce a novel computational model based on dynamic neural fields able to simulate decision dynamics and generate localization decisions, trial by trial, adapting to varying degrees of discrepancy between audio and visual stimulations. Neural fields are commonly used to model neural processes at a mesoscopic scale-for instance, neurophysiological activity in the superior colliculus. Our model is fit to human psychophysical data of the ventriloquist effect, additionally testing the influence of retinotopic projection onto the superior colliculus and providing a quantitative performance comparison to the Bayesian reference model. While models perform equally on average, a qualitative analysis of free parameters in our model allows insights into the dynamics of the decision and the individual variations in perception caused by noise. We finally show that the increase in the number of free parameters does not result in overfitting and that the parameter space may be either reduced to fit specific criteria or exploited to perform well on more demanding tasks in the future. Indeed, beyond decision or localization tasks, our model opens the door to the simulation of behavioral dynamics, as well as saccade generation driven by multimodal stimulation.

From Analog to Digital Computing: Is Homo sapiens’ Brain on Its Way to Become a Turing Machine?

The abstract basis of modern computation is the formal description of a finite state machine, the Universal Turing Machine, based on manipulation of integers and logic symbols. In this contribution to the discourse on the computer-brain analogy, we discuss the extent to which analog computing, as performed by the mammalian brain, is like and unlike the digital computing of Universal Turing Machines. We begin with ordinary reality being a permanent dialog between continuous and discontinuous worlds. So it is with computing, which can be analog or digital, and is often mixed. The theory behind computers is essentially digital, but efficient simulations of phenomena can be performed by analog devices; indeed, any physical calculation requires implementation in the physical world and is therefore analog to some extent, despite being based on abstract logic and arithmetic. The mammalian brain, comprised of neuronal networks, functions as an analog device and has given rise to artificial neural networks that are implemented as digital algorithms but function as analog models would. Analog constructs compute with the implementation of a variety of feedback and feedforward loops. In contrast, digital algorithms allow the implementation of recursive processes that enable them to generate unparalleled emergent properties. We briefly illustrate how the cortical organization of neurons can integrate signals and make predictions analogically. While we conclude that brains are not digital computers, we speculate on the recent implementation of human writing in the brain as a possible digital path that slowly evolves the brain into a genuine (slow) Turing machine.

DeepGaze III: Modeling free-viewing human scanpaths with deep learning

Humans typically move their eyes in “scanpaths” of fixations linked by saccades. Here we present DeepGaze III, a new model that predicts the spatial location of consecutive fixations in a free-viewing scanpath over static images. DeepGaze III is a deep learning–based model that combines image information with information about the previous fixation history to predict where a participant might fixate next. As a high-capacity and flexible model, DeepGaze III captures many relevant patterns in the human scanpath data, setting a new state of the art in the MIT300 dataset and thereby providing insight into how much information in scanpaths across observers exists in the first place. We use this insight to assess the importance of mechanisms implemented in simpler, interpretable models for fixation selection. Due to its architecture, DeepGaze III allows us to disentangle several factors that play an important role in fixation selection, such as the interplay of scene content and scanpath history. The modular nature of DeepGaze III allows us to conduct ablation studies, which show that scene content has a stronger effect on fixation selection than previous scanpath history in our main dataset. In addition, we can use the model to identify scenes for which the relative importance of these sources of information differs most. These data-driven insights would be difficult to accomplish with simpler models that do not have the computational capacity to capture such patterns, demonstrating an example of how deep learning advances can be used to contribute to scientific understanding.

Learning to use Muscles

The human musculoskeletal system is highly complex mechanically. Its neural control must deal successfully with this complexity to perform the diverse, efficient, robust and usually graceful behaviors of which humans are capable. Most of those behaviors might be performed by many different subsets of its myriad possible states, so how does the nervous system decide which subset to use? One solution that has received much attention over the past 50 years would be for the nervous system to be fundamentally limited in the patterns of muscle activation that it can access, a concept known as muscle synergies or movement primitives. Another solution, based on engineering control methodology, is for the nervous system to compute the single optimal pattern of muscle activation for each task according to a cost function. This review points out why neither appears to be the solution used by humans. There is a third solution that is based on trial-and-error learning, recall and interpolation of sensorimotor programs that are good-enough rather than limited or optimal. The solution set acquired by an individual during the protracted development of motor skills starting in infancy forms the basis of motor habits, which are inherently low-dimensional. Such habits give rise to muscle usage patterns that are consistent with synergies but do not reflect fundamental limitations of the nervous system and can be shaped by training or disability. This habit-based strategy provides a robust substrate for the control of new musculoskeletal structures during evolution as well as for efficient learning, athletic training and rehabilitation therapy.

Afferent and Efferent Visual Markers of Alzheimer's Disease: A Review and Update in Early Stage Disease

Vision, which requires extensive neural involvement, is often impaired in Alzheimer's disease (AD). Over the last few decades, accumulating evidence has shown that various visual functions and structures are compromised in Alzheimer's dementia and when measured can detect those with dementia from those with normal aging. These visual changes involve both the afferent and efferent parts of the visual system, which correspond to the sensory and eye movement aspects of vision, respectively. There are fewer, but a growing number of studies, that focus on the detection of predementia stages. Visual biomarkers that detect these stages are paramount in the development of successful disease-modifying therapies by identifying appropriate research participants and in identifying those who would receive future therapies. This review provides a summary and update on common afferent and efferent visual markers of AD with a focus on mild cognitive impairment (MCI) and preclinical disease detection. We further propose future directions in this area. Given the ease of performing visual tests, the accessibility of the eye, and advances in ocular technology, visual measures have the potential to be effective, practical, and non-invasive biomarkers of AD.

Defining "active sensing" through an analysis of sensing energetics: homeoactive and alloactive sensing

The term "active sensing" has been defined in multiple ways. Most strictly, the term refers to sensing that uses self-generated energy to sample the environment (e.g., echolocation). More broadly, the definition includes all sensing that occurs when the sensor is moving (e.g., tactile stimuli obtained by an immobile versus moving fingertip) and, broader still, includes all sensing guided by attention or intent (e.g., purposeful eye movements). The present work offers a framework to help disambiguate aspects of the "active sensing" terminology and reveals properties of tactile sensing unique among all modalities. The framework begins with the well-described "sensorimotor loop," which expresses the perceptual process as a cycle involving four subsystems: environment, sensor, nervous system, and actuator. Using system dynamics, we examine how information flows through the loop. This "sensory-energetic loop" reveals two distinct sensing mechanisms that subdivide active sensing into homeoactive and alloactive sensing. In homeoactive sensing, the animal can change the state of the environment, while in alloactive sensing the animal can alter only the sensor's configurational parameters and thus the mapping between input and output. Given these new definitions, examination of the sensory-energetic loop helps identify two unique characteristics of tactile sensing: 1) in tactile systems, alloactive and homeoactive sensing merge to a mutually controlled sensing mechanism, and 2) tactile sensing may require fundamentally different predictions to anticipate reafferent input. We expect this framework may help resolve ambiguities in the active sensing community and form a basis for future theoretical and experimental work regarding alloactive and homeoactive sensing.

The Role of Neuronal Oscillations in Visual Active Sensing

Visual perception is most often studied as a "passive" process in which an observer fixates steadily at point in space so that stimuli can be delivered to the system with spatial precision. Analysis of neuronal signals related to vision is generally keyed to stimulus onset, stimulus movement, etc.; i.e., events external to the observer. In natural "active" vision, however, information is systematically acquired by using eye movements including rapid (saccadic) eye movements, as well as smooth ocular pursuit of moving objects and slower drifts. Here we consider the use of alternating saccades and fixations to gather information from a visual scene. The underlying motor sampling plan contains highly reliable information regarding "where" and "when" the eyes will land, this information can be used predictively to modify firing properties of neurons precisely at the time when this "contextual" information is most useful - when a volley of retinal input enters the system at the onset of each fixation. Analyses focusing on neural events leading to and resulting from shifts in fixation, as well as visual events external to the observer, can provide a more complete and mechanistic understanding of visual information processing. Studies thus far suggest that active vision may be a fundamentally different from that process we usually study with more traditional passive viewing paradigms. In this Perspective we note that active saccadic sampling behavior imposes robust temporal patterning on the activity of neuron ensembles and large-scale neural dynamics throughout the brain's visual pathways whose mechanistic effects on information processing are not yet fully understood. The spatio-temporal sequence of eye movements elicits a succession of temporally predictable quasi-rhythmic sensory inputs, whose encoding is enhanced by entrainment of low frequency oscillations to the rate of eye movements. Review of the pertinent findings underscores the fact that temporal coordination between motor and visual cortices is critical for understanding neural dynamics of active vision and posits that phase entrainment of neuronal oscillations plays a mechanistic role in this process.

Modeling Saccadic Action Selection: Cortical and Basal Ganglia Signals Coalesce in the Superior Colliculus

The distributed nature of information processing in the brain creates a complex variety of decision making behavior. Likewise, computational models of saccadic decision making behavior are numerous and diverse. Here we present a generative model of saccadic action selection in the context of competitive decision making in the superior colliculus (SC) in order to investigate how independent neural signals may converge to interact and guide saccade selection, and to test if systematic variations can better replicate the variability in responses that are part of normal human behavior. The model was tasked with performing pro- and anti-saccades in order to replicate specific attributes of healthy human saccade behavior. Participants (ages 18-39) were instructed to either look toward (pro-saccade, well-practiced automated response) or away from (anti-saccade, combination of inhibitory and voluntary responses) a peripheral visual stimulus. They generated express and regular latency saccades in the pro-saccade task. In the anti-saccade task, correct reaction times were longer and participants occasionally looked at the stimulus (direction error) at either express or regular latencies. To gain a better understanding of the underlying neural processes that lead to saccadic action selection and response inhibition, we implemented 8 inputs inspired by systems neuroscience. These inputs reflected known sensory, automated, voluntary, and inhibitory components of cortical and basal ganglia activity that coalesces in the intermediate layers of the SC (SCi). The model produced bimodal reaction time distributions, where express and regular latency saccades had distinct modes, for both correct pro-saccades and direction errors in the anti-saccade task. Importantly, express and regular latency direction errors resulted from interactions of different inputs in the model. Express latency direction errors were due to a lack of pre-emptive fixation and inhibitory activity, which aloud sensory and automated inputs to initiate a stimulus-driven saccade. Regular latency errors occurred when the automated motor signals were stronger than the voluntary motor signals. While previous models have emulated fewer aspects of these behavioral findings, the focus of the simulations here is on the interaction of a wide variety of physiologically-based information integration producing a richer set of natural behavioral variability.

Sustained sensorimotor control as intermittent decisions about prediction errors: computational framework and application to ground vehicle steering

A conceptual and computational framework is proposed for modelling of human sensorimotor control and is exemplified for the sensorimotor task of steering a car. The framework emphasises control intermittency and extends on existing models by suggesting that the nervous system implements intermittent control using a combination of (1) motor primitives, (2) prediction of sensory outcomes of motor actions, and (3) evidence accumulation of prediction errors. It is shown that approximate but useful sensory predictions in the intermittent control context can be constructed without detailed forward models, as a superposition of simple prediction primitives, resembling neurobiologically observed corollary discharges. The proposed mathematical framework allows straightforward extension to intermittent behaviour from existing one-dimensional continuous models in the linear control and ecological psychology traditions. Empirical data from a driving simulator are used in model-fitting analyses to test some of the framework's main theoretical predictions: it is shown that human steering control, in routine lane-keeping and in a demanding near-limit task, is better described as a sequence of discrete stepwise control adjustments, than as continuous control. Results on the possible roles of sensory prediction in control adjustment amplitudes, and of evidence accumulation mechanisms in control onset timing, show trends that match the theoretical predictions; these warrant further investigation. The results for the accumulation-based model align with other recent literature, in a possibly converging case against the type of threshold mechanisms that are often assumed in existing models of intermittent control.

Integrating Brain and Biomechanical Models-A New Paradigm for Understanding Neuro-muscular Control

To date, realistic models of how the central nervous system governs behavior have been restricted in scope to the brain, brainstem or spinal column, as if these existed as disembodied organs. Further, the model is often exercised in relation to an in vivo physiological experiment with input comprising an impulse, a periodic signal or constant activation, and output as a pattern of neural activity in one or more neural populations. Any link to behavior is inferred only indirectly via these activity patterns. We argue that to discover the principles of operation of neural systems, it is necessary to express their behavior in terms of physical movements of a realistic motor system, and to supply inputs that mimic sensory experience. To do this with confidence, we must connect our brain models to neuro-muscular models and provide relevant visual and proprioceptive feedback signals, thereby closing the loop of the simulation. This paper describes an effort to develop just such an integrated brain and biomechanical system using a number of pre-existing models. It describes a model of the saccadic oculomotor system incorporating a neuromuscular model of the eye and its six extraocular muscles. The position of the eye determines how illumination of a retinotopic input population projects information about the location of a saccade target into the system. A pre-existing saccadic burst generator model was incorporated into the system, which generated motoneuron activity patterns suitable for driving the biomechanical eye. The model was demonstrated to make accurate saccades to a target luminance under a set of environmental constraints. Challenges encountered in the development of this model showed the importance of this integrated modeling approach. Thus, we exposed shortcomings in individual model components which were only apparent when these were supplied with the more plausible inputs available in a closed loop design. Consequently we were able to suggest missing functionality which the system would require to reproduce more realistic behavior. The construction of such closed-loop animal models constitutes a new paradigm of computational neurobehavior and promises a more thoroughgoing approach to our understanding of the brain's function as a controller for movement and behavior.

Audiovisual Rehabilitation in Hemianopia: A Model-Based Theoretical Investigation

Hemianopic patients exhibit visual detection improvement in the blind field when audiovisual stimuli are given in spatiotemporally coincidence. Beyond this “online” multisensory improvement, there is evidence of long-lasting, “offline” effects induced by audiovisual training: patients show improved visual detection and orientation after they were trained to detect and saccade toward visual targets given in spatiotemporal proximity with auditory stimuli. These effects are ascribed to the Superior Colliculus (SC), which is spared in these patients and plays a pivotal role in audiovisual integration and oculomotor behavior. Recently, we developed a neural network model of audiovisual cortico-collicular loops, including interconnected areas representing the retina, striate and extrastriate visual cortices, auditory cortex, and SC. The network simulated unilateral V1 lesion with possible spared tissue and reproduced “online” effects. Here, we extend the previous network to shed light on circuits, plastic mechanisms, and synaptic reorganization that can mediate the training effects and functionally implement visual rehabilitation. The network is enriched by the oculomotor SC-brainstem route, and Hebbian mechanisms of synaptic plasticity, and is used to test different training paradigms (audiovisual/visual stimulation in eye-movements/fixed-eyes condition) on simulated patients. Results predict different training effects and associate them to synaptic changes in specific circuits. Thanks to the SC multisensory enhancement, the audiovisual training is able to effectively strengthen the retina-SC route, which in turn can foster reinforcement of the SC-brainstem route (this occurs only in eye-movements condition) and reinforcement of the SC-extrastriate route (this occurs in presence of survived V1 tissue, regardless of eye condition). The retina-SC-brainstem circuit may mediate compensatory effects: the model assumes that reinforcement of this circuit can translate visual stimuli into short-latency saccades, possibly moving the stimuli into visual detection regions. The retina-SC-extrastriate circuit is related to restitutive effects: visual stimuli can directly elicit visual detection with no need for eye movements. Model predictions and assumptions are critically discussed in view of existing behavioral and neurophysiological data, forecasting that other oculomotor compensatory mechanisms, beyond short-latency saccades, are likely involved, and stimulating future experimental and theoretical investigations.

Behavioural and computational varieties of response inhibition in eye movements

Response inhibition is the ability to override a planned or an already initiated response. It is the hallmark of executive control as its deficits favour impulsive behaviours, which may be detrimental to an individual's life. This article reviews behavioural and computational guises of response inhibition. It focuses only on inhibition of oculomotor responses. It first reviews behavioural paradigms of response inhibition in eye movement research, namely the countermanding and antisaccade paradigms, both proven to be useful tools for the study of response inhibition in cognitive neuroscience and psychopathology. Then, it briefly reviews the neural mechanisms of response inhibition in these two behavioural paradigms. Computational models that embody a hypothesis and/or a theory of mechanisms underlying performance in both behavioural paradigms as well as provide a critical analysis of strengths and weaknesses of these models are discussed. All models assume the race of decision processes. The decision process in each paradigm that wins the race depends on different mechanisms. It has been shown that response latency is a stochastic process and has been proven to be an important measure of the cognitive control processes involved in response stopping in healthy and patient groups. Then, the inhibitory deficits in different brain diseases are reviewed, including schizophrenia and obsessive-compulsive disorder. Finally, new directions are suggested to improve the performance of models of response inhibition by drawing inspiration from successes of models in other domains.This article is part of the themed issue 'Movement suppression: brain mechanisms for stopping and stillness'.

A spiking neural network model of the midbrain superior colliculus that generates saccadic motor commands

Single-unit recordings suggest that the midbrain superior colliculus (SC) acts as an optimal controller for saccadic gaze shifts. The SC is proposed to be the site within the visuomotor system where the nonlinear spatial-to-temporal transformation is carried out: the population encodes the intended saccade vector by its location in the motor map (spatial), and its trajectory and velocity by the distribution of firing rates (temporal). The neurons' burst profiles vary systematically with their anatomical positions and intended saccade vectors, to account for the nonlinear main-sequence kinematics of saccades. Yet, the underlying collicular mechanisms that could result in these firing patterns are inaccessible to current neurobiological techniques. Here, we propose a simple spiking neural network model that reproduces the spike trains of saccade-related cells in the intermediate and deep SC layers during saccades. The model assumes that SC neurons have distinct biophysical properties for spike generation that depend on their anatomical position in combination with a center-surround lateral connectivity. Both factors are needed to account for the observed firing patterns. Our model offers a basis for neuronal algorithms for spatiotemporal transformations and bio-inspired optimal controllers.

A Model of the Superior Colliculus Predicts Fixation Locations during Scene Viewing and Visual Search

Modern computational models of attention predict fixations using saliency maps and target maps, which prioritize locations for fixation based on feature contrast and target goals, respectively. But whereas many such models are biologically plausible, none have looked to the oculomotor system for design constraints or parameter specification. Conversely, although most models of saccade programming are tightly coupled to underlying neurophysiology, none have been tested using real-world stimuli and tasks. We combined the strengths of these two approaches in MASC, a model of attention in the superior colliculus (SC) that captures known neurophysiological constraints on saccade programming. We show that MASC predicted the fixation locations of humans freely viewing naturalistic scenes and performing exemplar and categorical search tasks, a breadth achieved by no other existing model. Moreover, it did this as well or better than its more specialized state-of-the-art competitors. MASC's predictive success stems from its inclusion of high-level but core principles of SC organization: an over-representation of foveal information, size-invariant population codes, cascaded population averaging over distorted visual and motor maps, and competition between motor point images for saccade programming, all of which cause further modulation of priority (attention) after projection of saliency and target maps to the SC. Only by incorporating these organizing brain principles into our models can we fully understand the transformation of complex visual information into the saccade programs underlying movements of overt attention. With MASC, a theoretical footing now exists to generate and test computationally explicit predictions of behavioral and neural responses in visually complex real-world contexts.SIGNIFICANCE STATEMENT The superior colliculus (SC) performs a visual-to-motor transformation vital to overt attention, but existing SC models cannot predict saccades to visually complex real-world stimuli. We introduce a brain-inspired SC model that outperforms state-of-the-art image-based competitors in predicting the sequences of fixations made by humans performing a range of everyday tasks (scene viewing and exemplar and categorical search), making clear the value of looking to the brain for model design. This work is significant in that it will drive new research by making computationally explicit predictions of SC neural population activity in response to naturalistic stimuli and tasks. It will also serve as a blueprint for the construction of other brain-inspired models, helping to usher in the next generation of truly intelligent autonomous systems.

Eye movements in the wild: Oculomotor control, gaze behavior & frames of reference

Understanding the brain's capacity to encode complex visual information from a scene and to transform it into a coherent perception of 3D space and into well-coordinated motor commands are among the outstanding questions in the study of integrative brain function. Eye movement methodologies have allowed us to begin addressing these questions in increasingly naturalistic tasks, where eye and body movements are ubiquitous and, therefore, the applicability of most traditional neuroscience methods restricted. This review explores foundational issues in (1) how oculomotor and motor control in lab experiments extrapolates into more complex settings and (2) how real-world gaze behavior in turn decomposes into more elementary eye movement patterns. We review the received typology of oculomotor patterns in laboratory tasks, and how they map onto naturalistic gaze behavior (or not). We discuss the multiple coordinate systems needed to represent visual gaze strategies, how the choice of reference frame affects the description of eye movements, and the related but conceptually distinct issue of coordinate transformations between internal representations within the brain.

Cell-Type-Specific Control of Brainstem Locomotor Circuits by Basal Ganglia

The basal ganglia (BG) are critical for adaptive motor control, but the circuit principles underlying their pathway-specific modulation of target regions are not well understood. Here, we dissect the mechanisms underlying BG direct and indirect pathway-mediated control of the mesencephalic locomotor region (MLR), a brainstem target of BG that is critical for locomotion. We optogenetically dissect the locomotor function of the three neurochemically distinct cell types within the MLR: glutamatergic, GABAergic, and cholinergic neurons. We find that the glutamatergic subpopulation encodes locomotor state and speed, is necessary and sufficient for locomotion, and is selectively innervated by BG. We further show activation and suppression, respectively, of MLR glutamatergic neurons by direct and indirect pathways, which is required for bidirectional control of locomotion by BG circuits. These findings provide a fundamental understanding of how BG can initiate or suppress a motor program through cell-type-specific regulation of neurons linked to specific actions.

Effects of long and short simulated flights on the saccadic eye movement velocity of aviators

Aircrew fatigue is a major contributor to operational errors in civil and military aviation. Objective detection of pilot fatigue is thus critical to prevent aviation catastrophes. Previous work has linked fatigue to changes in oculomotor dynamics, but few studies have studied this relationship in critical safety environments. Here we measured the eye movements of US Marine Corps combat helicopter pilots before and after simulated flight missions of different durations.We found a decrease in saccadic velocities after long simulated flights compared to short simulated flights. These results suggest that saccadic velocity could serve as a biomarker of aviator fatigue.

Eye movements in Alzheimer's disease

A growing body of literature has investigated changes in eye movements as a result of Alzheimer's disease (AD). When compared to healthy, age-matched controls, patients display a number of remarkable alterations to oculomotor function and viewing behavior. In this article, we review AD-related changes to fundamental eye movements, such as saccades and smooth pursuit motion, in addition to changes to eye movement patterns during more complex tasks like visual search and scene exploration. We discuss the cognitive mechanisms that underlie these changes and consider the clinical significance of eye movement behavior, with a focus on eye movements in mild cognitive impairment. We conclude with directions for future research.

Saccade learning with concurrent cortical and subcortical basal ganglia loops

The Basal Ganglia (BG) is a central structure involved in multiple cortical and subcortical loops. Some of these loops are believed to be responsible for saccade target selection. We study here how the very specific structural relationships of these saccadic loops can affect the ability of learning spatial and feature-based tasks. We propose a model of saccade generation with reinforcement learning capabilities based on our previous BG and superior colliculus models. It is structured around the interactions of two parallel cortico-basal loops and one tecto-basal loop. The two cortical loops separately deal with spatial and non-spatial information to select targets in a concurrent way. The subcortical loop is used to make the final target selection leading to the production of the saccade. These different loops may work in concert or disturb each other regarding reward maximization. Interactions between these loops and their learning capabilities are tested on different saccade tasks. The results show the ability of this model to correctly learn basic target selection based on different criteria (spatial or not). Moreover the model reproduces and explains training dependent express saccades toward targets based on a spatial criterion. Finally, the model predicts that in absence of prefrontal control, the spatial loop should dominate.

A psychology based approach for longitudinal development in cognitive robotics

A major challenge in robotics is the ability to learn, from novel experiences, new behavior that is useful for achieving new goals and skills. Autonomous systems must be able to learn solely through the environment, thus ruling out a priori task knowledge, tuning, extensive training, or other forms of pre-programming. Learning must also be cumulative and incremental, as complex skills are built on top of primitive skills. Additionally, it must be driven by intrinsic motivation because formative experience is gained through autonomous activity, even in the absence of extrinsic goals or tasks. This paper presents an approach to these issues through robotic implementations inspired by the learning behavior of human infants. We describe an approach to developmental learning and present results from a demonstration of longitudinal development on an iCub humanoid robot. The results cover the rapid emergence of staged behavior, the role of constraints in development, the effect of bootstrapping between stages, and the use of a schema memory of experiential fragments in learning new skills. The context is a longitudinal experiment in which the robot advanced from uncontrolled motor babbling to skilled hand/eye integrated reaching and basic manipulation of objects. This approach offers promise for further fast and effective sensory-motor learning techniques for robotic learning.

Eye movements discriminate fatigue due to chronotypical factors and time spent on task--a double dissociation

Systematic differences in circadian rhythmicity are thought to be a substantial factor determining inter-individual differences in fatigue and cognitive performance. The synchronicity effect (when time of testing coincides with the respective circadian peak period) seems to play an important role. Eye movements have been shown to be a reliable indicator of fatigue due to sleep deprivation or time spent on cognitive tasks. However, eye movements have not been used so far to investigate the circadian synchronicity effect and the resulting differences in fatigue. The aim of the present study was to assess how different oculomotor parameters in a free visual exploration task are influenced by: a) fatigue due to chronotypical factors (being a 'morning type' or an 'evening type'); b) fatigue due to the time spent on task. Eighteen healthy participants performed a free visual exploration task of naturalistic pictures while their eye movements were recorded. The task was performed twice, once at their optimal and once at their non-optimal time of the day. Moreover, participants rated their subjective fatigue. The non-optimal time of the day triggered a significant and stable increase in the mean visual fixation duration during the free visual exploration task for both chronotypes. The increase in the mean visual fixation duration correlated with the difference in subjectively perceived fatigue at optimal and non-optimal times of the day. Conversely, the mean saccadic speed significantly and progressively decreased throughout the duration of the task, but was not influenced by the optimal or non-optimal time of the day for both chronotypes. The results suggest that different oculomotor parameters are discriminative for fatigue due to different sources. A decrease in saccadic speed seems to reflect fatigue due to time spent on task, whereas an increase in mean fixation duration a lack of synchronicity between chronotype and time of the day.

Production, control, and visual guidance of saccadic eye movements

Primate vision is served by rapid shifts of gaze called saccades. This review will survey current knowledge and particular problems concerning the neural control and guidance of gaze shifts.

The parietal cortex and saccade planning: lessons from human lesion studies

The parietal cortex is a critical interface for attention and integration of multiple sensory signals that can be used for the implementation of motor plans. Many neurons in this region exhibit strong attention-, reach-, grasp- or saccade-related activity. Here, we review human lesion studies supporting the critical role of the parietal cortex in saccade planning. Studies of patients with unilateral parietal damage and spatial neglect reveal characteristic spatially lateralized deficits of saccade programming when multiple stimuli compete for attention. However, these patients also show bilateral impairments of saccade initiation and control that are difficult to explain in the context of their lateralized deficits of visual attention. These findings are reminiscent of the deficits of oculomotor control observed in patients with Bálint's syndrome consecutive to bilateral parietal damage. We propose that some oculomotor deficits following parietal damage are compatible with a decisive role of the parietal cortex in saccade planning under conditions of sensory competition, while other deficits reflect disinhibition of low-level structures of the oculomotor network in the absence of top-down parietal modulation.

The mechanism of saccade motor pattern generation investigated by a large-scale spiking neuron model of the superior colliculus

The subcortical saccade-generating system consists of the retina, superior colliculus, cerebellum and brainstem motoneuron areas. The superior colliculus is the site of sensory-motor convergence within this basic visuomotor loop preserved throughout the vertebrates. While the system has been extensively studied, there are still several outstanding questions regarding how and where the saccade eye movement profile is generated and the contribution of respective parts within this system. Here we construct a spiking neuron model of the whole intermediate layer of the superior colliculus based on the latest anatomy and physiology data. The model consists of conductance-based spiking neurons with quasi-visual, burst, buildup, local inhibitory, and deep layer inhibitory neurons. The visual input is given from the superficial superior colliculus and the burst neurons send the output to the brainstem oculomotor nuclei. Gating input from the basal ganglia and an integral feedback from the reticular formation are also included.

We implement the model in the NEST simulator and show that the activity profile of bursting neurons can be reproduced by a combination of NMDA-type and cholinergic excitatory synaptic inputs and integrative inhibitory feedback. The model shows that the spreading neural activity observed in vivo can keep track of the collicular output over time and reset the system at the end of a saccade through activation of deep layer inhibitory neurons. We identify the model parameters according to neural recording data and show that the resulting model recreates the saccade size-velocity curves known as the saccadic main sequence in behavioral studies. The present model is consistent with theories that the superior colliculus takes a principal role in generating the temporal profiles of saccadic eye movements, rather than just specifying the end points of eye movements.

Visuo-oculomotor deficiency at early-stage idiopathic scoliosis in adolescent girls

STUDY DESIGN

Cross-sectional study.

OBJECTIVE

To determine whether adolescent idiopathic scoliosis (AIS) at onset is associated with oculomotor dysfunction and whether these oculomotor anomalies are correlated to the amplitude of the spine deformation.

SUMMARY OF BACKGROUND DATA

AIS is related to abnormalities of postural control. To date, few studies have focused on visuo-oculomotor and vestibulo-ocular functions at early-stage AIS.

METHODS

Fifty-three adolescent girls were diagnosed with AIS (mean age: 11.6 ± 2.1 yr) on clinical and radiological criteria (mean Cobb angle: 14.8° ± 5.0°). Visuo-oculomotor and vestibulo-ocular functions were studied with video-oculography, including saccades, smooth pursuit, caloric test, and pendular rotation, with visual vestibular ocular reflex and vestibulo-ocular reflex sequences. Two patient groups were defined according to the mean Cobb angle: group 1 included 29 patients with a Cobb angle from 5° to 14° and group 2 included 24 patients with a Cobb angle from 15° to 25°.

RESULTS

The group 2 showed different saccade characteristics than group 1: higher latencies for saccade sequences characterized by temporal uncertainty and predictive direction; lower velocity regardless of the type of the saccades. No difference was observed for saccadic accuracy and smooth-pursuit gain. For the visual vestibular ocular reflex, group 2 showed lower total maximal slow-phase velocity than group 1, whereas the vestibulo-ocular reflex (tested in dark) did not differ between groups. No difference was observed concerning the caloric vestibular test.

CONCLUSION

Patients with a Cobb angle of 15° or more presented normal vestibulo-ocular responses but altered visuo-oculomotor functions, especially for the saccadic latency and velocity. This could be the result of a dysfunction of oculomotor pathways at cerebellar and/or brainstem level. These central disorders may be incriminated in the development of AIS.

Optimal control of saccades by spatial-temporal activity patterns in the monkey superior colliculus

A major challenge in computational neurobiology is to understand how populations of noisy, broadly-tuned neurons produce accurate goal-directed actions such as saccades. Saccades are high-velocity eye movements that have stereotyped, nonlinear kinematics; their duration increases with amplitude, while peak eye-velocity saturates for large saccades. Recent theories suggest that these characteristics reflect a deliberate strategy that optimizes a speed-accuracy tradeoff in the presence of signal-dependent noise in the neural control signals. Here we argue that the midbrain superior colliculus (SC), a key sensorimotor interface that contains a topographically-organized map of saccade vectors, is in an ideal position to implement such an optimization principle. Most models attribute the nonlinear saccade kinematics to saturation in the brainstem pulse generator downstream from the SC. However, there is little data to support this assumption. We now present new neurophysiological evidence for an alternative scheme, which proposes that these properties reside in the spatial-temporal dynamics of SC activity. As predicted by this scheme, we found a remarkably systematic organization in the burst properties of saccade-related neurons along the rostral-to-caudal (i.e., amplitude-coding) dimension of the SC motor map: peak firing-rates systematically decrease for cells encoding larger saccades, while burst durations and skewness increase, suggesting that this spatial gradient underlies the increase in duration and skewness of the eye velocity profiles with amplitude. We also show that all neurons in the recruited population synchronize their burst profiles, indicating that the burst-timing of each cell is determined by the planned saccade vector in which it participates, rather than by its anatomical location. Together with the observation that saccade-related SC cells indeed show signal-dependent noise, this precisely tuned organization of SC burst activity strongly supports the notion of an optimal motor-control principle embedded in the SC motor map as it fully accounts for the straight trajectories and kinematic nonlinearity of saccades.

Learning where to attend with deep architectures for image tracking

We discuss an attentional model for simultaneous object tracking and recognition that is driven by gaze data. Motivated by theories of perception, the model consists of two interacting pathways, identity and control, intended to mirror the what and where pathways in neuroscience models. The identity pathway models object appearance and performs classification using deep (factored)-restricted Boltzmann machines. At each point in time, the observations consist of foveated images, with decaying resolution toward the periphery of the gaze. The control pathway models the location, orientation, scale, and speed of the attended object. The posterior distribution of these states is estimated with particle filtering. Deeper in the control pathway, we encounter an attentional mechanism that learns to select gazes so as to minimize tracking uncertainty. Unlike in our previous work, we introduce gaze selection strategies that operate in the presence of partial information and on a continuous action space. We show that a straightforward extension of the existing approach to the partial information setting results in poor performance, and we propose an alternative method based on modeling the reward surface as a gaussian process. This approach gives good performance in the presence of partial information and allows us to expand the action space from a small, discrete set of fixation points to a continuous domain.

Triggering mechanisms in microsaccade and saccade generation: a novel proposal

Saccades are rapid eye movements that change the line of sight between successive points of fixation. Even as we attempt to fixate our gaze precisely, small rapid eye movements called microsaccades interrupt fixation one or two times each second. Although the neural pathway controlling saccade generation is well understood, the specific mechanism for triggering microsaccades is unknown. Here, we review the evidence suggesting that microsaccades and saccades are generated by the same neural pathway. We also discuss current models of how the saccadic system produces microsaccades. Finally, we propose a new mechanism for triggering both microsaccades and saccades, based on a circuit formed by omnipause and long-lead burst neurons and driven by activity in the superior colliculus. Our model differs from previous proposals in that it does not require superior colliculus activity to surpass a particular threshold to trigger microsaccades and saccades. Rather, we propose that the reciprocal inhibition between omnipause and long-lead burst neurons gates each microsaccadic or saccadic event, triggering the eye movement whenever the activity in the long-lead burst neurons overcomes the inhibition from the omnipause neurons.

Neural dynamics of saccadic and smooth pursuit eye movement coordination during visual tracking of unpredictably moving targets

How does the brain coordinate saccadic and smooth pursuit eye movements to track objects that move in unpredictable directions and speeds? Saccadic eye movements rapidly foveate peripheral visual or auditory targets, and smooth pursuit eye movements keep the fovea pointed toward an attended moving target. Analyses of tracking data in monkeys and humans reveal systematic deviations from predictions of the simplest model of saccade-pursuit interactions, which would use no interactions other than common target selection and recruitment of shared motoneurons. Instead, saccadic and smooth pursuit movements cooperate to cancel errors of gaze position and velocity, and thus to maximize target visibility through time. How are these two systems coordinated to promote visual localization and identification of moving targets? How are saccades calibrated to correctly foveate a target despite its continued motion during the saccade? The neural model proposed here answers these questions. Modeled interactions encompass motion processing areas MT, MST, FPA, DLPN and NRTP; saccade planning and execution areas FEF, LIP, and SC; the saccadic generator in the brain stem; and the cerebellum. Simulations illustrate the model's ability to functionally explain and quantitatively simulate anatomical, neurophysiological and behavioral data about coordinated saccade-pursuit tracking.

Mechanisms for generating and compensating for the smallest possible saccades

Microsaccades are small eye movements that occur during gaze fixation. Although taking place only when we attempt to stabilize gaze position, microsaccades can be understood by relating them to the larger voluntary saccades, which abruptly shift gaze position. Starting from this approach to microsaccade analysis, I show how it can lead to significant insight about the generation and functional role of these eye movements. Like larger saccades, microsaccades are now known to be generated by brainstem structures involved not only in compiling motor commands for eye movements, but also in identifying and selecting salient target locations in the visual environment. In addition, these small eye movements both influence and are influenced by sensory and cognitive processes in various areas of the brain, and in a manner that is similar to the interactions between larger saccades and sensory or cognitive processes. By approaching the study of microsaccades from the perspective of what has been learned about their larger counterparts, we are now in a position to make greater strides in our understanding of the function of the smallest possible saccadic eye movements.

Neurally constrained modeling of perceptual decision making

Stochastic accumulator models account for response time in perceptual decision-making tasks by assuming that perceptual evidence accumulates to a threshold. The present investigation mapped the firing rate of frontal eye field (FEF) visual neurons onto perceptual evidence and the firing rate of FEF movement neurons onto evidence accumulation to test alternative models of how evidence is combined in the accumulation process. The models were evaluated on their ability to predict both response time distributions and movement neuron activity observed in monkeys performing a visual search task. Models that assume gating of perceptual evidence to the accumulating units provide the best account of both behavioral and neural data. These results identify discrete stages of processing with anatomically distinct neural populations and rule out several alternative architectures. The results also illustrate the use of neurophysiological data as a model selection tool and establish a novel framework to bridge computational and neural levels of explanation.

Robotic Vision with the Conformal Camera: Modeling Perisaccadic Perception

Humans make about 3 saccades per second at the eyeball's speed of 700 deg/sec to reposition the high-acuity fovea on the targets of interest to build up understanding of a scene. The brain's visuosaccadic circuitry uses the oculomotor command of each impending saccade to shift receptive fields (RFs) to cortical locations before the eyes take them there, giving a continuous and stable view of the world. We have developed a model for image representation based on projective Fourier transform (PFT) intended for robotic vision, which may efficiently process visual information during the motion of a camera with silicon retina that resembles saccadic eye movements. Here, the related neuroscience background is presented, effectiveness of the conformal camera's non-Euclidean geometry in intermediate-level vision is discussed, and the algorithmic steps in modeling perisaccadic perception with PFT are proposed. Our modeling utilizes basic properties of PFT. First, PFT is computable by FFT in complex logarithmic coordinates that also approximate the retinotopy. Second, the shift of RFs in retinotopic (logarithmic) coordinates is modeled by the shift property of discrete Fourier transform. The perisaccadic mislocalization observed by human subjects in laboratory experiments is the consequence of the fact that RFs' shifts are in logarithmic coordinates.

An updated time-optimal 3rd-order linear saccadic eye plant model

A linear third-order model of the ocular motor plant for horizontal saccadic eye movements is presented consisting of a linear ocular motor plant and a time-optimal saccadic controller based on physiological considerations. The ocular motor plant consists of the eyeball and two extraocular muscles. All parameters and initial conditions are estimated or measured from physiological data. The neural inputs are described by pulse-slide-step waveforms with a post inhibitory rebound burst and based on a time-optimal controller. Model parameters are estimated using the system identification technique. The static and dynamic behaviors of the model are in excellent agreement with the experimental data.

The human frontal oculomotor cortical areas contribute asymmetrically to motor planning in a gap saccade task

Background

Saccadic eye movements are used to rapidly align the fovea with the image of objects of interest in peripheral vision. We have recently shown that in children there is a high preponderance of quick latency but poorly planned saccades that consistently fall short of the target goal. The characteristics of these multiple saccades are consistent with a lack of proper inhibitory control of cortical oculomotor areas on the brainstem saccade generation circuitry.

Methodology/Principal Findings

In the present paper, we directly tested this assumption by using single pulse transcranial magnetic stimulation (TMS) to transiently disrupt neuronal activity in the frontal eye fields (FEF) and supplementary eye fields (SEF) in adults performing a gap saccade task. The results showed that the incidence of multiple saccades was increased for ispiversive but not contraversive directions for the right and left FEF, the left SEF, but not for the right SEF. Moreover, this disruption was most substantial during the ∼50 ms period around the appearance of the peripheral target. A control condition in which the dorsal motor cortex was stimulated demonstrated that this was not due to any non-specific effects of the TMS influencing the spatial distribution of attention.

Conclusions/Significance

Taken together, the results are consistent with a direction-dependent role of the FEF and left SEF in delaying the release of saccadic eye movements until they have been fully planned.