Brain circuits underlying visual stability across eye movements-converging evidence for a neuro-computational model of area LIP

A large-scale neurocomputational model of spatial cognition integrating memory with vision

We introduce a large-scale neurocomputational model of spatial cognition called 'Spacecog', which integrates recent findings from mechanistic models of visual and spatial perception. As a high-level cognitive ability, spatial cognition requires the processing of behaviourally relevant features in complex environments and, importantly, the updating of this information during processes of eye and body movement. The Spacecog model achieves this by interfacing spatial memory and imagery with mechanisms of object localisation, saccade execution, and attention through coordinate transformations in parietal areas of the brain. We evaluate the model in a realistic virtual environment where our neurocognitive model steers an agent to perform complex visuospatial tasks. Our modelling approach opens up new possibilities in the assessment of neuropsychological data and human spatial cognition.

Motor-related signals support localization invariance for stable visual perception

Our ability to perceive a stable visual world in the presence of continuous movements of the body, head, and eyes has puzzled researchers in the neuroscience field for a long time. We reformulated this problem in the context of hierarchical convolutional neural networks (CNNs)—whose architectures have been inspired by the hierarchical signal processing of the mammalian visual system—and examined perceptual stability as an optimization process that identifies image-defining features for accurate image classification in the presence of movements. Movement signals, multiplexed with visual inputs along overlapping convolutional layers, aided classification invariance of shifted images by making the classification faster to learn and more robust relative to input noise. Classification invariance was reflected in activity manifolds associated with image categories emerging in late CNN layers and with network units acquiring movement-associated activity modulations as observed experimentally during saccadic eye movements. Our findings provide a computational framework that unifies a multitude of biological observations on perceptual stability under optimality principles for image classification in artificial neural networks.

Stable visual perception during eye and body movements suggests neural algorithms that convert location information—"where” type of signals—across multiple frames of reference, for instance, from retinocentric to craniocentric coordinates. Accordingly, numerous theoretical studies have proposed biologically plausible computational processes to achieve such transformations. However, how coordinate transformations can then be used by the hierarchy of cortical visual areas to produce stable perception remains largely unknown. Here, we explore the hypothesis that perception equates to the activity states of networks trained to classify “features” (e.g., objects, salient components) in the visual scene, and perceptual stability equates to robust classification of these features relative to self-generated movements, that is, a “what” type of information processing. We demonstrate in CNNs that neural signals related to eye and body movements support accurate image classification by making “where” type of computations—localization invariances—faster to learn and more robust relative to input perturbations. Therefore, by equating perception to the activity states of classifier networks, we provide a simple unifying mechanistic framework to explain the role movement signals in support of stable perception in dynamic interactions with the environment.

Cerebellar Signals Drive Motor Adjustments and Visual Perceptual Changes during Forward and Backward Adaptation of Reactive Saccades

Saccadic adaptation ($SA$) is a cerebellar-dependent learning of motor commands ($MC$), which aims at preserving saccade accuracy. Since $SA$ alters visual localization during fixation and even more so across saccades, it could also involve changes of target and/or saccade visuospatial representations, the latter ($CDv$) resulting from a motor-to-visual transformation (forward dynamics model) of the corollary discharge of the $MC$. In the present study, we investigated if, in addition to its established role in adaptive adjustment of $MC$, the cerebellum could contribute to the adaptation-associated perceptual changes. Transfer of backward and forward adaptation to spatial perceptual performance (during ocular fixation and trans-saccadically) was assessed in eight cerebellar patients and eight healthy volunteers. In healthy participants, both types of $SA$ altered $MC$ as well as internal representations of the saccade target and of the saccadic eye displacement. In patients, adaptation-related adjustments of $MC$ and adaptation transfer to localization were strongly reduced relative to healthy participants, unraveling abnormal adaptation-related changes of target and $CDv$. Importantly, the estimated changes of $CDv$ were totally abolished following forward session but mainly preserved in backward session, suggesting that an internal model ensuring trans-saccadic localization could be located in the adaptation-related cerebellar networks or in downstream networks, respectively.

Saccadic suppression in schizophrenia

About 40% of schizophrenia patients report discrete visual disturbances which could occur if saccadic suppression, the decrease of visual sensitivity around saccade onset, is impaired. Two mechanisms contribute to saccadic suppression: efference copy processing and backwards masking. Both are reportedly altered in schizophrenia. However, saccadic suppression has not been investigated in schizophrenia. 17 schizophrenia patients and 18 healthy controls performed a saccadic suppression task using a Gabor stimulus with individually adjusted contrast, which was presented within an interval 300 ms around saccade onset. Visual disturbance scores were higher in patients than controls, but saccadic suppression strength and time course were similar in both groups with lower saccadic suppression rates being similarly related to smaller saccade amplitudes. Saccade amplitudes in the saccadic suppression task were reduced in patients, in contrast to unaltered amplitudes during a saccade control task. Notably, smaller saccade amplitudes were related to higher visual disturbances scores in patients. Saccadic suppression performance was unrelated to symptom expression and antipsychotic medication. Unaltered saccadic suppression in patients suggests sufficiently intact efference copy processing and backward masking as required for this task. Instead, visual disturbances in patients may be related to restricted saccadic amplitudes arising from cognitive load while completing a task.

The posterior parietal cortex processes visuo-spatial and extra-retinal information for saccadic remapping: A case study

Optimally collecting information and controlling behaviour require that we constantly scan our visual environment through eye movements. How the dynamic interaction between short-lived retinal images and extra-retinal signals of eye motion results in our subjective experience of visual stability remains a major issue in Cognitive Neuroscience. The present study aimed to assess and determine the nature of the contribution of the posterior parietal cortex (PPC) to the saccadic remapping mechanisms which contribute to such perceptual visual constancy. Perceptual responses in transsaccadic visual localization tasks were measured in a patient presenting with a PPC lesion and manifesting optic ataxia in the left hemifield with no neglect. Two perceptual localization tasks, each with versus without an intervening saccade, were used: the saccadic suppression of displacement (SSD) task (Ostendorf, Liebermann, & Ploner, 2010) and the peri-saccadic flash localization (LOC) task (Zimmerman & Lappe, 2010). Compared to a group of age-matched healthy subjects, the patient showed a specific pattern of perceptual deficits in the ataxic (left) hemifield. First, a significant impairment occurred in the stationary eye conditions, attesting for an alteration of visuo-spatial encoding. Second, in the saccade conditions, an additional perceptual deficit (an error of ~5° along the saccade direction) was observed in both tasks and mainly in conditions where extra-retinal signals are thought to be critically involved, revealing a constant underestimation by extra-retinal signals of the saccade size, despite preserved saccade accuracy. These findings highlight a crucial role of the PPC in saccadic remapping processes underlying perceptual visual constancy and provide empirical evidence for models such as Ziesche and Hamker's (2014).

Visuomotor learning from postdictive motor error

Sensorimotor learning adapts motor output to maintain movement accuracy. For saccadic eye movements, learning also alters space perception, suggesting a dissociation between the performed saccade and its internal representation derived from corollary discharge (CD). This is critical since learning is commonly believed to be driven by CD-based visual prediction error. We estimate the internal saccade representation through pre- and trans-saccadic target localization, showing that it decouples from the actual saccade during learning. We present a model that explains motor and perceptual changes by collective plasticity of spatial target percept, motor command, and a forward dynamics model that transforms CD from motor into visuospatial coordinates. We show that learning does not follow visual prediction error but instead a postdictive update of space after saccade landing. We conclude that trans-saccadic space perception guides motor learning via CD-based postdiction of motor error under the assumption of a stable world.

Intra-saccadic motion streaks as cues to linking object locations across saccades

When visual objects shift rapidly across the retina, they produce motion blur. Intra-saccadic visual signals, caused incessantly by our own saccades, are thought to be eliminated at early stages of visual processing. Here we investigate whether they are still available to the visual system and could—in principle—be used as cues for localizing objects as they change locations on the retina. Using a high-speed projection system, we developed a trans-saccadic identification task in which brief but continuous intra-saccadic object motion was key to successful performance. Observers made a saccade to a target stimulus that moved rapidly either up or down, strictly during the eye movement. Just as the target reached its final position, an identical distractor stimulus appeared on the opposite side, resulting in a display of two identical stimuli upon saccade landing. Observers had to identify the original target using the only available clue: the target's intra-saccadic movement. In an additional replay condition, we presented the observers’ own intra-saccadic retinal stimulus trajectories during fixation. Compared to the replay condition, task performance was impaired during saccades but recovered fully when a post-saccadic blank was introduced. Reverse regression analyses and confirmatory experiments showed that performance increased markedly when targets had long movement durations, low spatial frequencies, and orientations parallel to their retinal trajectory—features that promote intra-saccadic motion streaks. Although the potential functional role of intra-saccadic visual signals is still unclear, our results suggest that they could provide cues to tracking objects that rapidly change locations across saccades.

Perisaccadic perceptual mislocalization is different for upward saccades

Saccadic eye movements, which dramatically alter retinal images, are associated with robust perimovement perceptual alterations. Such alterations, thought to reflect brain mechanisms for maintaining perceptual stability in the face of saccade-induced retinal image disruptions, are often studied by asking subjects to localize brief stimuli presented around the time of horizontal saccades. However, other saccade directions are not usually explored. Motivated by recently discovered asymmetries in upper and lower visual field representations in the superior colliculus, a structure important for both saccade generation and visual analysis, we observed significant differences in perisaccadic perceptual alterations for upward saccades relative to other saccade directions. We also found that, even for purely horizontal saccades, perceptual alterations differ for upper vs. lower retinotopic stimulus locations. Our results, coupled with conceptual modeling, suggest that perisaccadic perceptual alterations might critically depend on neural circuits, such as superior colliculus, that asymmetrically represent the upper and lower visual fields. NEW & NOTEWORTHY Brief visual stimuli are robustly mislocalized around the time of saccades. Such mislocalization is thought to arise because oculomotor and visual neural maps distort space through foveal magnification. However, other neural asymmetries, such as upper visual field magnification in the superior colliculus, may also exist, raising the possibility that interactions between saccades and visual stimuli would depend on saccade direction. We confirmed this behaviorally by exploring and characterizing perisaccadic perception for upward saccades.

Learning Peri-saccadic Remapping of Receptive Field from Experience in Lateral Intraparietal Area

Our eyes move constantly at a frequency of 3-5 times per second. These movements, called saccades, induce the sweeping of visual images on the retina, yet we perceive the world as stable. It has been suggested that the brain achieves this visual stability via predictive remapping of neuronal receptive field (RF). A recent experimental study disclosed details of this remapping process in the lateral intraparietal area (LIP), that is, about the time of the saccade, the neuronal RF expands along the saccadic trajectory temporally, covering the current RF (CRF), the future RF (FRF), and the region the eye will sweep through during the saccade. A cortical wave (CW) model was also proposed, which attributes the RF remapping as a consequence of neural activity propagating in the cortex, triggered jointly by a visual stimulus and the corollary discharge (CD) signal responsible for the saccade. In this study, we investigate how this CW model is learned naturally from visual experiences at the development of the brain. We build a two-layer network, with one layer consisting of LIP neurons and the other superior colliculus (SC) neurons. Initially, neuronal connections are random and non-selective. A saccade will cause a static visual image to sweep through the retina passively, creating the effect of the visual stimulus moving in the opposite direction of the saccade. According to the spiking-time-dependent-plasticity rule, the connection path in the opposite direction of the saccade between LIP neurons and the connection path from SC to LIP are enhanced. Over many such visual experiences, the CW model is developed, which generates the peri-saccadic RF remapping in LIP as observed in the experiment.

Neural Network Evidence for the Coupling of Presaccadic Visual Remapping to Predictive Eye Position Updating

As we look around a scene, we perceive it as continuous and stable even though each saccadic eye movement changes the visual input to the retinas. How the brain achieves this perceptual stabilization is unknown, but a major hypothesis is that it relies on presaccadic remapping, a process in which neurons shift their visual sensitivity to a new location in the scene just before each saccade. This hypothesis is difficult to test in vivo because complete, selective inactivation of remapping is currently intractable. We tested it in silico with a hierarchical, sheet-based neural network model of the visual and oculomotor system. The model generated saccadic commands to move a video camera abruptly. Visual input from the camera and internal copies of the saccadic movement commands, or corollary discharge, converged at a map-level simulation of the frontal eye field (FEF), a primate brain area known to receive such inputs. FEF output was combined with eye position signals to yield a suitable coordinate frame for guiding arm movements of a robot. Our operational definition of perceptual stability was "useful stability," quantified as continuously accurate pointing to a visual object despite camera saccades. During training, the emergence of useful stability was correlated tightly with the emergence of presaccadic remapping in the FEF. Remapping depended on corollary discharge but its timing was synchronized to the updating of eye position. When coupled to predictive eye position signals, remapping served to stabilize the target representation for continuously accurate pointing. Graded inactivations of pathways in the model replicated, and helped to interpret, previous in vivo experiments. The results support the hypothesis that visual stability requires presaccadic remapping, provide explanations for the function and timing of remapping, and offer testable hypotheses for in vivo studies. We conclude that remapping allows for seamless coordinate frame transformations and quick actions despite visual afferent lags. With visual remapping in place for behavior, it may be exploited for perceptual continuity.

Saccadic Corollary Discharge Underlies Stable Visual Perception

Saccadic eye movements direct the high-resolution foveae of our retinas toward objects of interest. With each saccade, the image jumps on the retina, causing a discontinuity in visual input. Our visual perception, however, remains stable. Philosophers and scientists over centuries have proposed that visual stability depends upon an internal neuronal signal that is a copy of the neuronal signal driving the eye movement, now referred to as a corollary discharge (CD) or efference copy. In the old world monkey, such a CD circuit for saccades has been identified extending from superior colliculus through MD thalamus to frontal cortex, but there is little evidence that this circuit actually contributes to visual perception. We tested the influence of this CD circuit on visual perception by first training macaque monkeys to report their perceived eye direction, and then reversibly inactivating the CD as it passes through the thalamus. We found that the monkey's perception changed; during CD inactivation, there was a difference between where the monkey perceived its eyes to be directed and where they were actually directed. Perception and saccade were decoupled. We established that the perceived eye direction at the end of the saccade was not derived from proprioceptive input from eye muscles, and was not altered by contextual visual information. We conclude that the CD provides internal information contributing to the brain's creation of perceived visual stability. More specifically, the CD might provide the internal saccade vector used to unite separate retinal images into a stable visual scene.

SIGNIFICANCE STATEMENT

Visual stability is one of the most remarkable aspects of human vision. The eyes move rapidly several times per second, displacing the retinal image each time. The brain compensates for this disruption, keeping our visual perception stable. A major hypothesis explaining this stability invokes a signal within the brain, a corollary discharge, that informs visual regions of the brain when and where the eyes are about to move. Such a corollary discharge circuit for eye movements has been identified in macaque monkey. We now show that selectively inactivating this brain circuit alters the monkey's visual perception. We conclude that this corollary discharge provides a critical signal that can be used to unite jumping retinal images into a consistent visual scene.

The Dorsal Visual System Predicts Future and Remembers Past Eye Position

Eye movements are essential to primate vision but introduce potentially disruptive displacements of the retinal image. To maintain stable vision, the brain is thought to rely on neurons that carry both visual signals and information about the current direction of gaze in their firing rates. We have shown previously that these neurons provide an accurate representation of eye position during fixation, but whether they are updated fast enough during saccadic eye movements to support real-time vision remains controversial. Here we show that not only do these neurons carry a fast and accurate eye-position signal, but also that they support in parallel a range of time-lagged variants, including predictive and post dictive signals. We recorded extracellular activity in four areas of the macaque dorsal visual cortex during a saccade task, including the lateral and ventral intraparietal areas (LIP, VIP), and the middle temporal (MT) and medial superior temporal (MST) areas. As reported previously, neurons showed tonic eye-position-related activity during fixation. In addition, they showed a variety of transient changes in activity around the time of saccades, including relative suppression, enhancement, and pre-saccadic bursts for one saccade direction over another. We show that a hypothetical neuron that pools this rich population activity through a weighted sum can produce an output that mimics the true spatiotemporal dynamics of the eye. Further, with different pooling weights, this downstream eye position signal (EPS) could be updated long before (<100 ms) or after (<200 ms) an eye movement. The results suggest a flexible coding scheme in which downstream computations have access to past, current, and future eye positions simultaneously, providing a basis for visual stability and delay-free visually-guided behavior.

Peri-saccadic compression to two locations in a two-target choice saccade task

When visual stimuli are presented at the onset of a saccadic eye movement they are seen compressed onto the target location of the saccade. This peri-saccadic compression is believed to result from internal feedback pathways between oculomotor and visual areas of the brain. This feedback enhances vision around the saccade target at the expense of localization ability in other regions of the visual field. Although saccades can be targeted at only one object at a time, often multiple potential targets are available in a visual scene, and the oculomotor system has to choose which target to look at. If two targets are available, preparatory activity builds-up at both target locations in oculomotor maps. Here we show that, in this situation, two foci of compression develop, independent of which of the two targets is eventually chosen for the saccade. Our results suggest that theories that use oculomotor feedback as efference copy signals for upcoming eye movements should take the possibility into account that multiple feedback signals from potential targets may occur in parallel before the execution of a saccade.

The spatial profile of mask-induced compression for perception and action

Stimuli briefly flashed just before a saccade are perceived closer to the saccade target, a phenomenon known as saccadic compression of space. We have recently demonstrated that similar mislocalizations of flashed stimuli can be observed in the absence of saccades: brief probes were attracted towards a visual reference when followed by a mask. To examine the spatial profile of this new phenomenon of masked-induced compression, here we used a pair of references that draw the probe into the gap between them. Strong compression was found when we masked the probe and presented it following a reference pair, whereas little or no compression occurred for the probe without the reference pair or without the mask. When the two references were arranged vertically, horizontal mislocalizations prevailed. That is, probes presented to the left or right of the vertically arranged references were "drawn in" to be seen aligned with the references. In contrast, when we arranged the two references horizontally, we found vertical compression for stimuli presented above or below the references. Finally, when participants were to indicate the perceived probe location by making an eye movement towards it, saccade landing positions were compressed in a similar fashion as perceptual judgments, confirming the robustness of mask-induced compression. Our findings challenge pure oculomotor accounts of saccadic compression of space that assume a vital role for saccade-specific signals such as corollary discharge or the updating of eye position. Instead, we suggest that saccade- and mask-induced compression both reflect how the visual system deals with disruptions.

Masking produces compression of space and time in the absence of eye movements

Whenever the visual stream is abruptly disturbed by eye movements, blinks, masks, or flashes of light, the visual system needs to retrieve the new locations of current targets and to reconstruct the timing of events to straddle the interruption. This process may introduce position and timing errors. We here report that very similar errors are seen in human subjects across three different paradigms when disturbances are caused by either eye movements, as is well known, or, as we now show, masking. We suggest that the characteristic effects of eye movements on position and time, spatial and temporal compression and saccadic suppression of displacement, are consequences of the interruption and the subsequent reconnection and are seen also when visual input is masked without any eye movements. Our data show that compression and suppression effects are not solely a product of ocular motor activity but instead can be properties of a correspondence process that links the targets of interest across interruptions in visual input, no matter what their source.