A model of the mechanism for the perceived location of a single flash and two successive flashes presented around the time of a saccade

Motor-related signals support localization invariance for stable visual perception

Our ability to perceive a stable visual world in the presence of continuous movements of the body, head, and eyes has puzzled researchers in the neuroscience field for a long time. We reformulated this problem in the context of hierarchical convolutional neural networks (CNNs)-whose architectures have been inspired by the hierarchical signal processing of the mammalian visual system-and examined perceptual stability as an optimization process that identifies image-defining features for accurate image classification in the presence of movements. Movement signals, multiplexed with visual inputs along overlapping convolutional layers, aided classification invariance of shifted images by making the classification faster to learn and more robust relative to input noise. Classification invariance was reflected in activity manifolds associated with image categories emerging in late CNN layers and with network units acquiring movement-associated activity modulations as observed experimentally during saccadic eye movements. Our findings provide a computational framework that unifies a multitude of biological observations on perceptual stability under optimality principles for image classification in artificial neural networks.

Vision During Saccadic Eye Movements

The perceptual consequences of eye movements are manifold: Each large saccade is accompanied by a drop of sensitivity to luminance-contrast, low-frequency stimuli, impacting both conscious vision and involuntary responses, including pupillary constrictions. They also produce transient distortions of space, time, and number, which cannot be attributed to the mere motion on the retinae. All these are signs that the visual system evokes active processes to predict and counteract the consequences of saccades. We propose that a key mechanism is the reorganization of spatiotemporal visual fields, which transiently increases the temporal and spatial uncertainty of visual representations just before and during saccades. On one hand, this accounts for the spatiotemporal distortions of visual perception; on the other hand, it implements a mechanism for fusing pre- and postsaccadic stimuli. This, together with the active suppression of motion signals, ensures the stability and continuity of our visual experience.

Afferent motor feedback determines the perceived location of tactile stimuli in the external space presented to the moving arm

People make systematic errors when localizing a brief tactile stimulus in the external space presented on the index finger while moving the arm. Although these errors likely arise in the spatiotemporal integration of the tactile input and information about arm position, the underlying arm position information used in this process is not known. In this study, we tested the contributions of afferent proprioceptive feedback and predictive arm position signals by comparing localization errors during passive vs. active arm movements. In the active trials, participants were instructed to localize a tactile stimulus in the external space that was presented to the index finger near the time of a self-generated arm movement. In the passive trials, each of the active trials was passively replayed in randomized order, using a robotic device. Our results provide evidence that the localization error patterns of the passive trials are similar to the active trials and, moreover, did not lag but rather led the active trials, which suggests that proprioceptive feedback makes an important contribution to tactile localization. To further test which kinematic property of this afferent feedback signal drives the underlying computations, we examined the localization errors with movements that had differently skewed velocity profiles but overall the same displacement. This revealed a difference in the localization patterns, which we explain by a probabilistic model in which temporal uncertainty about the stimulus is converted into a spatial likelihood, depending on the actual velocity of the arm rather than involving an efferent, preprogrammed movement.NEW & NOTEWORTHY We show that proprioceptive feedback of arm motion rather than efferent motor signals contributes to tactile localization during an arm movement. Data further show that localization errors depend on arm velocity, not displacement per se, suggesting that instantaneous velocity feedback plays a role in the underlying computations. Model simulation using Bayesian inference suggests that these errors depend not only on spatial but also on temporal uncertainties of sensory and motor signals.

Neural Network Evidence for the Coupling of Presaccadic Visual Remapping to Predictive Eye Position Updating

As we look around a scene, we perceive it as continuous and stable even though each saccadic eye movement changes the visual input to the retinas. How the brain achieves this perceptual stabilization is unknown, but a major hypothesis is that it relies on presaccadic remapping, a process in which neurons shift their visual sensitivity to a new location in the scene just before each saccade. This hypothesis is difficult to test in vivo because complete, selective inactivation of remapping is currently intractable. We tested it in silico with a hierarchical, sheet-based neural network model of the visual and oculomotor system. The model generated saccadic commands to move a video camera abruptly. Visual input from the camera and internal copies of the saccadic movement commands, or corollary discharge, converged at a map-level simulation of the frontal eye field (FEF), a primate brain area known to receive such inputs. FEF output was combined with eye position signals to yield a suitable coordinate frame for guiding arm movements of a robot. Our operational definition of perceptual stability was "useful stability," quantified as continuously accurate pointing to a visual object despite camera saccades. During training, the emergence of useful stability was correlated tightly with the emergence of presaccadic remapping in the FEF. Remapping depended on corollary discharge but its timing was synchronized to the updating of eye position. When coupled to predictive eye position signals, remapping served to stabilize the target representation for continuously accurate pointing. Graded inactivations of pathways in the model replicated, and helped to interpret, previous in vivo experiments. The results support the hypothesis that visual stability requires presaccadic remapping, provide explanations for the function and timing of remapping, and offer testable hypotheses for in vivo studies. We conclude that remapping allows for seamless coordinate frame transformations and quick actions despite visual afferent lags. With visual remapping in place for behavior, it may be exploited for perceptual continuity.

Spatiotemporal integration for tactile localization during arm movements: a probabilistic approach

It has been shown that people make systematic errors in the localization of a brief tactile stimulus that is delivered to the index finger while they are making an arm movement. Here we modeled these spatial errors with a probabilistic approach, assuming that they follow from temporal uncertainty about the occurrence of the stimulus. In the model, this temporal uncertainty converts into a spatial likelihood about the external stimulus location, depending on arm velocity. We tested the prediction of the model that the localization errors depend on arm velocity. Participants (n = 8) were instructed to localize a tactile stimulus that was presented to their index finger while they were making either slow- or fast-targeted arm movements. Our results confirm the model's prediction that participants make larger localization errors when making faster arm movements. The model, which was used to fit the errors for both slow and fast arm movements simultaneously, accounted very well for all the characteristics of these data with temporal uncertainty in stimulus processing as the only free parameter. We conclude that spatial errors in dynamic tactile perception stem from the temporal precision with which tactile inputs are processed.

Eye movements: the past 25 years

This article reviews the past 25 years of research on eye movements (1986-2011). Emphasis is on three oculomotor behaviors: gaze control, smooth pursuit and saccades, and on their interactions with vision. Focus over the past 25 years has remained on the fundamental and classical questions: What are the mechanisms that keep gaze stable with either stationary or moving targets? How does the motion of the image on the retina affect vision? Where do we look - and why - when performing a complex task? How can the world appear clear and stable despite continual movements of the eyes? The past 25 years of investigation of these questions has seen progress and transformations at all levels due to new approaches (behavioral, neural and theoretical) aimed at studying how eye movements cope with real-world visual and cognitive demands. The work has led to a better understanding of how prediction, learning and attention work with sensory signals to contribute to the effective operation of eye movements in visually rich environments.

Computational models of spatial updating in peri-saccadic perception

Perceptual phenomena that occur around the time of a saccade, such as peri-saccadic mislocalization or saccadic suppression of displacement, have often been linked to mechanisms of spatial stability. These phenomena are usually regarded as errors in processes of trans-saccadic spatial transformations and they provide important tools to study these processes. However, a true understanding of the underlying brain processes that participate in the preparation for a saccade and in the transfer of information across it requires a closer, more quantitative approach that links different perceptual phenomena with each other and with the functional requirements of ensuring spatial stability. We review a number of computational models of peri-saccadic spatial perception that provide steps in that direction. Although most models are concerned with only specific phenomena, some generalization and interconnection between them can be obtained from a comparison. Our analysis shows how different perceptual effects can coherently be brought together and linked back to neuronal mechanisms on the way to explaining vision across saccades.

The perception of motion smear during eye and head movements

Because the visual system integrates information across time, an image that moves on the retina would be expected to be perceived as smeared. In this article, we summarize the previous evidence that human observers perceive a smaller extent of smear when retinal image motion results from an eye or head movement, compared to when a physically moving target generates comparable image motion while the eyes and head are still. This evidence indicates that the reduction of perceived motion smear is asymmetrical, occurring only for targets that move against the direction of an eye or head movement. In addition, we present new data to show that no reduction of perceived motion smear occurs for targets that move in either direction during a visually-induced perception of self motion. We propose that low-level extra-retinal eye- and head-movement signals are responsible for the reduction of perceived motion smear, by decreasing the duration of the temporal impulse response. Although retinal as well as extra-retinal mechanisms can reduce the extent of perceived motion smear, available evidence suggests that improved visual functioning may occur only when an extra-retinal mechanism reduces the perception of smear.

Perisaccadic stereo depth with zero retinal disparity

When an object is viewed binocularly, unequal perspective projections of the two eyes' half images (binocular disparity) provide a cue for the sensation of stereo depth. For almost 200 years, binocular disparity has remained synonymous with retinal disparity, which is computed by subtracting the distance of each half image from its respective fovea. However, binocular disparity could also be coded in headcentric instead of retinal coordinates, by combining eye position and retinal image position in each eye and representing disparity as differences between visual directions of half images relative to the head. Although these two disparity-coding schemes suggest very different neural mechanisms, both offer identical predictions for stereopsis in almost every viewing condition, making it difficult to empirically distinguish between them. We designed a novel stimulus that uses perisaccadic spatial distortion to generate inconsistency between headcentric and retinal disparity. Foveal half images flashed asynchronously just before a horizontal saccade have zero retinal disparity, yet they produce a sensation of depth consistent with a nonzero headcentric disparity. Furthermore, this headcentric disparity can cancel and reverse the perceived depth stimulated with nonzero retinal disparity. This is the first demonstration that a coding scheme other than retinal disparity has a role in human stereopsis.

Head movements modulate visual responsiveness in the absence of gaze shifts

Visuospatial attention is strongly associated with saccades. Given that gaze shifts are often accomplished by combined eye-head movements, attention may also be coupled to head movements. We showed that simply turning the head without shifting the gaze is sufficient to cause a transient unbalance in responding to a visual stimulus. Manual responses to a stimulus flashed shortly before the onset of a horizontal head movement were faster in congruent trials, when the head moved towards the stimulus, than in incongruent trials, when the head moved away from the stimulus. These effects are similar to those observed for saccades. We take this as evidence for a tight link between visuospatial attention and head movements, even when the gaze does not shift.

Perisaccadic mislocalization of visual targets by head-free gaze shifts: visual or motor?

Such perisaccadic mislocalization is maximal in the direction of the saccade and varies systematically with the target-saccade onset delay. We have recently shown that under head-fixed conditions perisaccadic errors do not follow the quantitative predictions of current visuomotor models that explain these mislocalizations in terms of spatial updating. These models all assume sluggish eye-movement feedback and therefore predict that errors should vary systematically with the amplitude and kinematics of the intervening saccade. Instead, we reported that errors depend only weakly on the saccade amplitude. An alternative explanation for the data is that around the saccade the perceived target location undergoes a uniform transient shift in the saccade direction, but that the oculomotor feedback is, on average, accurate. This "visual shift" hypothesis predicts that errors will also remain insensitive to kinematic variability within much larger head-free gaze shifts. Here we test this prediction by presenting a brief visual probe near the onset of gaze saccades between 40 and 70 degrees amplitude. According to models with inaccurate gaze-motor feedback, the expected perisaccadic errors for such gaze shifts should be as large as 30 degrees and depend heavily on the kinematics of the gaze shift. In contrast, we found that the actual peak errors were similar to those reported for much smaller saccadic eye movements, i.e., on average about 10 degrees, and that neither gaze-shift amplitude nor kinematics plays a systematic role. Our data further corroborate the visual origin of perisaccadic mislocalization under open-loop conditions and strengthen the idea that efferent feedback signals in the gaze-control system are fast and accurate.

Body-tilt and visual verticality perception during multiple cycles of roll rotation

To assess the effects of degrading canal cues for dynamic spatial orientation in human observers, we tested how judgments about visual-line orientation in space (subjective visual vertical task, SVV) and estimates of instantaneous body tilt (subjective body-tilt task, SBT) develop in the course of three cycles of constant-velocity roll rotation. These abilities were tested across the entire tilt range in separate experiments. For comparison, we also obtained SVV data during static roll tilt. We found that as tilt increased, dynamic SVV responses became strongly biased toward the head pole of the body axis (A-effect), as if body tilt was underestimated. However, on entering the range of near-inverse tilts, SVV responses adopted a bimodal pattern, alternating between A-effects (biased toward head-pole) and E-effects (biased toward feet-pole). Apart from an onset effect, this tilt-dependent pattern of systematic SVV errors repeated itself in subsequent rotation cycles with little sign of worsening performance. Static SVV responses were qualitatively similar and consistent with previous reports but showed smaller A-effects. By contrast, dynamic SBT errors were small and unimodal, indicating that errors in visual-verticality estimates were not caused by errors in body-tilt estimation. We discuss these results in terms of predictions from a canal-otolith interaction model extended with a leaky integrator and an egocentric bias mechanism. We conclude that the egocentric-bias mechanism becomes more manifest during constant velocity roll-rotation and that perceptual errors due to incorrect disambiguation of the otolith signal are small despite the decay of canal signals.