Optimal inference explains dimension-specific contractions of spatial perception

Decoding Trans-Saccadic Prediction Error

We are constantly sampling our environment by moving our eyes, but our subjective experience of the world is stable and constant. Stimulus displacement during or shortly after a saccade often goes unnoticed, a phenomenon called the saccadic suppression of displacement. Although we fail to notice such displacements, our oculomotor system computes the prediction errors and adequately adjusts the gaze and future saccadic execution, a phenomenon known as saccadic adaptation. In the present study, we aimed to find a brain signature of the trans-saccadic prediction error that informs the motor system but not explicit perception. We asked participants (either sex) to report whether a visual target was displaced during a saccade while recording electroencephalography (EEG). Using multivariate pattern analysis, we were able to differentiate displacements from no displacements, even when participants failed to report the displacement. In other words, we found that trans-saccadic prediction error is represented in the EEG signal 100 ms after the displacement presentation, mainly in occipital and parieto-occipital channels, even in the absence of explicit perception of the displacement.SIGNIFICANCE STATEMENT Stability in vision occurs even while performing saccades. One suggested mechanism for this counterintuitive visual phenomenon is that external displacement is suppressed during the retinal remapping caused by a saccade. Here, we shed light on the mechanisms of trans-saccadic stability by showing that displacement information is not entirely suppressed and specifically present in the early stages of visual processing. Such a signal is relevant and computed for oculomotor adjustment despite being neglected for perception.

Motor-related signals support localization invariance for stable visual perception

Our ability to perceive a stable visual world in the presence of continuous movements of the body, head, and eyes has puzzled researchers in the neuroscience field for a long time. We reformulated this problem in the context of hierarchical convolutional neural networks (CNNs)—whose architectures have been inspired by the hierarchical signal processing of the mammalian visual system—and examined perceptual stability as an optimization process that identifies image-defining features for accurate image classification in the presence of movements. Movement signals, multiplexed with visual inputs along overlapping convolutional layers, aided classification invariance of shifted images by making the classification faster to learn and more robust relative to input noise. Classification invariance was reflected in activity manifolds associated with image categories emerging in late CNN layers and with network units acquiring movement-associated activity modulations as observed experimentally during saccadic eye movements. Our findings provide a computational framework that unifies a multitude of biological observations on perceptual stability under optimality principles for image classification in artificial neural networks.

Stable visual perception during eye and body movements suggests neural algorithms that convert location information—"where” type of signals—across multiple frames of reference, for instance, from retinocentric to craniocentric coordinates. Accordingly, numerous theoretical studies have proposed biologically plausible computational processes to achieve such transformations. However, how coordinate transformations can then be used by the hierarchy of cortical visual areas to produce stable perception remains largely unknown. Here, we explore the hypothesis that perception equates to the activity states of networks trained to classify “features” (e.g., objects, salient components) in the visual scene, and perceptual stability equates to robust classification of these features relative to self-generated movements, that is, a “what” type of information processing. We demonstrate in CNNs that neural signals related to eye and body movements support accurate image classification by making “where” type of computations—localization invariances—faster to learn and more robust relative to input perturbations. Therefore, by equating perception to the activity states of classifier networks, we provide a simple unifying mechanistic framework to explain the role movement signals in support of stable perception in dynamic interactions with the environment.

A bias in saccadic suppression of shape change

Processing of visual information in the central (foveal) and peripheral visual field is vastly different. To achieve a homogeneous representation of the visual world across eye movements, the visual system needs to compensate for these differences. By introducing subtle changes between peripheral and foveal inputs across saccades, one can test this compensation. We morphed shapes between a triangle and a circle and presented two different change directions (circularity decrease or increase) at varying magnitudes across a saccade. In a change-discrimination task, observers disproportionally often reported percepts of circularity increase. To test the relationship with visual-field differences, we measured perception when shapes were exclusively presented either in the periphery (before a saccade), or in the fovea (after a saccade). We found that overall shapes were perceived as more circular before than after a saccade and the more pronounced this difference was for a participant, the smaller was their circularity-increase bias in the change-discrimination task. We propose that visual-field differences have a direct and an indirect influence on transsaccadic perception of shape change. The direct influence is based on the distinct appearance of shape in the central and peripheral visual field in a trial, causing an increase of the perceptual magnitude of circularity-decrease changes. The indirect influence is based on long-term build-up of transsaccadic expectations; if a change is opposite (circularity increase) to the expectation (circularity decrease), it should elicit a strong error signal facilitating change detection. We discuss the concept of transsaccadic expectations and theoretical implications for transsaccadic perception of other feature changes.

Transsaccadic visual perception of foveal compared to peripheral environmental changes

The maintenance of stable visual perception across eye movements is hypothesized to be aided by extraretinal information (e.g., corollary discharge [CD]). Previous studies have focused on the benefits of this information for perception at the fovea. However, there is little information on the extent that CD benefits peripheral visual perception. Here we systematically examined the extent that CD supports the ability to perceive transsaccadic changes at the fovea compared to peripheral changes. Human subjects made saccades to targets positioned at different amplitudes (4° or 8°) and directions (rightward or upward). On each trial there was a reference point located either at (fovea) or 4° away (periphery) from the target. During the saccade the target and reference disappeared and, after a blank period, the reference reappeared at a shifted location. Subjects reported the perceived shift direction, and we determined the perceptual threshold for detection and estimate of the reference location. We also simulated the detection and location if subjects solely relied on the visual error of the shifted reference experienced after the saccade. The comparison of the reference location under these two conditions showed that overall the perceptual estimate was approximately 53% more accurate and 30% less variable than estimates based solely on visual information at the fovea. These values for peripheral shifts were consistently lower than that at the fovea: 34% more accurate and 9% less variable. Overall, the results suggest that CD information does support stable visual perception in the periphery, but is consistently less beneficial compared to the fovea.

Saccadic Suppression of Displacement Does Not Reflect a Saccade-Specific Bias to Assume Stability

Across saccades, small displacements of a visual target are harder to detect and their directions more difficult to discriminate than during steady fixation. Prominent theories of this effect, known as saccadic suppression of displacement, propose that it is due to a bias to assume object stability across saccades. Recent studies comparing the saccadic effect to masking effects suggest that suppression of displacement is not saccade-specific. Further evidence for this account is presented from two experiments where participants judged the size of displacements on a continuous scale in saccade and mask conditions, with and without blanking. Saccades and masks both reduced the proportion of correctly perceived displacements and increased the proportion of missed displacements. Blanking improved performance in both conditions by reducing the proportion of missed displacements. Thus, if suppression of displacement reflects a bias for stability, it is not a saccade-specific bias, but a more general stability assumption revealed under conditions of impoverished vision. Specifically, I discuss the potentially decisive role of motion or other transient signals for displacement perception. Without transients or motion, the quality of relative position signals is poor, and saccadic and mask-induced suppression of displacement reflects performance when the decision has to be made on these signals alone. Blanking may improve those position signals by providing a transient onset or a longer time to encode the pre-saccadic target position.

Working memory in action: inspecting the systematic and unsystematic errors of spatial memory across saccades

Our ability to interact with the world depends on memory buffers that flexibly store and process information for short periods of time. Current working memory research, however, mainly uses tasks that avoid eye movements, whereas in daily life we need to remember information across saccades. Because saccades disrupt perception and attention, the brain might use special transsaccadic memory systems. Therefore, to compare working memory systems between and across saccades, the current study devised transsaccadic memory tasks that evaluated the influence of memory load on several kinds of systematic and unsystematic spatial errors, and tested whether these measures predicted performance in more established working memory paradigms. Experiment 1 used a line intersection task that had people integrate lines shown before and after saccades, and it administered a 2-back task. Experiments 2 and 3 asked people to point at one of several locations within a memory array flashed before an eye movement, and we tested change detection and 2-back performance. We found that unsystematic transsaccadic errors increased with memory load and were correlated with 2-back performance. Systematic errors produced similar results, although effects varied as a function of the geometric layout of the memory arrays. Surprisingly, transsaccadic errors did not predict change detection performance despite the latter being a widely accepted measure of working memory capacity. Our results suggest that working memory systems between and across saccades share, in part, similar neural resources. Nevertheless, our data highlight the importance of investigating working memory across saccades.

Functions of Memory Across Saccadic Eye Movements

Several times per second, humans make rapid eye movements called saccades which redirect their gaze to sample new regions of external space. Saccades present unique challenges to both perceptual and motor systems. During the movement, the visual input is smeared across the retina and severely degraded. Once completed, the projection of the world onto the retina has undergone a large-scale spatial transformation. The vector of this transformation, and the new orientation of the eye in the external world, is uncertain. Memory for the pre-saccadic visual input is thought to play a central role in compensating for the disruption caused by saccades. Here, we review evidence that memory contributes to (1) detecting and identifying changes in the world that occur during a saccade, (2) bridging the gap in input so that visual processing does not have to start anew, and (3) correcting saccade errors and recalibrating the oculomotor system to ensure accuracy of future saccades. We argue that visual working memory (VWM) is the most likely candidate system to underlie these behaviours and assess the consequences of VWM's strict resource limitations for transsaccadic processing. We conclude that a full understanding of these processes will require progress on broader unsolved problems in psychology and neuroscience, in particular how the brain solves the object correspondence problem, to what extent prior beliefs influence visual perception, and how disparate signals arriving with different delays are integrated.

How post-saccadic target blanking affects the detection of stimulus displacements across saccades

When a visual stimulus is displaced during a saccade the displacement is often not noticed unless it is large compared to the amplitude of the eye movement. Displacement detection is improved, however, if a blank intervenes between saccade target offset and the presentation of the displaced post-saccadic stimulus. This has been interpreted as evidence that precise information about eye position and accurate memory for the position of the pre-saccadic target are available immediately after saccade offset, but are overridden by the presence of the post-saccadic stimulus if it is present when the eyes land. In the current set of experiments we examined in more detail how blanking contributes to the increase in displacement sensitivity. In two experiments we showed that the presentation of a blank interval between saccade offset and the presentation of the displaced stimulus improved people's ability to detect that the stimulus had been displaced and also their ability to judge the direction that it had been displaced, but only for displacements opposite to the direction of the saccade (backward displacements). A third experiment suggested that this improvement in the detection of backward displacements was due in part to subjects misremembering the saccade target location as being closer to the initial fixation point than it actually was immediately after the saccade but remembering its location more veridically 50 ms later. This has the effect of improving the detection of displacements as well as their direction of displacement, but preferentially for backwards vs. forward displacements.

Quantifying the spatial extent of the corollary discharge benefit to transsaccadic visual perception

Extraretinal information, such as corollary discharge (CD), is hypothesized to help compensate for saccade-induced visual input disruptions. However, support for this hypothesis is largely for one-dimensional transsaccadic visual changes, with little comprehensive information on the spatial characteristics. Here we systematically mapped the two-dimensional extent of this compensation by quantifying the insensitivity to different displacement metrics. Human subjects made saccades to targets positioned at different amplitudes (4° or 8°) and directions (rightward, oblique, or upward). After the saccade the initial target disappeared and, after a blank period, reappeared at a shifted location-a collinear, diagonal, or orthogonal displacement. Subjects reported the perceived shift direction, and we determined the displacement detection based on the perceptual judgments. The two-dimensional insensitivity fields resulting from the perceptual thresholds had spatial features similar to the saccadic eye movement variability: 1) scaled with movement amplitude, 2) oriented (less sensitive to the change) along the saccade vector, and 3) approximately constant in shape when normalized by movement amplitude. In addition, comparing the postsaccadic perceptual estimate of the presaccadic target location to that based solely on the postsaccade visual error showed that overall the perceptual estimate was approximately 50% more accurate and 35% less variable than estimates based solely on this visual information. However, this relationship was not uniform: The benefit of extraretinal information was observed largely for displacements with a component parallel to the saccade vector. These results suggest a graded use of extraretinal information when forming the postsaccadic perceptual evaluation of transsaccadic environmental changes.

Suppression and reversal of motion perception around the time of the saccade

We make fast, “saccadic” eye movements to capture finely resolved foveal snapshots of the world but these saccades cause motion artefacts. The artefacts go unnoticed, perhaps because the brain suppresses them through subcortical oculomotor signals feeding back into visual cortex. Opposing views, however, claim that passive mechanisms suffice: saccadic shearing forces might render the retina insensitive to the artefacts or post-saccadic snapshots might mask them before they enter consciousness. Crucially, only active suppression could explain perceptual changes that precede saccades but existing evidence for presaccadic misperception are ill-suited for addressing this issue: Previous studies have found misperceptions of space for objects briefly flashed before saccades, but perhaps only because observers confused the timing of flashes and saccades before they could be tested (“postdiction”), and presaccadic motion perception might have appeared to decline because motion stimuli persisted past eye movement onset. Here we addressed these concerns using briefly flashed two-frame animations (50 ms) to probe people’s motion sensitivity during and around saccades. We found that sensitivity declined before saccade onset, even when the probe appeared entirely outside the saccade, and this sensitivity decline was present for motion in every direction relative to saccade, ruling out problems with postdiction. Intriguingly, brief periods during the saccade produced negative sensitivity as if motion was reversed, arguably due to postsaccadic enhancement. These data suggest that motion perception is minimized during saccades through active suppression, complementing neurophysiological findings of colliculo-pulvinar projections that suppress the cortical middle temporal area around the time of the saccade.

Evidence for Optimal Integration of Visual Feature Representations across Saccades

We explore the visual world through saccadic eye movements, but saccades also present a challenge to visual processing by shifting externally stable objects from one retinal location to another. The brain could solve this problem in two ways: by overwriting preceding input and starting afresh with each fixation or by maintaining a representation of presaccadic visual features in working memory and updating it with new information from the remapped location. Crucially, when multiple objects are present in a scene the planning of eye movements profoundly affects the precision of their working memory representations, transferring limited memory resources from fixation toward the saccade target. Here we show that when humans make saccades, it results in an update of not just the precision of representations but also their contents. When multiple item colors are shifted imperceptibly during a saccade the perceived colors are found to fall between presaccadic and postsaccadic values, with the weight given to each input varying continuously with item location, and fixed relative to saccade parameters. Increasing sensory uncertainty, by adding color noise, biases updating toward the more reliable input, which is consistent with an optimal integration of presaccadic working memory with a postsaccadic updating signal. We recover this update signal and show it to be tightly focused on the vicinity of the saccade target. These results reveal how the nervous system accumulates detailed visual information from multiple views of the same object or scene.

SIGNIFICANCE STATEMENT

This study examines the consequences of saccadic eye movements for the internal representation of visual objects. A saccade shifts the image of a stable visual object from one part of the retina to another. We show that visual representations are built up over these different views of the same object, by combining information obtained before and after each saccade. The weights given to presaccadic and postsaccadic information are determined by the relative reliability of each input. This provides evidence that the visual system combines inputs over time in a statistically optimal way.

The effect of saccade metrics on the corollary discharge contribution to perceived eye location

Corollary discharge (CD) is hypothesized to provide the movement information (direction and amplitude) required to compensate for the saccade-induced disruptions to visual input. Here, we investigated to what extent these conveyed metrics influence perceptual stability in human subjects with a target-displacement detection task. Subjects made saccades to targets located at different amplitudes (4°, 6°, or 8°) and directions (horizontal or vertical). During the saccade, the target disappeared and then reappeared at a shifted location either in the same direction or opposite to the movement vector. Subjects reported the target displacement direction, and from these reports we determined the perceptual threshold for shift detection and estimate of target location. Our results indicate that the thresholds for all amplitudes and directions generally scaled with saccade amplitude. Additionally, subjects on average produced hypometric saccades with an estimated CD gain <1. Finally, we examined the contribution of different error signals to perceptual performance, the saccade error (movement-to-movement variability in saccade amplitude) and visual error (distance between the fovea and the shifted target location). Perceptual judgment was not influenced by the fluctuations in movement amplitude, and performance was largely the same across movement directions for different magnitudes of visual error. Importantly, subjects reported the correct direction of target displacement above chance level for very small visual errors (<0.75°), even when these errors were opposite the target-shift direction. Collectively, these results suggest that the CD-based compensatory mechanisms for visual disruptions are highly accurate and comparable for saccades with different metrics.

Masking produces compression of space and time in the absence of eye movements

Whenever the visual stream is abruptly disturbed by eye movements, blinks, masks, or flashes of light, the visual system needs to retrieve the new locations of current targets and to reconstruct the timing of events to straddle the interruption. This process may introduce position and timing errors. We here report that very similar errors are seen in human subjects across three different paradigms when disturbances are caused by either eye movements, as is well known, or, as we now show, masking. We suggest that the characteristic effects of eye movements on position and time, spatial and temporal compression and saccadic suppression of displacement, are consequences of the interruption and the subsequent reconnection and are seen also when visual input is masked without any eye movements. Our data show that compression and suppression effects are not solely a product of ocular motor activity but instead can be properties of a correspondence process that links the targets of interest across interruptions in visual input, no matter what their source.

Computational models of spatial updating in peri-saccadic perception

Perceptual phenomena that occur around the time of a saccade, such as peri-saccadic mislocalization or saccadic suppression of displacement, have often been linked to mechanisms of spatial stability. These phenomena are usually regarded as errors in processes of trans-saccadic spatial transformations and they provide important tools to study these processes. However, a true understanding of the underlying brain processes that participate in the preparation for a saccade and in the transfer of information across it requires a closer, more quantitative approach that links different perceptual phenomena with each other and with the functional requirements of ensuring spatial stability. We review a number of computational models of peri-saccadic spatial perception that provide steps in that direction. Although most models are concerned with only specific phenomena, some generalization and interconnection between them can be obtained from a comparison. Our analysis shows how different perceptual effects can coherently be brought together and linked back to neuronal mechanisms on the way to explaining vision across saccades.

Effects of saccadic adaptation on visual localization before and during saccades

Short-term saccadic adaptation is a mechanism that adjusts saccade amplitude to accurately reach an intended saccade target. Short-term saccadic adaptation induces a shift of perceived localization of objects flashed before the saccade. This shift, being detectable only before an adapted saccade, disappears at some time around saccade onset. Up to now, the exact time course of this effect has remained unknown. In previous experiments, the mislocalization caused by this adaptation-induced shift was overlapping with the mislocalization caused by a different, saccade-related localization error, the peri-saccadic compression. Due to peri-saccadic compression, objects flashed immediately at saccade onset appear compressed towards the saccade target. First, we tested whether the adaptation-induced shift and the peri-saccadic compression were either independent or related processes. We performed experiments with two different luminance-contrast conditions to separate the adaptation-induced shift and the peri-saccadic compression. Human participants had to indicate the perceived location of briefly presented stimuli before, during or after an adapted saccade. Adaptation-induced shift occurred similarly in either contrast condition, with or without peri-saccadic compression. Second, after validating the premise of both processes being independent and superimposing, we aimed at characterizing the time course of the adaptation-induced shift in more detail. Being present up to 1 s before an adapted saccade, the adaptation-induced shift begins to gradually decline from about 150 ms before saccade onset, and ceases during the saccade. A final experiment revealed that visual references make a major contribution to adaptation-induced mislocalization.

Perisaccadic mislocalization of visual targets by head-free gaze shifts: visual or motor?

Such perisaccadic mislocalization is maximal in the direction of the saccade and varies systematically with the target-saccade onset delay. We have recently shown that under head-fixed conditions perisaccadic errors do not follow the quantitative predictions of current visuomotor models that explain these mislocalizations in terms of spatial updating. These models all assume sluggish eye-movement feedback and therefore predict that errors should vary systematically with the amplitude and kinematics of the intervening saccade. Instead, we reported that errors depend only weakly on the saccade amplitude. An alternative explanation for the data is that around the saccade the perceived target location undergoes a uniform transient shift in the saccade direction, but that the oculomotor feedback is, on average, accurate. This "visual shift" hypothesis predicts that errors will also remain insensitive to kinematic variability within much larger head-free gaze shifts. Here we test this prediction by presenting a brief visual probe near the onset of gaze saccades between 40 and 70 degrees amplitude. According to models with inaccurate gaze-motor feedback, the expected perisaccadic errors for such gaze shifts should be as large as 30 degrees and depend heavily on the kinematics of the gaze shift. In contrast, we found that the actual peak errors were similar to those reported for much smaller saccadic eye movements, i.e., on average about 10 degrees, and that neither gaze-shift amplitude nor kinematics plays a systematic role. Our data further corroborate the visual origin of perisaccadic mislocalization under open-loop conditions and strengthen the idea that efferent feedback signals in the gaze-control system are fast and accurate.