Perception of visual space at the time of pro- and anti-saccades

Neural correlates of perisaccadic visual mislocalization in extrastriate cortex

When interacting with the visual world using saccadic eye movements (saccades), the perceived location of visual stimuli becomes biased, a phenomenon called perisaccadic mislocalization. However, the neural mechanism underlying this altered visuospatial perception and its potential link to other perisaccadic perceptual phenomena have not been established. Using the electrophysiological recording of extrastriate areas in four male macaque monkeys, combined with a computational model, we were able to quantify spatial bias around the saccade target (ST) based on the perisaccadic dynamics of extrastriate spatiotemporal sensitivity captured by a statistical model. This approach could predict the perisaccadic spatial bias around the ST, consistent with behavioral data, and revealed the precise neuronal response components underlying representational bias. These findings also establish the crucial role of increased sensitivity near the ST for neurons with receptive fields far from the ST in driving the ST spatial bias. Moreover, we showed that, by allocating more resources for visual target representation, visual areas enhance their representation of the ST location, even at the expense of transient distortions in spatial representation. This potential neural basis for perisaccadic ST representation also supports a general role for extrastriate neurons in creating the perception of stimulus location.

Time-varying generalized linear models: characterizing and decoding neuronal dynamics in higher visual areas

To create a behaviorally relevant representation of the visual world, neurons in higher visual areas exhibit dynamic response changes to account for the time-varying interactions between external (e.g., visual input) and internal (e.g., reward value) factors. The resulting high-dimensional representational space poses challenges for precisely quantifying individual factors' contributions to the representation and readout of sensory information during a behavior. The widely used point process generalized linear model (GLM) approach provides a powerful framework for a quantitative description of neuronal processing as a function of various sensory and non-sensory inputs (encoding) as well as linking particular response components to particular behaviors (decoding), at the level of single trials and individual neurons. However, most existing variations of GLMs assume the neural systems to be time-invariant, making them inadequate for modeling nonstationary characteristics of neuronal sensitivity in higher visual areas. In this review, we summarize some of the existing GLM variations, with a focus on time-varying extensions. We highlight their applications to understanding neural representations in higher visual areas and decoding transient neuronal sensitivity as well as linking physiology to behavior through manipulation of model components. This time-varying class of statistical models provide valuable insights into the neural basis of various visual behaviors in higher visual areas and hold significant potential for uncovering the fundamental computational principles that govern neuronal processing underlying various behaviors in different regions of the brain.

Neural correlates of perisaccadic visual mislocalization in extrastriate cortex

When interacting with the visual world using saccadic eye movements (saccades), the perceived location of visual stimuli becomes biased, a phenomenon called perisaccadic mislocalization, which is indeed an exemplar of the brain's dynamic representation of the visual world. However, the neural mechanism underlying this altered visuospatial perception and its potential link to other perisaccadic perceptual phenomena have not been established. Using a combined experimental and computational approach, we were able to quantify spatial bias around the saccade target (ST) based on the perisaccadic dynamics of extrastriate spatiotemporal sensitivity captured by statistical models. This approach could predict the perisaccadic spatial bias around the ST, consistent with the psychophysical studies, and revealed the precise neuronal response components underlying representational bias. These findings also established the crucial role of response remapping toward ST representation for neurons with receptive fields far from the ST in driving the ST spatial bias. Moreover, we showed that, by allocating more resources for visual target representation, visual areas enhance their representation of the ST location, even at the expense of transient distortions in spatial representation. This potential neural basis for perisaccadic ST representation, also supports a general role for extrastriate neurons in creating the perception of stimulus location.

Compression of Space for Low Visibility Probes

Stimuli briefly flashed just before a saccade are perceived closer to the saccade target, a phenomenon known as perisaccadic compression of space (Ross et al., 1997). More recently, we have demonstrated that brief probes are attracted towards a visual reference when followed by a mask, even in the absence of saccades (Zimmermann et al., 2014a). Here, we ask whether spatial compression depends on the transient disruptions of the visual input stream caused by either a mask or a saccade. Both of these degrade the probe visibility but we show that low probe visibility alone causes compression in the absence of any disruption. In a first experiment, we varied the regions of the screen covered by a transient mask, including areas where no stimulus was presented and a condition without masking. In all conditions, we adjusted probe contrast to make the probe equally hard to detect. Compression effects were found in all conditions. To obtain compression without a mask, the probe had to be presented at much lower contrasts than with masking. Comparing mislocalizations at different probe detection rates across masking, saccades and low contrast conditions without mask or saccade, Experiment 2 confirmed this observation and showed a strong influence of probe contrast on compression. Finally, in Experiment 3, we found that compression decreased as probe duration increased both for masks and saccades although here we did find some evidence that factors other than simply visibility as we measured it contribute to compression. Our experiments suggest that compression reflects how the visual system localizes weak targets in the context of highly visible stimuli.

Peri-saccadic compression to two locations in a two-target choice saccade task

When visual stimuli are presented at the onset of a saccadic eye movement they are seen compressed onto the target location of the saccade. This peri-saccadic compression is believed to result from internal feedback pathways between oculomotor and visual areas of the brain. This feedback enhances vision around the saccade target at the expense of localization ability in other regions of the visual field. Although saccades can be targeted at only one object at a time, often multiple potential targets are available in a visual scene, and the oculomotor system has to choose which target to look at. If two targets are available, preparatory activity builds-up at both target locations in oculomotor maps. Here we show that, in this situation, two foci of compression develop, independent of which of the two targets is eventually chosen for the saccade. Our results suggest that theories that use oculomotor feedback as efference copy signals for upcoming eye movements should take the possibility into account that multiple feedback signals from potential targets may occur in parallel before the execution of a saccade.

Effect of saccade automaticity on perisaccadic space compression

Briefly presented stimuli occurring just before or during a saccadic eye movement are mislocalized, leading to a compression of visual space toward the target of the saccade. In most cases this has been measured in subjects over-trained to perform a stereotyped and unnatural task where saccades are repeatedly driven to the same location, marked by a highly salient abrupt onset. Here, we asked to what extent the pattern of perisaccadic mislocalization depends on this specific context. We addressed this question by studying perisaccadic localization in a set of participants with no prior experience in eye-movement research, measuring localization performance as they practiced the saccade task. Localization was marginally affected by practice over the course of the experiment and it was indistinguishable from the performance of expert observers. The mislocalization also remained similar when the expert observers were tested in a condition leading to less stereotypical saccadic behavior-with no abrupt onset marking the saccade target location. These results indicate that perisaccadic compression is a robust behavior, insensitive to the specific paradigm used to drive saccades and to the level of practice with the saccade task.

The Role of Temporal Information in Perisaccadic Mislocalization

In dynamic environments, it is crucial to accurately consider the timing of information. For instance, during saccades the eyes rotate so fast that even small temporal errors in relating retinal stimulation by flashed stimuli to extra-retinal information about the eyes' orientations will give rise to substantial errors in where the stimuli are judged to be. If spatial localization involves judging the eyes' orientations at the estimated time of the flash, we should be able to manipulate the pattern of mislocalization by altering the estimated time of the flash. We reasoned that if we presented a relevant flash within a short rapid sequence of irrelevant flashes, participants' estimates of when the relevant flash was presented might be shifted towards the centre of the sequence. In a first experiment, we presented five bars at different positions around the time of a saccade. Four of the bars were black. Either the second or the fourth bar in the sequence was red. The task was to localize the red bar. We found that when the red bar was presented second in the sequence, it was judged to be further in the direction of the saccade than when it was presented fourth in the sequence. Could this be because the red bar was processed faster when more black bars preceded it? In a second experiment, a red bar was either presented alone or followed by two black bars. When two black bars followed it, it was judged to be further in the direction of the saccade. We conclude that the spatial localization of flashed stimuli involves judging the eye orientation at the estimated time of the flash.

The spatial profile of mask-induced compression for perception and action

Stimuli briefly flashed just before a saccade are perceived closer to the saccade target, a phenomenon known as saccadic compression of space. We have recently demonstrated that similar mislocalizations of flashed stimuli can be observed in the absence of saccades: brief probes were attracted towards a visual reference when followed by a mask. To examine the spatial profile of this new phenomenon of masked-induced compression, here we used a pair of references that draw the probe into the gap between them. Strong compression was found when we masked the probe and presented it following a reference pair, whereas little or no compression occurred for the probe without the reference pair or without the mask. When the two references were arranged vertically, horizontal mislocalizations prevailed. That is, probes presented to the left or right of the vertically arranged references were "drawn in" to be seen aligned with the references. In contrast, when we arranged the two references horizontally, we found vertical compression for stimuli presented above or below the references. Finally, when participants were to indicate the perceived probe location by making an eye movement towards it, saccade landing positions were compressed in a similar fashion as perceptual judgments, confirming the robustness of mask-induced compression. Our findings challenge pure oculomotor accounts of saccadic compression of space that assume a vital role for saccade-specific signals such as corollary discharge or the updating of eye position. Instead, we suggest that saccade- and mask-induced compression both reflect how the visual system deals with disruptions.

Does perisaccadic compression require foveal vision?

People make systematic errors when localizing a stimulus that is presented briefly near the time of a saccade. These errors have been interpreted as compression towards the position that is fixated after the saccade. Normally, fixating a position means that its image falls on the fovea. Macular degeneration (MD) damages the central retina, obliterating foveal vision. Many people with MD adopt a new retinal locus for fixation, called the preferred retinal locus (PRL). If the compression of space during the saccade is a special characteristic of the fovea, possibly due to the high density of cones that is found in the fovea, one might expect people lacking central vision to show no compression of space around the time of a saccade. If the compression of space during the saccade is related to the position that is fixated after the saccade, one would expect compression towards the PRL, despite the lack of a high density of cones in this area. We found that a person with MD showed a clear compression towards her PRL. We conclude that perisaccadic compression is related to the position that is fixated after the saccade rather than to the high density of receptors in the fovea.

Masking produces compression of space and time in the absence of eye movements

Whenever the visual stream is abruptly disturbed by eye movements, blinks, masks, or flashes of light, the visual system needs to retrieve the new locations of current targets and to reconstruct the timing of events to straddle the interruption. This process may introduce position and timing errors. We here report that very similar errors are seen in human subjects across three different paradigms when disturbances are caused by either eye movements, as is well known, or, as we now show, masking. We suggest that the characteristic effects of eye movements on position and time, spatial and temporal compression and saccadic suppression of displacement, are consequences of the interruption and the subsequent reconnection and are seen also when visual input is masked without any eye movements. Our data show that compression and suppression effects are not solely a product of ocular motor activity but instead can be properties of a correspondence process that links the targets of interest across interruptions in visual input, no matter what their source.

Visual mislocalization during saccade sequences

Visual objects briefly presented around the time of saccadic eye movements are perceived compressed towards the saccade target. Here, we investigated perisaccadic mislocalization with a double-step saccade paradigm, measuring localization of small probe dots briefly flashed at various times around the sequence of the two saccades. At onset of the first saccade, probe dots were mislocalized towards the first and, to a lesser extent, also towards the second saccade target. However, there was very little mislocalization at the onset of the second saccade. When we increased the presentation duration of the saccade targets prior to onset of the saccade sequence, perisaccadic mislocalization did occur at the onset of the second saccade.

Is mislocalization during saccades related to the position of the saccade target within the image or to the gaze position at the end of the saccade?

A stimulus that is flashed around the time of a saccade tends to be mislocalized in the direction of the saccade target. Our question is whether the mislocalization is related to the position of the saccade target within the image or to the gaze position at the end of the saccade. We separated the two with a visual illusion that influences the perceived distance to the target of the saccade and thus saccade endpoint without affecting the perceived position of the saccade target within the image. We asked participants to make horizontal saccades from the left to the right end of the shaft of a Müller-Lyer figure. Around the time of the saccade, we flashed a bar at one of five possible positions and asked participants to indicate its location by touching the screen. As expected, participants made shorter saccades along the fins-in (<->) configuration than along the fins-out (>-<) configuration of the figure. The illusion also influenced the mislocalization pattern during saccades, with flashes presented with the fins-out configuration being perceived beyond flashes presented with the fins-in configuration. The difference between the patterns of mislocalization for bars flashed during the saccade for the two configurations corresponded quantitatively with a prediction based on compression towards the saccade endpoint considering the magnitude of the effect of the illusion on saccade amplitude. We conclude that mislocalization is related to the eye position at the end of the saccade, rather than to the position of the saccade target within the image.

Computational models of spatial updating in peri-saccadic perception

Perceptual phenomena that occur around the time of a saccade, such as peri-saccadic mislocalization or saccadic suppression of displacement, have often been linked to mechanisms of spatial stability. These phenomena are usually regarded as errors in processes of trans-saccadic spatial transformations and they provide important tools to study these processes. However, a true understanding of the underlying brain processes that participate in the preparation for a saccade and in the transfer of information across it requires a closer, more quantitative approach that links different perceptual phenomena with each other and with the functional requirements of ensuring spatial stability. We review a number of computational models of peri-saccadic spatial perception that provide steps in that direction. Although most models are concerned with only specific phenomena, some generalization and interconnection between them can be obtained from a comparison. Our analysis shows how different perceptual effects can coherently be brought together and linked back to neuronal mechanisms on the way to explaining vision across saccades.

Spatiotopic coding and remapping in humans

How our perceptual experience of the world remains stable and continuous in the face of continuous rapid eye movements still remains a mystery. This review discusses some recent progress towards understanding the neural and psychophysical processes that accompany these eye movements. We firstly report recent evidence from imaging studies in humans showing that many brain regions are tuned in spatiotopic coordinates, but only for items that are actively attended. We then describe a series of experiments measuring the spatial and temporal phenomena that occur around the time of saccades, and discuss how these could be related to visual stability. Finally, we introduce the concept of the spatio-temporal receptive field to describe the local spatiotopicity exhibited by many neurons when the eyes move.

Inconsistency between peri-saccadic mislocalization and compression: evidence for separate "what" and "where" visual systems

The view of two separate "what" and "where" visual systems is supported by compelling neurophysiological evidence. However, very little direct psychophysical evidence has been presented to suggest that the two functions can be separated in neurologically intact persons. Using a peri-saccadic perception paradigm in which bars of different lengths were flashed around saccade onset, we directly measured the perceived object size (a "what" attribute) and location (a "where" attribute). We found that the perceived object location shifted toward the saccade target to show strongly compressed localization, whereas the perceived object size was not compressed accordingly. This dissociation indicates that the perceived size is not determined by spatial localization of the object boundary, providing direct psychophysical evidence to support that "what" and "where" attributes of objects are indeed processed separately.

Perisaccadic localization of auditory stimuli

Interaction with the outside world requires the knowledge about where objects are with respect to one's own body. Such spatial information is represented in various topographic maps in different sensory systems. From a computational point of view, however, a single, modality-invariant map of the incoming sensory signals appears to be a more efficient strategy for spatial representations. If such a single supra-modal map existed and were used for perceptual purposes, localization characteristics should be similar across modalities. Previous studies had shown mislocalization of brief visual stimuli presented in the temporal vicinity of saccadic eye-movements. Here, we tested, if such mislocalizations could also be found for auditory stimuli. We presented brief noise bursts before, during, and after visually guided saccades. Indeed, we found localization errors for these auditory stimuli. The spatio-temporal pattern of this mislocalization, however, clearly differed from the one found for visual stimuli. The spatial error also depended on the exact type of eye-movement (visually guided vs. memory guided saccades). Finally, results obtained in fixational control paradigms under different conditions suggest that auditory localization can be strongly influenced by both static and dynamic visual stimuli. Visual localization on the other hand is not influenced by distracting visual stimuli but can be inaccurate in the temporal vicinity of eye-movements. Taken together, our results argue against a single, modality-independent spatial representation of sensory signals.

Behavioral Evidence of Separate Adaptation Mechanisms Controlling Saccade Amplitude Lengthening and Shortening

The accuracy of saccadic eye movements is maintained over the long term by adaptation mechanisms that decrease or increase saccade amplitude. It is still unknown whether these opposite adaptive changes rely on common mechanisms. Here, a double-step target paradigm was used to adaptively decrease (backward second target step) or increase (forward step) the amplitude of reactive saccades in one direction only. To test which sensorimotor transformation stages are subjected to these adaptive changes, we measured their transfer to antisaccades in which sensory and motor vectors are spatially dissociated. In the backward adaptation condition, all subjects showed a significant amplitude decrease for adapted prosaccades and a significant transfer of adaptation to antisaccades performed in the adapted direction, but not to oppositely directed antisaccades elicited by a target jump in the adapted direction. In the forward adaptation condition, only 14 of 19 subjects showed a significant amplitude increase for prosaccades and no significant adaptation transfer to antisaccades was detected in either the adapted or nonadapted direction. These findings suggest that, whereas the level(s) of forward adaptation cannot be resolved, the mechanisms involved in backward adaptation of reactive saccades take place at a sensorimotor level downstream from the vector inversion process of antisaccades and differ markedly from those involved in forward adaptation.

Changes in orientation discrimination at the time of saccadic eye movements

Perceptual performance has been known to change around the time of saccadic eye movement. In the current study, we measured the accuracy and sensitivity of orientation discrimination of bar stimuli presented during fixation and before saccadic eye movements. Human participants compared the orientations of the test and reference bar stimuli with the head erect in a two-interval forced choice task. For the targets presented during steady fixation, the accuracy and sensitivity of orientation discrimination were better near the cardinal than oblique axes, a perceptual anisotropy known as the oblique effect. For the targets presented during the 100 ms interval immediately before a saccade was executed, the anisotropy decreased mainly due to reduction in sensitivity for cardinal orientations. Directing attention to the goal location of the impending saccade emulated the saccadic effects on orientation discrimination for the targets at saccadic goal, suggesting that the saccadic effects on orientation discrimination are partly mediated by the shift of spatial attention that accompanies the saccade. These results were in line with the anti-oblique effect that perceptual judgment of motion direction along the oblique angle becomes relatively accurate for motion targets presented before saccadic eye movements [Lee, J., & Lee, C. (2005). Changes in visual motion perception before saccadic eye movements. Vision Research, 45(11), 1447-1457].

The effect of optokinetic nystagmus on the perceived position of briefly flashed targets

Stimuli flashed briefly around the time of an impending saccade are mislocalized in the direction of the saccade and also compressed towards the saccadic target. Similarly, targets flashed during pursuit eye movements are mislocalized in the direction of pursuit. Here, we investigate the effects of optokinetic nystagmus (OKN) on visual localization. Subjects passively viewed a wide-field drifting grating that elicited strong OKN, comprising the characteristic slow-phase tracking movement interspersed with corrected "saccade-like" fast-phase movements. Subjects reported the apparent position of salient bars flashed briefly at various positions on the screen. In general, bars were misperceived in the direction of the slow-phase tracking movement. Bars flashed around the onset of the fast-phase movements were subject to much less mislocalization, pointing to a competing shift in the direction of the fast-phase, as occurs with saccades. However, as distinct from saccades, there was no evidence for spatial compression around the time of the corrective fast-phase OKN. The results suggest that OKN cause perceptual mislocalizations similar to those of smooth pursuit and saccades, but there are some differences in the nature of the mislocalizations, pointing to different perceptual mechanisms associated with the different types of eye movements.

Mislocalization of perceived saccade target position induced by perisaccadic visual stimulation

The perceptual localization of objects flashed at the time of a saccade often shows large spatial distortions. These perisaccadic mislocalizations exhibit different spatial patterns depending on the experimental condition. In darkness, when only extraretinal information is available, mislocalization is spatially uniform. In light and when visual references are available, mislocalization is directed toward the saccade target, resembling a compression of visual space. These patterns are derived from measurements of the absolute perceived position of the flashed object in egocentric space. Here, we report that also the perceived location of the saccade target is altered when an object is flashed perisaccadically. The mislocalization of the target depends on the presentation time of the flashed object and is directed toward the position of the flash. The resulting compression of the relative distance between target and flash is similar in darkness and in light and can also be found during fixation. When the localization of the flashed object is described relative to the perceived location of the saccade target, spatial compression becomes similar in many experimental conditions. We therefore suggest that perisaccadic compression relies on an encoding of relative spatial locations of objects rather than on localizations in egocentric space.

Perisaccadic mislocalization without saccadic eye movements

Despite frequent saccadic gaze shifts we perceive the surrounding visual world as stable. It has been proposed that the brain uses extraretinal eye position signals to cancel out saccade-induced retinal image motion. Nevertheless, stimuli flashed briefly around the onset of a saccade are grossly mislocalized, resulting in a shift and, under certain conditions, an additional compression of visual space. Perisaccadic mislocalization has been related to a spatio-temporal misalignment of an extraretinal eye position signal with the corresponding saccade. Here, we investigated perceptual mislocalization of human observers both in saccade and fixation conditions. In the latter conditions, the retinal stimulation during saccadic eye movements was simulated by a fast saccade-like shift of the stimulus display. We show that the spatio-temporal pattern of both the shift and compression components of perceptual mislocalization can be surprisingly similar before real and simulated saccades. Our findings suggest that the full pattern of perisaccadic mislocalization can also occur in conditions which are unlikely to involve changes of an extraretinal eye position signal. Instead, we suggest that, under the conditions of our experiments, the arising difficulty to establish a stable percept of a briefly flashed stimulus within a given visual reference frame yields mislocalizations before fast retinal image motion. The availability of visual references appears to exert a major influence on the relative contributions of shift and compression components to mislocalization across the visual field.

Remapping the Remembered Target Location for Anti-Saccades in Human Posterior Parietal Cortex

We used functional magnetic resonance imaging (fMRI) to investigate the role of the human posterior parietal cortex (PPC) in anti-saccades. To do so, we exploited the laterality of a subregion of the PPC for remembered target location. Using an event-related design, we tracked fMRI signal changes in this region while subjects remembered the location of a flashed target, then were instructed to plan either an anti- or pro-saccade to that location, and finally were instructed to execute the movement. At first, the region responded preferentially to the memory of a target location presented in the contralateral visual field. However, when an anti-cue specified a saccadic response into the opposite visual field, we observed a dynamic shift in cortical activity from one hemisphere to the other. This shows that this region within the human posterior parietal cortex codes the target location for an upcoming saccade, rather than the location of the remembered visual stimulus in an anti-saccade task.

Perisaccadic perception of continuous flickers

To realize perceptual space constancy, the visual system compensates for the retinal displacement caused by eye movements. It has been reported that the compensation process does not function perfectly around the time of a saccade--a perisaccadic flash is systematically mislocalized. However, observations made with transient flash stimuli do not necessarily indicate a general perisaccadic failure of space constancy. To investigate how the visual system realizes perisaccadic space constancy for continuous stimuli, we examined the time course of localization for a perisaccadic 500 Hz flicker with systematic variation of the onset timing, the offset timing and the duration. If each flash in the flicker is localized individually in the same way as a single flash, the apparent position and length of the flicker should be predicted from the time course of mislocalization of a perisaccadic flash. However, the results did not support this prediction in many respects. A dot array (of half the length of the retinal image) was perceived when the flicker was presented during a saccade, while only a single dot was perceived when the flicker was presented only before or after the saccade. A flash in a flicker was localized at a different position, depending on the onset timing, the offset timing and the duration of the flicker, even if the flash was presented at the same timing to the saccade. In general, our results support a two-stage localization in which the local geometrical configuration is first generated primarily based on the retinal information, and then localized as a whole in the egocentric or exocentric space. The localization is based on the eye position signal sampled at a time temporally distant from the saccade, which enables precise localization and space constancy for continuous stimuli.

Contrast dependency of sacadic compression and suppression

In the occurrence of a saccadic eye movement vision becomes suppressed. Supra-threshold visual stimuli that are briefly presented at that time become perceptually compressed towards the saccade target (saccadic compression) and shifted in saccade direction (saccadic shift). We show that the strength of saccadic compression, like the strength of saccadic suppression, varies with stimulus contrast. Low contrast stimuli lead to stronger compression than high contrast stimuli. The similarity of contrast dependence and time course suggests that saccadic compression is related to saccadic suppression. Because the saccadic shift did not depend on contrast we suggest that shift and compression are different effects.