Visual stability and the motion aftereffect: a psychophysical study revealing spatial updating

Predictive remapping of visual features beyond saccadic targets

Visual stability is thought to be mediated by predictive remapping of the relevant object information from its current, presaccadic location to its future, postsaccadic location on the retina. However, it is heavily debated whether and what feature information is predictively remapped during the presaccadic interval. Here we examined the spatial and featural properties of predictive remapping in a set of three psychophysical studies. We made use of an orientation-adaptation paradigm, in which we induced a tilt aftereffect by prolonged exposure to an oriented adaptor stimulus. Following this adaptation phase, a test stimulus was presented shortly before saccade onset. We found strong evidence for predictive remapping of the features of this test stimulus presented shortly before saccade onset, evidenced by a large tilt aftereffect elicited when the adaptor was positioned at the postsaccadic retinal location of the test stimulus. Conversely, the adaptation state itself, caused by the exposure to the adaptor stimulus, was not predictively remapped. Furthermore, we establish that predictive remapping also occurs for stimuli that are not saccade targets, pointing toward a forward remapping process operating across the whole visual field. Together, our findings suggest that predictive feature remapping of object information plays an important role in mediating visual stability.

Effect of spatial attention on spatiotopic visual motion perception

We almost never experience visual instability, despite retinal image instability induced by eye movements. How the stability of visual perception is maintained through spatiotopic representation remains a matter of debate. The discrepancies observed in the findings of existing neuroscience studies regarding spatiotopic representation partly originate from differences in regard to how attention is deployed to stimuli. In this study, we psychophysically examined whether spatial attention is needed to perceive spatiotopic visual motion. For this purpose, we used visual motion priming, which is a phenomenon in which a preceding priming stimulus modulates the perceived moving direction of an ambiguous test stimulus, such as a drifting grating that phase shifts by 180°. To examine the priming effect in different coordinates, participants performed a saccade soon after the offset of a primer. The participants were tasked with judging the direction of a subsequently presented test stimulus. To control the effect of spatial attention, the participants were asked to conduct a concurrent dot contrast-change detection task after the saccade. Positive priming was prominent in spatiotopic conditions, whereas negative priming was dominant in retinotopic conditions. At least a 600-ms interval between the priming and test stimuli was needed to observe positive priming in spatiotopic coordinates. When spatial attention was directed away from the location of the test stimulus, spatiotopic positive motion priming completely disappeared; meanwhile, the spatiotopic positive motion priming at shorter interstimulus intervals was enhanced when spatial attention was directed to the location of the test stimulus. These results provide evidence that an attentional resource is requisite for developing spatiotopic representation more quickly.

Perceptual learning while preparing saccades

Traditional perceptual learning protocols rely almost exclusively on long periods of uninterrupted fixation. Taking a first step towards understanding perceptual learning in natural vision, we had observers report the orientation of a briefly flashed stimulus (clockwise or counterclockwise from a reference orientation) presented strictly during saccade preparation at a location offset from the saccade target. For each observer, the saccade direction, stimulus location, and orientation remained the same throughout training. Subsequently, we assessed performance during fixation in three transfer sessions, either at the trained or at an untrained location, and either using an untrained (Experiment 1) or the trained (Experiment 2) stimulus orientation. We modeled the evolution of contrast thresholds (i.e., the stimulus contrast necessary to discriminate its orientation correctly 75% of the time) as an exponential learning curve, and quantified departures from this curve in transfer sessions using two new, complementary measures of transfer costs (i.e., performance decrements after the transition into the Transfer phase). We observed robust perceptual learning and associated transfer costs for untrained locations and orientations. We also assessed if spatial transfer costs were reduced for the remapped location of the pre-saccadic stimulus-the location the stimulus would have had (but never had) after the saccade. Although the pattern of results at that location differed somewhat from that at the control location, we found no clear evidence for perceptual learning at remapped locations. Using novel, model-based ways to assess learning and transfer costs, our results show that location and feature specificity, hallmarks of perceptual learning, subsist if the target stimulus is presented strictly during saccade preparation throughout training.

Spatiotopic updating across saccades revealed by spatially-specific fMRI adaptation

Brain representations of visual space are predominantly eye-centred (retinotopic) yet our experience of the world is largely world-centred (spatiotopic). A long-standing question is how the brain creates continuity between these reference frames across successive eye movements (saccades). Here we use functional magnetic resonance imaging (fMRI) to address whether spatially specific repetition suppression (RS) is evident during trans-saccadic perception. We presented two successive Gabor patches (S1 and S2) in either the upper or lower visual field, left or right of fixation. Spatial congruency was manipulated by having S1 and S2 occur in the same or different upper/lower visual field. On half the trials, a saccade was cued between S1 and S2, placing spatiotopic and retinotopic reference frames in opposition. Equivalent RS was observed in the posterior parietal cortex and frontal eye fields when S1-S2 were spatiotopically congruent, irrespective of whether retinotopic and spatiotopic coordinates were in accord or were placed in opposition by a saccade. Additionally the post-saccadic response to S2 demonstrated spatially-specific RS in retinotopic visual regions, with stronger RS in extrastriate than striate cortex. Collectively, these results are consistent with a robust trans-saccadic spatial updating mechanism for object position that directly influences even the earliest levels of visual processing.

Spatiotopic updating facilitates perception immediately after saccades

As the neural representation of visual information is initially coded in retinotopic coordinates, eye movements (saccades) pose a major problem for visual stability. If no visual information were maintained across saccades, retinotopic representations would have to be rebuilt after each saccade. It is currently strongly debated what kind of information (if any at all) is accumulated across saccades, and when this information becomes available after a saccade. Here, we use a motion illusion to examine the accumulation of visual information across saccades. In this illusion, an annulus with a random texture slowly rotates, and is then replaced with a second texture (motion transient). With increasing rotation durations, observers consistently perceive the transient as large rotational jumps in the direction opposite to rotation direction (backward jumps). We first show that accumulated motion information is updated spatiotopically across saccades. Then, we show that this accumulated information is readily available after a saccade, immediately biasing postsaccadic perception. The current findings suggest that presaccadic information is used to facilitate postsaccadic perception and are in support of a forward model of transsaccadic perception, aiming at anticipating the consequences of eye movements and operating within the narrow perisaccadic time window.

Nonretinotopic visual processing in the brain

A basic principle in visual neuroscience is the retinotopic organization of neural receptive fields. Here, we review behavioral, neurophysiological, and neuroimaging evidence for nonretinotopic processing of visual stimuli. A number of behavioral studies have shown perception depending on object or external-space coordinate systems, in addition to retinal coordinates. Both single-cell neurophysiology and neuroimaging have provided evidence for the modulation of neural firing by gaze position and processing of visual information based on craniotopic or spatiotopic coordinates. Transient remapping of the spatial and temporal properties of neurons contingent on saccadic eye movements has been demonstrated in visual cortex, as well as frontal and parietal areas involved in saliency/priority maps, and is a good candidate to mediate some of the spatial invariance demonstrated by perception. Recent studies suggest that spatiotopic selectivity depends on a low spatial resolution system of maps that operates over a longer time frame than retinotopic processing and is strongly modulated by high-level cognitive factors such as attention. The interaction of an initial and rapid retinotopic processing stage, tied to new fixations, and a longer lasting but less precise nonretinotopic level of visual representation could underlie the perception of both a detailed and a stable visual world across saccadic eye movements.

Perisaccadic Updating of Visual Representations and Attentional States: Linking Behavior and Neurophysiology

During natural vision, saccadic eye movements lead to frequent retinal image changes that result in different neuronal subpopulations representing the same visual feature across fixations. Despite these potentially disruptive changes to the neural representation, our visual percept is remarkably stable. Visual receptive field remapping, characterized as an anticipatory shift in the position of a neuron's spatial receptive field immediately before saccades, has been proposed as one possible neural substrate for visual stability. Many of the specific properties of remapping, e.g., the exact direction of remapping relative to the saccade vector and the precise mechanisms by which remapping could instantiate stability, remain a matter of debate. Recent studies have also shown that visual attention, like perception itself, can be sustained across saccades, suggesting that the attentional control system can also compensate for eye movements. Classical remapping could have an attentional component, or there could be a distinct attentional analog of visual remapping. At this time we do not yet fully understand how the stability of attentional representations relates to perisaccadic receptive field shifts. In this review, we develop a vocabulary for discussing perisaccadic shifts in receptive field location and perisaccadic shifts of attentional focus, review and synthesize behavioral and neurophysiological studies of perisaccadic perception and perisaccadic attention, and identify open questions that remain to be experimentally addressed.

Transsaccadic processing: stability, integration, and the potential role of remapping

While our frequent saccades allow us to sample the complex visual environment in a highly efficient manner, they also raise certain challenges for interpreting and acting upon visual input. In the present, selective review, we discuss key findings from the domains of cognitive psychology, visual perception, and neuroscience concerning two such challenges: (1) maintaining the phenomenal experience of visual stability despite our rapidly shifting gaze, and (2) integrating visual information across discrete fixations. In the first two sections of the article, we focus primarily on behavioral findings. Next, we examine the possibility that a neural phenomenon known as predictive remapping may provide an explanation for aspects of transsaccadic processing. In this section of the article, we delineate and critically evaluate multiple proposals about the potential role of predictive remapping in light of both theoretical principles and empirical findings.

Saccades and shifting receptive fields: anticipating consequences or selecting targets?

Saccadic eye movements cause frequent and substantial displacements of the retinal image, but those displacements go unnoticed. It has been widely assumed that this perceived stability emerges from the shifting of visual receptive fields from their current, presaccadic locations to their future, postsaccadic locations in anticipation of the retinal consequences of saccades. Although evidence consistent with this anticipatory remapping has accumulated over the years, more recent work suggests an alternative view. In this opinion article, we examine the evidence of presaccadic receptive field shifts and their relationship to the perceptual changes that accompany saccades. We argue that both reflect the selection of targets for saccades rather than the anticipation of a displaced retinal image.

Characterizing ensemble statistics: mean size is represented across multiple frames of reference

The visual system represents the overall statistical, not individual, properties of sets. Here we tested the spatial nature of ensemble statistics. We used a mean-size adaptation paradigm (Corbett et al. in Visual Cognition, 20, 211-231, 2012) to examine whether average size is encoded in multiple reference frames. We adapted observers to patches of small- and large-sized dots in opposite regions of the display (left/right or top/bottom) and then tested their perceptions of the sizes of single test dots presented in regions that corresponded to retinotopic, spatiotopic, and hemispheric coordinates within the adapting displays. We observed retinotopic, spatiotopic, and hemispheric adaptation aftereffects, such that participants perceived a test dot as being larger when it was presented in the area adapted to the patch of small dots than when it was presented in the area adapted to large dots. This aftereffect also transferred between eyes. Our results demonstrate that mean size is represented across multiple spatial frames of reference, supporting the proposal that ensemble statistics play a fundamental role in maintaining perceptual stability.

The background is remapped across saccades

Physiological studies have found that neurons prepare for impending eye movements, showing anticipatory responses to stimuli presented at the location of the post-saccadic receptive fields (RFs) (Wurtz in Vis Res 48:2070-2089, 2008). These studies proposed that visual neurons with shifting RFs prepared for the stimuli they would process after an impending saccade. Additionally, psychophysical studies have shown behavioral consequences of those anticipatory responses, including the transfer of aftereffects (Melcher in Nat Neurosci 10:903-907, 2007) and the remapping of attention (Rolfs et al. in Nat Neurosci 14:252-258, 2011). As the physiological studies proposed, the shifting RF mechanism explains the transfer of aftereffects. Recently, a new mechanism based on activation transfer via a saliency map was proposed, which accounted for the remapping of attention (Cavanagh et al. in Trends Cogn Sci 14:147-153, 2010). We hypothesized that there would be different aspects of the remapping corresponding to these different neural mechanisms. This study found that the information in the background was remapped to a similar extent as the figure, provided that the visual context remained stable. We manipulated the status of the figure and the ground in the saliency map and showed that the manipulation modulated the remapping of the figure and the ground in different ways. These results suggest that the visual system has an ability to remap the background as well as the figure, but lacks the ability to modulate the remapping of the background based on the visual context, and that different neural mechanisms might work together to maintain visual stability across saccades.

A reinvestigation of the reference frame of the tilt-adaptation aftereffect

The tilt-adaptation aftereffect (TAE) is the phenomenon that prolonged perception of a tilted ‘adapter’ stimulus affects the perceived tilt of a subsequent ‘tester’ stimulus. Although it is clear that TAE is strongest when adapter and tester are presented at the same location, the reference frame of the effect is debated. Some authors have reported that TAE is spatiotopic (world centred): It occurs when adapter and tester are presented at the same display location, even when this corresponds to different retinal locations. Others have reported that TAE is exclusively retinotopic (eye centred): It occurs only when adapter and tester are presented at the same retinal location, even when this corresponds to different display locations. Because this issue is crucial for models of transsaccadic perception, we reinvestigated the reference frame of TAE. We report that TAE is exclusively retinotopic, supporting the notion that there is no transsaccadic integration of low-level visual information.