Context dependence of receptive field remapping in superior colliculus

Feature-selective responses in macaque visual cortex follow eye movements during natural vision

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

Grouping cells in primate visual cortex

Our perception of how objects are laid out in visual scenes is remarkably stable, despite rapid shifts in the patterns of light that fall on the retina with each saccade. One mechanism that may help establish perceptual stability is border ownership assignment. Studies in macaque area V2 have identified border ownership neurons that signal which side of a border belongs to a foreground surface. This signal persists for hundreds of milliseconds after border ownership has been rendered ambiguous by deleting the stimulus features that distinguish foreground from background. Remarkably, this signal survives eye movements: border ownership neurons also exhibit border ownership signals de novo when an eye movement places the newly ambiguous border within their receptive field. The grouping cell hypothesis proposes the existence of hypothetical grouping cells in a downstream brain area. These cells would compute persistent proto-object representations and therefore have the properties to endow cells in upstream brain areas with selectivity for border ownership. Such grouping cells have been predicted to show a centripetal and persistent pattern of preferred side of ownership for a border placed parallel to the perimeter of their classical receptive field, and such a centripetal ownership preference pattern should also occur de novo in these same cells if an ambiguous border lands in their receptive field after a saccade. It is unknown if grouping cells exist. Here we used laminar multielectrodes in area V4 - the main source of feedback to V2 - of behaving macaques to determine whether such grouping cells exist. Consistent with the model prediction we find a substantial population of neurons with these properties, in all laminar compartments, and they exhibit a response latency that is short enough to act as the source that endows neurons in V2 with selectivity for border ownership. While grouping cell activity provides information about the location of foreground surfaces, these neurons are, counterintuitively, not as strongly tuned for luminance contrast polarity, a feature of those surfaces, as are border ownership cells. Our data suggest a division of labor in which these newly discovered grouping cells provide spatiotemporal continuity of segmented surfaces whereas border ownership cells link this location information with surface features such as luminance contrast.

Inhibitory tagging in the superior colliculus during visual search

Inhibitory tagging is an important feature of many models of saccade target selection, in particular those that are based on the notion of a neural priority map. The superior colliculus (SC) has been suggested as a potential site of such a map, yet it is unknown whether inhibitory tagging is represented in the SC during visual search. In this study, we tested the hypothesis that SC neurons represent inhibitory tagging during search, as might be expected if they contribute to a priority map. To do so, we recorded the activity of SC neurons in a multisaccade visual-search task. On each trial, a single reward-bearing target was embedded in an array of physically identical, potentially reward-bearing targets and physically distinct, non-reward-bearing distractors. The task was to fixate the reward-bearing target. We found that, in the context of this task, the activity of many SC neurons was greater when their response field stimulus was a target than when it was a distractor and was reduced when it had been previously fixated relative to when it had not. Moreover, we found that the previous-fixation-related reduction of activity was larger for targets than for distractors and decreased with increasing time (or number of saccades) since fixation. Taken together, the results suggest that fixated stimuli are transiently inhibited in the SC during search, consistent with the notion that inhibitory tagging plays an important role in visual search and that SC neurons represent this inhibition as part of a priority map used for saccade target selection.NEW & NOTEWORTHY Searching a cluttered scene for an object of interest is a ubiquitous task in everyday life, which we often perform relatively quickly and efficiently. It has been suggested that to achieve such speed and efficiency an inhibitory-tagging mechanism inhibits saccades to objects in the scene once they have been searched and rejected. Here, we demonstrate that the superior colliculus represents this type of inhibition during search, consistent with its role in saccade target selection.

Sensory tuning in neuronal movement commands

Movement control is critical for successful interaction with our environment. However, movement does not occur in complete isolation of sensation, and this is particularly true of eye movements. Here, we show that the neuronal eye movement commands emitted by the superior colliculus (SC), a structure classically associated with oculomotor control, encompass a robust visual sensory representation of eye movement targets. Thus, similar saccades toward different images are associated with different saccade-related "motor" bursts. Such sensory tuning in SC saccade motor commands appeared for all image manipulations that we tested, from simple visual features to real-life object images, and it was also strongest in the most motor neurons in the deeper collicular layers. Visual-feature discrimination performance in the motor commands was also stronger than in visual responses. Comparing SC motor command feature discrimination performance to that in the primary visual cortex during steady-state gaze fixation revealed that collicular motor bursts possess a reliable perisaccadic sensory representation of the peripheral saccade target's visual appearance, exactly when retinal input is expected to be most uncertain. Our results demonstrate that SC neuronal movement commands likely serve a fundamentally sensory function.

Bilateral increase in MEG planar gradients prior to saccade onset

Every time we move our eyes, the retinal locations of objects change. To distinguish the changes caused by eye movements from actual external motion of the objects, the visual system is thought to anticipate the consequences of eye movements (saccades). Single neuron recordings have indeed demonstrated changes in receptive fields before saccade onset. Although some EEG studies with human participants have also demonstrated a pre-saccadic increased potential over the hemisphere that will process a stimulus after a saccade, results have been mixed. Here, we used magnetoencephalography to investigate the timing and lateralization of visually evoked planar gradients before saccade onset. We modelled the gradients from trials with both a saccade and a stimulus as the linear combination of the gradients from two conditions with either only a saccade or only a stimulus. We reasoned that any residual gradients in the condition with both a saccade and a stimulus must be uniquely linked to visually-evoked neural activity before a saccade. We observed a widespread increase in residual planar gradients. Interestingly, this increase was bilateral, showing activity both contralateral and ipsilateral to the stimulus, i.e. over the hemisphere that would process the stimulus after saccade offset. This pattern of results is consistent with predictive pre-saccadic changes involving both the current and the future receptive fields involved in processing an attended object, well before the start of the eye movement. The active, sensorimotor coupling of vision and the oculomotor system may underlie the seamless subjective experience of stable and continuous perception.

Spatiotopic and retinotopic memory in the context of natural images

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.

A sensory memory to preserve visual representations across eye movements

Saccadic eye movements (saccades) disrupt the continuous flow of visual information, yet our perception of the visual world remains uninterrupted. Here we assess the representation of the visual scene across saccades from single-trial spike trains of extrastriate visual areas, using a combined electrophysiology and statistical modeling approach. Using a model-based decoder we generate a high temporal resolution readout of visual information, and identify the specific changes in neurons' spatiotemporal sensitivity that underly an integrated perisaccadic representation of visual space. Our results show that by maintaining a memory of the visual scene, extrastriate neurons produce an uninterrupted representation of the visual world. Extrastriate neurons exhibit a late response enhancement close to the time of saccade onset, which preserves the latest pre-saccadic information until the post-saccadic flow of retinal information resumes. These results show how our brain exploits available information to maintain a representation of the scene while visual inputs are disrupted.

Visual Remapping

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The functional roles of neural remapping in cortex

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

Perisaccadic encoding of temporal information in macaque area V4

The accurate processing of temporal information is of critical importance in everyday life. Yet, psychophysical studies in humans have shown that the perception of time is distorted around saccadic eye movements. The neural correlates of this misperception are still poorly understood. Behavioral and neural evidence suggest that it is tightly linked to other known perisaccadic modulations of visual perception. To further our understanding of how temporal processing is affected by saccades, we studied the representations of brief visual time intervals during fixation and saccades in area V4 of two awake macaques. We presented random sequences of vertical bar stimuli and extracted neural responses to double-pulse stimulation at varying interstimulus intervals. Our results show that temporal information about very brief intervals of as brief as 20 ms is reliably represented in the multiunit activity in area V4. Response latencies were not systematically modulated by the saccade. However, a general increase in perisaccadic activity altered the ratio of response amplitudes within stimulus pairs compared with fixation. In line with previous studies showing that the perception of brief time intervals is partly based on response levels, this may be seen as a possible correlate of the perisaccadic misperception of time.NEW & NOTEWORTHY We investigated for the first time how temporal information on very brief timescales is represented in area V4 around the time of saccadic eye movements. Overall, the responses showed an unexpectedly precise representation of time intervals. Our finding of a perisaccadic modulation of relative response amplitudes introduces a new possible correlate of saccade-related perceptual distortions of time.

Low-Level Visual Information Is Maintained across Saccades, Allowing for a Postsaccadic Handoff between Visual Areas

Experience seems continuous and detailed despite saccadic eye movements changing retinal input several times per second. There is debate whether neural signals related to updating across saccades contain information about stimulus features, or only location pointers without visual details. We investigated the time course of low-level visual information processing across saccades by decoding the spatial frequency of a stationary stimulus that changed from one visual hemifield to the other because of a horizontal saccadic eye movement. We recorded magnetoencephalography while human subjects (both sexes) monitored the orientation of a grating stimulus, making spatial frequency task irrelevant. Separate trials, in which subjects maintained fixation, were used to train a classifier, whose performance was then tested on saccade trials. Decoding performance showed that spatial frequency information of the presaccadic stimulus remained present for ∼200 ms after the saccade, transcending retinotopic specificity. Postsaccadic information ramped up rapidly after saccade offset. There was an overlap of over 100 ms during which decoding was significant from both presaccadic and postsaccadic processing areas. This suggests that the apparent richness of perception across saccades may be supported by the continuous availability of low-level information with a "soft handoff" of information during the initial processing sweep of the new fixation.SIGNIFICANCE STATEMENT Saccades create frequent discontinuities in visual input, yet perception appears stable and continuous. How is this discontinuous input processed resulting in visual stability? Previous studies have focused on presaccadic remapping. Here we examined the time course of processing of low-level visual information (spatial frequency) across saccades with magnetoencephalography. The results suggest that spatial frequency information is not predictively remapped but also is not discarded. Instead, they suggest a soft handoff over time between different visual areas, making this information continuously available across the saccade. Information about the presaccadic stimulus remains available, while the information about the postsaccadic stimulus has also become available. The simultaneous availability of both the presaccadic and postsaccadic information could enable rich and continuous perception across saccades.

Perisaccadic remapping: What? How? Why?

About 25 years ago, the discovery of receptive field (RF) remapping in the parietal cortex of nonhuman primates revealed that visual RFs, widely assumed to have a fixed retinotopic organization, can change position before every saccade. Measuring such changes can be deceptively difficult. As a result, studies that followed have generated a fascinating but somewhat confusing picture of the phenomenon. In this review, we describe how observations of RF remapping depend on the spatial and temporal sampling of visual RFs and saccade directions. Further, we summarize some of the theories of how remapping might occur in neural circuitry. Finally, based on neurophysiological and psychophysical observations, we discuss the ways in which remapping information might facilitate computations in downstream brain areas.

Saccades Trigger Predictive Updating of Attentional Topography in Area V4

During natural behavior, saccades and attention act together to allocate limited neural resources. Attention is generally mediated by retinotopic visual neurons; therefore, specific neurons representing attended features change with each saccade. We investigated the neural mechanisms that allow attentional targeting in the face of saccades. Specifically, we looked for predictive changes in attentional modulation state or receptive field position that could stabilize attentional representations across saccades in area V4, known to be necessary for attention-dependent behavior. We recorded from neurons in monkeys performing a novel spatiotopic attention task, in which performance depended on accurate saccade compensation. Measurements of attentional modulation revealed a predictive attentional "hand-off" corresponding to a presaccadic transfer of attentional state from neurons inside the attentional focus before the saccade to those that will be inside the focus after the saccade. The predictive nature of the hand-off ensures that attentional brain maps are properly configured immediately after each saccade.

Memory for retinotopic locations is more accurate than memory for spatiotopic locations, even for visually guided reaching

To interact successfully with objects, we must maintain stable representations of their locations in the world. However, their images on the retina may be displaced several times per second by large, rapid eye movements. A number of studies have demonstrated that visual processing is heavily influenced by gaze-centered (retinotopic) information, including a recent finding that memory for an object's location is more accurate and precise in gaze-centered (retinotopic) than world-centered (spatiotopic) coordinates (Golomb & Kanwisher, 2012b). This effect is somewhat surprising, given our intuition that behavior is successfully guided by spatiotopic representations. In the present experiment, we asked whether the visual system may rely on a more spatiotopic memory store depending on the mode of responding. Specifically, we tested whether reaching toward and tapping directly on an object's location could improve memory for its spatiotopic location. Participants performed a spatial working memory task under four conditions: retinotopic vs. spatiotopic task, and computer mouse click vs. touchscreen reaching response. When participants responded by clicking with a mouse on the screen, we replicated Golomb & Kanwisher's original results, finding that memory was more accurate in retinotopic than spatiotopic coordinates and that the accuracy of spatiotopic memory deteriorated substantially more than retinotopic memory with additional eye movements during the memory delay. Critically, we found the same pattern of results when participants responded by using their finger to reach and tap the remembered location on the monitor. These results further support the hypothesis that spatial memory is natively retinotopic; we found no evidence that engaging the motor system improves spatiotopic memory across saccades.

Corollary Discharge Contributions to Perceptual Continuity Across Saccades

Our vision depends upon shifting our high-resolution fovea to objects of interest in the visual field. Each saccade displaces the image on the retina, which should produce a chaotic scene with jerks occurring several times per second. It does not. This review examines how an internal signal in the primate brain (a corollary discharge) contributes to visual continuity across saccades. The article begins with a review of evidence for a corollary discharge in the monkey and evidence from inactivation experiments that it contributes to perception. The next section examines a specific neuronal mechanism for visual continuity, based on corollary discharge that is referred to as visual remapping. Both the basic characteristics of this anticipatory remapping and the factors that control it are enumerated. The last section considers hypotheses relating remapping to the perceived visual continuity across saccades, including remapping's contribution to perceived visual stability across saccades.

Remapping locations and features across saccades: a dual-spotlight theory of attentional updating

How do we maintain visual stability across eye movements? Much work has focused on how visual information is rapidly updated to maintain spatiotopic representations. However, predictive spatial remapping is only part of the story. Here I review key findings, recent debates, and open questions regarding remapping and its implications for visual attention and perception. This review focuses on two key questions: when does remapping occur, and what is the impact on feature perception? Findings are reviewed within the framework of a two-stage, or dual- spotlight, remapping process, where spatial attention must be both updated to the new location (fast, predictive stage) and withdrawn from the previous retinotopic location (slow, post-saccadic stage), with a particular focus on the link between spatial and feature information across eye movements.

Mechanisms of Saccadic Suppression in Primate Cortical Area V4

Psychophysical studies have shown that subjects are often unaware of visual stimuli presented around the time of an eye movement. This saccadic suppression is thought to be a mechanism for maintaining perceptual stability. The brain might accomplish saccadic suppression by reducing the gain of visual responses to specific stimuli or by simply suppressing firing uniformly for all stimuli. Moreover, the suppression might be identical across the visual field or concentrated at specific points. To evaluate these possibilities, we recorded from individual neurons in cortical area V4 of nonhuman primates trained to execute saccadic eye movements. We found that both modes of suppression were evident in the visual responses of these neurons and that the two modes showed different spatial and temporal profiles: while gain changes started earlier and were more widely distributed across visual space, nonspecific suppression was found more often in the peripheral visual field, after the completion of the saccade. Peripheral suppression was also associated with increased noise correlations and stronger local field potential oscillations in the α frequency band. This pattern of results suggests that saccadic suppression shares some of the circuitry responsible for allocating voluntary attention.

SIGNIFICANCE STATEMENT

We explore our surroundings by looking at things, but each eye movement that we make causes an abrupt shift of the visual input. Why doesn't the world look like a film recorded on a shaky camera? The answer in part is a brain mechanism called saccadic suppression, which reduces the responses of visual neurons around the time of each eye movement. Here we reveal several new properties of the underlying mechanisms. First, the suppression operates differently in the central and peripheral visual fields. Second, it appears to be controlled by oscillations in the local field potentials at frequencies traditionally associated with attention. These results suggest that saccadic suppression shares the brain circuits responsible for actively ignoring irrelevant stimuli.

Circuits for presaccadic visual remapping

Saccadic eye movements rapidly displace the image of the world that is projected onto the retinas. In anticipation of each saccade, many neurons in the visual system shift their receptive fields. This presaccadic change in visual sensitivity, known as remapping, was first documented in the parietal cortex and has been studied in many other brain regions. Remapping requires information about upcoming saccades via corollary discharge. Analyses of neurons in a corollary discharge pathway that targets the frontal eye field (FEF) suggest that remapping may be assembled in the FEF's local microcircuitry. Complementary data from reversible inactivation, neural recording, and modeling studies provide evidence that remapping contributes to transsaccadic continuity of action and perception. Multiple forms of remapping have been reported in the FEF and other brain areas, however, and questions remain about the reasons for these differences. In this review of recent progress, we identify three hypotheses that may help to guide further investigations into the structure and function of circuits for remapping.

No Evidence for Automatic Remapping of Stimulus Features or Location Found with fMRI

The input to our visual system shifts every time we move our eyes. To maintain a stable percept of the world, visual representations must be updated with each saccade. Near the time of a saccade, neurons in several visual areas become sensitive to the regions of visual space that their receptive fields occupy after the saccade. This process, known as remapping, transfers information from one set of neurons to another, and may provide a mechanism for visual stability. However, it is not clear whether remapping transfers information about stimulus features in addition to information about stimulus location. To investigate this issue, we recorded blood-oxygen-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses while human subjects viewed images of faces and houses (two visual categories with many feature differences). Immediately after some image presentations, subjects made a saccade that moved the previously stimulated location to the opposite side of the visual field. We then used a combination of univariate analyses and multivariate pattern analyses to test whether information about stimulus location and stimulus features were remapped to the ipsilateral hemisphere after the saccades. We found no reliable indication of stimulus feature remapping in any region. However, we also found no reliable indication of stimulus location remapping, despite the fact that our paradigm was highly similar to previous fMRI studies of remapping. The absence of location remapping in our study precludes strong conclusions regarding feature remapping. However, these results also suggest that measurement of location remapping with fMRI depends strongly on the details of the experimental paradigm used. We highlight differences in our approach from the original fMRI studies of remapping, discuss potential reasons for the failure to generalize prior location remapping results, and suggest directions for future research.

A Probabilistic Approach to Receptive Field Mapping in the Frontal Eye Fields

Studies of the neuronal mechanisms of perisaccadic vision often lack the resolution needed to determine important changes in receptive field (RF) structure. Such limited analytical power can lead to inaccurate descriptions of visuomotor processing. To address this issue, we developed a precise, probabilistic technique that uses a generalized linear model (GLM) for mapping the visual RFs of frontal eye field (FEF) neurons during stable fixation (Mayo et al., 2015). We previously found that full-field RF maps could be obtained using 1–8 dot stimuli presented at frame rates of 10–150 ms. FEF responses were generally robust to changes in the number of stimuli presented or the rate of presentation, which allowed us to visualize RFs over a range of spatial and temporal resolutions. Here, we compare the quality of RFs obtained over different stimulus and GLM parameters to facilitate future work on the detailed mapping of FEF RFs. We first evaluate the interactions between the number of stimuli presented per trial, the total number of trials, and the quality of RF mapping. Next, we vary the spatial resolution of our approach to illustrate the tradeoff between visualizing RF sub-structure and sampling at high resolutions. We then evaluate local smoothing as a possible correction for situations where under-sampling occurs. Finally, we provide a preliminary demonstration of the usefulness of a probabilistic approach for visualizing full-field perisaccadic RF shifts. Our results present a powerful, and perhaps necessary, framework for studying perisaccadic vision that is applicable to FEF and possibly other visuomotor regions of the brain.

Eye position effects on the remapped memory trace of visual motion in cortical area MST

After a saccade, most MST neurons respond to moving visual stimuli that had existed in their post-saccadic receptive fields and turned off before the saccade ("trans-saccadic memory remapping"). Neuronal responses in higher visual processing areas are known to be modulated in relation to gaze angle to represent image location in spatiotopic coordinates. In the present study, we investigated the eye position effects after saccades and found that the gaze angle modulated the visual sensitivity of MST neurons after saccades both to the actually existing visual stimuli and to the visual memory traces remapped by the saccades. We suggest that two mechanisms, trans-saccadic memory remapping and gaze modulation, work cooperatively in individual MST neurons to represent a continuous visual world.

Perisaccadic Updating of Visual Representations and Attentional States: Linking Behavior and Neurophysiology

During natural vision, saccadic eye movements lead to frequent retinal image changes that result in different neuronal subpopulations representing the same visual feature across fixations. Despite these potentially disruptive changes to the neural representation, our visual percept is remarkably stable. Visual receptive field remapping, characterized as an anticipatory shift in the position of a neuron's spatial receptive field immediately before saccades, has been proposed as one possible neural substrate for visual stability. Many of the specific properties of remapping, e.g., the exact direction of remapping relative to the saccade vector and the precise mechanisms by which remapping could instantiate stability, remain a matter of debate. Recent studies have also shown that visual attention, like perception itself, can be sustained across saccades, suggesting that the attentional control system can also compensate for eye movements. Classical remapping could have an attentional component, or there could be a distinct attentional analog of visual remapping. At this time we do not yet fully understand how the stability of attentional representations relates to perisaccadic receptive field shifts. In this review, we develop a vocabulary for discussing perisaccadic shifts in receptive field location and perisaccadic shifts of attentional focus, review and synthesize behavioral and neurophysiological studies of perisaccadic perception and perisaccadic attention, and identify open questions that remain to be experimentally addressed.

Two distinct types of remapping in primate cortical area V4

Visual neurons typically receive information from a limited portion of the retina, and such receptive fields are a key organizing principle for much of visual cortex. At the same time, there is strong evidence that receptive fields transiently shift around the time of saccades. The nature of the shift is controversial: Previous studies have found shifts consistent with a role for perceptual constancy; other studies suggest a role in the allocation of spatial attention. Here we present evidence that both the previously documented functions exist in individual neurons in primate cortical area V4. Remapping associated with perceptual constancy occurs for saccades in all directions, while attentional shifts mainly occur for neurons with receptive fields in the same hemifield as the saccade end point. The latter are relatively sluggish and can be observed even during saccade planning. Overall these results suggest a complex interplay of visual and extraretinal influences during the execution of saccades.

Visual receptive fields are known to change positions around the time of a saccade, but the nature of this remapping is unclear. Here Neupane and colleagues show that neurons in area V4 of the visual cortex exhibit two types of remapping, one consistent with a role in maintaining perceptual stability, and a second that seems to reflect shifts of attention.

Remapping, Spatial Stability, and Temporal Continuity: From the Pre-Saccadic to Postsaccadic Representation of Visual Space in LIP

As our eyes move, we have a strong percept that the world is stable in space and time; however, the signals in cortex coming from the retina change with each eye movement. It is not known how this changing input produces the visual percept we experience, although the predictive remapping of receptive fields has been described as a likely candidate. To explain how remapping accounts for perceptual stability, we examined responses of neurons in the lateral intraparietal area while animals performed a visual foraging task. When a stimulus was brought into the response field of a neuron that exhibited remapping, the onset of the postsaccadic representation occurred shortly after the saccade ends. Whenever a stimulus was taken out of the response field, the presaccadic representation abruptly ended shortly after the eyes stopped moving. In the 38% (20/52) of neurons that exhibited remapping, there was no more than 30 ms between the end of the presaccadic representation and the start of the postsaccadic representation and, in some neurons, and the population as a whole, it was continuous. We conclude by describing how this seamless shift from a presaccadic to postsaccadic representation could contribute to spatial stability and temporal continuity.

A sensorimotor role for traveling waves in primate visual cortex

Traveling waves of neural activity are frequently observed to occur in concert with the presentation of a sensory stimulus or the execution of a movement. Although such waves have been studied for decades, little is known about their function. Here we show that traveling waves in the primate extrastriate visual cortex provide a means of integrating sensory and motor signals. Specifically, we describe a traveling wave of local field potential (LFP) activity in cortical area V4 of macaque monkeys that is triggered by the execution of saccadic eye movements. These waves sweep across the V4 retinotopic map, following a consistent path from the foveal to the peripheral representations of space; their amplitudes correlate with the direction and size of each saccade. Moreover, these waves are associated with a reorganization of the postsaccadic neuronal firing patterns, which follow a similar retinotopic progression, potentially prioritizing the processing of behaviorally relevant stimuli.

Predictive saccade target selection in superior colliculus during visual search

Searching for a visual object naturally involves sequences of gaze fixations, during which the current foveal image is analyzed and the next object to inspect is selected as a saccade target. Fixation durations during such sequences are short, suggesting that saccades may be concurrently processed. Therefore, the selection of the next saccade target may occur before the current saccade target is acquired. To test this hypothesis, we trained four female rhesus monkeys (Macaca mulatta) to perform a multiple-fixation visual conjunction search task. We simultaneously recorded the activity of sensorimotor neurons in the midbrain superior colliculus (SC) in two monkeys. In this task, monkeys made multiple fixations before foveating the target. Fixation durations were significantly shorter than the latency of the initial responses to the search display, with approximately one-quarter being shorter than the shortest response latencies. The time at which SC sensorimotor activity discriminated the target from distracters occurred significantly earlier for the selection of subsequent fixations than for the selection of the first fixation. Target selection during subsequent fixations occurred even before the visual afferent delay in more than half of the neuronal sample, suggesting that the process of selection can encompass at least two future saccade targets. This predictive selection was present even when differences in saccade latencies were taken into account. Altogether, these findings demonstrate how neural representations on the visual salience map are processed in parallel, thus facilitating visual search.

Neurons in cortical area MST remap the memory trace of visual motion across saccadic eye movements

Perception of a stable visual world despite eye motion requires integration of visual information across saccadic eye movements. To investigate how the visual system deals with localization of moving visual stimuli across saccades, we observed spatiotemporal changes of receptive fields (RFs) of motion-sensitive neurons across periods of saccades in the middle temporal (MT) and medial superior temporal (MST) areas. We found that the location of the RFs moved with shifts of eye position due to saccades, indicating that motion-sensitive neurons in both areas have retinotopic RFs across saccades. Different characteristic responses emerged when the moving visual stimulus was turned off before the saccades. For MT neurons, virtually no response was observed after the saccade, suggesting that the responses of these neurons simply reflect the reafferent visual information. In contrast, most MST neurons increased their firing rates when a saccade brought the location of the visual stimulus into their RFs, where the visual stimulus itself no longer existed. These findings suggest that the responses of such MST neurons after saccades were evoked by a memory of the stimulus that had preexisted in the postsaccadic RFs ("memory remapping"). A delayed-saccade paradigm further revealed that memory remapping in MST was linked to the saccade itself, rather than to a shift in attention. Thus, the visual motion information across saccades was integrated in spatiotopic coordinates and represented in the activity of MST neurons. This is likely to contribute to the perception of a stable visual world in the presence of eye movements.

Anticipatory remapping of attentional priority across the entire visual field

It has been suggested that one way we may create a stable percept of the visual world across multiple eye movements is to pass information from one set of neurons to another around the time of each eye movement. Previous studies have shown that some neurons in the lateral intraparietal area (LIP) exhibit anticipatory remapping: these neurons produce a visual response to a stimulus that will enter their receptive field after a saccade but before it actually does so. LIP responses during fixation are thought to represent attentional priority, behavioral relevance, or value. In this study, we test whether the remapped response represents this attentional priority by examining the activity of LIP neurons while animals perform a visual foraging task. We find that the population responds more to a target than to a distractor before the saccade even begins to bring the stimulus into the receptive field. Within 20 ms of the saccade ending, the responses in almost one-third of LIP neurons closely resemble the responses that will emerge during stable fixation. Finally, we show that, in these neurons and in the population as a whole, this remapping occurs for all stimuli in all locations across the visual field and for both long and short saccades. We conclude that this complete remapping of attentional priority across the visual field could underlie spatial stability across saccades.

Perisaccadic remapping and rescaling of visual responses in macaque superior colliculus

Visual neurons have spatial receptive fields that encode the positions of objects relative to the fovea. Because foveate animals execute frequent saccadic eye movements, this position information is constantly changing, even though the visual world is generally stationary. Interestingly, visual receptive fields in many brain regions have been found to exhibit changes in strength, size, or position around the time of each saccade, and these changes have often been suggested to be involved in the maintenance of perceptual stability. Crucial to the circuitry underlying perisaccadic changes in visual receptive fields is the superior colliculus (SC), a brainstem structure responsible for integrating visual and oculomotor signals. In this work we have studied the time-course of receptive field changes in the SC. We find that the distribution of the latencies of SC responses to stimuli placed outside the fixation receptive field is bimodal: The first mode is comprised of early responses that are temporally locked to the onset of the visual probe stimulus and stronger for probes placed closer to the classical receptive field. We suggest that such responses are therefore consistent with a perisaccadic rescaling, or enhancement, of weak visual responses within a fixed spatial receptive field. The second mode is more similar to the remapping that has been reported in the cortex, as responses are time-locked to saccade onset and stronger for stimuli placed in the postsaccadic receptive field location. We suggest that these two temporal phases of spatial updating may represent different sources of input to the SC.

Eye movements reset visual perception

Human vision uses saccadic eye movements to rapidly shift the sensitive foveal portion of our retina to objects of interest. For vision to function properly amidst these ballistic eye movements, a mechanism is needed to extract discrete percepts on each fixation from the continuous stream of neural activity that spans fixations. The speed of visual parsing is crucial because human behaviors ranging from reading to driving to sports rely on rapid visual analysis. We find that a brain signal associated with moving the eyes appears to play a role in resetting visual analysis on each fixation, a process that may aid in parsing the neural signal. We quantified the degree to which the perception of tilt is influenced by the tilt of a stimulus on a preceding fixation. Two key conditions were compared, one in which a saccade moved the eyes from one stimulus to the next and a second simulated saccade condition in which the stimuli moved in the same manner but the subjects did not move their eyes. We find that there is a brief period of time at the start of each fixation during which the tilt of the previous stimulus influences perception (in a direction opposite to the tilt aftereffect)--perception is not instantaneously reset when a fixation starts. Importantly, the results show that this perceptual bias is much greater, with nearly identical visual input, when saccades are simulated. This finding suggests that, in real-saccade conditions, some signal related to the eye movement may be involved in the reset phenomenon. While proprioceptive information from the extraocular muscles is conceivably a factor, the fast speed of the effect we observe suggests that a more likely mechanism is a corollary discharge signal associated with eye movement.

Allocation of attention across saccades

Whenever the eyes move, spatial attention must keep track of the locations of targets as they shift on the retina. This study investigated transsaccadic updating of visual attention to cued targets. While observers prepared a saccade, we flashed an irrelevant, but salient, color cue in their visual periphery and measured the allocation of spatial attention before and after the saccade using a tilt discrimination task. We found that just before the saccade, attention was allocated to the cue's future retinal location, its predictively "remapped" location. Attention was sustained at the cue's location in the world across the saccade, despite the change of retinal position whereas it decayed quickly at the retinal location of the cue, after the eye landed. By extinguishing the color cue across the saccade, we further demonstrate that the visual system relies only on predictive allocation of spatial attention, as the presence of the cue after the saccade did not substantially affect attentional allocation. These behavioral results support and extend physiological evidence showing predictive activation of visual neurons when an attended stimulus will fall in their receptive field after a saccade. Our results show that tracking of spatial locations across saccades is a plausible consequence of physiological remapping.

Spatiotemporal structure of visual receptive fields in macaque superior colliculus

Saccades are useful for directing the high-acuity fovea to visual targets that are of behavioral relevance. The selection of visual targets for eye movements involves the superior colliculus (SC), where many neurons respond to visual stimuli. Many of these neurons are also activated before and during saccades of specific directions and amplitudes. Although the role of the SC in controlling eye movements has been thoroughly examined, far less is known about the nature of the visual responses in this area. We have, therefore, recorded from neurons in the intermediate layers of the macaque SC, while using a sparse-noise mapping procedure to obtain a detailed characterization of the spatiotemporal structure of visual receptive fields. We find that SC responses to flashed visual stimuli start roughly 50 ms after the onset of the stimulus and last for on average ∼70 ms. About 50% of these neurons are strongly suppressed by visual stimuli flashed at certain locations flanking the excitatory center, and the spatiotemporal pattern of suppression exerts a predictable influence on the timing of saccades. This suppression may, therefore, contribute to the filtering of distractor stimuli during target selection. We also find that saccades affect the processing of visual stimuli by SC neurons in a manner that is quite similar to the saccadic suppression and postsaccadic enhancement that has been observed in the cortex and in perception. However, in contrast to what has been observed in the cortex, decreased visual sensitivity was generally associated with increased firing rates, while increased sensitivity was associated with decreased firing rates. Overall, these results suggest that the processing of visual stimuli by SC receptive fields can influence oculomotor behavior and that oculomotor signals originating in the SC can shape perisaccadic visual perception.

Division of labor in frontal eye field neurons during presaccadic remapping of visual receptive fields

Our percept of visual stability across saccadic eye movements may be mediated by presaccadic remapping. Just before a saccade, neurons that remap become visually responsive at a future field (FF), which anticipates the saccade vector. Hence, the neurons use corollary discharge of saccades. Many of the neurons also decrease their response at the receptive field (RF). Presaccadic remapping occurs in several brain areas including the frontal eye field (FEF), which receives corollary discharge of saccades in its layer IV from a collicular-thalamic pathway. We studied, at two levels, the microcircuitry of remapping in the FEF. At the laminar level, we compared remapping between layers IV and V. At the cellular level, we compared remapping between different neuron types of layer IV. In the FEF in four monkeys (Macaca mulatta), we identified 27 layer IV neurons with orthodromic stimulation and 57 layer V neurons with antidromic stimulation from the superior colliculus. With the use of established criteria, we classified the layer IV neurons as putative excitatory (n = 11), putative inhibitory (n = 12), or ambiguous (n = 4). We found that just before a saccade, putative excitatory neurons increased their visual response at the RF, putative inhibitory neurons showed no change, and ambiguous neurons increased their visual response at the FF. None of the neurons showed presaccadic visual changes at both RF and FF. In contrast, neurons in layer V showed full remapping (at both the RF and FF). Our data suggest that elemental signals for remapping are distributed across neuron types in early cortical processing and combined in later stages of cortical microcircuitry.