Anticipatory remapping of attentional priority across the entire visual field

Gaze and attention: Mechanisms underlying the therapeutic effect of optokinetic stimulation in spatial neglect

Left smooth pursuit eye movement training in response to large-field visual motion (optokinetic stimulation) has become a promising rehabilitation method in left spatial inattention or neglect. The mechanisms underlying the therapeutic effect, however, remain unknown. During optokinetic stimulation, there is an error in visual localisation ahead of the line of sight. This could indicate a change in the brain's estimate of one's own direction of gaze. We hypothesized that optokinetic stimulation changes the brain's estimate of gaze. Because this estimate is critical for coding the locus of attention in the visual space relative to the body and across sensory modalities, its change might underlie the change in spatial attention. Here, we report that in healthy participants optokinetic stimulation causes not only a directional bias in the proprioceptive signal from the extraocular muscles, but also a corresponding shift of the locus of attention. Both changes outlasted the period of stimulation. This result forms a step in investigating a causal link between the adaptation in the sensorimotor gaze signals and the recovery in spatial neglect.

Recurrent neural networks that learn multi-step visual routines with reinforcement learning

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

Inversion of pop-out for a distracting feature dimension in monkey visual cortex

During visual search, it is important to reduce the interference of distracting objects in the scene. The neuronal responses elicited by the search target stimulus are typically enhanced. However, it is equally important to suppress the representations of distracting stimuli, especially if they are salient and capture attention. We trained monkeys to make an eye movement to a unique "pop-out" shape stimulus among an array of distracting stimuli. One of these distractors had a salient color that varied across trials and differed from the color of the other stimuli, causing it to also pop-out. The monkeys were able to select the pop-out shape target with high accuracy and actively avoided the pop-out color distractor. This behavioral pattern was reflected in the activity of neurons in area V4. Responses to the shape targets were enhanced, while the activity evoked by the pop-out color distractor was only briefly enhanced, directly followed by a sustained period of pronounced suppression. These behavioral and neuronal results demonstrate a cortical selection mechanism that rapidly inverts a pop-out signal to "pop-in" for an entire feature dimension thereby facilitating goal-directed visual search in the presence of salient distractors.

Oculomotor corollary discharge signaling is related to repetitive behavior in children with autism spectrum disorder

Corollary discharge (CD) signals are "copies" of motor signals sent to sensory regions that allow animals to adjust sensory consequences of self-generated actions. Autism spectrum disorder (ASD) is characterized by sensory and motor deficits, which may be underpinned by altered CD signaling. We evaluated oculomotor CD using the blanking task, which measures the influence of saccades on visual perception, in 30 children with ASD and 35 typically developing (TD) children. Participants were instructed to make a saccade to a visual target. Upon saccade initiation, the presaccadic target disappeared and reappeared to the left or right of the original position. Participants indicated the direction of the jump. With intact CD, participants can make accurate perceptual judgements. Otherwise, participants may use saccade landing site as a proxy of the presaccadic target and use it to inform perception. We used multilevel modeling to examine the influence of saccade landing site on trans-saccadic perceptual judgements. We found that, compared with TD participants, children with ASD were more sensitive to target displacement and less reliant on saccade landing site when spatial uncertainty of the post-saccadic target was high. This pattern was driven by ASD participants with less severe restricted and repetitive behaviors. These results suggest a relationship between altered CD signaling and core ASD symptoms.

Transsaccadic integration of visual information is predictive, attention-based, and spatially precise

Eye movements produce shifts in the positions of objects in the retinal image, but observers are able to integrate these shifting retinal images into a coherent representation of visual space. This ability is thought to be mediated by attention-dependent saccade-related neural activity that is used by the visual system to anticipate the retinal consequences of impending eye movements. Previous investigations of the perceptual consequences of this predictive activity typically infer attentional allocation using indirect measures such as accuracy or reaction time. Here, we investigated the perceptual consequences of saccades using an objective measure of attentional allocation, reverse correlation. Human observers executed a saccade while monitoring a flickering target object flanked by flickering distractors and reported whether the average luminance of the target was lighter or darker than the background. Successful task performance required subjects to integrate visual information across the saccade. A reverse correlation analysis yielded a spatiotemporal "psychophysical kernel" characterizing how different parts of the stimulus contributed to the luminance decision throughout each trial. Just before the saccade, observers integrated luminance information from a distractor located at the post-saccadic retinal position of the target, indicating a predictive perceptual updating of the target. Observers did not integrate information from distractors placed in alternative locations, even when they were nearer to the target object. We also observed simultaneous predictive perceptual updating for two spatially distinct targets. These findings suggest both that shifting neural representations mediate the coherent representation of visual space, and that these shifts have significant consequences for transsaccadic perception.

Visual Remapping

Our visual system is fundamentally retinotopic. When viewing a stable scene, each eye movement shifts object features and locations on the retina. Thus, sensory representations must be updated, or remapped, across saccades to align presaccadic and postsaccadic inputs. The earliest remapping studies focused on anticipatory, presaccadic shifts of neuronal spatial receptive fields. Over time, it has become clear that there are multiple forms of remapping and that different forms of remapping may be mediated by different neural mechanisms. This review attempts to organize the various forms of remapping into a functional taxonomy based on experimental data and ongoing debates about forward versus convergent remapping, presaccadic versus postsaccadic remapping, and spatial versus attentional remapping. We integrate findings from primate neurophysiological, human neuroimaging and behavioral, and computational modeling studies. We conclude by discussing persistent open questions related to remapping, with specific attention to binding of spatial and featural information during remapping and speculations about remapping's functional significance. Expected final online publication date for the Annual Review of Vision Science, Volume 7 is September 2021. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

The functional roles of neural remapping in cortex

Remapping is a property of some cortical and subcortical neurons that update their responses around the time of an eye movement to account for the shift of stimuli on the retina due to the saccade. Physiologically, remapping is traditionally tested by briefly presenting a single stimulus around the time of the saccade and looking at the onset of the response and the locations in space to which the neuron is responsive. Here we suggest that a better way to understand the functional role of remapping is to look at the time at which the neural signal emerges when saccades are made across a stable scene. Based on data obtained using this approach, we suggest that remapping in the lateral intraparietal area is sufficient to play a role in maintaining visual stability across saccades, whereas in the frontal eye field, remapped activity carries information that affects future saccadic choices and, in a separate subset of neurons, is used to maintain a map of locations in the scene that have been previously fixated.

The roles of the lateral intraparietal area and frontal eye field in guiding eye movements in free viewing search behavior

The lateral intraparietal area (LIP) and frontal eye field (FEF) have been shown to play significant roles in oculomotor control, yet most studies have found that the two areas behave similarly. To identify the unique roles each area plays in guiding eye movements, we recorded 200 LIP neurons and 231 FEF neurons from four animals performing a free viewing visual foraging task. We analyzed how neuronal responses were modulated by stimulus identity and the animals' choice of where to make a saccade. We additionally analyzed the comodulation of the sensory signals and the choice signal to identify how the sensory signals drove the choice. We found a clearly defined division of labor: LIP provided a stable map integrating task rules and stimulus identity, whereas FEF responses were dynamic, representing more complex information and, just before the saccade, were integrated with task rules and stimulus identity to decide where to move the eye.NEW & NOTEWORTHY The lateral intrapareital area (LIP) and frontal eye field (FEF) are known to contribute to guiding eye movements, but little is known about the unique roles that each area plays. Using a free viewing visual search task, we found that LIP provides a stable map of the visual world, integrating task rules and stimulus identity. FEF activity is consistently modulated by more complex information but, just before the saccade, integrates all the information to make the final decision about where to move.

Perisaccadic remapping: What? How? Why?

About 25 years ago, the discovery of receptive field (RF) remapping in the parietal cortex of nonhuman primates revealed that visual RFs, widely assumed to have a fixed retinotopic organization, can change position before every saccade. Measuring such changes can be deceptively difficult. As a result, studies that followed have generated a fascinating but somewhat confusing picture of the phenomenon. In this review, we describe how observations of RF remapping depend on the spatial and temporal sampling of visual RFs and saccade directions. Further, we summarize some of the theories of how remapping might occur in neural circuitry. Finally, based on neurophysiological and psychophysical observations, we discuss the ways in which remapping information might facilitate computations in downstream brain areas.

Behavioral and neuronal study of inhibition of return in barn owls

Inhibition of return (IOR) is the reduction of detection speed and/or detection accuracy of a target in a recently attended location. This phenomenon, which has been discovered and studied thoroughly in humans, is believed to reflect a brain mechanism for controlling the allocation of spatial attention in a manner that enhances efficient search. Findings showing that IOR is robust, apparent at a very early age and seemingly dependent on midbrain activity suggest that IOR is a universal attentional mechanism in vertebrates. However, studies in non-mammalian species are scarce. To explore this hypothesis comparatively, we tested for IOR in barn owls (Tyto alba) using the classical Posner cueing paradigm. Two barn owls were trained to initiate a trial by fixating on the center of a computer screen and then turning their gaze to the location of a target. A short, non-informative cue appeared before the target, either at a location predicting the target (valid) or a location not predicting the target (invalid). In one barn owl, the response times (RT) to the valid targets compared to the invalid targets shifted from facilitation (lower RTs) to inhibition (higher RTs) when increasing the time lag between the cue and the target. The second owl mostly failed to maintain fixation and responded to the cue before the target onset. However, when including in the analysis only the trials in which the owl maintained fixation, an inhibition in the valid trials could be detected. To search for the neural correlates of IOR, we recorded multiunit responses in the optic tectum (OT) of four head-fixed owls passively viewing a cueing paradigm as in the behavioral experiments. At short cue to target lags (<100 ms), neural responses to the target in the receptive field (RF) were usually enhanced if the cue appeared earlier inside the RF (valid) and were suppressed if the cue appeared earlier outside the RF (invalid). This was reversed at longer lags: neural responses were suppressed in the valid conditions and were unaffected in the invalid conditions. The findings support the notion that IOR is a basic mechanism in the evolution of vertebrate behavior and suggest that the effect appears as a result of the interaction between lateral and forward inhibition in the tectal circuitry.

What Neuroscientific Studies Tell Us about Inhibition of Return

An inhibitory aftermath of orienting, inhibition of return (IOR), has intrigued scholars since its discovery about 40 years ago. Since then, the phenomenon has been subjected to a wide range of neuroscientific methods and the results of these are reviewed in this paper. These include direct manipulations of brain structures (which occur naturally in brain damage and disease or experimentally as in TMS and lesion studies) and measurements of brain activity (in humans using EEG and fMRI and in animals using single unit recording). A variety of less direct methods (e.g., computational modeling, developmental studies, etc.) have also been used. The findings from this wide range of methods support the critical role of subcortical and cortical oculomotor pathways in the generation and nature of IOR.

Transsaccadic integration benefits are not limited to the saccade target

Across saccades, humans can integrate the low-resolution presaccadic information of an upcoming saccade target with the high-resolution postsaccadic information. There is converging evidence to suggest that transsaccadic integration occurs at the saccade target. However, given divergent evidence on the spatial specificity of related mechanisms such as attention, visual working memory, and remapping, it is unclear whether integration is also possible at locations other than the saccade target. We tested the spatial profile of transsaccadic integration, by testing perceptual performance at six locations around the saccade target and between the saccade target and initial fixation. Results show that integration benefits do not differ between the saccade target and surrounding locations. Transsaccadic integration benefits are not specific to the saccade target and can occur at other locations when they are behaviorally relevant, although there is a trend for worse performance for the location above initial fixation compared with those in the direction of the saccade. This suggests that transsaccadic integration may be a more general mechanism used to reconcile task-relevant pre- and postsaccadic information at attended locations other than the saccade target.

NEW & NOTEWORTHY This study shows that integration of pre- and postsaccadic information across saccades is not restricted to the saccade target. We found performance benefits of transsaccadic integration at attended locations other than the saccade target, and these benefits did not differ from those found at the saccade target. This suggests that transsaccadic integration may be a more general mechanism used to reconcile pre- and postsaccadic information at task-relevant locations.

Saccades Trigger Predictive Updating of Attentional Topography in Area V4

During natural behavior, saccades and attention act together to allocate limited neural resources. Attention is generally mediated by retinotopic visual neurons; therefore, specific neurons representing attended features change with each saccade. We investigated the neural mechanisms that allow attentional targeting in the face of saccades. Specifically, we looked for predictive changes in attentional modulation state or receptive field position that could stabilize attentional representations across saccades in area V4, known to be necessary for attention-dependent behavior. We recorded from neurons in monkeys performing a novel spatiotopic attention task, in which performance depended on accurate saccade compensation. Measurements of attentional modulation revealed a predictive attentional "hand-off" corresponding to a presaccadic transfer of attentional state from neurons inside the attentional focus before the saccade to those that will be inside the focus after the saccade. The predictive nature of the hand-off ensures that attentional brain maps are properly configured immediately after each saccade.

Corollary Discharge Contributions to Perceptual Continuity Across Saccades

Our vision depends upon shifting our high-resolution fovea to objects of interest in the visual field. Each saccade displaces the image on the retina, which should produce a chaotic scene with jerks occurring several times per second. It does not. This review examines how an internal signal in the primate brain (a corollary discharge) contributes to visual continuity across saccades. The article begins with a review of evidence for a corollary discharge in the monkey and evidence from inactivation experiments that it contributes to perception. The next section examines a specific neuronal mechanism for visual continuity, based on corollary discharge that is referred to as visual remapping. Both the basic characteristics of this anticipatory remapping and the factors that control it are enumerated. The last section considers hypotheses relating remapping to the perceived visual continuity across saccades, including remapping's contribution to perceived visual stability across saccades.

Updating spatial working memory in a dynamic visual environment

The present review describes recent developments regarding the role of the eye movement system in representing spatial information and keeping track of locations of relevant objects. First, we discuss the active vision perspective and why eye movements are considered crucial for perception and attention. The second part focuses on the question of how the oculomotor system is used to represent spatial attentional priority, and the role of the oculomotor system in maintenance of this spatial information. Lastly, we discuss recent findings demonstrating rapid updating of information across saccadic eye movements. We argue that the eye movement system plays a key role in maintaining and rapidly updating spatial information. Furthermore, we suggest that rapid updating emerges primarily to make sure actions are minimally affected by intervening eye movements, allowing us to efficiently interact with the world around us.

Time course of spatiotopic updating across saccades

Humans move their eyes several times per second, yet we perceive the outside world as continuous despite the sudden disruptions created by each eye movement. To date, the mechanism that the brain employs to achieve visual continuity across eye movements remains unclear. While it has been proposed that the oculomotor system quickly updates and informs the visual system about the upcoming eye movement, behavioral studies investigating the time course of this updating suggest the involvement of a slow mechanism, estimated to take more than 500 ms to operate effectively. This is a surprisingly slow estimate, because both the visual system and the oculomotor system process information faster. If spatiotopic updating is indeed this slow, it cannot contribute to perceptual continuity, because it is outside the temporal regime of typical oculomotor behavior. Here, we argue that the behavioral paradigms that have been used previously are suboptimal to measure the speed of spatiotopic updating. In this study, we used a fast gaze-contingent paradigm, using high phi as a continuous stimulus across eye movements. We observed fast spatiotopic updating within 150 ms after stimulus onset. The results suggest the involvement of a fast updating mechanism that predictively influences visual perception after an eye movement. The temporal characteristics of this mechanism are compatible with the rate at which saccadic eye movements are typically observed in natural viewing.

Target-distractor competition cannot be resolved across a saccade

When a distractor is presented in close spatial proximity to a target, a saccade tends to land in between the two objects rather than on the target. This robust phenomenon (also referred to as the global effect) is thought to reflect unresolved competition between target and distractor. It is unclear whether this landing bias persists across saccades since a saccade displaces the retinotopic representations of target and distractor. In the present study participants made successive saccades towards two saccadic targets which were presented simultaneously with an irrelevant distractor in close proximity to the second saccade target. The second saccade was either visually-guided or memory-guided. For the memory-guided trials, the second saccade showed a landing bias towards the location of the distractor, despite the disappearance of the distractor after the first saccade. In contrast, for the visually-guided trials, the bias was corrected and the landing bias was eliminated, even for saccades with the shortest intersaccadic intervals. This suggests that the biased saccade plan was remapped across the first saccade. Therefore, we conclude that the target-distractor competition was not resolved across a saccade, but can be resolved based on visual information that is available after a saccade.

Saccade-synchronized rapid attention shifts in macaque visual cortical area MT

While making saccadic eye-movements to scan a visual scene, humans and monkeys are able to keep track of relevant visual stimuli by maintaining spatial attention on them. This ability requires a shift of attentional modulation from the neuronal population representing the relevant stimulus pre-saccadically to the one representing it post-saccadically. For optimal performance, this trans-saccadic attention shift should be rapid and saccade-synchronized. Whether this is so is not known. We trained two rhesus monkeys to make saccades while maintaining covert attention at a fixed spatial location. We show that the trans-saccadic attention shift in cortical visual medial temporal (MT) area is well synchronized to saccades. Attentional modulation crosses over from the pre-saccadic to the post-saccadic neuronal representation by about 50 ms after a saccade. Taking response latency into account, the trans-saccadic attention shift is well timed to maintain spatial attention on relevant stimuli, so that they can be optimally tracked and processed across saccades.

Saccades result in remapping the neural representation of a target object as well as its attentional modulation. Here the authors show that the trans-saccadic attentional shift is precisely synchronized with the saccade resulting in optimal maintenance of the locus of spatial attention.

Distractor displacements during saccades are reflected in the time-course of saccade curvature

Every time we make a saccade we form a prediction about where objects are going to be when the eye lands. This is crucial since the oculomotor system is retinotopically organized and every saccade drastically changes the projection of objects on the retina. We investigated how quickly the oculomotor system accommodates new spatial information when a distractor is displaced during a saccade. Participants performed sequences of horizontal and vertical saccades and oculomotor competition was induced by presenting a task-irrelevant distractor before the first saccade. On half of the trials the distractor remained in the same location after the first saccade and on the other half the distractor moved during the first saccade. Curvature of the second saccade was used to track target-distractor competition. At short intersaccadic intervals, saccades curved away from the original distractor location, confirming that in the oculomotor system spatiotopic representations emerge rapidly and automatically. Approximately 190 ms after the first saccade, second saccades curved away from the new distractor location. These results show that after a saccade the oculomotor system is initially driven by the spatial prediction made before the saccade, but it is able to quickly update these spatial predictions based on new visual information.

Chances and challenges for an active visual search perspective

Using fixations as the fundamental unit of visual search is an appealing gear change in a paradigm that has long dominated attention research. To truly inform theories of search, however, additional challenges must be faced, including (1) an empirically motivated definition of fixation in the presence of fixational saccades and (2) the biases and limitations of transsaccadic perception and memory.

Coherent alpha oscillations link current and future receptive fields during saccades

Oscillations are ubiquitous in the brain, and they can powerfully influence neural coding. In particular, when oscillations at distinct sites are coherent, they provide a means of gating the flow of neural signals between different cortical regions. Coherent oscillations also occur within individual brain regions, although the purpose of this coherence is not well understood. Here, we report that within a single brain region, coherent alpha oscillations link stimulus representations as they change in space and time. Specifically, in primate cortical area V4, alpha coherence links sites that encode the retinal location of a visual stimulus before and after a saccade. These coherence changes exhibit properties similar to those of receptive field remapping, a phenomenon in which individual neurons change their receptive fields according to the metrics of each saccade. In particular, alpha coherence, like remapping, is highly dependent on the saccade vector and the spatial arrangement of current and future receptive fields. Moreover, although visual stimulation plays a modulatory role, it is neither necessary nor sufficient to elicit alpha coherence. Indeed, a similar pattern of coherence is observed even when saccades are made in darkness. Together, these results show that the pattern of alpha coherence across the retinotopic map in V4 matches many of the properties of receptive field remapping. Thus, oscillatory coherence might play a role in constructing the stable representation of visual space that is an essential aspect of conscious perception.

Circuits for presaccadic visual remapping

Saccadic eye movements rapidly displace the image of the world that is projected onto the retinas. In anticipation of each saccade, many neurons in the visual system shift their receptive fields. This presaccadic change in visual sensitivity, known as remapping, was first documented in the parietal cortex and has been studied in many other brain regions. Remapping requires information about upcoming saccades via corollary discharge. Analyses of neurons in a corollary discharge pathway that targets the frontal eye field (FEF) suggest that remapping may be assembled in the FEF's local microcircuitry. Complementary data from reversible inactivation, neural recording, and modeling studies provide evidence that remapping contributes to transsaccadic continuity of action and perception. Multiple forms of remapping have been reported in the FEF and other brain areas, however, and questions remain about the reasons for these differences. In this review of recent progress, we identify three hypotheses that may help to guide further investigations into the structure and function of circuits for remapping.

No Evidence for Automatic Remapping of Stimulus Features or Location Found with fMRI

The input to our visual system shifts every time we move our eyes. To maintain a stable percept of the world, visual representations must be updated with each saccade. Near the time of a saccade, neurons in several visual areas become sensitive to the regions of visual space that their receptive fields occupy after the saccade. This process, known as remapping, transfers information from one set of neurons to another, and may provide a mechanism for visual stability. However, it is not clear whether remapping transfers information about stimulus features in addition to information about stimulus location. To investigate this issue, we recorded blood-oxygen-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses while human subjects viewed images of faces and houses (two visual categories with many feature differences). Immediately after some image presentations, subjects made a saccade that moved the previously stimulated location to the opposite side of the visual field. We then used a combination of univariate analyses and multivariate pattern analyses to test whether information about stimulus location and stimulus features were remapped to the ipsilateral hemisphere after the saccades. We found no reliable indication of stimulus feature remapping in any region. However, we also found no reliable indication of stimulus location remapping, despite the fact that our paradigm was highly similar to previous fMRI studies of remapping. The absence of location remapping in our study precludes strong conclusions regarding feature remapping. However, these results also suggest that measurement of location remapping with fMRI depends strongly on the details of the experimental paradigm used. We highlight differences in our approach from the original fMRI studies of remapping, discuss potential reasons for the failure to generalize prior location remapping results, and suggest directions for future research.

Neural Network Evidence for the Coupling of Presaccadic Visual Remapping to Predictive Eye Position Updating

As we look around a scene, we perceive it as continuous and stable even though each saccadic eye movement changes the visual input to the retinas. How the brain achieves this perceptual stabilization is unknown, but a major hypothesis is that it relies on presaccadic remapping, a process in which neurons shift their visual sensitivity to a new location in the scene just before each saccade. This hypothesis is difficult to test in vivo because complete, selective inactivation of remapping is currently intractable. We tested it in silico with a hierarchical, sheet-based neural network model of the visual and oculomotor system. The model generated saccadic commands to move a video camera abruptly. Visual input from the camera and internal copies of the saccadic movement commands, or corollary discharge, converged at a map-level simulation of the frontal eye field (FEF), a primate brain area known to receive such inputs. FEF output was combined with eye position signals to yield a suitable coordinate frame for guiding arm movements of a robot. Our operational definition of perceptual stability was "useful stability," quantified as continuously accurate pointing to a visual object despite camera saccades. During training, the emergence of useful stability was correlated tightly with the emergence of presaccadic remapping in the FEF. Remapping depended on corollary discharge but its timing was synchronized to the updating of eye position. When coupled to predictive eye position signals, remapping served to stabilize the target representation for continuously accurate pointing. Graded inactivations of pathways in the model replicated, and helped to interpret, previous in vivo experiments. The results support the hypothesis that visual stability requires presaccadic remapping, provide explanations for the function and timing of remapping, and offer testable hypotheses for in vivo studies. We conclude that remapping allows for seamless coordinate frame transformations and quick actions despite visual afferent lags. With visual remapping in place for behavior, it may be exploited for perceptual continuity.

An Attention-Sensitive Memory Trace in Macaque MT Following Saccadic Eye Movements

We experience a visually stable world despite frequent retinal image displacements induced by eye, head, and body movements. The neural mechanisms underlying this remain unclear. One mechanism that may contribute is transsaccadic remapping, in which the responses of some neurons in various attentional, oculomotor, and visual brain areas appear to anticipate the consequences of saccades. The functional role of transsaccadic remapping is actively debated, and many of its key properties remain unknown. Here, recording from two monkeys trained to make a saccade while directing attention to one of two spatial locations, we show that neurons in the middle temporal area (MT), a key locus in the motion-processing pathway of humans and macaques, show a form of transsaccadic remapping called a memory trace. The memory trace in MT neurons is enhanced by the allocation of top-down spatial attention. Our data provide the first demonstration, to our knowledge, of the influence of top-down attention on the memory trace anywhere in the brain. We find evidence only for a small and transient effect of motion direction on the memory trace (and in only one of two monkeys), arguing against a role for MT in the theoretically critical yet empirically contentious phenomenon of spatiotopic feature-comparison and adaptation transfer across saccades. Our data support the hypothesis that transsaccadic remapping represents the shift of attentional pointers in a retinotopic map, so that relevant locations can be tracked and rapidly processed across saccades. Our results resolve important issues concerning the perisaccadic representation of visual stimuli in the dorsal stream and demonstrate a significant role for top-down attention in modulating this representation.

How does the brain keep track of specific attended features after eye movements? A new study of the macaque brain implicates the middle temporal (MT) area in the remapping of attentional pointers across saccades.

Humans experience a visually stable world despite the fact that eye, head, and body movements cause frequent shifts of the image on the retina. Humans and monkeys are also able to keep track of visual stimuli across such movements. One mechanism that may contribute to these abilities is “transsaccadic remapping,” in which the responses of some neurons in various attentional, oculomotor, and visual brain areas appear to anticipate the consequences of saccades. A current hypothesis proposes that the brain maintains “attentional pointers” to the locations of relevant stimuli and that, via transsaccadic remapping, it rapidly relocates these pointers to compensate for intervening eye movements. Whether stimulus features are also remapped across saccades (along with their location) remains unclear. Here, we show the presence of transsaccadic remapping in a macaque monkey brain area critical for visual motion processing, the middle temporal area (MT). This remapped response is stronger for an attended stimulus. We find only weak evidence for motion-direction information in the remapped response. These results support the attentional pointer hypothesis and demonstrate for the first time, to our knowledge, the impact of top-down attention on transsaccadic remapping in the brain.

Perisaccadic Updating of Visual Representations and Attentional States: Linking Behavior and Neurophysiology

During natural vision, saccadic eye movements lead to frequent retinal image changes that result in different neuronal subpopulations representing the same visual feature across fixations. Despite these potentially disruptive changes to the neural representation, our visual percept is remarkably stable. Visual receptive field remapping, characterized as an anticipatory shift in the position of a neuron's spatial receptive field immediately before saccades, has been proposed as one possible neural substrate for visual stability. Many of the specific properties of remapping, e.g., the exact direction of remapping relative to the saccade vector and the precise mechanisms by which remapping could instantiate stability, remain a matter of debate. Recent studies have also shown that visual attention, like perception itself, can be sustained across saccades, suggesting that the attentional control system can also compensate for eye movements. Classical remapping could have an attentional component, or there could be a distinct attentional analog of visual remapping. At this time we do not yet fully understand how the stability of attentional representations relates to perisaccadic receptive field shifts. In this review, we develop a vocabulary for discussing perisaccadic shifts in receptive field location and perisaccadic shifts of attentional focus, review and synthesize behavioral and neurophysiological studies of perisaccadic perception and perisaccadic attention, and identify open questions that remain to be experimentally addressed.

Attention in Active Vision: A Perspective on Perceptual Continuity Across Saccades

Alfred L. Yarbus was among the first to demonstrate that eye movements actively serve our perceptual and cognitive goals, a crucial recognition that is at the heart of today's research on active vision. He realized that not the changes in fixation stick in memory but the changes in shifts of attention. Indeed, oculomotor control is tightly coupled to functions as fundamental as attention and memory. This tight relationship offers an intriguing perspective on transsaccadic perceptual continuity, which we experience despite the fact that saccades cause rapid shifts of the image across the retina. Here, I elaborate this perspective based on a series of psychophysical findings. First, saccade preparation shapes the visual system's priorities; it enhances visual performance and perceived stimulus intensity at the targets of the eye movement. Second, before saccades, the deployment of visual attention is updated, predictively facilitating perception at those retinal locations that will be relevant once the eyes land. Third, saccadic eye movements strongly affect the contents of visual memory, highlighting their crucial role for which parts of a scene we remember or forget. Together, these results provide insights on how attentional processes enable the visual system to cope with the retinal consequences of saccades.

Remapping, Spatial Stability, and Temporal Continuity: From the Pre-Saccadic to Postsaccadic Representation of Visual Space in LIP

As our eyes move, we have a strong percept that the world is stable in space and time; however, the signals in cortex coming from the retina change with each eye movement. It is not known how this changing input produces the visual percept we experience, although the predictive remapping of receptive fields has been described as a likely candidate. To explain how remapping accounts for perceptual stability, we examined responses of neurons in the lateral intraparietal area while animals performed a visual foraging task. When a stimulus was brought into the response field of a neuron that exhibited remapping, the onset of the postsaccadic representation occurred shortly after the saccade ends. Whenever a stimulus was taken out of the response field, the presaccadic representation abruptly ended shortly after the eyes stopped moving. In the 38% (20/52) of neurons that exhibited remapping, there was no more than 30 ms between the end of the presaccadic representation and the start of the postsaccadic representation and, in some neurons, and the population as a whole, it was continuous. We conclude by describing how this seamless shift from a presaccadic to postsaccadic representation could contribute to spatial stability and temporal continuity.

Transsaccadic processing: stability, integration, and the potential role of remapping

While our frequent saccades allow us to sample the complex visual environment in a highly efficient manner, they also raise certain challenges for interpreting and acting upon visual input. In the present, selective review, we discuss key findings from the domains of cognitive psychology, visual perception, and neuroscience concerning two such challenges: (1) maintaining the phenomenal experience of visual stability despite our rapidly shifting gaze, and (2) integrating visual information across discrete fixations. In the first two sections of the article, we focus primarily on behavioral findings. Next, we examine the possibility that a neural phenomenon known as predictive remapping may provide an explanation for aspects of transsaccadic processing. In this section of the article, we delineate and critically evaluate multiple proposals about the potential role of predictive remapping in light of both theoretical principles and empirical findings.

Retinotopic patterns of background connectivity between V1 and fronto-parietal cortex are modulated by task demands

Attention facilitates the processing of task-relevant visual information and suppresses interference from task-irrelevant information. Modulations of neural activity in visual cortex depend on attention, and likely result from signals originating in fronto-parietal and cingulo-opercular regions of cortex. Here, we tested the hypothesis that attentional facilitation of visual processing is accomplished in part by changes in how brain networks involved in attentional control interact with sectors of V1 that represent different retinal eccentricities. We measured the strength of background connectivity between fronto-parietal and cingulo-opercular regions with different eccentricity sectors in V1 using functional MRI data that were collected while participants performed tasks involving attention to either a centrally presented visual stimulus or a simultaneously presented auditory stimulus. We found that when the visual stimulus was attended, background connectivity between V1 and the left frontal eye fields (FEF), left intraparietal sulcus (IPS), and right IPS varied strongly across different eccentricity sectors in V1 so that foveal sectors were more strongly connected than peripheral sectors. This retinotopic gradient was weaker when the visual stimulus was ignored, indicating that it was driven by attentional effects. Greater task-driven differences between foveal and peripheral sectors in background connectivity to these regions were associated with better performance on the visual task and faster response times on correct trials. These findings are consistent with the notion that attention drives the configuration of task-specific functional pathways that enable the prioritized processing of task-relevant visual information, and show that the prioritization of visual information by attentional processes may be encoded in the retinotopic gradient of connectivty between V1 and fronto-parietal regions.

Attentional trade-offs maintain the tracking of moving objects across saccades

In many situations like playing sports or driving a car, we keep track of moving objects, despite the frequent eye movements that drastically interrupt their retinal motion trajectory. Here we report evidence that transsaccadic tracking relies on trade-offs of attentional resources from a tracked object's motion path to its remapped location. While participants covertly tracked a moving object, we presented pulses of coherent motion at different locations to probe the allocation of spatial attention along the object's entire motion path. Changes in the sensitivity for these pulses showed that during fixation attention shifted smoothly in anticipation of the tracked object's displacement. However, just before a saccade, attentional resources were withdrawn from the object's current motion path and reflexively drawn to the retinal location the object would have after saccade. This finding demonstrates the predictive choice the visual system makes to maintain the tracking of moving objects across saccades.

Role of Oculoproprioception in Coding the Locus of Attention

The most common neural representations for spatial attention encode locations retinotopically, relative to center of gaze. To keep track of visual objects across saccades or to orient toward sounds, retinotopic representations must be combined with information about the rotation of one's own eyes in the orbits. Although gaze input is critical for a correct allocation of attention, the source of this input has so far remained unidentified. Two main signals are available: corollary discharge (copy of oculomotor command) and oculoproprioception (feedback from extraocular muscles). Here we asked whether the oculoproprioceptive signal relayed from the somatosensory cortex contributes to coding the locus of attention. We used continuous theta burst stimulation (cTBS) over a human oculoproprioceptive area in the postcentral gyrus (S1EYE). S1EYE-cTBS reduces proprioceptive processing, causing ∼1° underestimation of gaze angle. Participants discriminated visual targets whose location was cued in a nonvisual modality. Throughout the visual space, S1EYE-cTBS shifted the locus of attention away from the cue by ∼1°, in the same direction and by the same magnitude as the oculoproprioceptive bias. This systematic shift cannot be attributed to visual mislocalization. Accuracy of open-loop pointing to the same visual targets, a function thought to rely mainly on the corollary discharge, was unchanged. We argue that oculoproprioception is selective for attention maps. By identifying a potential substrate for the coupling between eye and attention, this study contributes to the theoretical models for spatial attention.

Saccades and shifting receptive fields: anticipating consequences or selecting targets?

Saccadic eye movements cause frequent and substantial displacements of the retinal image, but those displacements go unnoticed. It has been widely assumed that this perceived stability emerges from the shifting of visual receptive fields from their current, presaccadic locations to their future, postsaccadic locations in anticipation of the retinal consequences of saccades. Although evidence consistent with this anticipatory remapping has accumulated over the years, more recent work suggests an alternative view. In this opinion article, we examine the evidence of presaccadic receptive field shifts and their relationship to the perceptual changes that accompany saccades. We argue that both reflect the selection of targets for saccades rather than the anticipation of a displaced retinal image.

Extrafoveal preview benefit during free-viewing visual search in the monkey

Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

Shape selectivity and remapping in dorsal stream visual area LIP

We explore the visual world by making rapid eye movements (saccades) to focus on objects and locations of interest. Despite abrupt retinal image shifts, we see the world as stable. Remapping contributes to visual stability by updating the internal image with every saccade. Neurons in macaque lateral intraparietal cortex (LIP) and other brain areas update information about salient locations around the time of a saccade. The depth of information transfer remains to be thoroughly investigated. Area LIP, as part of the dorsal visual stream, is regarded as a spatially selective area, yet there is evidence that LIP neurons also encode object features. We sought to determine whether LIP remaps shape information. This knowledge is important for understanding what information is retained from each glance. We identified 82 remapping neurons. First, we presented shapes within the receptive field and tested for shape selectivity in a fixation task. Among the remapping neurons, 28 neurons (34%) were selective for shape. Second, we presented the same shapes in the future location of the receptive field around the time of the saccade and tested for shape selectivity during remapping. Thirty-one (38%) neurons were selective for shape. Of 11 neurons that were shape selective in both tasks, 5 showed significant correlation between shape selectivity in the two tasks. Across the population, there was a weak but significant correlation between responses to shape in the two tasks. Our results provide neurophysiological evidence that remapped responses in area LIP can encode shape information as well as spatial information.

Evidence for differential top-down and bottom-up suppression in posterior parietal cortex

When searching for an object, we usually avoid items that are visually different from the target and objects or places that have been searched already. Previous studies have shown that neural activity in the lateral intraparietal area (LIP) can be used to guide this behaviour; responses to task irrelevant stimuli or to stimuli that have been fixated previously in the trial are reduced compared with responses to potential targets. Here, we test the hypothesis that these reduced responses have a different genesis. Two animals were trained on a visual foraging task, in which they had to find a target among a number of physically identical potential targets (T) and task irrelevant distractors. We recorded neural activity and local field potentials (LFPs) in LIP while the animals performed the task. We found that LFP power was similar for potential targets and distractors but was greater in the alpha and low beta bands when a previously fixated T was in the response field. We interpret these data to suggest that the reduced single-unit response to distractors is a bottom-up feed-forward result of processing in earlier areas and the reduced response to previously fixated Ts is a result of active top-down suppression.