Peri-saccadic natural vision

A computational account of transsaccadic attentional allocation based on visual gain fields

Coordination of goal-directed behavior depends on the brain's ability to recover the locations of relevant objects in the world. In humans, the visual system encodes the spatial organization of sensory inputs, but neurons in early visual areas map objects according to their retinal positions, rather than where they are in the world. How the brain computes world-referenced spatial information across eye movements has been widely researched and debated. Here, we tested whether shifts of covert attention are sufficiently precise in space and time to track an object's real-world location across eye movements. We found that observers' attentional selectivity is remarkably precise and is barely perturbed by the execution of saccades. Inspired by recent neurophysiological discoveries, we developed an observer model that rapidly estimates the real-world locations of objects and allocates attention within this reference frame. The model recapitulates the human data and provides a parsimonious explanation for previously reported phenomena in which observers allocate attention to task-irrelevant locations across eye movements. Our findings reveal that visual attention operates in real-world coordinates, which can be computed rapidly at the earliest stages of cortical processing.

Oculomotor behavior can be adjusted on the basis of artificial feedback signals indicating externally caused errors

Whether a saccade is accurate and has reached the target cannot be evaluated during its execution, but relies on post-saccadic feedback. If the eye has missed the target object, a secondary corrective saccade has to be made to align the fovea with the target. If a systematic post-saccadic error occurs, adaptive changes to the oculomotor behavior are made, such as shortening or lengthening the saccade amplitude. Systematic post-saccadic errors are typically attributed internally to erroneous motor commands. The corresponding adaptive changes to the motor command reduce the error and the need for secondary corrective saccades, and, in doing so, restore accuracy and efficiency. However, adaptive changes to the oculomotor behavior also occur if a change in saccade amplitude is beneficial for task performance, or if it is rewarded. Oculomotor learning thus is more complex than reducing a post-saccadic position error. In the current study, we used a novel oculomotor learning paradigm and investigated whether human participants are able to adapt their oculomotor behavior to improve task performance even when they attribute the error externally. The task was to indicate the intended target object among several objects to a simulated human-machine interface by making eye movements. The participants were informed that the system itself could make errors. The decoding process depended on a distorted landing point of the saccade, resulting in decoding errors. Two different types of visual feedback were added to the post-saccadic scene and we compared how participants used the different feedback types to adjust their oculomotor behavior to avoid errors. We found that task performance improved over time, regardless of the type of feedback. Thus, error feedback from the simulated human-machine interface was used for post-saccadic error evaluation. This indicates that 1) artificial visual feedback signals and 2) externally caused errors might drive adaptive changes to oculomotor behavior.

Quantifying task-related gaze

Competing theories attempt to explain what guides eye movements when exploring natural scenes: bottom-up image salience and top-down semantic salience. In one study, we apply language-based analyses to quantify the well-known observation that task influences gaze in natural scenes. Subjects viewed ten scenes as if they were performing one of two tasks. We found that the semantic similarity between the task and the labels of objects in the scenes captured the task-dependence of gaze (t(39) = 13.083; p < 0.001). In another study, we examined whether image salience or semantic salience better predicts gaze during a search task, and if viewing strategies are affected by searching for targets of high or low semantic relevance to the scene. Subjects searched 100 scenes for a high- or low-relevance object. We found that image salience becomes a worse predictor of gaze across successive fixations, while semantic salience remains a consistent predictor (X2(1, N=40) = 75.148, p < .001). Furthermore, we found that semantic salience decreased as object relevance decreased (t(39) = 2.304; p = .027). These results suggest that semantic salience is a useful predictor of gaze during task-related scene viewing, and that even in target-absent trials, gaze is modulated by the relevance of a search target to the scene in which it might be located.

Visual search patterns during exploration of naturalistic scenes are driven by saliency cues in individuals with cerebral visual impairment

We investigated the relative influence of image salience and image semantics during the visual search of naturalistic scenes, comparing performance in individuals with cerebral visual impairment (CVI) and controls with neurotypical development. Participants searched for a prompted target presented as either an image or text cue. Success rate and reaction time were collected, and gaze behavior was recorded with an eye tracker. A receiver operating characteristic (ROC) analysis compared the distribution of individual gaze landings based on predictions of image salience (using Graph-Based Visual Saliency) and image semantics (using Global Vectors for Word Representations combined with Linguistic Analysis of Semantic Salience) models. CVI participants were less likely and were slower in finding the target. Their visual search behavior was also associated with a larger visual search area and greater number of fixations. ROC scores were also lower in CVI compared to controls for both model predictions. Furthermore, search strategies in the CVI group were not affected by cue type, although search times and accuracy showed a significant correlation with verbal IQ scores for text-cued searches. These results suggest that visual search patterns in CVI are driven mainly by image salience and provide further characterization of higher-order processing deficits observed in this population.

Pathways for Naturalistic Looking Behavior in Primate I: Behavioral Characteristics and Brainstem Circuits

Organisms control their visual worlds by moving their eyes, heads, and bodies. This control of "gaze" or "looking" is key to survival and intelligence, but our investigation of the underlying neural mechanisms in natural conditions is hindered by technical limitations. Recent advances have enabled measurement of both brain and behavior in freely moving animals in complex environments, expanding on historical head-fixed laboratory investigations. We juxtapose looking behavior as traditionally measured in the laboratory against looking behavior in naturalistic conditions, finding that behavior changes when animals are free to move or when stimuli have depth or sound. We specifically focus on the brainstem circuits driving gaze shifts and gaze stabilization. The overarching goal of this review is to reconcile historical understanding of the differential neural circuits for different "classes" of gaze shift with two inconvenient truths. (1) "classes" of gaze behavior are artificial. (2) The neural circuits historically identified to control each "class" of behavior do not operate in isolation during natural behavior. Instead, multiple pathways combine adaptively and non-linearly depending on individual experience. While the neural circuits for reflexive and voluntary gaze behaviors traverse somewhat independent brainstem and spinal cord circuits, both can be modulated by feedback, meaning that most gaze behaviors are learned rather than hardcoded. Despite this flexibility, there are broadly enumerable neural pathways commonly adopted among primate gaze systems. Parallel pathways which carry simultaneous evolutionary and homeostatic drives converge in superior colliculus, a layered midbrain structure which integrates and relays these volitional signals to brainstem gaze-control circuits.

Despite Impaired Binocular Function, Binocular Disparity Integration Across the Visual Field Is Spared in Normal Aging and Glaucoma

Purpose

To examine how binocularly asymmetric glaucomatous visual field damage affects binocular disparity processing across the visual field.

Methods

We recruited 18 patients with primary open-angle glaucoma, 16 age-matched controls, and 13 young controls. Participants underwent standard clinical assessments of binocular visual acuity, binocular contrast sensitivity, stereoacuity, and perimetry. We employed a previously validated psychophysical procedure to measure how sensitivity to binocular disparity varied across spatial frequencies and visual field sectors (i.e., with full-field stimuli spanning the central 21° of the visual field and with stimuli restricted to annular regions spanning 0°-3°, 3°-9°, or 9°-21°). We employed measurements with annular stimuli to model different possible scenarios regarding how disparity information is combined across visual field sectors. We adjudicated between potential mechanisms by comparing model predictions to the patterns observed with full-field stimuli.

Results

Perimetry confirmed that patients with glaucoma exhibited binocularly asymmetric visual field damage (P < 0.001). Across participant groups, foveal regions preferentially processed disparities at finer spatial scales, whereas periphery regions were tuned for coarser scales (P < 0.001). Disparity sensitivity also decreased from fovea to periphery (P < 0.001) and across participant groups (Ps < 0.01). Finally, similar to controls, patients with glaucoma exhibited near-optimal disparity integration, specifically at low spatial frequencies (P < 0.001).

Conclusions

Contrary to the conventional view that glaucoma spares central vision, we find that glaucomatous damage causes a widespread loss of disparity sensitivity across both foveal and peripheral regions. Despite these losses, cortical integration mechanisms appear to be well preserved, suggesting that patients with glaucoma make the best possible use of their remaining binocular function.

Spatial structure, phase, and the contrast of natural images

The sensitivity of the human visual system is thought to be shaped by environmental statistics. A major endeavor in vision science, therefore, is to uncover the image statistics that predict perceptual and cognitive function. When searching for targets in natural images, for example, it has recently been proposed that target detection is inversely related to the spatial similarity of the target to its local background. We tested this hypothesis by measuring observers' sensitivity to targets that were blended with natural image backgrounds. Targets were designed to have a spatial structure that was either similar or dissimilar to the background. Contrary to masking from similarity, we found that observers were most sensitive to targets that were most similar to their backgrounds. We hypothesized that a coincidence of phase alignment between target and background results in a local contrast signal that facilitates detection when target-background similarity is high. We confirmed this prediction in a second experiment. Indeed, we show that, by solely manipulating the phase of a target relative to its background, the target can be rendered easily visible or undetectable. Our study thus reveals that, in addition to its structural similarity, the phase of the target relative to the background must be considered when predicting detection sensitivity in natural images.

During natural viewing, neural processing of visual targets continues throughout saccades

Relatively little is known about visual processing during free-viewing visual search in realistic dynamic environments. Free-viewing is characterized by frequent saccades. During saccades, visual processing is thought to be suppressed, yet we know that the presaccadic visual content can modulate postsaccadic processing. To better understand these processes in a realistic setting, we study here saccades and neural responses elicited by the appearance of visual targets in a realistic virtual environment. While subjects were being driven through a 3D virtual town, they were asked to discriminate between targets that appear on the road. Using a system identification approach, we separated overlapping and correlated activity evoked by visual targets, saccades, and button presses. We found that the presence of a target enhances early occipital as well as late frontocentral saccade-related responses. The earlier potential, shortly after 125 ms post-saccade onset, was enhanced for targets that appeared in the peripheral vision as compared to the central vision, suggesting that fast peripheral processing initiated before saccade onset. The later potential, at 195 ms post-saccade onset, was strongly modulated by the visibility of the target. Together these results suggest that, during natural viewing, neural processing of the presaccadic visual stimulus continues throughout the saccade, apparently unencumbered by saccadic suppression.

Selective postsaccadic enhancement of motion perception

Saccadic eye movements can drastically affect motion perception: during saccades, the stationary surround is swept rapidly across the retina and contrast sensitivity is suppressed. However, after saccades, contrast sensitivity is enhanced for color and high-spatial frequency stimuli and reflexive tracking movements known as ocular following responses (OFR) are enhanced in response to large field motion. Additionally, OFR and postsaccadic enhancement of neural activity in primate motion processing areas are well correlated. It is not yet known how this postsaccadic enhancement arises. Therefore, we tested if the enhancement can be explained by changes in the balance of centre-surround antagonism in motion processing, where spatial summation is favoured at low contrasts and surround suppression is favoured at high contrasts. We found motion perception was selectively enhanced immediately after saccades for high spatial frequency stimuli, consistent with previously reported selective postsaccadic enhancement of contrast sensitivity for flashed high spatial frequency stimuli. The observed enhancement was also associated with changes in spatial summation and suppression, as well as contrast facilitation and inhibition, suggesting that motion processing is augmented to maximise visual perception immediately after saccades. The results highlight that spatial and contrast properties of underlying neural mechanisms for motion processing can be affected by an antecedent saccade for highly detailed stimuli and are in line with studies that show behavioural and neuronal enhancement of motion processing in non-human primates.

Age effects on saccadic suppression of luminance and color

Saccadic eye movements modulate visual perception: they initiate and terminate high acuity vision at a certain location in space, but before and during their execution visual contrast sensitivity is strongly attenuated for 100 to 200 ms. Transient perisaccadic perceptual distortions are assumed to be an important mechanism to maintain visual stability. Little is known about age effects on saccadic suppression, even though for healthy adults other major age-related changes are well documented, like a decrease of visual contrast sensitivity for intermediate and high spatial frequencies or an increase of saccade latencies. Here, we tested saccadic suppression of luminance and isoluminant chromatic flashes in 100 participants from eight to 78 years. To estimate the effect of saccadic suppression on contrast sensitivity, we used a two-alternative forced choice (2AFC) design and an adaptive staircase procedure to modulate the luminance or chromatic contrast of a flashed detection target during fixation and 15 ms after saccade onset. The target was a single horizontal luminance or chromatic line flashed 2° above or below the fixation or saccade target. Compared to fixation, average perisaccadic contrast sensitivity decreased significantly by 66% for luminance and by 36% for color. A significant correlation was found for the strength of saccadic suppression of luminance and color. However, a small age effect was found only for the strength of saccadic suppression of luminance, which increased from 64% to 70% from young to old age. We conclude that saccadic suppression for luminance and color is present in most participants independent of their age and that mechanisms of suppression stay relatively stable during healthy aging.

Blink- and saccade-related suppression effects in early visual areas of the human brain: Intracranial EEG investigations during natural viewing conditions

Blinks and saccades, both ubiquitous in natural viewing conditions, cause rapid changes of visual inputs that are hardly consciously perceived. The neural dynamics in early visual areas of the human brain underlying this remarkable visual stability are still incompletely understood. We used electrocorticography (ECoG) from electrodes directly implanted on the human early visual areas V1, V2, V3d/v, V4d/v and the fusiform gyrus to investigate blink- and saccade-related neuronal suppression effects during non-experimental, free viewing conditions. We found a characteristic, biphasic, broadband gamma power decrease-increase pattern in all investigated visual areas. During saccades, a decrease in gamma power clearly preceded eye movement onset, at least in V1. This may indicate that cortical information processing is actively suppressed in human early visual areas before and during saccades, which then possibly mediates perceptual visual suppression. The following eye movement offset-related increase in gamma power may indicate the recovery of visual perception and the resumption of visual processing.

A dichoptic feedback-based oculomotor training method to manipulate interocular alignment

Strabismus is a prevalent impairment of binocular alignment that is associated with a spectrum of perceptual deficits and social disadvantages. Current treatments for strabismus involve ocular alignment through surgical or optical methods and may include vision therapy exercises. In the present study, we explore the potential of real-time dichoptic visual feedback that may be used to quantify and manipulate interocular alignment. A gaze-contingent ring was presented independently to each eye of 11 normally-sighted observers as they fixated a target dot presented only to their dominant eye. Their task was to center the rings within 2° of the target for at least 1 s, with feedback provided by the sizes of the rings. By offsetting the ring in the non-dominant eye temporally or nasally, this task required convergence or divergence, respectively, of the non-dominant eye. Eight of 11 observers attained 5° asymmetric convergence and 3 of 11 attained 3° asymmetric divergence. The results suggest that real-time gaze-contingent feedback may be used to quantify and transiently simulate strabismus and holds promise as a method to augment existing therapies for oculomotor alignment disorders.

Perisaccadic remapping: What? How? Why?

About 25 years ago, the discovery of receptive field (RF) remapping in the parietal cortex of nonhuman primates revealed that visual RFs, widely assumed to have a fixed retinotopic organization, can change position before every saccade. Measuring such changes can be deceptively difficult. As a result, studies that followed have generated a fascinating but somewhat confusing picture of the phenomenon. In this review, we describe how observations of RF remapping depend on the spatial and temporal sampling of visual RFs and saccade directions. Further, we summarize some of the theories of how remapping might occur in neural circuitry. Finally, based on neurophysiological and psychophysical observations, we discuss the ways in which remapping information might facilitate computations in downstream brain areas.

Near-optimal combination of disparity across a log-polar scaled visual field

The human visual system is foveated: we can see fine spatial details in central vision, whereas resolution is poor in our peripheral visual field, and this loss of resolution follows an approximately logarithmic decrease. Additionally, our brain organizes visual input in polar coordinates. Therefore, the image projection occurring between retina and primary visual cortex can be mathematically described by the log-polar transform. Here, we test and model how this space-variant visual processing affects how we process binocular disparity, a key component of human depth perception. We observe that the fovea preferentially processes disparities at fine spatial scales, whereas the visual periphery is tuned for coarse spatial scales, in line with the naturally occurring distributions of depths and disparities in the real-world. We further show that the visual system integrates disparity information across the visual field, in a near-optimal fashion. We develop a foveated, log-polar model that mimics the processing of depth information in primary visual cortex and that can process disparity directly in the cortical domain representation. This model takes real images as input and recreates the observed topography of human disparity sensitivity. Our findings support the notion that our foveated, binocular visual system has been moulded by the statistics of our visual environment.

The spatial and temporal properties of attentional selectivity for saccades and reaches

The preparation and execution of saccades and goal-directed movements elicits an accompanying shift in attention at the locus of the impending movement. However, some key aspects of the spatiotemporal profile of this attentional shift between eye and hand movements are not resolved. While there is evidence that attention is improved at the target location when making a reach, it is not clear how attention shifts over space and time around the movement target as a saccade and a reach are made to that target. Determining this spread of attention is an important aspect in understanding how attentional resources are used in relation to movement planning and guidance in real world tasks. We compared performance on a perceptual discrimination paradigm during a saccade-alone task, reach-alone task, and a saccade-plus-reach task to map the temporal profile of the premotor attentional shift at the goal of the movement and at three surrounding locations. We measured performance relative to a valid baseline level to determine whether motor planning induces additional attentional facilitation compared to mere covert attention. Sensitivity increased relative to movement onset at the target and at the surrounding locations, for both the saccade-alone and saccade-plus-reach conditions. The results suggest that the temporal profile of the attentional shift is similar for the two tasks involving saccades (saccade-alone and saccade-plus-reach tasks), but is very different when the influence of the saccade is removed. In this case, performance in the saccade-plus-reach task reflects the lower sensitivity observed when a reach-alone task is being conducted. In addition, the spatial profile of this spread of attention is not symmetrical around the target. This suggests that when a saccade and reach are being planned together, the saccade drives the attentional shift, and the reach-alone carries little attentional weight.

Predictive remapping of visual features beyond saccadic targets

Visual stability is thought to be mediated by predictive remapping of the relevant object information from its current, presaccadic location to its future, postsaccadic location on the retina. However, it is heavily debated whether and what feature information is predictively remapped during the presaccadic interval. Here we examined the spatial and featural properties of predictive remapping in a set of three psychophysical studies. We made use of an orientation-adaptation paradigm, in which we induced a tilt aftereffect by prolonged exposure to an oriented adaptor stimulus. Following this adaptation phase, a test stimulus was presented shortly before saccade onset. We found strong evidence for predictive remapping of the features of this test stimulus presented shortly before saccade onset, evidenced by a large tilt aftereffect elicited when the adaptor was positioned at the postsaccadic retinal location of the test stimulus. Conversely, the adaptation state itself, caused by the exposure to the adaptor stimulus, was not predictively remapped. Furthermore, we establish that predictive remapping also occurs for stimuli that are not saccade targets, pointing toward a forward remapping process operating across the whole visual field. Together, our findings suggest that predictive feature remapping of object information plays an important role in mediating visual stability.

Perceptual learning while preparing saccades

Traditional perceptual learning protocols rely almost exclusively on long periods of uninterrupted fixation. Taking a first step towards understanding perceptual learning in natural vision, we had observers report the orientation of a briefly flashed stimulus (clockwise or counterclockwise from a reference orientation) presented strictly during saccade preparation at a location offset from the saccade target. For each observer, the saccade direction, stimulus location, and orientation remained the same throughout training. Subsequently, we assessed performance during fixation in three transfer sessions, either at the trained or at an untrained location, and either using an untrained (Experiment 1) or the trained (Experiment 2) stimulus orientation. We modeled the evolution of contrast thresholds (i.e., the stimulus contrast necessary to discriminate its orientation correctly 75% of the time) as an exponential learning curve, and quantified departures from this curve in transfer sessions using two new, complementary measures of transfer costs (i.e., performance decrements after the transition into the Transfer phase). We observed robust perceptual learning and associated transfer costs for untrained locations and orientations. We also assessed if spatial transfer costs were reduced for the remapped location of the pre-saccadic stimulus-the location the stimulus would have had (but never had) after the saccade. Although the pattern of results at that location differed somewhat from that at the control location, we found no clear evidence for perceptual learning at remapped locations. Using novel, model-based ways to assess learning and transfer costs, our results show that location and feature specificity, hallmarks of perceptual learning, subsist if the target stimulus is presented strictly during saccade preparation throughout training.

Vision During Saccadic Eye Movements

The perceptual consequences of eye movements are manifold: Each large saccade is accompanied by a drop of sensitivity to luminance-contrast, low-frequency stimuli, impacting both conscious vision and involuntary responses, including pupillary constrictions. They also produce transient distortions of space, time, and number, which cannot be attributed to the mere motion on the retinae. All these are signs that the visual system evokes active processes to predict and counteract the consequences of saccades. We propose that a key mechanism is the reorganization of spatiotemporal visual fields, which transiently increases the temporal and spatial uncertainty of visual representations just before and during saccades. On one hand, this accounts for the spatiotemporal distortions of visual perception; on the other hand, it implements a mechanism for fusing pre- and postsaccadic stimuli. This, together with the active suppression of motion signals, ensures the stability and continuity of our visual experience.

Motion Masking by Stationary Objects: A Study of Simulated Saccades

Saccades are crucial to visual information intake by re-orienting the fovea to regions of interest in the visual scene. However, they cause drastic disruptions of the retinal input by shifting the retinal image at very high speeds. The resulting motion and smear are barely noticed, a phenomenon known as saccadic omission. Here, we studied the perception of motion during simulated saccades while observers fixated, moving naturalistic visual scenes across the retina with saccadic speed profiles using a very high temporal frequency display. We found that the mere presence of static pre- and post-saccadic images significantly reduces the perceived amplitude of motion but does not eliminate it entirely. This masking of motion perception could make the intra-saccadic stimulus much less salient and thus easier to ignore.

Selective modulation of visual sensitivity during fixation

During periods of steady fixation, we make small-amplitude ocular movements, termed microsaccades, at a rate of 1–2 every second. Early studies provided evidence that visual sensitivity is reduced during microsaccades—akin to the well-established suppression associated with larger saccades. However, the results of more recent work suggest that microsaccades may alter retinal input in a manner that enhances visual sensitivity to some stimuli. Here we parametrically varied the spatial frequency of a stimulus during a detection task and tracked contrast sensitivity as a function of time relative to microsaccades. Our data reveal two distinct modulations of sensitivity: suppression during the eye movement itself and facilitation after the eye has stopped moving. The magnitude of suppression and facilitation of visual sensitivity is related to the spatial content of the stimulus: suppression is greatest for low spatial frequencies, while sensitivity is enhanced most for stimuli of 1–2 cycles/°, spatial frequencies at which we are already most sensitive in the absence of eye movements. We present a model in which the tuning of suppression and facilitation is explained by delayed lateral inhibition between spatial frequency channels. Our data show that eye movements actively modulate visual sensitivity even during fixation: the detectability of images at different spatial scales can be increased or decreased depending on when the image occurs relative to a microsaccade.

NEW & NOTEWORTHY Given the frequency with which we make microsaccades during periods of fixation, it is vital that we understand how they affect visual processing. We demonstrate two selective modulations of contrast sensitivity that are time-locked to the occurrence of a microsaccade: suppression of low spatial frequencies during each eye movement and enhancement of higher spatial frequencies after the eye has stopped moving. These complementary changes may arise naturally because of sluggish gain control between spatial channels.

Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research

The Tobii Eyex Controller is a new low-cost binocular eye tracker marketed for integration in gaming and consumer applications. The manufacturers claim that the system was conceived for natural eye gaze interaction, does not require continuous recalibration, and allows moderate head movements. The Controller is provided with a SDK to foster the development of new eye tracking applications. We review the characteristics of the device for its possible use in scientific research. We develop and evaluate an open source Matlab Toolkit that can be employed to interface with the EyeX device for gaze recording in behavioral experiments. The Toolkit provides calibration procedures tailored to both binocular and monocular experiments, as well as procedures to evaluate other eye tracking devices. The observed performance of the EyeX (i.e. accuracy < 0.6°, precision < 0.25°, latency < 50 ms and sampling frequency ≈55 Hz), is sufficient for some classes of research application. The device can be successfully employed to measure fixation parameters, saccadic, smooth pursuit and vergence eye movements. However, the relatively low sampling rate and moderate precision limit the suitability of the EyeX for monitoring micro-saccadic eye movements or for real-time gaze-contingent stimulus control. For these applications, research grade, high-cost eye tracking technology may still be necessary. Therefore, despite its limitations with respect to high-end devices, the EyeX has the potential to further the dissemination of eye tracking technology to a broad audience, and could be a valuable asset in consumer and gaming applications as well as a subset of basic and clinical research settings.

Mechanisms of Saccadic Suppression in Primate Cortical Area V4

Psychophysical studies have shown that subjects are often unaware of visual stimuli presented around the time of an eye movement. This saccadic suppression is thought to be a mechanism for maintaining perceptual stability. The brain might accomplish saccadic suppression by reducing the gain of visual responses to specific stimuli or by simply suppressing firing uniformly for all stimuli. Moreover, the suppression might be identical across the visual field or concentrated at specific points. To evaluate these possibilities, we recorded from individual neurons in cortical area V4 of nonhuman primates trained to execute saccadic eye movements. We found that both modes of suppression were evident in the visual responses of these neurons and that the two modes showed different spatial and temporal profiles: while gain changes started earlier and were more widely distributed across visual space, nonspecific suppression was found more often in the peripheral visual field, after the completion of the saccade. Peripheral suppression was also associated with increased noise correlations and stronger local field potential oscillations in the α frequency band. This pattern of results suggests that saccadic suppression shares some of the circuitry responsible for allocating voluntary attention.

SIGNIFICANCE STATEMENT

We explore our surroundings by looking at things, but each eye movement that we make causes an abrupt shift of the visual input. Why doesn't the world look like a film recorded on a shaky camera? The answer in part is a brain mechanism called saccadic suppression, which reduces the responses of visual neurons around the time of each eye movement. Here we reveal several new properties of the underlying mechanisms. First, the suppression operates differently in the central and peripheral visual fields. Second, it appears to be controlled by oscillations in the local field potentials at frequencies traditionally associated with attention. These results suggest that saccadic suppression shares the brain circuits responsible for actively ignoring irrelevant stimuli.

Saccadic Suppression Is Embedded Within Extended Oscillatory Modulation of Sensitivity

Action and perception are intimately coupled systems. One clear case is saccadic suppression, the reduced visibility around the time of saccades, which is important in mediating visual stability; another is the oscillatory modulation of visibility synchronized with hand action. To suppress effectively the spurious retinal motion generated by the eye movements, it is crucial that saccadic suppression and saccadic onset be temporally synchronous. However, the mechanisms that determine this temporal synchrony are unknown. We investigated the effect of saccades on contrast discrimination sensitivity over a long period stretching over >1 s before and after saccade execution. Human subjects made horizontal saccades at will to two stationary saccadic targets separated by 20°. At a random interval, a brief Gabor patch was displayed between the two fixations in either the upper or lower visual field and the subject had to detect its location. Strong saccadic suppression was measured between -50 and 50 ms from saccadic onset. However, the suppression was systematically embedded in a trough of oscillations of contrast sensitivity that fluctuated rhythmically in the delta range (at ∼3 Hz), commencing ∼1 s before saccade execution and lasting for up to 1 s after the saccade. The results show that saccadic preparation and visual sensitivity oscillations are coupled and the coupling might be instrumental in temporally aligning the initiation of the saccade with the visual suppression.SIGNIFICANCE STATEMENT Saccades are known to produce a suppression of contrast sensitivity at saccadic onset and an enhancement after saccadic offset. Here, we show that these dynamics are systematically embedded in visual oscillations of contrast sensitivity that fluctuate rhythmically in the delta range (at ∼3 Hz), commencing ∼1 s before saccade execution and lasting for up to 1 s after the saccade. The results show that saccadic preparation and visual sensitivity oscillations are coupled and the coupling might be instrumental in aligning temporally the initiation of the saccade with the visual suppression.

Contrast sensitivity, V1 neural activity, and natural vision

Contrast sensitivity is fundamental to natural visual processing and an important tool for characterizing both visual function and clinical disorders. We simultaneously measured contrast sensitivity and neural contrast response functions and compared measurements in common laboratory conditions with naturalistic conditions. In typical experiments, a subject holds fixation and a stimulus is flashed on, whereas in natural vision, saccades bring stimuli into view. Motivated by our previous V1 findings, we tested the hypothesis that perceptual contrast sensitivity is lower in natural vision and that this effect is associated with corresponding changes in V1 activity. We found that contrast sensitivity and V1 activity are correlated and that the relationship is similar in laboratory and naturalistic paradigms. However, in the more natural situation, contrast sensitivity is reduced up to 25% compared with that in a standard fixation paradigm, particularly at lower spatial frequencies, and this effect correlates with significant reductions in V1 responses. Our data suggest that these reductions in natural vision result from fast adaptation on one fixation that lowers the response on a subsequent fixation. This is the first demonstration of rapid, natural-image adaptation that carries across saccades, a process that appears to constantly influence visual sensitivity in natural vision.

NEW & NOTEWORTHY

Visual sensitivity and activity in brain area V1 were studied in a paradigm that included saccadic eye movements and natural visual input. V1 responses and contrast sensitivity were significantly reduced compared with results in common laboratory paradigms. The parallel neural and perceptual effects of eye movements and stimulus complexity appear to be due to a form of rapid adaptation that carries across saccades.

Visual attention is available at a task-relevant location rapidly after a saccade

Maintaining attention at a task-relevant spatial location while making eye-movements necessitates a rapid, saccade-synchronized shift of attentional modulation from the neuronal population representing the task-relevant location before the saccade to the one representing it after the saccade. Currently, the precise time at which spatial attention becomes fully allocated to the task-relevant location after the saccade remains unclear. Using a fine-grained temporal analysis of human peri-saccadic detection performance in an attention task, we show that spatial attention is fully available at the task-relevant location within 30 milliseconds after the saccade. Subjects tracked the attentional target veridically throughout our task: i.e. they almost never responded to non-target stimuli. Spatial attention and saccadic processing therefore co-ordinate well to ensure that relevant locations are attentionally enhanced soon after the beginning of each eye fixation.

DOI:http://dx.doi.org/10.7554/eLife.18009.001

When we look at a scene, our gaze does not move continuously across it. Instead, our eyes move discontinuously, shifting gaze rapidly from point to point to focus on different locations in the scene. These eye movements are known as saccades, and during them the brain temporarily and selectively stops processing visual information.

In the brain, a particular area of a scene is represented by different neurons before and after a saccade. Paying attention to a relevant location in a scene across an eye movement therefore requires the brain to shift its attentional effects from the neurons that represented that location in the scene before the saccade to the set of neurons that do so after the saccade. Ideally, this shift should happen rapidly and be synchronized with the eye movement.

Exactly how long it takes for attention to emerge at a relevant location after a saccade was not clear because attention had not been recorded on a fine enough time-scale immediately after an eye movement. Yao et al. have now addressed this issue in a series of experiments that asked volunteers to focus their eyes on a fixed point. The volunteers had to follow the point with their eyes as it jumped to a new location, and at the same time had to look out for a change in the movement of a pattern of random dots.

The results reveal that attention is fully available at the relevant location within 30 milliseconds after the saccade. In fact, the 30-millisecond delay in the emergence of attention matches the period during which vision is suppressed during a saccade. Thus, the change in the brain’s focus of attention coordinates with the saccadic eye movement to ensure that attention can be fixed on a relevant location as soon as possible after the eye movement ends. More studies are now needed to investigate how the brain coordinates its attention and eye-movement processes to synchronize the shift in attention with the eye movement.

DOI:http://dx.doi.org/10.7554/eLife.18009.002

Monocular and Binocular Contributions to Oculomotor Plasticity

Most eye movements in the real-world redirect the foveae to objects at a new depth and thus require the co-ordination of monocular saccade amplitudes and binocular vergence eye movements. Additionally to maintain the accuracy of these oculomotor control processes across the lifespan, ongoing calibration is required to compensate for errors in foveal landing positions. Such oculomotor plasticity has generally been studied under conditions in which both eyes receive a common error signal, which cannot resolve the long-standing debate regarding whether both eyes are innervated by a common cortical signal or by a separate signal for each eye. Here we examine oculomotor plasticity when error signals are independently manipulated in each eye, which can occur naturally owing to aging changes in each eye’s orbit and extra-ocular muscles, or in oculomotor dysfunctions. We find that both rapid saccades and slow vergence eye movements are continuously recalibrated independently of one another and corrections can occur in opposite directions in each eye. Whereas existing models assume a single cortical representation of space employed for the control of both eyes, our findings provide evidence for independent monoculomotor and binoculomotor plasticities and dissociable spatial mapping for each eye.

Perisaccadic Updating of Visual Representations and Attentional States: Linking Behavior and Neurophysiology

During natural vision, saccadic eye movements lead to frequent retinal image changes that result in different neuronal subpopulations representing the same visual feature across fixations. Despite these potentially disruptive changes to the neural representation, our visual percept is remarkably stable. Visual receptive field remapping, characterized as an anticipatory shift in the position of a neuron's spatial receptive field immediately before saccades, has been proposed as one possible neural substrate for visual stability. Many of the specific properties of remapping, e.g., the exact direction of remapping relative to the saccade vector and the precise mechanisms by which remapping could instantiate stability, remain a matter of debate. Recent studies have also shown that visual attention, like perception itself, can be sustained across saccades, suggesting that the attentional control system can also compensate for eye movements. Classical remapping could have an attentional component, or there could be a distinct attentional analog of visual remapping. At this time we do not yet fully understand how the stability of attentional representations relates to perisaccadic receptive field shifts. In this review, we develop a vocabulary for discussing perisaccadic shifts in receptive field location and perisaccadic shifts of attentional focus, review and synthesize behavioral and neurophysiological studies of perisaccadic perception and perisaccadic attention, and identify open questions that remain to be experimentally addressed.

ScreenMasker: An Open-source Gaze-contingent Screen Masking Environment

The moving-window paradigm, based on gazecontingent technic, traditionally used in a studies of the visual perceptual span. There is a strong demand for new environments that could be employed by non-technical researchers. We have developed an easy-to-use tool with a graphical user interface (GUI) allowing both execution and control of visual gaze-contingency studies. This work describes ScreenMasker, an environment that allows create gaze-contingent textured displays used together with stimuli presentation software. ScreenMasker has an architecture that meets the requirements of low-latency real-time eye-movement experiments. It also provides a variety of settings and functions. Effective rendering times and performance are ensured by means of GPU processing under CUDA technology. Performance tests show ScreenMasker's latency to be 67-74 ms on a typical office computer, and high-end 144-Hz screen latencies of about 25-28 ms. ScreenMasker is an open-source system distributed under the GNU Lesser General Public License and is available at https://github.com/PaulOrlov/ScreenMasker .

Rhythmic oscillations of visual contrast sensitivity synchronized with action

It is well known that the motor and the sensory systems structure sensory data collection and cooperate to achieve an efficient integration and exchange of information. Increasing evidence suggests that both motor and sensory functions are regulated by rhythmic processes reflecting alternating states of neuronal excitability, and these may be involved in mediating sensory-motor interactions. Here we show an oscillatory fluctuation in early visual processing time locked with the execution of voluntary action, and, crucially, even for visual stimuli irrelevant to the motor task. Human participants were asked to perform a reaching movement toward a display and judge the orientation of a Gabor patch, near contrast threshold, briefly presented at random times before and during the reaching movement. When the data are temporally aligned to the onset of movement, visual contrast sensitivity oscillates with periodicity within the theta band. Importantly, the oscillations emerge during the motor planning stage, ∼500 ms before movement onset. We suggest that brain oscillatory dynamics may mediate an automatic coupling between early motor planning and early visual processing, possibly instrumental in linking and closing up the visual-motor control loop.

Sensitivity to gaze-contingent contrast increments in naturalistic movies: An exploratory report and model comparison

Sensitivity to luminance contrast is a prerequisite for all but the simplest visual systems. To examine contrast increment detection performance in a way that approximates the natural environmental input of the human visual system, we presented contrast increments gaze-contingently within naturalistic video freely viewed by observers. A band-limited contrast increment was applied to a local region of the video relative to the observer's current gaze point, and the observer made a forced-choice response to the location of the target (≈25,000 trials across five observers). We present exploratory analyses showing that performance improved as a function of the magnitude of the increment and depended on the direction of eye movements relative to the target location, the timing of eye movements relative to target presentation, and the spatiotemporal image structure at the target location. Contrast discrimination performance can be modeled by assuming that the underlying contrast response is an accelerating nonlinearity (arising from a nonlinear transducer or gain control). We implemented one such model and examined the posterior over model parameters, estimated using Markov-chain Monte Carlo methods. The parameters were poorly constrained by our data; parameters constrained using strong priors taken from previous research showed poor cross-validated prediction performance. Atheoretical logistic regression models were better constrained and provided similar prediction performance to the nonlinear transducer model. Finally, we explored the properties of an extended logistic regression that incorporates both eye movement and image content features. Models of contrast transduction may be better constrained by incorporating data from both artificial and natural contrast perception settings.

Integrating retinotopic features in spatiotopic coordinates

The receptive fields of early visual neurons are anchored in retinotopic coordinates (Hubel and Wiesel, 1962). Eye movements shift these receptive fields and therefore require that different populations of neurons encode an object's constituent features across saccades. Whether feature groupings are preserved across successive fixations or processing starts anew with each fixation has been hotly debated (Melcher and Morrone, 2003; Melcher, 2005, 2010; Knapen et al., 2009; Cavanagh et al., 2010a,b; Morris et al., 2010). Here we show that feature integration initially occurs within retinotopic coordinates, but is then conserved within a spatiotopic coordinate frame independent of where the features fall on the retinas. With human observers, we first found that the relative timing of visual features plays a critical role in determining the spatial area over which features are grouped. We exploited this temporal dependence of feature integration to show that features co-occurring within 45 ms remain grouped across eye movements. Our results thus challenge purely feedforward models of feature integration (Pelli, 2008; Freeman and Simoncelli, 2011) that begin de novo after every eye movement, and implicate the involvement of brain areas beyond early visual cortex. The strong temporal dependence we quantify and its link with trans-saccadic object perception instead suggest that feature integration depends, at least in part, on feedback from higher brain areas (Mumford, 1992; Rao and Ballard, 1999; Di Lollo et al., 2000; Moore and Armstrong, 2003; Stanford et al., 2010).

Suppression of Face Perception during Saccadic Eye Movements

Lack of awareness of a stimulus briefly presented during saccadic eye movement is known as saccadic omission. Studying the reduced visibility of visual stimuli around the time of saccade-known as saccadic suppression-is a key step to investigate saccadic omission. To date, almost all studies have been focused on the reduced visibility of simple stimuli such as flashes and bars. The extension of the results from simple stimuli to more complex objects has been neglected. In two experimental tasks, we measured the subjective and objective awareness of a briefly presented face stimuli during saccadic eye movement. In the first task, we measured the subjective awareness of the visual stimuli and showed that in most of the trials there is no conscious awareness of the faces. In the second task, we measured objective sensitivity in a two-alternative forced choice (2AFC) face detection task, which demonstrated chance-level performance. Here, we provide the first evidence of complete suppression of complex visual stimuli during the saccadic eye movement.