Selection within visual memory representations activates the oculomotor system

Oculomotor feature discrimination is cortically mediated

Eye movements are often directed toward stimuli with specific features. Decades of neurophysiological research has determined that this behavior is subserved by a feature-reweighting of the neural activation encoding potential eye movements. Despite the considerable body of research examining feature-based target selection, no comprehensive theoretical account of the feature-reweighting mechanism has yet been proposed. Given that such a theory is fundamental to our understanding of the nature of oculomotor processing, we propose an oculomotor feature-reweighting mechanism here. We first summarize the considerable anatomical and functional evidence suggesting that oculomotor substrates that encode potential eye movements rely on the visual cortices for feature information. Next, we highlight the results from our recent behavioral experiments demonstrating that feature information manifests in the oculomotor system in order of featural complexity, regardless of whether the feature information is task-relevant. Based on the available evidence, we propose an oculomotor feature-reweighting mechanism whereby (1) visual information is projected into the oculomotor system only after a visual representation manifests in the highest stage of the cortical visual processing hierarchy necessary to represent the relevant features and (2) these dynamically recruited cortical module(s) then perform feature discrimination via shifting neural feature representations, while also maintaining parity between the feature representations in cortical and oculomotor substrates by dynamically reweighting oculomotor vectors. Finally, we discuss how our behavioral experiments may extend to other areas in vision science and its possible clinical applications.

Memory-driven capture occurs for individual features of an object

Items held in working memory (WM) capture attention (memory-driven capture). People can selectively prioritize specific object features in WM. Here, we examined whether feature-specific prioritization within WM modulates memory-driven capture. In Experiment 1, after remembering the color and orientation of a triangle, participants were instructed, via retro-cue, whether the color, the orientation, or both features were relevant. To measure capture, we asked participants to execute a subsequent search task, and we compared performance in displays that did and did not contain the memory-matching feature. Color attracted attention only when it was relevant. No capture by orientation was found. In Experiment 2, we presented the retro-cue at one of the four locations of the search display to direct attention to specific objects. We found capture by color and this capture was larger when it was indicated as relevant. Crucially, orientation also attracted attention, but only when it was relevant. These findings provide evidence for reciprocal interaction between internal prioritization and external attention on the features level. Specifically, internal feature-specific prioritization modulates memory-driven capture but this capture also depends on the salience of the features.

Visual information is required to reduce the global effect

When a distractor appears in close proximity to a saccade target, the saccadic end point is biased towards the distractor. This so-called global effect reduces with the latency of the saccade if the saccade is visually guided. We recently reported that the global effect does not reduce with the latency of a double-step memory-guided saccade. The aim of this study was to investigate why the global effect in memory-guided saccades does not show the typically observed reduction with saccadic latency. One possibility is that reduction of the global effect requires continuous access to visual information about target and distractor locations, which is lacking in the case of a memory-guided saccade. Alternatively, participants may be inclined to routinely preprogram a memory-guided saccade at the moment the visual information disappears, with the result that a memory-guided saccade is typically programmed on the basis of an earlier representation than necessary. To distinguish between these alternatives, two potential targets were presented, and participants were asked to make a saccade to one of them after a delay. In one condition, the target identity was precued, allowing preprogramming of the saccade, while in another condition, it was revealed by a retro cue after the delay. The global effect remained present in both conditions. Increasing visual exposure of target and distractor led to a reduction of the global effect, irrespective of whether participants could preprogram a saccade or not. The results suggest that continuous access to visual information is required in order to eliminate the global effect.

Visual working memory and action: Functional links and bi-directional influences

Working memory bridges perception to action over extended delays, enabling flexible goal-directed behaviour. To date, studies of visual working memory – concerned with detailed visual representations such as shape and colour – have considered visual memory predominantly in the context of visual task demands, such as visual identification and search. Another key purpose of visual working memory is to directly inform and guide upcoming actions. Taking this as a starting point, I review emerging evidence for the pervasive bi-directional links between visual working memory and (planned) action, and discuss these links from the perspective of their common goal of enabling flexible and precise behaviour.

Updating spatial working memory in a dynamic visual environment

The present review describes recent developments regarding the role of the eye movement system in representing spatial information and keeping track of locations of relevant objects. First, we discuss the active vision perspective and why eye movements are considered crucial for perception and attention. The second part focuses on the question of how the oculomotor system is used to represent spatial attentional priority, and the role of the oculomotor system in maintenance of this spatial information. Lastly, we discuss recent findings demonstrating rapid updating of information across saccadic eye movements. We argue that the eye movement system plays a key role in maintaining and rapidly updating spatial information. Furthermore, we suggest that rapid updating emerges primarily to make sure actions are minimally affected by intervening eye movements, allowing us to efficiently interact with the world around us.

Higher order, multifeatural object encoding by the oculomotor system

Previous behavioral and physiological research has demonstrated that as the behavioral relevance of potential saccade goals increases, they elicit more competition during target selection processing as evidenced by increased saccade curvature and neural activity. However, these effects have only been demonstrated for lower order feature singletons, and it remains unclear whether more complicated featural differences between higher order objects also elicit vector modulation. Therefore, we measured human saccades curvature elicited by distractors bilaterally flanking a target during a visual search saccade task and systematically varied subsets of features shared between the two distractors and the target, referred to as objective similarity (OS). Our results demonstrate that saccades deviated away from the distractor highest in OS to the target and that there was a linear relationship between the magnitude of saccade deviation and the number of feature differences between the most similar distractor and the target. Furthermore, an analysis of curvature over the time course of the saccade demonstrated that curvature only occurred in the first 20-30 ms of the movement. Given the multifeatural complexity of the novel stimuli, these results suggest that saccadic target selection processing involves dynamically reweighting vector representations for movement planning to several possible targets based on their behavioral relevance. NEW & NOTEWORTHY We demonstrate that small featural differences between unfamiliar, higher order object representations modulate vector weights during saccadic target selection processing. Such effects have previously only been demonstrated for familiar, simple feature singletons (e.g., color) in which features characterize entire objects. The complexity and novelty of our stimuli suggest that the oculomotor system dynamically receives visual/cognitive information processed in the higher order representational networks of the cortical visual processing hierarchy and integrates this information for saccadic movement planning.

Perceptual Color Space Representations in the Oculomotor System Are Modulated by Surround Suppression and Biased Selection

The oculomotor system utilizes color extensively for planning saccades. Therefore, we examined how the oculomotor system actually encodes color and several factors that modulate these representations: attention-based surround suppression and inherent biases in selecting and encoding color categories. We measured saccade trajectories while human participants performed a memory-guided saccade task with color targets and distractors and examined whether oculomotor target selection processing was functionally related to the CIE (x,y) color space distances between color stimuli and whether there were hierarchical differences between color categories in the strength and speed of encoding potential saccade goals. We observed that saccade planning was modulated by the CIE (x,y) distances between stimuli thus demonstrating that color is encoded in perceptual color space by the oculomotor system. Furthermore, these representations were modulated by (1) cueing attention to a particular color thereby eliciting surround suppression in oculomotor color space and (2) inherent selection and encoding biases based on color category independent of cueing and perceptual discriminability. Since surround suppression emerges from recurrent feedback attenuation of sensory projections, observing oculomotor surround suppression suggested that oculomotor encoding of behavioral relevance results from integrating sensory and cognitive signals that are pre-attenuated based on task demands and that the oculomotor system therefore does not functionally contribute to this process. Second, although perceptual discriminability did partially account for oculomotor processing differences between color categories, we also observed preferential processing of the red color category across various behavioral metrics. This is consistent with numerous previous studies and could not be simply explained by perceptual discriminability. Since we utilized a memory-guided saccade task, this indicates that the biased processing of the red color category does not rely on sustained sensory input and must therefore involve cortical areas associated with the highest levels of visual processing involved in visual working memory.

Rapid updating of spatial working memory across saccades

Each time we make an eye movement, positions of objects on the retina change. In order to keep track of relevant objects their positions have to be updated. The situation becomes even more complex if the object is no longer present in the world and has to be held in memory. In the present study, we used saccadic curvature to investigate the time-course of updating a memorized location across saccades. Previous studies have shown that a memorized location competes with a saccade target for selection on the oculomotor map, which leads to saccades curving away from it. In our study participants performed a sequence of two saccades while keeping a location in memory. The trajectory of the second saccade was used to measure when the memorized location was updated after the first saccade. The results showed that the memorized location was rapidly updated with the eyes curving away from its spatial coordinates within 130 ms after the first eye movement. The time-course of updating was comparable to the updating of an exogenously attended location, and depended on how well the location was memorized.

Peripheral target identification performance modulates eye movements

We often shift our eyes to an interesting stimulus, but it is important to inhibit that eye movement in some environments (e.g., a no-look pass in basketball). Here, we investigated participants' ability to inhibit eye movements when they had to process a peripheral target with a requirement to maintain strict fixation. An array of eight letters composed of four characters was briefly presented and a directional cue was centrally presented to indicate the target location. The stimulus onset asynchrony (SOA) between the cue and the stimulus array was chosen from six values, consisting of pre-cue conditions (-400 and -200ms), a simultaneous cue condition (0ms), and post-cue conditions (200, 400, and 800ms). We found the following: 1) participants shifted their eyes toward the cued location even though the stimulus array was absent at the onset of eye movements, but the eye movement amplitude was smaller than the actual location of the target; 2) eye movements occurred approximately 150ms after the onset of stimulus array in the pre-cue conditions and 250ms after cue onset in the simultaneous and post-cue conditions; and 3) eye movement onsets were delayed and their amplitudes were smaller in correct trials than incorrect trials. These results indicate that the inhibitory process controlling eye movements also compete for cognitive resources like other cognitive processes.

The Role of the Oculomotor System in Updating Visual-Spatial Working Memory across Saccades

Visual-spatial working memory (VSWM) helps us to maintain and manipulate visual information in the absence of sensory input. It has been proposed that VSWM is an emergent property of the oculomotor system. In the present study we investigated the role of the oculomotor system in updating of spatial working memory representations across saccades. Participants had to maintain a location in memory while making a saccade to a different location. During the saccade the target was displaced, which went unnoticed by the participants. After executing the saccade, participants had to indicate the memorized location. If memory updating fully relies on cancellation driven by extraretinal oculomotor signals, the displacement should have no effect on the perceived location of the memorized stimulus. However, if postsaccadic retinal information about the location of the saccade target is used, the perceived location will be shifted according to the target displacement. As it has been suggested that maintenance of accurate spatial representations across saccades is especially important for action control, we used different ways of reporting the location held in memory; a match-to-sample task, a mouse click or by making another saccade. The results showed a small systematic target displacement bias in all response modalities. Parametric manipulation of the distance between the to-be-memorized stimulus and saccade target revealed that target displacement bias increased over time and changed its spatial profile from being initially centered on locations around the saccade target to becoming spatially global. Taken together results suggest that we neither rely exclusively on extraretinal nor on retinal information in updating working memory representations across saccades. The relative contribution of retinal signals is not fixed but depends on both the time available to integrate these signals as well as the distance between the saccade target and the remembered location.

Selection in spatial working memory is independent of perceptual selective attention, but they interact in a shared spatial priority map

We examined the relationship between the attentional selection of perceptual information and of information in working memory (WM) through four experiments, using a spatial WM-updating task. Participants remembered the locations of two objects in a matrix and worked through a sequence of updating operations, each mentally shifting one dot to a new location according to an arrow cue. Repeatedly updating the same object in two successive steps is typically faster than switching to the other object; this object switch cost reflects the shifting of attention in WM. In Experiment 1, the arrows were presented in random peripheral locations, drawing perceptual attention away from the selected object in WM. This manipulation did not eliminate the object switch cost, indicating that the mechanisms of perceptual selection do not underlie selection in WM. Experiments 2a and 2b corroborated the independence of selection observed in Experiment 1, but showed a benefit to reaction times when the placement of the arrow cue was aligned with the locations of relevant objects in WM. Experiment 2c showed that the same benefit also occurs when participants are not able to mark an updating location through eye fixations. Together, these data can be accounted for by a framework in which perceptual selection and selection in WM are separate mechanisms that interact through a shared spatial priority map.

Attentional cueing by cross-modal congruency produces both facilitation and inhibition on short-term visual recognition

The attentional modulation of performance in a memory task, comparable to the one obtained in a perceptual task, is at the focus of contemporary research. We hypothesized that a biphasic effect (namely, facilitation followed by inhibition) can be obtained in visual working memory when attention is cued towards one item of the memorandum and participants must recognize a delayed probe as being identical to any item of the memorandum. In every trial, a delayed spiky/curvy probe appeared centrally, to be matched with the same-category shape maintained in visual working memory which could be either physically identical (positive trials) or only categorically similar (negative trials). To orient the participant's attention towards a selected portion of a two-item memorandum, a (tzk/wow) sound was played simultaneously with two lateral visual shapes (one spiky and one curved). Our results indicate that an exogenous attentional shift during perception of the memorandum, induced by a congruent audio-visual pairing, first facilitates and then inhibits the recognition of a cued item (but not of a non-cued item) stored in visual working memory. A coherent pattern of individual differences emerged, indicating that the amount of early facilitation in congruent-sound trials was negatively correlated with recognition sensitivity in no-sound trials (suggesting that the inverse effectiveness rule may also apply to memory) and positively correlated with later inhibition, as well as with the self-reported susceptibility to memory failures.

Target-distractor competition in the oculomotor system is spatiotopic

In natural scenes, multiple visual stimuli compete for selection; however, each saccade displaces the stimulus representations in retinotopicaly organized visual and oculomotor maps. In the present study, we used saccade curvature to investigate whether oculomotor competition across eye movements is represented in retinotopic or spatiotopic coordinates. Participants performed a sequence of saccades and we induced oculomotor competition by briefly presenting a task-irrelevant distractor at different times during the saccade sequence. Despite the intervening saccade, the second saccade curved away from a spatial representation of the distractor that was presented before the first saccade. Furthermore, the degree of saccade curvature increased with the salience of the distractor presented before the first saccade. The results suggest that spatiotopic representations of target-distractor competition are crucial for successful interaction with objects of interest despite the intervening eye movements.

Updating visual-spatial working memory during object movement

Working memory enables temporary maintenance and manipulation of information for immediate access by cognitive processes. The present study investigates how spatial information stored in working memory is updated during object movement. Participants had to remember a particular location on an object which, after a retention interval, started to move. The question was whether the memorized location was updated with the movement of the object or whether after object movement it remained represented in retinotopic coordinates. We used saccade trajectories to examine how memorized locations were represented. The results showed that immediately after the object stopped moving, there was both a retinotopic and an object-centered representation. However, 200ms later, the activity at the retinotopic location decayed, making the memory representation fully object-centered. Our results suggest that memorized locations are updated from retinotopic to object-centered coordinates during, or shortly after object movement.

Saccades curve away from previously inhibited locations: evidence for the role of priming in oculomotor competition

The oculomotor system serves as the basis for representing concurrently competing motor programs. Here, we examine whether the oculomotor system also keeps track of the outcome of competition between target and distractor on the previous trial. Participants had to perform a simple task of making a saccade toward a predefined direction. On two-thirds of the trials, an irrelevant distractor was presented to either the left or right of the fixation. On one-third of the trials, no distractor was present. The results show that on trials without a distractor, saccades curved away from the empty location that was occupied by a distractor on the previous trial. This result was replicated and extended to cases when different saccade directions were used. In addition, we show that repetition of distractor location on the distractor-present trials results in a stronger curvature away and in a shorter saccade latency to the target. Taken together, these results provide strong evidence that the oculomotor system automatically codes and retains locations that had been ignored in the past to bias future behavior.

Nonspatial Cueing of Tactile STM Causes Shift of Spatial Attention

The focus of attention can be flexibly altered in mnemonic representations of past sensory events. We investigated the neural mechanisms of selection in tactile STM by applying vibrotactile sample stimuli of different intensities to both hands, followed by a symmetrically shaped visual retro-cue. The retro-cue indicated whether the weak or strong sample was relevant for subsequent comparison with a single tactile test stimulus. Locations of tactile stimuli were randomized, and the required response did not depend upon the spatial relation between cued sample and test stimulus. Selection between spatially segregated items in tactile STM was mirrored in lateralized activity following visual retro-cues (N2pc) and influenced encoding of task-irrelevant tactile probe stimuli (N140). Our findings support four major conclusions. First, retrospective selection results in transient shifts of spatial attention. Second, retrospective selection is functionally dissociable from attention-based rehearsal of locations. Third, selection mechanisms are linked across processing stages, as attention shifts in STM influence encoding of sensory signals. Fourth, selection in tactile STM recruits attentional control mechanisms that are, at least partially, supramodal.

Shifting attention within memory representations involves early visual areas

Prior studies have shown that spatial attention modulates early visual cortex retinotopically, resulting in enhanced processing of external perceptual representations. However, it is not clear whether the same visual areas are modulated when attention is focused on, and shifted within a working memory representation. In the current fMRI study participants were asked to memorize an array containing four stimuli. After a delay, participants were presented with a verbal cue instructing them to actively maintain the location of one of the stimuli in working memory. Additionally, on a number of trials a second verbal cue instructed participants to switch attention to the location of another stimulus within the memorized representation. Results of the study showed that changes in the BOLD pattern closely followed the locus of attention within the working memory representation. A decrease in BOLD-activity (V1–V3) was observed at ROIs coding a memory location when participants switched away from this location, whereas an increase was observed when participants switched towards this location. Continuous increased activity was obtained at the memorized location when participants did not switch. This study shows that shifting attention within memory representations activates the earliest parts of visual cortex (including V1) in a retinotopic fashion. We conclude that even in the absence of visual stimulation, early visual areas support shifting of attention within memorized representations, similar to when attention is shifted in the outside world. The relationship between visual working memory and visual mental imagery is discussed in light of the current findings.