The role of visual working memory (VWM) in the control of gaze during visual search

Prefrontal working memory signal primarily controls phase-coded information within extrastriate cortex

In order to understand how prefrontal cortex provides the benefits of working memory (WM) for visual processing we examined the influence of WM on the representation of visual signals in V4 neurons in two macaque monkeys. We found that WM induces strong β oscillations in V4 and that the timing of action potentials relative to this oscillation reflects sensory information- i.e., a phase coding of visual information. Pharmacologically inactivating the Frontal Eye Field part of prefrontal cortex, we confirmed the necessity of prefrontal signals for the WM-driven boost in phase coding of visual information. Indeed, changes in the average firing rate of V4 neurons could be accounted for by WM-induced oscillatory changes. We present a network model to describe how WM signals can recruit sensory areas primarily by inducing oscillations within these areas and discuss the implications of these findings for a sensory recruitment theory of WM through coherence.

Attenuation of spatial bias with target template variation

This study investigated the impact of target template variation or consistency on attentional bias in location probability learning. Participants conducted a visual search task to find a heterogeneous shape among a homogeneous set of distractors. The target and distractor shapes were either fixed throughout the experiment (target-consistent group) or unpredictably varied on each trial (target-variant group). The target was often presented in one possible search region, unbeknownst to the participants. When the target template was consistent throughout the biased visual search, spatial attention was persistently biased toward the frequent target location. However, when the target template was inconsistent and varied during the biased search, the spatial bias was attenuated so that attention was less prioritized to a frequent target location. The results suggest that the alternative use of target templates may interfere with the emergence of a persistent spatial bias. The regularity-based spatial bias depends on the number of attentional shifts to the frequent target location, but also on search-relevant contexts.

When the mind's eye prevails: The Internal Dominance over External Attention (IDEA) hypothesis

Throughout the 20th century, the psychological literature has considered attention as being primarily directed at the outside world. More recent theories conceive attention as also operating on internal information, and mounting evidence suggests a single, shared attentional focus between external and internal information. Such sharing implies a cognitive architecture where attention needs to be continuously shifted between prioritizing either external or internal information, but the fundamental principles underlying this attentional balancing act are currently unknown. Here, we propose and evaluate one such principle in the shape of the Internal Dominance over External Attention (IDEA) hypothesis: Contrary to the traditional view of attention as being primarily externally oriented, IDEA asserts that attention is inherently biased toward internal information. We provide a theoretical account for why such an internal attention bias may have evolved and examine findings from a wide range of literatures speaking to the balancing of external versus internal attention, including research on working memory, attention switching, visual search, mind wandering, sustained attention, and meditation. We argue that major findings in these disparate research lines can be coherently understood under IDEA. Finally, we consider tentative neurocognitive mechanisms contributing to IDEA and examine the practical implications of more deliberate control over this bias in the context of psychopathology. It is hoped that this novel hypothesis motivates cross-talk between the reviewed research lines and future empirical studies directly examining the mechanisms that steer attention either inward or outward on a moment-by-moment basis.

Guided Search 6.0: An updated model of visual search

This paper describes Guided Search 6.0 (GS6), a revised model of visual search. When we encounter a scene, we can see something everywhere. However, we cannot recognize more than a few items at a time. Attention is used to select items so that their features can be "bound" into recognizable objects. Attention is "guided" so that items can be processed in an intelligent order. In GS6, this guidance comes from five sources of preattentive information: (1) top-down and (2) bottom-up feature guidance, (3) prior history (e.g., priming), (4) reward, and (5) scene syntax and semantics. These sources are combined into a spatial "priority map," a dynamic attentional landscape that evolves over the course of search. Selective attention is guided to the most active location in the priority map approximately 20 times per second. Guidance will not be uniform across the visual field. It will favor items near the point of fixation. Three types of functional visual field (FVFs) describe the nature of these foveal biases. There is a resolution FVF, an FVF governing exploratory eye movements, and an FVF governing covert deployments of attention. To be identified as targets or rejected as distractors, items must be compared to target templates held in memory. The binding and recognition of an attended object is modeled as a diffusion process taking > 150 ms/item. Since selection occurs more frequently than that, it follows that multiple items are undergoing recognition at the same time, though asynchronously, making GS6 a hybrid of serial and parallel processes. In GS6, if a target is not found, search terminates when an accumulating quitting signal reaches a threshold. Setting of that threshold is adaptive, allowing feedback about performance to shape subsequent searches. Simulation shows that the combination of asynchronous diffusion and a quitting signal can produce the basic patterns of response time and error data from a range of search experiments.

Working memory gates visual input to primate prefrontal neurons

Visually guided behavior relies on the integration of sensory input and information held in working memory (WM). Yet it remains unclear how this is accomplished at the level of neural circuits. We studied the direct visual cortical inputs to neurons within a visuomotor area of prefrontal cortex in behaving monkeys. We show that the efficacy of visual input to prefrontal cortex is gated by information held in WM. Surprisingly, visual input to prefrontal neurons was found to target those with both visual and motor properties, rather than preferentially targeting other visual neurons. Furthermore, activity evoked from visual cortex was larger in magnitude, more synchronous, and more rapid, when monkeys remembered locations that matched the location of visual input. These results indicate that WM directly influences the circuitry that transforms visual input into visually guided behavior.

Shared resources between visual attention and visual working memory are allocated through rhythmic sampling

Attention and visual working memory (VWM) are among the most theoretically detailed and empirically tested constructs in human cognition. Nevertheless, the nature of the interrelation between selective attention and VWM still presents a fundamental controversy: Do they rely on the same cognitive resources or not? The present study aims at disentangling this issue by capitalizing on recent evidence showing that attention is a rhythmic phenomenon, oscillating over short time windows. Using a dual-task approach, we combined a classic VWM task with a visual detection task in which we densely sampled detection performance during the time between the memory and the test array. Our results show that an increment in VWM load was related to reduced detection of near-threshold visual stimuli. Importantly, we observed an oscillatory pattern in detection at ~7.5 Hz in the low VWM load conditions, which decreased towards ~5 Hz in the high VWM load condition. These findings suggest that the frequency of this sampling rhythm changes according to the allocation of attentional resources to either the VWM or the detection task. This pattern of results is consistent with a central sampling attentional rhythm which allocates shared attentional resources both to the flow of external visual stimulation and to the internal maintenance of visual information.

Visual and verbal working memory loads interfere with scene-viewing

Working memory is thought to be divided into distinct visual and verbal subsystems. Studies of visual working memory frequently use verbal working memory tasks as control conditions and/or use articulatory suppression to ensure that visual load is not transferred to verbal working memory. Using these verbal tasks relies on the assumption that the verbal working memory load will not interfere with the same processes as visual working memory. In the present study, participants maintained a visual or verbal working memory load as they simultaneously viewed scenes while their eye movements were recorded. Because eye movements and visual working memory are closely linked, we anticipated the visual load would interfere with scene-viewing (and vice versa), while the verbal load would not. Surprisingly, both visual and verbal memory loads interfered with scene-viewing behavior, while eye movements during scene-viewing did not significantly interfere with performance on either memory task. These results suggest that a verbal working memory load can interfere with eye movements in a visual task.

Oculomotor capture by search-irrelevant features in visual working memory: on the crucial role of target-distractor similarity

When searching for varying targets in the environment, a target template has to be maintained in visual working memory (VWM). Recently, we showed that search-irrelevant features of a VWM template bias attention in an object-based manner, so that objects sharing such features with a VWM template capture the eyes involuntarily. Here, we investigated whether target-distractor similarity modulates capture strength. Participants saccaded to a target accompanied by a distractor. A single feature (e.g., shape) defined the target in each trial indicated by a cue, and the cue also varied in one irrelevant feature (e.g., color). The distractor matched the cue's irrelevant feature in half of the trials. Nine experiments showed that target-distractor similarity consistently influenced the degree of oculomotor capture. High target-distractor dissimilarity in the search-relevant feature reduced capture by the irrelevant feature (Experiments 1, 3, 6, 7). However, capture was reduced by high target-distractor similarity in the search-irrelevant feature (Experiments 1, 4, 5, 8). Strong oculomotor capture was observed if target-distractor similarity was reasonably low in the relevant and high in the irrelevant feature, irrespective of whether color or shape were relevant (Experiments 2 and 5). These findings argue for involuntary and object-based, top-down control by VWM templates, whereas its manifestation in oculomotor capture depends crucially on target-distractor similarity in relevant and irrelevant feature dimensions of the search object.

Visual working memory and action: Functional links and bi-directional influences

Working memory bridges perception to action over extended delays, enabling flexible goal-directed behaviour. To date, studies of visual working memory – concerned with detailed visual representations such as shape and colour – have considered visual memory predominantly in the context of visual task demands, such as visual identification and search. Another key purpose of visual working memory is to directly inform and guide upcoming actions. Taking this as a starting point, I review emerging evidence for the pervasive bi-directional links between visual working memory and (planned) action, and discuss these links from the perspective of their common goal of enabling flexible and precise behaviour.

Feature-based guidance of attention during post-saccadic selection

Current models of trans-saccadic perception propose that, after a saccade, the saccade target object must be localized among objects near the landing position. However, the nature of the attentional mechanisms supporting this process is currently under debate. In the present study, we tested whether surface properties of the saccade target object automatically bias post-saccadic selection using a variant of the visual search task. Participants executed a saccade to a shape-singleton target in a circular array. During this primary saccade, the array sometimes rotated so that the eyes landed between the target and an adjacent distractor, requiring gaze correction. In addition, each object in the array had an incidental color value. On Switch trials, the target and adjacent distractor switched colors. The accuracy and latency of gaze correction to the target (measures that provide a direct index of target localization) were compared with a control condition in which no color switch occurred (No-switch trials). Gaze correction to the target was substantially impaired in the Switch condition. This result was obtained even when participants had substantial incentive to avoid encoding the color of the saccade target. In addition, similar effects were observed when the roles of the two feature dimensions (color and shape) were reversed. The results indicate that saccade target features are automatically encoded before a saccade, are retained in visual working memory across the saccade, and instantiate a feature-based selection operation when the eyes land, biasing attention toward objects that match target features.

On-item fixations during serial encoding do not affect spatial working memory

Ample evidence suggests that there is overlap between the eye-movement system and spatial working memory. Such overlapping structures or capacities may result in interference on the one hand and beneficial support on the other. We investigated eye-movement control during encoding of verbal or spatial information, keeping the display the same between tasks. Saccades to to-be-encoded items were scarce during spatial encoding in comparison with verbal encoding. However, despite replicating this difference across different tasks (serial, free recall) and presentation modalities (simultaneous, sequential presentation), we found no relation between item fixations and memory performance-that is, no costs or benefits. Inducing a change from covert to overt encoding did not affect spatial memory performance as well. In contrast, regressive fixations on prior items, that were no longer on the screen, were associated with increased spatial memory performance. Regressions occurred mainly at the end of the encoding period and were targeted at the first presented item. Our results suggest a dissociation between two types of fixations that accompany serial spatial memory: On-item fixations are epiphenomenal; regressions indicate rehearsal or output preparation.

Using Eye Movements to Understand how Security Screeners Search for Threats in X-Ray Baggage

There has been an increasing drive to understand failures in searches for weapons and explosives in X-ray baggage screening. Tracking eye movements during the search has produced new insights into the guidance of attention during the search, and the identification of targets once they are fixated. Here, we review the eye-movement literature that has emerged on this front over the last fifteen years, including a discussion of the problems that real-world searchers face when trying to detect targets that could do serious harm to people and infrastructure.

Feature-based guidance of attention by visual working memory is applied independently of remembered object location

Visual working memory (VWM) has been implicated both in the online representation of object tokens (in the object-file framework) and in the top-down guidance of attention during visual search, implementing a feature template. It is well established that object representations in VWM are structured by location, with access to the content of VWM modulated by position consistency. In the present study, we examined whether this property generalizes to the guidance of attention. Specifically, in two experiments, we probed whether the guidance of spatial attention from features in VWM is modulated by the position of the object from which these features were encoded. Participants remembered an object with an incidental color. Items in a subsequent search array could match either the color of the remembered object, the location, or both. Robust benefits of color match (when the matching item was the target) and costs (when the matching items was a distractor) were observed. Critically, the magnitude of neither effect was influenced by spatial correspondence. The results demonstrate that features in VWM influence attentional priority maps in a manner that does not necessarily inherit the spatial structure of the object representations in which those features are maintained.

Neural Dynamics of Cognitive Control over Working Memory Capture of Attention

The contents of working memory (WM) guide visual attention toward matching features, with visual search being faster when the target and a feature of an item held in WM spatially overlap (validly cued) than when they occur at different locations (invalidly cued). Recent behavioral studies have indicated that attentional capture by WM content can be modulated by cognitive control: When WM cues are reliably helpful to visual search (predictably valid), capture is enhanced, but when reliably detrimental (predictably invalid), capture is attenuated. The neural mechanisms underlying this effect are not well understood, however. Here, we leveraged the high temporal resolution of ERPs time-locked to the onset of the search display to determine how and at what processing stage cognitive control modulates the search process. We manipulated predictability by grouping trials into unpredictable (50% valid/invalid) and predictable (100% valid, 100% invalid) blocks. Behavioral results confirmed that predictability modulated WM-related capture. Comparison of ERPs to the search arrays showed that the N2pc, a posteriorly distributed signature of initial attentional orienting toward a lateralized target, was not impacted by target validity predictability. However, a longer latency, more anterior, lateralized effect-here, termed the "contralateral attention-related negativity"-was reduced under predictable conditions. This reduction interacted with validity, with substantially greater reduction for invalid than valid trials. These data suggest cognitive control over attentional capture by WM content does not affect the initial attentional-orienting process but can reduce the need to marshal later control mechanisms for processing relevant items in the visual world.

Guidance and selection history in hybrid foraging visual search

In Hybrid Foraging tasks, observers search for multiple instances of several types of target. Collecting all the dirty laundry and kitchenware out of a child's room would be a real-world example. How are such foraging episodes structured? A series of four experiments shows that selection of one item from the display makes it more likely that the next item will be of the same type. This pattern holds if the targets are defined by basic features like color and shape but not if they are defined by their identity (e.g., the letters p & d). Additionally, switching between target types during search is expensive in time, with longer response times between successive selections if the target type changes than if they are the same. Finally, the decision to leave a screen/patch for the next screen in these foraging tasks is imperfectly consistent with the predictions of optimal foraging theory. The results of these hybrid foraging studies cast new light on the ways in which prior selection history guides subsequent visual search in general.

Prospectively reinstated memory drives conscious access of matching visual input

Maintaining information in visual working memory (VWM) biases attentional selection of concurrent visual input, by favoring VWM-matching over VWM-mismatching visual input. Recently, it was shown that this bias disappears when the same item is memorized on consecutive occasions (as memoranda presumably transit from VWM to long-term memory), but reemerges when observers anticipate to memorize a novel item on a subsequent trial. Here, we aimed to conceptually replicate and extend this intriguing finding, by investigating whether prospectively reinstated memory drives conscious access of memory-matching visual input. We measured the time it took for participants to detect interocularly suppressed target stimuli, which were either from the same color category as a concurrently memorized color or not. Our results showed that the advantage of memory-matching targets in overcoming suppression progresses non-monotonically across consecutive memorizations of the same color ('repetitions'): the advantage for memory-matching visual input initially declined to asymptote, before being fully revived on the last repetition. This revival was not observed in a control experiment in which targets were not interocularly suppressed. The results suggest that, as observers anticipate to memorize a novel item imminently, VWM usage is prospectively reinstated, causing memory-matching visual input to gain accelerated access to consciousness again.

Evidence for the world as an external memory: A trade-off between internal and external visual memory storage

We use visual working memory (VWM) to maintain the visual features of objects in our world. Although the capacity of VWM is limited, it is unlikely that this limit will pose a problem in daily life, as visual information can be supplemented with input from our external visual world by using eye movements. In the current study, we influenced the trade-off between eye movements and VWM utilization by introducing a cost to a saccade. Higher costs were created by adding a delay in stimulus availability to a copying task. We show that increased saccade cost results in less saccades towards the model and an increased dwell time on the model. These results suggest a shift from making eye movements towards taxing internal VWM. Our findings reveal that the trade-off between executing eye-movements and building an internal representation of our world is based on an adaptive mechanism, governed by cost-efficiency.

Functions of Memory Across Saccadic Eye Movements

Several times per second, humans make rapid eye movements called saccades which redirect their gaze to sample new regions of external space. Saccades present unique challenges to both perceptual and motor systems. During the movement, the visual input is smeared across the retina and severely degraded. Once completed, the projection of the world onto the retina has undergone a large-scale spatial transformation. The vector of this transformation, and the new orientation of the eye in the external world, is uncertain. Memory for the pre-saccadic visual input is thought to play a central role in compensating for the disruption caused by saccades. Here, we review evidence that memory contributes to (1) detecting and identifying changes in the world that occur during a saccade, (2) bridging the gap in input so that visual processing does not have to start anew, and (3) correcting saccade errors and recalibrating the oculomotor system to ensure accuracy of future saccades. We argue that visual working memory (VWM) is the most likely candidate system to underlie these behaviours and assess the consequences of VWM's strict resource limitations for transsaccadic processing. We conclude that a full understanding of these processes will require progress on broader unsolved problems in psychology and neuroscience, in particular how the brain solves the object correspondence problem, to what extent prior beliefs influence visual perception, and how disparate signals arriving with different delays are integrated.

Eye Movements in the "Morris Maze" Spatial Working Memory Task Reveal Deficits in Strategic Planning

Analysis of eye movements can provide insights into processes underlying performance of cognitive tasks. We recorded eye movements in healthy participants and people with idiopathic Parkinson disease during a token foraging task based on the spatial working memory component of the widely used Cambridge Neuropsychological Test Automated Battery. Participants selected boxes (using a mouse click) to reveal hidden tokens. Tokens were never hidden under a box where one had been found before, such that memory had to be used to guide box selections. A key measure of performance in the task is between search errors (BSEs) in which a box where a token has been found is selected again. Eye movements were found to be most commonly directed toward the next box to be clicked on, but fixations also occurred at rates higher than expected by chance on boxes farther ahead or back along the search path. Looking ahead and looking back in this way was found to correlate negatively with BSEs and was significantly reduced in patients with Parkinson disease. Refixating boxes where tokens had already been found correlated with BSEs and the severity of Parkinson disease symptoms. It is concluded that eye movements can provide an index of cognitive planning in the task. Refixations on locations where a token has been found may also provide a sensitive indicator of visuospatial memory integrity. Eye movement measures derived from the spatial working memory task may prove useful in the assessment of executive functions as well as neurological and psychiatric diseases in the future.

Optimal trans-saccadic integration relies on visual working memory

Saccadic eye movements alter the visual processing of objects of interest by bringing them from the periphery, where there is only low-resolution vision, to the high-resolution fovea. Evidence suggests that people are able to achieve trans-saccadic integration in a near-optimal manner; however the mechanisms underlying integration are still unclear. Visual working memory (VWM) is sustained across a saccade, and it has been suggested that this memory resource is used to store and compare the pre- and post- saccadic percepts. This study directly tested the hypothesis that VWM is necessary for optimal trans-saccadic integration, by introducing memory load during a saccade, and testing subsequent integration performance on feature similar and dissimilar stimuli. Results show that integration performance was impaired when there was an additional memory task. Additionally, performance on the memory task was affected by feature-specific integration stimuli. Our results suggest that VWM supports the integration of pre- and post- saccadic stimuli because integration performance is impaired under VWM load.

Fast-backward replay of sequentially memorized items in humans

Storing temporal sequences of events (i.e., sequence memory) is fundamental to many cognitive functions. However, it is unknown how the sequence order information is maintained and represented in working memory and its behavioral significance, particularly in human subjects. We recorded electroencephalography (EEG) in combination with a temporal response function (TRF) method to dissociate item-specific neuronal reactivations. We demonstrate that serially remembered items are successively reactivated during memory retention. The sequential replay displays two interesting properties compared to the actual sequence. First, the item-by-item reactivation is compressed within a 200 – 400 ms window, suggesting that external events are associated within a plasticity-relevant window to facilitate memory consolidation. Second, the replay is in a temporally reversed order and is strongly related to the recency effect in behavior. This fast-backward replay, previously revealed in rat hippocampus and demonstrated here in human cortical activities, might constitute a general neural mechanism for sequence memory and learning.

Have you ever played the ‘Memory Maze Challenge’ game, or its predecessor from the 1980s, ‘Simon’? Players must memorize a sequence of colored lights, and then reproduce the sequence by tapping the colors on a pad. The sequence becomes longer with each trial, making the task more and more difficult. One wrong response and the game is over.

Storing and retrieving sequences is key to many cognitive processes, from following speech to hitting a tennis ball to recalling what you did last week. Such tasks require memorizing the order in which items occur as well as the items themselves. But how do we hold this information in memory? Huang et al. reveal the answer by using scalp electrodes to record the brain activity of healthy volunteers as they memorize and then recall a sequence.

Memorizing, or encoding, each of the items in the sequence triggered a distinct pattern of brain activity. As the volunteers held the sequence in memory, their brains replayed these activity patterns one after the other. But this replay showed two non-intuitive features. First, it was speeded up relative to the original encoding. In fact, the brain compressed the entire sequence into about 200 to 400 milliseconds. Second, the brain replayed the sequence backwards. The activity pattern corresponding to the last item was replayed first, while that corresponding to the first item was replayed last. This ‘fast-backward’ replay may explain why we tend to recall items at the end of a list better than those in the middle, a phenomenon known as the recency effect.

The results of Huang et al. suggest that when we hold a list of items in memory, the brain does not replay the list in its original form, like an echo. Instead, the brain restructures and reorganizes the list, compressing and reversing it. This process, which is also seen in rodents, helps the brain to incorporate the list of items into existing neuronal networks for memory storage.

Visual short-term memory guides infants' visual attention

Adults' visual attention is guided by the contents of visual short-term memory (VSTM). Here we asked whether 10-month-old infants' (N = 41) visual attention is also guided by the information stored in VSTM. In two experiments, we modified the one-shot change detection task (Oakes, Baumgartner, Barrett, Messenger, & Luck, 2013) to create a simplified cued visual search task to ask how information stored in VSTM influences where infants look. A single sample item (e.g., a colored circle) was presented at fixation for 500 ms, followed by a brief (300 ms) retention interval and then a test array consisting of two items, one on each side of fixation. One item in the test array matched the sample stimulus and the other did not. Infants were more likely to look at the non-matching item than at the matching item, demonstrating that the information stored rapidly in VSTM guided subsequent looking behavior.

The architecture of interaction between visual working memory and visual attention

In five experiments, we examined whether a task-irrelevant item in visual working memory (VWM) interacts with perceptual selection when VWM must also be used to maintain a template representation of a search target. This question is critical to distinguishing between competing theories specifying the architecture of interaction between VWM and attention. The single-item template hypothesis (SIT) posits that only a single item in VWM can be maintained in a state that interacts with attention. Thus, the secondary item should be inert with respect to attentional guidance. The multiple-item template hypothesis (MIT) posits that multiple items can be maintained in a state that interacts with attention; thus, both the target representation and the secondary item should be capable of guiding selection. This question has been addressed previously in attention capture studies, but the results have been ambiguous. Here, we modified these earlier paradigms to optimize sensitivity to capture. Capture by a distractor matching the secondary item in VWM was observed consistently across multiple types of search task (abstract arrays and natural scenes), multiple dependent measures (search reaction time (RT) and oculomotor capture), multiple memory dimensions (color and shape), and multiple search stimulus dimensions (color, shape, common objects), providing strong support for the MIT. (PsycINFO Database Record

Visuospatial Working Memory as a Fundamental Component of the Eye Movement System

Humans make frequent movements of the eyes (saccades) to explore the visual environment. Here we argue that visuo-spatial working memory (VSWM) is a fundamental component of the eye movement system. Memory representations in VSWM are functionally integrated at all stages of orienting, from selection of the target, to maintenance of visual features across the saccade, to processes supporting the experience of perceptual continuity after the saccade, to the correction of gaze when the eyes fail to land on the intended object. VSWM is finely tuned to meet the challenges of active vision.

The bilateral field advantage effect in memory precision

Previous research has demonstrated that visual working memory performance is better when visual items are allocated in both left and right visual fields than within only one hemifield. This phenomenon is called the bilateral field advantage (BFA). The BFA is thought to be driven by an enhanced probability of storage, rather than by greater precision. In the present experiments, we sought to test whether the BFA can also extend to precision when the parameters of the task are modified. Using a moderate number of to-be-remembered items and 400 ms presentation time, we found better precision in the bilateral condition than in the unilateral condition. The classic BFA was still found in the form of an enhanced probability of storage, when presentation time was 200 ms. Thus, the BFA appears to convey both enhanced precision and greater probability of storage. The BFA is most likely due to the allocation of more attentional resources, when items are presented in both left and right visual fields.

Interactions between visual working memory representations

We investigated whether the representations of different objects are maintained independently in working memory or interact with each other. Observers were shown two sequentially presented orientations and required to reproduce each orientation after a delay. The sequential presentation minimized perceptual interactions so that we could isolate interactions between memory representations per se. We found that similar orientations were repelled from each other whereas dissimilar orientations were attracted to each other. In addition, when one of the items was given greater attentional priority by means of a cue, the representation of the high-priority item was not influenced very much by the orientation of the low-priority item, but the representation of the low-priority item was strongly influenced by the orientation of the high-priority item. This indicates that attention modulates the interactions between working memory representations. In addition, errors in the reported orientations of the two objects were positively correlated under some conditions, suggesting that representations of distinct objects may become grouped together in memory. Together, these results demonstrate that working-memory representations are not independent but instead interact with each other in a manner that depends on attentional priority.

Whatever you do, don't look at the...: Evaluating guidance by an exclusionary attentional template

People can use a target template consisting of one or more features to guide attention and gaze to matching objects in a search array. But can we also use feature information to guide attention away from known irrelevant items? Some studies found a benefit from foreknowledge of a distractor feature, whereas others found a cost. Importantly, previous work has largely relied on end-of-trial manual responses; it is unclear how feature-guided avoidance might unfold as candidate objects are inspected. In the current experiments, participants were cued with a distractor feature to avoid, then performed a visual search task while eye movements were recorded. Participants initially fixated a to-be-avoided object more frequently than predicted by chance, but they also demonstrated avoidance of cue-matching objects later in the trial. When provided more time between cue stimulus and search array, participants continued to be initially captured by a cued-color item. Furthermore, avoidance of cue-matching objects later in the trial was not contingent on initial capture by a cue-matching object. These results suggest that the conflicting findings in previous negative-cue experiments may be explained by a mixture of two independent processes: initial attentional capture by memory-matching items and later avoidance of known irrelevant items. (PsycINFO Database Record

An eye tracking investigation of color-location binding in infants' visual short-term memory

Two experiments examined 8- and 10-month-old infants' (N = 71) binding of object identity (color) and location information in visual short-term memory (VSTM) using a one-shot change detection task. Building on previous work using the simultaneous streams change detection task, we confirmed that 8- and 10-month-old infants are sensitive to changes in binding between identity and location in VSTM. Further, we demonstrated that infants recognize specifically what changed in these events. Thus, infants' VSTM for binding is robust and can be observed in different procedures and with different stimuli.

Contrasting gist-based and template-based guidance during real-world visual search

Visual search through real-world scenes is guided both by a representation of target features and by knowledge of the sematic properties of the scene (derived from scene gist recognition). In 3 experiments, we compared the relative roles of these 2 sources of guidance. Participants searched for a target object in the presence of a critical distractor object. The color of the critical distractor either matched or mismatched (a) the color of an item maintained in visual working memory for a secondary task (Experiment 1), or (b) the color of the target, cued by a picture before search commenced (Experiments 2 and 3). Capture of gaze by a matching distractor served as an index of template guidance. There were 4 main findings: (a) The distractor match effect was observed from the first saccade on the scene, (b) it was independent of the availability of scene-level gist-based guidance, (c) it was independent of whether the distractor appeared in a plausible location for the target, and (d) it was preserved even when gist-based guidance was available before scene onset. Moreover, gist-based, semantic guidance of gaze to target-plausible regions of the scene was delayed relative to template-based guidance. These results suggest that feature-based template guidance is not limited to plausible scene regions after an initial, scene-level analysis. (PsycINFO Database Record

Visual Working Memory Enhances the Neural Response to Matching Visual Input

Visual working memory (VWM) is used to maintain visual information available for subsequent goal-directed behavior. The content of VWM has been shown to affect the behavioral response to concurrent visual input, suggesting that visual representations originating from VWM and from sensory input draw upon a shared neural substrate (i.e., a sensory recruitment stance on VWM storage). Here, we hypothesized that visual information maintained in VWM would enhance the neural response to concurrent visual input that matches the content of VWM. To test this hypothesis, we measured fMRI BOLD responses to task-irrelevant stimuli acquired from 15 human participants (three males) performing a concurrent delayed match-to-sample task. In this task, observers were sequentially presented with two shape stimuli and a retro-cue indicating which of the two shapes should be memorized for subsequent recognition. During the retention interval, a task-irrelevant shape (the probe) was briefly presented in the peripheral visual field, which could either match or mismatch the shape category of the memorized stimulus. We show that this probe stimulus elicited a stronger BOLD response, and allowed for increased shape-classification performance, when it matched rather than mismatched the concurrently memorized content, despite identical visual stimulation. Our results demonstrate that VWM enhances the neural response to concurrent visual input in a content-specific way. This finding is consistent with the view that neural populations involved in sensory processing are recruited for VWM storage, and it provides a common explanation for a plethora of behavioral studies in which VWM-matching visual input elicits a stronger behavioral and perceptual response.SIGNIFICANCE STATEMENT Humans heavily rely on visual information to interact with their environment and frequently must memorize such information for later use. Visual working memory allows for maintaining such visual information in the mind's eye after termination of its retinal input. It is hypothesized that information maintained in visual working memory relies on the same neural populations that process visual input. Accordingly, the content of visual working memory is known to affect our conscious perception of concurrent visual input. Here, we demonstrate for the first time that visual input elicits an enhanced neural response when it matches the content of visual working memory, both in terms of signal strength and information content.

Dissociating models of visual working memory by reaction-time distribution analysis

There have been heated debates on whether visual working memory (VWM) represents information in discrete-slots or a reservoir of flexible-resources. However, one key aspect of the models has gone unnoticed, the speed of processing when stored information in memory is assessed for accuracy. The present study evaluated contrasting predictions from the two models regarding the change detection decision times spent on the assessment of stored information by estimating the ex-Gaussian parameters from change detection RT distributions across different set sizes (2, 4, 6, or 8). The estimation showed that the Gaussian components μ and σ became larger as the set size increased from 2 to 4, but stayed constant as it reached 6 and 8, with an exponential component τ increasing at above-capacity set sizes. Moreover, we found that an individual's capacity limit correlates with the memory set size where the Gaussian μ reaches a plateau. These results indicate that the decision time for assessing in-memory items is constant regardless of memory set sizes whereas the time for the remaining not-in-memory items increases as the set size exceeds VWM storage capacity. The findings suggest that the discrete-slot model explains the observed RT distributions better than the flexible-resource model.

Object files across eye movements: Previous fixations affect the latencies of corrective saccades

One of the factors contributing to a seamless visual experience is object correspondence-that is, the integration of pre- and postsaccadic visual object information into one representation. Previous research had suggested that before the execution of a saccade, a target object is loaded into visual working memory and subsequently is used to locate the target object after the saccade. Until now, studies on object correspondence have not taken previous fixations into account. In the present study, we investigated the influence of previously fixated information on object correspondence. To this end, we adapted a gaze correction paradigm in which a saccade was executed toward either a previously fixated or a novel target. During the saccade, the stimuli were displaced such that the participant's gaze landed between the target stimulus and a distractor. Participants then executed a corrective saccade to the target. The results indicated that these corrective saccades had lower latencies toward previously fixated than toward nonfixated targets, indicating object-specific facilitation. In two follow-up experiments, we showed that presaccadic spatial and object (surface feature) information can contribute separately to the execution of a corrective saccade, as well as in conjunction. Whereas the execution of a corrective saccade to a previously fixated target object at a previously fixated location is slowed down (i.e., inhibition of return), corrective saccades toward either a previously fixated target object or a previously fixated location are facilitated. We concluded that corrective saccades are executed on the basis of object files rather than of unintegrated feature information.

Spatial inhibition of return as a function of fixation history, task, and spatial references

In oculomotor selection, each saccade is thought to be automatically biased toward uninspected locations, inhibiting the inefficient behavior of repeatedly refixating the same objects. This automatic bias is related to inhibition of return (IOR). Although IOR seems an appealing property that increases efficiency in visual search, such a mechanism would not be efficient in other tasks. Indeed, evidence for additional, more flexible control over refixations has been provided. Here, we investigated whether task demands implicitly affect the rate of refixations. We measured the probability of refixations after series of six binary saccadic decisions under two conditions: visual search and free viewing. The rate of refixations seems influenced by two effects. One effect is related to the rate of intervening fixations, specifically, more refixations were observed with more intervening fixations. In addition, we observed an effect of task set, with fewer refixations in visual search than in free viewing. Importantly, the history-related effect was more pronounced when sufficient spatial references were provided, suggesting that this effect is dependent on spatiotopic encoding of previously fixated locations. This known history-related bias in gaze direction is not the primary influence on the refixation rate. Instead, multiple factors, such as task set and spatial references, assert strong influences as well.

The relationship between visual attention and visual working memory encoding: A dissociation between covert and overt orienting

There is substantial debate over whether visual working memory (VWM) and visual attention constitute a single system for the selection of task-relevant perceptual information or whether they are distinct systems that can be dissociated when their representational demands diverge. In the present study, we focused on the relationship between visual attention and the encoding of objects into VWM. Participants performed a color change-detection task. During the retention interval, a secondary object, irrelevant to the memory task, was presented. Participants were instructed either to execute an overt shift of gaze to this object (Experiments 1-3) or to attend it covertly (Experiments 4 and 5). Our goal was to determine whether these overt and covert shifts of attention disrupted the information held in VWM. We hypothesized that saccades, which typically introduce a memorial demand to bridge perceptual disruption, would lead to automatic encoding of the secondary object. However, purely covert shifts of attention, which introduce no such demand, would not result in automatic memory encoding. The results supported these predictions. Saccades to the secondary object produced substantial interference with VWM performance, but covert shifts of attention to this object produced no interference with VWM performance. These results challenge prevailing theories that consider attention and VWM to reflect a common mechanism. In addition, they indicate that the relationship between attention and VWM is dependent on the memorial demands of the orienting behavior. (PsycINFO Database Record

Detection of Stimulus Displacements Across Saccades is Capacity-Limited and Biased in Favor of the Saccade Target

Retinal image displacements caused by saccadic eye movements are generally unnoticed. Recent theories have proposed that perceptual stability across saccades depends on a local evaluation process centered on the saccade target object rather than on remapping and evaluating the positions of all objects in a display. In three experiments, we examined whether objects other than the saccade target also influence perceptual stability by measuring displacement detection thresholds across saccades for saccade targets and a variable number of non-saccade objects. We found that the positions of multiple objects are maintained across saccades, but with variable precision, with the saccade target object having priority in the perception of displacement, most likely because it is the focus of attention before the saccade and resides near the fovea after the saccade. The perception of displacement of objects that are not the saccade target is affected by acuity limitations, attentional limitations, and limitations on memory capacity. Unlike previous studies that have found that a postsaccadic blank improves the detection of displacement direction across saccades, we found that postsaccadic blanking hurt the detection of displacement per se by increasing false alarms. Overall, our results are consistent with the hypothesis that visual working memory underlies the perception of stability across saccades.

Stimulus familiarity improves consolidation of visual working memory representations

Short-term consolidation is a process that stabilizes visual working memory (VWM) representations so that they are less susceptible to interference. The current study examined this process, specifically if training on specific shapes facilitated the consolidation of visual representations in working memory. Three experiments using two different training tasks compared performance between trained and novel stimuli using the backward masking paradigm. Experiment 1 used a four alternative forced choice task and found an overall advantage for trained shapes as well as evidence for faster consolidation for trained shapes and this cannot be explained by verbal labeling of the trained items (Experiment 3). Experiment 2 used a change-detection training task and showed no overall benefit of training but did show evidence of transfer of training to novel shapes. Taken together, these results show that long-term visual representations can facilitate VWM processing, but the type of training task used impacts the degree to which the long-term representations will affect VWM.

How memory mechanisms are a key component in the guidance of our eye movements: evidence from the global effect

Investigating eye movements has been a promising approach to uncover the role of visual working memory in early attentional processes. Prior research has already demonstrated that eye movements in search tasks are more easily drawn toward stimuli that show similarities to working memory content, as compared with neutral stimuli. Previous saccade tasks, however, have always required a selection process, thereby automatically recruiting working memory. The present study was an attempt to confirm the role of working memory in oculomotor selection in an unbiased saccade task that rendered memory mechanisms irrelevant. Participants executed a saccade in a display with two elements, without any instruction to aim for one particular element. The results show that when two objects appear simultaneously, a working memory match attracts the first saccade more profoundly than do mismatch objects, an effect that was present throughout the saccade latency distribution. These findings demonstrate that memory plays a fundamental biasing role in the earliest competitive processes in the selection of visual objects, even when working memory is not recruited during selection.

Evidence for negative feature guidance in visual search is explained by spatial recoding

Theories of attention and visual search explain how attention is guided toward objects with known target features. But can attention be directed away from objects with a feature known to be associated only with distractors? Most studies have found that the demand to maintain the to-be-avoided feature in visual working memory biases attention toward matching objects rather than away from them. In contrast, Arita, Carlisle, and Woodman (2012) claimed that attention can be configured to selectively avoid objects that match a cued distractor color, and they reported evidence that this type of negative cue generates search benefits. However, the colors of the search array items in Arita et al. (2012) were segregated by hemifield (e.g., blue items on the left, red on the right), which allowed for a strategy of translating the feature-cue information into a simple spatial template (e.g., avoid right, or attend left). In the present study, we replicated the negative cue benefit using the Arita et al. (2012), method (albeit within a subset of participants who reliably used the color cues to guide attention). Then, we eliminated the benefit by using search arrays that could not be grouped by hemifield. Our results suggest that feature-guided avoidance is implemented only indirectly, in this case by translating feature-cue information into a spatial template.

Target templates: the precision of mental representations affects attentional guidance and decision-making in visual search

When people look for things in the environment, they use target templates-mental representations of the objects they are attempting to locate-to guide attention and to assess incoming visual input as potential targets. However, unlike laboratory participants, searchers in the real world rarely have perfect knowledge regarding the potential appearance of targets. In seven experiments, we examined how the precision of target templates affects the ability to conduct visual search. Specifically, we degraded template precision in two ways: 1) by contaminating searchers' templates with inaccurate features, and 2) by introducing extraneous features to the template that were unhelpful. We recorded eye movements to allow inferences regarding the relative extents to which attentional guidance and decision-making are hindered by template imprecision. Our findings support a dual-function theory of the target template and highlight the importance of examining template precision in visual search.

Visual-spatial attention aids the maintenance of object representations in visual working memory

Theories have proposed that the maintenance of object representations in visual working memory is aided by a spatial rehearsal mechanism. In this study, we used two different approaches to test the hypothesis that overt and covert visual-spatial attention mechanisms contribute to the maintenance of object representations in visual working memory. First, we tracked observers' eye movements while they remembered a variable number of objects during change-detection tasks. We observed that during the blank retention interval, participants spontaneously shifted gaze to the locations that the objects had occupied in the memory array. Next, we hypothesized that if attention mechanisms contribute to the maintenance of object representations, then drawing attention away from the object locations during the retention interval should impair object memory during these change-detection tasks. Supporting this prediction, we found that attending to the fixation point in anticipation of a brief probe stimulus during the retention interval reduced change-detection accuracy, even on the trials in which no probe occurred. These findings support models of working memory in which visual-spatial selection mechanisms contribute to the maintenance of object representations.

Visual working memory capacity for color is independent of representation resolution

Background

The relationship between visual working memory (VWM) capacity and resolution of representation have been extensively investigated. Several recent ERP studies using orientation (or arrow) stimuli suggest that there is an inverse relationship between VWM capacity and representation resolution. However, different results have been obtained in studies using color stimuli. This could be due to important differences in the experimental paradigms used in previous studies.

Methodology/Principal Findings

We examined whether the same relationship between capacity and resolution holds for color information. Participants performed a color change detection task while their electroencephalography was recorded. We manipulated representation resolution by asking participants to detect either a salient change (low-resolution) or a subtle change (high-resolution) in color. We used an ERP component known as contralateral delay activity (CDA) to index the amount of information maintained in VWM. The result demonstrated the same pattern for both low- and high-resolution conditions, with no difference between conditions.

Conclusions/Significance

This result suggests that VWM always represents a fixed number of approximately 3–4 colors regardless of the resolution of representation.