Object-based Encoding in Visual Working Memory: Evidence from Memory-driven Attentional Capture

The object as the unit for state switching in visual working memory

This study aimed to determine the unit for switching representational states in visual working memory (VWM). Two opposing hypotheses were investigated: (a) the unit of switching being a feature (feature-based hypothesis), and (b) the unit of switching being an object (object-based hypothesis). Participants (N = 180) were instructed to hold two features from either one or two objects in their VWM. The memory-driven attentional capture effect, suggesting that actively held information in VWM can cause attention to be drawn towards matched distractors, was employed to assess representational states of the first and second probed colors (indicated by a retro-cue). The results showed that only the feature indicated to be probed first could elicit memory related capture for the condition of separate objects. Importantly, features from an integrated object could guide attention regardless of the probe order. These findings were observed across three experiments involving features of different dimensions, same dimensions, or perceptual objects defined by Gestalt principles. They provide convergent evidence supporting the object-based hypothesis by indicating that features within a single object cannot exist in different states.

Unraveling the binding problem in working memory: insights from the hierarchical binding model

The binding problem is a crucial issue in the study of working memory (WM) and remains a central topic of debate among various WM models. Over the past decade, we have explored feature binding within WM, guided by the Hierarchical Binding Model (HBM). This model suggests that WM binding occurs in two stages: an initial implicit binding involving rapid, coarse feature processing, followed by explicit binding where focused attention refines these features via a reentry process. We found that implicit binding is closely related to the attentional processing of features during the perceptual stage. Basic features that can be rapidly and coarsely processed in parallel through spread attention are involuntarily extracted into WM along with the target features, forming a rough bound representation. For explicit binding, we examined the role of attention in retaining explicit binding in WM, emphasizing the unique role of reentry in the HBM. Our findings indicate that WM binding requires additional object attention through the reentry process. These results demonstrate that both implicit and explicit bindings are integral to WM and that the HBM is effective in elucidating the binding mechanisms within WM.

Selectively maintaining an object's feature in visual working memory: A comparison between highly discriminable and fine-grained features

Selectively maintaining information is an essential function of visual working memory (VWM). Recent VWM studies have mainly focused on selective maintenance of objects, leaving the mechanisms of selectively maintaining an object's feature in VWM unknown. Based on the interactive model of perception and VWM, we hypothesized that there are distinct selective maintenance mechanisms for objects containing fine-grained features versus objects containing highly discriminable features. To test this hypothesis, we first required participants to memorize a dual-feature object (colored simple shapes vs. colored polygons), and informed them about the target feature via a retro-cue. Then a visual search task was added to examine the fate of the irrelevant feature. The selective maintenance of an object's feature predicted that the irrelevant feature should be removed from the active state of VWM and should not capture attention when presented as a distractor in the visual search task. We found that irrelevant simple shapes impaired performance in the visual search task (Experiment 1). However, irrelevant polygons did not affect visual search performance (Experiment 2), and this could not be explained by decay of polygons (Experiment 3) or by polygons not capturing attention (Experiment 4). These findings suggest that VWM adopts dissociable mechanisms to selectively maintain an object's feature, depending on the feature's perceptual characteristics.

Behavioral and electrophysiological evidence for the flexible recruitment of feature- and object-based processing in visual working memory comparison

Previous research is inconclusive on when visual working memory (VWM) can be object-based or feature-based. Prior event-related potential (ERP) studies using change detection tasks have found that amplitudes of the N200-an ERP index of VWM comparison- are sensitive to changes in both relevant and irrelevant features, suggesting a bias toward object-based processing. To test whether VWM comparison processing can operate in a feature-based manner, we aimed to create circumstances that would support feature-based processing by: 1) using a strong task-relevance manipulation, and 2) repeating features within a display. Participants completed two blocks of a change detection task for four-item displays in which they were told to respond to color changes (task relevant) but not shape changes (task irrelevant). The first block contained only task-relevant changes to create a strong task-relevance manipulation. In the second block, both relevant and irrelevant changes were present. In both blocks, half of the arrays contained within-display feature repetitions (e.g. two items of the same color or shape). We found that during the second block, N200 amplitudes were sensitive to task-relevant but not irrelevant features regardless of repetition status, consistent with feature-based processing. However, analyses of behavioral data and N200 latencies suggested that object-based processing was occurring at some stages of VWM processing on task-irrelevant feature change trials. In particular, task-irrelevant changes may be processed after no task-relevant feature change is revealed. Overall, the results from the current study suggest that the VWM processing is flexible and can be either object- or feature-based.

More attention with less working memory: The active inhibition of attended but outdated information

Attention has traditionally been regarded as a gateway to working memory, and almost all theoretical frameworks of attention and working memory assume that individuals always have a better memory for information that has received more attention. Here, we provide a series of counterintuitive demonstrations that show that paying more attention to a piece of information impedes, rather than enhances, the selection of this information into working memory. Experiments 1 to 5 provide converging evidence for an even weaker working memory trace of fully attended but outdated features, compared with baseline irrelevant features that were completely ignored. This indicates that the brain actively inhibits attended but outdated information to prevent it from entering working memory. Experiment 6 demonstrates that this inhibition processing is subject to executive control. These findings lead to a substantial reinterpretation of the relationship between attention and working memory.

Object-based selection in visual working memory

Attentional mechanisms in perception can operate over locations, features, or objects. However, people direct attention not only towards information in the external world, but also to information maintained in working memory. To what extent do perception and memory draw on similar selection properties? Here we examined whether principles of object-based attention can also hold true in visual working memory. Experiment 1 examined whether object structure guides selection independently of spatial distance. In a memory updating task, participants encoded two rectangular bars with colored ends before updating two colors during maintenance. Memory updates were faster for two equidistant colors on the same object than on different objects. Experiment 2 examined whether selection of a single object feature spreads to other features within the same object. Participants memorized two sequentially presented Gabors, and a retro-cue indicated which object and feature dimension (color or orientation) would be most relevant to the memory test. We found stronger effects of object selection than feature selection: accuracy was higher for the uncued feature in the same object than the cued feature in the other object. Together these findings demonstrate effects of object-based attention on visual working memory, at least when object-based representations are encouraged, and suggest shared attentional mechanisms across perception and memory.

Allocation of resources in working memory: Theoretical and empirical implications for visual search

Recently, working memory (WM) has been conceptualized as a limited resource, distributed flexibly and strategically between an unlimited number of representations. In addition to improving the precision of representations in WM, the allocation of resources may also shape how these representations act as attentional templates to guide visual search. Here, we reviewed recent evidence in favor of this assumption and proposed three main principles that govern the relationship between WM resources and template-guided visual search. First, the allocation of resources to an attentional template has an effect on visual search, as it may improve the guidance of visual attention, facilitate target recognition, and/or protect the attentional template against interference. Second, the allocation of the largest amount of resources to a representation in WM is not sufficient to give this representation the status of attentional template and thus, the ability to guide visual search. Third, the representation obtaining the status of attentional template, whether at encoding or during maintenance, receives an amount of WM resources proportional to its relevance for visual search. Thus defined, the resource hypothesis of visual search constitutes a parsimonious and powerful framework, which provides new perspectives on previous debates and complements existing models of template-guided visual search.

Memory-driven capture occurs for individual features of an object

Items held in working memory (WM) capture attention (memory-driven capture). People can selectively prioritize specific object features in WM. Here, we examined whether feature-specific prioritization within WM modulates memory-driven capture. In Experiment 1, after remembering the color and orientation of a triangle, participants were instructed, via retro-cue, whether the color, the orientation, or both features were relevant. To measure capture, we asked participants to execute a subsequent search task, and we compared performance in displays that did and did not contain the memory-matching feature. Color attracted attention only when it was relevant. No capture by orientation was found. In Experiment 2, we presented the retro-cue at one of the four locations of the search display to direct attention to specific objects. We found capture by color and this capture was larger when it was indicated as relevant. Crucially, orientation also attracted attention, but only when it was relevant. These findings provide evidence for reciprocal interaction between internal prioritization and external attention on the features level. Specifically, internal feature-specific prioritization modulates memory-driven capture but this capture also depends on the salience of the features.

Oculomotor capture by search-irrelevant features in visual working memory: on the crucial role of target-distractor similarity

When searching for varying targets in the environment, a target template has to be maintained in visual working memory (VWM). Recently, we showed that search-irrelevant features of a VWM template bias attention in an object-based manner, so that objects sharing such features with a VWM template capture the eyes involuntarily. Here, we investigated whether target-distractor similarity modulates capture strength. Participants saccaded to a target accompanied by a distractor. A single feature (e.g., shape) defined the target in each trial indicated by a cue, and the cue also varied in one irrelevant feature (e.g., color). The distractor matched the cue's irrelevant feature in half of the trials. Nine experiments showed that target-distractor similarity consistently influenced the degree of oculomotor capture. High target-distractor dissimilarity in the search-relevant feature reduced capture by the irrelevant feature (Experiments 1, 3, 6, 7). However, capture was reduced by high target-distractor similarity in the search-irrelevant feature (Experiments 1, 4, 5, 8). Strong oculomotor capture was observed if target-distractor similarity was reasonably low in the relevant and high in the irrelevant feature, irrespective of whether color or shape were relevant (Experiments 2 and 5). These findings argue for involuntary and object-based, top-down control by VWM templates, whereas its manifestation in oculomotor capture depends crucially on target-distractor similarity in relevant and irrelevant feature dimensions of the search object.

Opposing effects of stimulus-driven and memory-driven attention in visual search

When one searches for a specific target in a cluttered visual scene, a perceptually salient stimulus or a stimulus that matches working memory's contents is prioritized for attentional selection. In the present study, we aimed at clarifying under which circumstance stimulus-driven attention or memory-driven attention is more pronounced. We hypothesized that one crucial factor affecting stimulus-driven versus memory-driven attention is how a concurrent visual search task is performed. To address this issue, we employed two visual search tasks whose underlying mechanisms are known to be different: Landolt-C search and orientation feature search. One group of participants performed visual search tasks containing a memory-matching stimulus, and the other group conducted searches in the presence of a salient singleton distractor. The results showed that the effects of stimulus-driven and memory-driven attention differed, depending on the cognitive mechanisms underlying the visual search tasks. A memory-matching stimulus captured attention when participants performed the Landolt-C search, whereas this capture was diminished under feature search. In contrast, capture by the salient singleton distractor was found only under feature search. These results demonstrate that the nature of the underlying visual search tasks is an important factor for observing stimulus-driven versus memory-driven attention. Our results also provide a potential solution to resolve current debate regarding memory-driven attention in visual search.

Agent identity drives adaptive encoding of biological motion into working memory

To engage in normal social interactions, we have to encode human biological motions (BMs, e.g., walking and jumping), which is one of the most salient and biologically significant types of kinetic information encountered in everyday life, into working memory (WM). Critically, each BM in real life is produced by a distinct person, carrying a dynamic motion signature (i.e., identity). Whether agent identity influences the WM processing of BMs remains unknown. Here, we addressed this question by examining whether memorizing BMs with different identities promoted the WM processing of task-irrelevant clothing colors. Two opposing hypotheses were tested: (a) WM only stores the target action (element-based hypothesis) and (b) WM stores both action and irrelevant clothing color (event-based hypothesis), interpreting each BM as an event. We required participants to memorize actions that either performed by one agent or distinct agents, while ignoring clothing colors. Then we examined whether the irrelevant color was also stored in WM by probing a distracting effect: If the color was extracted into WM, the change of irrelevant color in the probe would lead to a significant distracting effect on action performance. We found that WM encoding of BMs was adaptive: Once the memorized actions had different identities, WM adopted an event-based encoding mode regardless of memory load and probe identity (Experiment 1, different-identity group of Experiment 2, and Experiment 3). However, WM used an element-based encoding mode when memorized-actions shared the same identity (same-identity group of Experiment 2) or were inverted (Experiment 4). Overall, these findings imply that agent identity information has a significant effect on the WM processing of BMs.

Task-Irrelevant Features in Visual Working Memory Influence Covert Attention: Evidence from a Partial Report Task

Selecting a target based on a representation in visual working memory (VWM) affords biasing covert attention towards objects with memory-matching features. Recently, we showed that even task-irrelevant features of a VWM template bias attention. Specifically, when participants had to saccade to a cued shape, distractors sharing the cue's search-irrelevant color captured the eyes. While a saccade always aims at one target location, multiple locations can be attended covertly. Here, we investigated whether covert attention is captured similarly as the eyes. In our partial report task, each trial started with a shape-defined search cue, followed by a fixation cross. Next, two colored shapes, each including a letter, appeared left and right from fixation, followed by masks. The letter inside that shape matching the preceding cue had to be reported. In Experiment 1, either target, distractor, both, or no object matched the cue's irrelevant color. Target-letter reports were most frequent in target-match trials and least frequent in distractor-match trials. Irrelevant cue and target color never matched in Experiment 2. Still, participants reported the distractor more often to the target's disadvantage, when cue and distractor color matched. Thus, irrelevant features of a VWM template can influence covert attention in an involuntarily object-based manner when searching for trial-wise varying targets.

Working Memory and Attention - A Conceptual Analysis and Review

There is broad agreement that working memory is closely related to attention. This article delineates several theoretical options for conceptualizing this link, and evaluates their viability in light of their theoretical implications and the empirical support they received. A first divide exists between the concept of attention as a limited resource, and the concept of attention as selective information processing. Theories conceptualizing attention as a resource assume that this resource is responsible for the limited capacity of working memory. Three versions of this idea have been proposed: Attention as a resource for storage and processing, a shared resource for perceptual attention and memory maintenance, and a resource for the control of attention. The first of these three is empirically well supported, but the other two are not. By contrast, when attention is understood as a selection mechanism, it is usually not invoked to explain the capacity limit of working memory - rather, researchers ask how different forms of attention interact with working memory, in two areas. The first pertains to attentional selection of the contents of working memory, controlled by mechanisms of filtering out irrelevant stimuli, and removing no-longer relevant representations from working memory. Within working memory contents, a single item is often selected into the focus of attention for processing. The second area pertains to the role of working memory in cognitive control. Working memory contributes to controlling perceptual attention - by holding templates for targets of perceptual selection - and controlling action - by holding task sets to implement our current goals.

The time course of encoding and maintenance of task-relevant versus irrelevant object features in working memory

Access to WM can be restricted on the basis of goal-relevant properties such as spatial location. However, the extent of voluntary control over which features of an attended multi-feature object are encoded and maintained in WM is debated. Some evidence suggests that attending to an object leads to obligatory storage of all of its features, whereas other evidence suggests that access to WM can be restricted to only goal-relevant features. Another possibility is that all features are initially encoded, but irrelevant features are removed from WM over time. To address these various possibilities, we used pattern classification of EEG signals to track the temporal evolution of representations reflecting the encoding and storage of task-relevant and irrelevant features in WM. In different blocks, participants remembered the orientation, color or both orientation and color of a colored, oriented grating. The color and orientation of the grating was randomly drawn from two distinct feature bins on each trial. To examine trial-specific activity reflecting storage of the object's features, a support vector machine (SVM) classifier was trained to classify what bin the stimulus features came from. Importantly, for orientation, the classifier produced reliably above-chance classification across the delay when orientation was task-relevant but not when it was task-irrelevant. Interestingly, orientation could be accurately classified on trials for which both orientation and color were remembered. Moreover, a separate measure corresponding to the probability of a feature belonging to the correct bin was significantly higher when orientation was task-relevant compared to task-irrelevant during encoding. Above-chance classification for color was only present during the initial 500 msec across all conditions. Our results suggest that although information about all of an object's features is present in the initial stimulus-evoked neural response, information about the task-irrelevant features is attenuated during stimulus encoding and is largely absent throughout the delay.

Involuntary top-down control by search-irrelevant features: Visual working memory biases attention in an object-based manner

Many everyday tasks involve successive visual-search episodes with changing targets. Converging evidence suggests that these targets are retained in visual working memory (VWM) and bias attention from there. It is unknown whether all or only search-relevant features of a VWM template bias attention during search. Bias signals might be configured exclusively to task-relevant features so that only search-relevant features bias attention. Alternatively, VWM might maintain objects in the form of bound features. Then, all template features will bias attention in an object-based manner, so that biasing effects are ranked by feature relevance. Here, we investigated whether search-irrelevant VWM template features bias attention. Participants had to saccade to a target opposite a distractor. A colored cue depicted the target prior to each search trial. The target was predefined only by its identity, while its color was irrelevant. When target and cue matched not only in identity (search-relevant) but also in color (search-irrelevant), saccades went more often and faster directly to the target than without any color match (Experiment 1). When introducing a cue-distractor color match (Experiment 2), direct target saccades were most likely when target and cue matched in the search-irrelevant color and least likely in case of a cue-distractor color match. When cue and target were never colored the same (Experiment 3), cue-colored distractors still captured the eyes more often than different-colored distractors despite color being search-irrelevant. As participants were informed about the misleading color, the result argues against a strategical and voluntary usage of color. Instead, search-irrelevant features biased attention obligatorily arguing for involuntary top-down control by object-based VWM templates.

Attribute amnesia is greatly reduced with novel stimuli

Attribute amnesia is the counterintuitive phenomenon where observers are unable to report a salient aspect of a stimulus (e.g., its colour or its identity) immediately after the stimulus was presented, despite both attending to and processing the stimulus. Almost all previous attribute amnesia studies used highly familiar stimuli. Our study investigated whether attribute amnesia would also occur for unfamiliar stimuli. We conducted four experiments using stimuli that were highly familiar (colours or repeated animal images) or that were unfamiliar to the observers (unique animal images). Our results revealed that attribute amnesia was present for both sets of familiar stimuli, colour (p < .001) and repeated animals (p = .001); but was greatly attenuated, and possibly eliminated, when the stimuli were unique animals (p = .02). Our data shows that attribute amnesia is greatly reduced for novel stimuli.