Does high memory load kick task-irrelevant information out of visual working memory?

The object as the unit for state switching in visual working memory

This study aimed to determine the unit for switching representational states in visual working memory (VWM). Two opposing hypotheses were investigated: (a) the unit of switching being a feature (feature-based hypothesis), and (b) the unit of switching being an object (object-based hypothesis). Participants (N = 180) were instructed to hold two features from either one or two objects in their VWM. The memory-driven attentional capture effect, suggesting that actively held information in VWM can cause attention to be drawn towards matched distractors, was employed to assess representational states of the first and second probed colors (indicated by a retro-cue). The results showed that only the feature indicated to be probed first could elicit memory related capture for the condition of separate objects. Importantly, features from an integrated object could guide attention regardless of the probe order. These findings were observed across three experiments involving features of different dimensions, same dimensions, or perceptual objects defined by Gestalt principles. They provide convergent evidence supporting the object-based hypothesis by indicating that features within a single object cannot exist in different states.

Unraveling the binding problem in working memory: insights from the hierarchical binding model

The binding problem is a crucial issue in the study of working memory (WM) and remains a central topic of debate among various WM models. Over the past decade, we have explored feature binding within WM, guided by the Hierarchical Binding Model (HBM). This model suggests that WM binding occurs in two stages: an initial implicit binding involving rapid, coarse feature processing, followed by explicit binding where focused attention refines these features via a reentry process. We found that implicit binding is closely related to the attentional processing of features during the perceptual stage. Basic features that can be rapidly and coarsely processed in parallel through spread attention are involuntarily extracted into WM along with the target features, forming a rough bound representation. For explicit binding, we examined the role of attention in retaining explicit binding in WM, emphasizing the unique role of reentry in the HBM. Our findings indicate that WM binding requires additional object attention through the reentry process. These results demonstrate that both implicit and explicit bindings are integral to WM and that the HBM is effective in elucidating the binding mechanisms within WM.

Object-based visual working memory: an object benefit for equidistant memory items presented within simple contours

Previous research has shown that more information can be stored in visual working memory (VWM) when multiple items belong to the same object. Here, in four experiments, we investigated the object effect on memory for spatially equidistant features by manipulating simple, task-irrelevant contours that combined these features. In Experiments 1, 3, and, 4, three grating orientations, and in Experiment 2, one color and two orientations, were presented simultaneously to be memorized. Mixture modeling was applied to estimate both the precision and the guess rates of recall errors. Overall results showed that two target features were remembered more accurately when both were part of the same object. Further analysis showed that the probability of recall increased in particular when both features were extracted from the same object. In Experiment 2, we found that the object effect was greater for features from orthogonal dimensions, but this came at the cost of lower memory precision. In Experiment 3, when we kept the locations of the features perfectly consistent over trials so that the participants could attend to these locations rather than the contour, we still found object benefits. Finally, in Experiment 4 when we manipulated the temporal order of the object and the memory features presentations, it was confirmed that the object benefit is unlikely to stem from the strategical usage of object information. These results suggested that the object benefit arises automatically, likely at an early perceptual level.

Automatically binding relevant and irrelevant features in visual working memory

It is generally assumed that, to save storage space, features are stored as integrated objects in visual working memory (VWM). Although such an object-based account does not always hold because features can be processed in parallel, a previous study has shown that different features can be automatically bound with their locations (task-irrelevant feature) into an integrated unit, resulting in improved memory performance. The present study was designed to further explore this phenomenon by investigating whether other features, which are not spatial in origin, can act as the binding cue to form such automatic binding. To test this, we used three different features as binding cues (i.e., colour, spatial frequency, and shape) over multiple separate experiments. The results consistently showed that when two features shared the same binding cue, memory performance was better relative to when each of those features had their own binding cue. We conclude that any task-irrelevant feature can act as a binding cue to automatically bind with task-relevant features even across different objects, resulting in memory enhancement.

Object-based encoding in visual working memory: A critical revisit

Visual working memory (VWM) is responsible for the temporal retention and manipulation of visual information. It has been suggested that VWM employs an object-based encoding (OBE) manner to extract highly discriminable information from visual perception: Whenever one feature dimension of the objects is selected for entry into VWM, the other task-irrelevant highly discriminable dimension is also extracted into VWM involuntarily. However, the task-irrelevant feature in OBE studies might reflect a high capacity fragile VWM (FVWM) trace that stores maskable sensory representations. To directly compare the VWM storage hypothesis and the FVWM storage hypothesis, we used a unique characteristic of FVWM that the representations in FVWM could be erased by backward masks presented at the original locations of the memory array. We required participants to memorise the orientations of three coloured bars while ignoring their colours, and presented backward masks during the VWM maintenance interval. In four experiments, we consistently observed that the OBE occurs regardless of the presentation of the backward masks, except when even the task-relevant features in VWM were significantly interrupted by immediate backward masks, suggesting that the task-irrelevant features of objects are stored in VWM rather than in FVWM.

Topological change induces an interference effect in visual working memory

The “irrelevant-change distracting effect” refers to the effect of changes in irrelevant features on the performance of the target feature, which has frequently been used to study information processing in visual working memory (VWM). In the current study, we reported a novel interference effect in VWM: the topological-change interference effect (TCIE). In a series of six experiments, we examined the influence of topological and nontopological changes as irrelevant features on VWM using a color change detection paradigm. The results revealed that only topological changes, although task irrelevant, could produce a significant interference effect. In contrast, nontopological changes did not produce any evident interference effect. Moreover, the TCIE was a stable and lasting effect, regardless of changes in locations, reporting methods, particular stimulus figures, the other salient feature dimensions and delay interval times. Therefore, our results support the notion that topological invariance that defines perceptual objects plays an essential role in maintaining representations in VWM.

The storage mechanism of dynamic relations in visual working memory

The ability to briefly hold and manipulate object relations provides a foundation for interacting with our complex environment. Previous studies on working memory focused more on objects than their relations, which is disproportionate in the context of their theoretical importance. This study examined dynamic relations of objects to better understand the storage mechanism (capacity and representation) using a self-developed modified change detection paradigm, where an object moved dynamically, based on its relation to other objects. Eighty-four university students participated in four experiments (21 each), wherein they observed dynamic relations between objects presented on a display, memorized them, and reported whether the memorized relations were identical to the probatory relations. Results showed that visual working memory had an upper limitation of holding dynamic relations. When relations were independent of each other, the limitation was two; whereas in non-independent cases where different relations shared an object, memory accuracy decreased with increasing relations complexity, rather than with the number of objects or relations. Thus, dynamic relations are probably not stored as object features, nor stored independent of objects. More likely, relations and objects are represented in visual working memory as an integrated perceptual unit.

Age Differences in the Efficiency of Filtering and Ignoring Distraction in Visual Working Memory

Healthy aging is associated with decline in the ability to maintain visual information in working memory (WM). We examined whether this decline can be explained by decreases in the ability to filter distraction during encoding or to ignore distraction during memory maintenance. Distraction consisted of irrelevant objects (Exp. 1) or irrelevant features of an object (Exp. 2). In Experiment 1, participants completed a spatial WM task requiring remembering locations on a grid. During encoding or during maintenance, irrelevant distractor positions were presented. In Experiment 2, participants encoded either single-feature (colors or orientations) or multifeature objects (colored triangles) and later reproduced one of these features using a continuous scale. In multifeature blocks, a precue appeared before encoding or a retrocue appeared during memory maintenance indicating with 100% certainty to the to-be-tested feature, thereby enabling filtering and ignoring of the irrelevant (not-cued) feature, respectively. There were no age-related deficits in the efficiency of filtering and ignoring distractor objects (Exp. 1) and of filtering irrelevant features (Exp. 2). Both younger and older adults could not ignore irrelevant features when cued with a retrocue. Overall, our results provide no evidence for an aging deficit in using attention to manage visual WM.

The role of attention in retaining the binding of integral features in working memory

Previous studies have suggested that retaining bindings in working memory (WM) requires more object-based attention than retaining constituent features. However, we still need to address the object-based attention hypothesis to determine both the generality (Does the object-based attention hypothesis of binding apply to feature bindings other than those tested?) and the reality (Was the observed effect in previous studies an artifact of the testing process?). We addressed these two issues by focusing on the binding of integral features, which was ignored in previous studies. Integral features can be manipulated independently but cannot be attended to or processed independently of each other, and they are primarily perceived in a more unitary fashion. Consequently, integral-feature bindings should be processed as integrated units without the help of extra object-based attention. We examined whether or not the object-based attention hypothesis applied to integral-feature bindings (generality), and these results enabled us to check the reality of the hypothesis. In line with our prediction, we found that a secondary task consuming object-based attention did not selectively impair the binding performance (Experiments 1, 2, 3, 5, and 7). The absence of selective binding impairment was not attributable to the use of an invalid secondary task (Experiment 4), failure to memorize the binding between length and width (Experiment 6), tapping the incorrect type of attention (Experiment 6), the feasibility of feature categorization (Experiment 7), or poor task performance (Experiment 7). Overall, these results suggest that the object-based attention hypothesis does not fit for the integral-feature bindings, and that the pivotal role of object-based attention reported by previous studies was reliable.

Event-based encoding of biological motion and location in visual working memory

We make use of discrete yet meaningful events to orient ourselves to the dynamic environment. Among these events, biological motion, referring to the movements of animate entities, is one of the most biologically salient. We usually encounter biological motions of multiple human beings taking place simultaneously at distinct locations. How we encode biological motions into visual working memory (VWM) to form a coherent experience of the external world and guide our social behaviour remains unclear. This study for the first time addressed the VWM encoding mechanism of biological motions and their corresponding locations. We tested an event-based encoding hypothesis for biological motion and location: When one element of an event is required to be memorised, the irrelevant element of an event will also be extracted into VWM. We presented participants with three biological motions at different locations and required them to memorise only the biological motions or their locations while ignoring the other dimension. We examined the event-based encoding by probing a distracting effect: If the event-based encoding took place, the change of irrelevant dimension in the probe would lead to a significant distraction and impair the performance of detecting target dimension. We found significant distracting effects, which lasted for 3 s but vanished at 6 s, regardless of the target dimension (biological motions vs. locations, Experiment 1) and the exposure time of memory array (1 s vs. 3 s, Experiment 2). These results together support an event-based encoding mechanism during VWM encoding of biological motions and their corresponding locations.

Feature-based guidance of attention during post-saccadic selection

Current models of trans-saccadic perception propose that, after a saccade, the saccade target object must be localized among objects near the landing position. However, the nature of the attentional mechanisms supporting this process is currently under debate. In the present study, we tested whether surface properties of the saccade target object automatically bias post-saccadic selection using a variant of the visual search task. Participants executed a saccade to a shape-singleton target in a circular array. During this primary saccade, the array sometimes rotated so that the eyes landed between the target and an adjacent distractor, requiring gaze correction. In addition, each object in the array had an incidental color value. On Switch trials, the target and adjacent distractor switched colors. The accuracy and latency of gaze correction to the target (measures that provide a direct index of target localization) were compared with a control condition in which no color switch occurred (No-switch trials). Gaze correction to the target was substantially impaired in the Switch condition. This result was obtained even when participants had substantial incentive to avoid encoding the color of the saccade target. In addition, similar effects were observed when the roles of the two feature dimensions (color and shape) were reversed. The results indicate that saccade target features are automatically encoded before a saccade, are retained in visual working memory across the saccade, and instantiate a feature-based selection operation when the eyes land, biasing attention toward objects that match target features.

Feature-based information filtering in visual working memory is impaired in Parkinson's disease

Increasing attention has been given to working memory (WM) impairment in Parkinson's disease (PD) patients. Previous studies revealed that the space-orientated feature-based filtering (target and distractors in distinct locations) was impaired in PD patients. However, the object-orientated feature-based filtering (target and distractor information pertaining to one object) ability in PD patients remains unclear. In this study, we examined the object-orientated feature-based filtering ability of 14 PD patients and 14 healthy controls in a change detection task under EEG monitoring. Participants were asked to remember the colors of two different objects while ignoring their shapes. Critically, the irrelevant feature could be changed in the probe. A failure in complete feature-based filtering would lead to an "irrelevant-change distracting effect," where the change of the irrelevant feature would impair the performance of the target feature, and lead to an enhanced anterior N2. We found that the distracting effect was larger in PD patients than in the control group in terms of d'; however, the N2 amplitude evoked by the irrelevant change was smaller in PD patients than in the control group. These results suggested that the object-orientated feature-based filtering ability was impaired in PD, which might derive from the deficit of their executive control.

Visual short-term memory for oriented, colored objects

A central question in the study of visual short-term memory (VSTM) has been whether its basic units are objects or features. Most studies addressing this question have used change detection tasks in which the feature value before the change is highly discriminable from the feature value after the change. This approach assumes that memory noise is negligible, which recent work has shown not to be the case. Here, we investigate VSTM for orientation and color within a noisy-memory framework, using change localization with a variable magnitude of change. A specific consequence of the noise is that it is necessary to model the inference (decision) stage. We find that (a) orientation and color have independent pools of memory resource (consistent with classic results); (b) an irrelevant feature dimension is either encoded but ignored during decision-making, or encoded with low precision and taken into account during decision-making; and (c) total resource available in a given feature dimension is lower in the presence of task-relevant stimuli that are neutral in that feature dimension. We propose a framework in which feature resource comes both in packaged and in targeted form.

Crowdsourced single-trial probes of visual working memory for irrelevant features

We measured the precision with which an irrelevant feature of a relevant object is stored in visual short-term memory. In each experiment, 600 online subjects each completed 30 trials in which the same feature (orientation or color) was relevant, followed by a single surprise trial in which the other feature was relevant. Pooling data across all subjects, we find in a delayed-estimation task but not in a change localization task that the irrelevant feature is retrieved, but with much lower precision than when the same feature is relevant: The irrelevant/relevant precision ratio was 3.8% for orientation and 20.4% for color.

Object-based Encoding in Visual Working Memory: Evidence from Memory-driven Attentional Capture

Visual working memory (VWM) adopts a specific manner of object-based encoding (OBE) to extract perceptual information: Whenever one feature-dimension is selected for entry into VWM, the others are also extracted. Currently most studies revealing OBE probed an 'irrelevant-change distracting effect', where changes of irrelevant-features dramatically affected the performance of the target feature. However, the existence of irrelevant-feature change may affect participants' processing manner, leading to a false-positive result. The current study conducted a strict examination of OBE in VWM, by probing whether irrelevant-features guided the deployment of attention in visual search. The participants memorized an object's colour yet ignored shape and concurrently performed a visual-search task. They searched for a target line among distractor lines, each embedded within a different object. One object in the search display could match the shape, colour, or both dimensions of the memory item, but this object never contained the target line. Relative to a neutral baseline, where there was no match between the memory and search displays, search time was significantly prolonged in all match conditions, regardless of whether the memory item was displayed for 100 or 1000 ms. These results suggest that task-irrelevant shape was extracted into VWM, supporting OBE in VWM.

Seeing without knowing: task relevance dissociates between visual awareness and recognition

We demonstrate that task relevance dissociates between visual awareness and knowledge activation to create a state of seeing without knowing-visual awareness of familiar stimuli without recognizing them. We rely on the fact that in order to experience a Kanizsa illusion, participants must be aware of its inducers. While people can indicate the orientation of the illusory rectangle with great ease (signifying that they have consciously experienced the illusion's inducers), almost 30% of them could not report the inducers' color. Thus, people can see, in the sense of phenomenally experiencing, but not know, in the sense of recognizing what the object is or activating appropriate knowledge about it. Experiment 2 tests whether relevance-based selection operates within objects and shows that, contrary to the pattern of results found with features of different objects in our previous studies and replicated in Experiment 1, selection does not occur when both relevant and irrelevant features belong to the same object. We discuss these findings in relation to the existing theories of consciousness and to attention and inattentional blindness, and the role of cognitive load, object-based attention, and the use of self-reports as measures of awareness.

Coarse-to-fine construction for high-resolution representation in visual working memory

Background

This study explored whether the high-resolution representations created by visual working memory (VWM) are constructed in a coarse-to-fine or all-or-none manner. The coarse-to-fine hypothesis suggests that coarse information precedes detailed information in entering VWM and that its resolution increases along with the processing time of the memory array, whereas the all-or-none hypothesis claims that either both enter into VWM simultaneously, or neither does.

Methodology/Principal Findings

We tested the two hypotheses by asking participants to remember two or four complex objects. An ERP component, contralateral delay activity (CDA), was used as the neural marker. CDA is higher for four objects than for two objects when coarse information is primarily extracted; yet, this CDA difference vanishes when detailed information is encoded. Experiment 1 manipulated the comparison difficulty of the task under a 500-ms exposure time to determine a condition in which the detailed information was maintained. No CDA difference was found between two and four objects, even in an easy-comparison condition. Thus, Experiment 2 manipulated the memory array’s exposure time under the easy-comparison condition and found a significant CDA difference at 100 ms while replicating Experiment 1′s results at 500 ms. In Experiment 3, the 500-ms memory array was blurred to block the detailed information; this manipulation reestablished a significant CDA difference.

Conclusions/Significance

These findings suggest that the creation of high-resolution representations in VWM is a coarse-to-fine process.