Different effects of color-based and location-based selection on visual working memory

Life motion signals modulate visual working memory

Previous research has demonstrated that biological motion (BM) cues can induce a reflexive attentional orienting effect, a phenomenon referred to as social attention. However, it remains undetermined whether BM cues can further affect higher-order cognitive processes, such as visual working memory (WM). By combining a modified central pre-cueing paradigm with a traditional WM change detection task, the current study investigated whether the walking direction of BM, as a non-predictive central cue, could modulate the encoding process of WM. Results revealed a significant improvement in WM performance for the items appearing at the location cued by the walking direction of BM. The observed effect disappeared when the BM cues were shown inverted, or when the critical biological characteristics of the cues were removed. Crucially, this effect could be extended to upright feet motion cues without global configuration, reflecting the key role of local BM signals in modulating WM. More importantly, such a BM-induced modulation effect was not observed with inanimate motion cues, although these cues can also elicit attentional effects. Our findings suggest that the attentional effect induced by life motion signals can penetrate to higher-order cognitive processes, and provide compelling evidence for the existence of "life motion detector" in the human brain from a high-level cognitive function perspective.

A direct comparison of attentional orienting to spatial and temporal positions in visual working memory

Different visual attributes effectively guide attention to specific items in visual working memory (VWM), ensuring that particularly important memory contents are readily available. Predictable temporal structures contribute to this efficient use of VWM: items are prospectively prioritized when they are expected to be needed. Occasionally, however, visual events only gain relevance through their timing after they have passed. We investigated retrospective attentional orienting based on temporal position by directly comparing it with orienting to spatial locations, which is typically considered the most powerful selection mechanism. In a colour-change-detection task, in which items appeared sequentially at different locations, symbolic number cues validly indicated the temporal or spatial location of the upcoming probe item either before encoding (precues; Experiment 1) or during maintenance (retrocues; Experiments 1-3). Temporal and spatial cues were physically identical and only differed in their mapping onto either temporal or spatial positions. Predictive cues yielded cueing benefits (i.e., higher accuracy and shorter reaction times) as compared with neutral cues, with larger benefits for precues than for retrocues. Importantly, spatial and temporal cueing benefits did not differ. Equivalent retrocueing benefits were also observed across different cue-probe intervals and irrespective of whether spatial or temporal position was used as retrieval cue, indicating that items were directly bound to temporal position and not prioritized via a space-based mechanism. These findings show that spatial and temporal properties can be used equally well to flexibly prioritise representations held in VWM and they highlight the functional similarities of space and time in VWM.

Decoding Color Visual Working Memory from EEG Signals Using Graph Convolutional Neural Networks

Color has an important role in object recognition and visual working memory (VWM). Decoding color VWM in the human brain is helpful to understand the mechanism of visual cognitive process and evaluate memory ability. Recently, several studies showed that color could be decoded from scalp electroencephalogram (EEG) signals during the encoding stage of VWM, which process visible information with strong neural coding. Whether color could be decoded from other VWM processing stages, especially the maintaining stage which processes invisible information, is still unknown. Here, we constructed an EEG color graph convolutional network model (ECo-GCN) to decode colors during different VWM stages. Based on graph convolutional networks, ECo-GCN considers the graph structure of EEG signals and may be more efficient in color decoding. We found that (1) decoding accuracies for colors during the encoding, early, and late maintaining stages were 81.58%, 79.36%, and 77.06%, respectively, exceeding those during the pre-stimuli stage (67.34%), and (2) the decoding accuracy during maintaining stage could predict participants' memory performance. The results suggest that EEG signals during the maintaining stage may be more sensitive than behavioral measurement to predict the VWM performance of human, and ECo-GCN provides an effective approach to explore human cognitive function.

Multivariate patterns and long-range temporal correlations of alpha oscillations are associated with flexible manipulation of visual working memory representations

The ability to flexibly manipulate memory representations is embedded in visual working memory (VWM) and can be tested using paradigms with retrospective cues. Although valid retrospective cues often facilitate memory recall, invalid ones may or may not result in performance costs. We investigated individual differences in utilising retrospective cues and evaluated how these individual differences are associated with brain oscillatory activity at rest. At the behavioural level, we operationalised flexibility as the ability to make effective use of retrospective cues or disregard them if required. At the neural level, we tested whether individual differences in such flexibility were associated with properties of resting-state alpha oscillatory activity (8-12 Hz). To capture distinct aspects of these brain oscillations, we evaluated their power spectral density and temporal dynamics using long-range temporal correlations (LRTCs). In addition, we performed multivariate patterns analysis (MVPA) to classify individuals' level of behavioural flexibility based on these neural measures. We observed that alpha power alone (magnitude) at rest was not associated with flexibility. However, we found that the participants' ability to manipulate VWM representations was correlated with alpha LRTC and could be decoded using MVPA on patterns of alpha power. Our findings suggest that alpha LRTC and multivariate patterns of alpha power at rest may underlie some of the individual differences in using retrospective cues in working memory tasks.

Individual differences in working memory capacity are unrelated to the magnitudes of retrocue benefits

Previous studies have associated visual working memory (VWM) capacity with the use of internal attention. Retrocues, which direct internal attention to a particular object or feature dimension, can improve VWM performance (i.e., retrocue benefit, RCB). However, so far, no study has investigated the relationship between VWM capacity and the magnitudes of RCBs obtained from object-based and dimension-based retrocues. The present study explored individual differences in the magnitudes of object- and dimension-based RCBs and their relationships with VWM capacity. Participants completed a VWM capacity measurement, an object-based cue task, and a dimension-based cue task. We confirmed that both object- and dimension-based retrocues could improve VWM performance. We also found a significant positive correlation between the magnitudes of object- and dimension-based RCB indexes, suggesting a partly overlapping mechanism between the use of object- and dimension-based retrocues. However, our results provided no evidence for a correlation between VWM capacity and the magnitudes of the object- or dimension-based RCBs. Although inadequate attention control is usually assumed to be associated with VWM capacity, the results suggest that the internal attention mechanism for using retrocues in VWM retention is independent of VWM capacity.

Orienting Attention to Short-Term Memory Representations via Sensory Modality and Semantic Category Retro-Cues

There is growing interest in characterizing the neural mechanisms underlying the interactions between attention and memory. Current theories posit that reflective attention to memory representations generally involves a fronto-parietal attentional control network. The present study aimed to test this idea by manipulating how a particular short-term memory (STM) representation is accessed, that is, based on its input sensory modality or semantic category, during functional magnetic resonance imaging (fMRI). Human participants performed a novel variant of the retro-cue paradigm, in which they were presented with both auditory and visual non-verbal stimuli followed by Modality, Semantic, or Uninformative retro-cues. Modality and, to a lesser extent, Semantic retro-cues facilitated response time relative to Uninformative retro-cues. The univariate and multivariate pattern analyses (MVPAs) of fMRI time-series revealed three key findings. First, the posterior parietal cortex (PPC), including portions of the intraparietal sulcus (IPS) and ventral angular gyrus (AG), had activation patterns that spatially overlapped for both modality-based and semantic-based reflective attention. Second, considering both the univariate and multivariate analyses, Semantic retro-cues were associated with a left-lateralized fronto-parietal network. Finally, the experimental design enabled us to examine how dividing attention cross-modally within STM modulates the brain regions involved in reflective attention. This analysis revealed that univariate activation within bilateral portions of the PPC increased when participants simultaneously attended both auditory and visual memory representations. Therefore, prefrontal and parietal regions are flexibly recruited during reflective attention, depending on the representational feature used to selectively access STM representations.

Association between cue lead time and template-for-rejection effect

The human visual system can actively prioritize task-relevant features to search for a target. Recent studies have reported cases in which the system may suppress irrelevant features by using a template for rejection. However, in those studies, the templates used for rejection were limited to the color domain, and they have yielded mixed results. Our literature review identified three differences among studies that may be responsible for such mixed results: differences in the spatial segmentation of items (i.e., segregated or intermixed across the display), differences in how features are defined and reported (i.e., combined or separate), and differences in cue lead times (short or long). Participants searched for a target-line segment in a shape and identified its orientation from among non-target line-shaped compound shapes that were preceded by one of three cue displays. Positive cues indicated that the target segment would appear in a shape, and negative cues that it would not appear in a shape. Neutral cues indicated that a particular shape would not appear in the current search display. The results demonstrated that reaction times were faster under the negative-cue condition than the neutral-cue condition, reflecting the effect of a shape-based template for rejection (Experiment 1). Experiment 2 replicated the absence of the effect in the shape domain. Experiment 3 indicated that the template-for-rejection effect occurred only when the cue lead time was relatively long, suggesting that time is required (approximately 2,400 ms or longer) for the visual system to form rejection templates. Experiment 4 excluded the possibility that a confound in the target-defining/reporting feature was involved. These results indicated that apparent inconsistencies in research on the template-for-rejection effect can be explained in terms of the time required for templates to be configured.

Task-dependent effects of voluntary space-based and involuntary feature-based attention on visual working memory

Previous research has shown that visual working memory (VWM) can be modulated by space-based or feature-based attentional selection. However, it remains unclear how the two modes of attention operate jointly to affect VWM, and in particular, if involuntary feature-based attention plays a role in VWM. In this study, a pre-cued change detection paradigm was employed to investigate the concurrent effects of space- and feature-based attention on VWM. Space-based attention was manipulated by informative spatial cueing and by varying the proximity between the test item and the cued (fixated) memory item, while feature-based attention was induced in an involuntary manner by having the test item to share the same color or shape with the cued item on a fraction of trials. The results showed that: (1) the memory performance for the cued items was always better than the uncued items, suggesting a beneficial effect of voluntary spatial attention; (2) with a brief duration of the memory array (250 ms), cue-test proximity benefited VWM in the shape judgment task but not in the color judgment task, whereas with a longer duration (1200 ms), no proximity effect was found for either task; (3) VWM was improved for the same-colored items regardless of the task and duration; (4) VWM was improved for the same-shaped items only in the shape judgment task with the longer duration of the memory array. A discrimination task further showed that the proximity effect associated with VWM reflects a perceptual bottleneck in memory encoding for shape but not for color with a brief display. Our results suggest that involuntary feature-based attention could be triggered by spatial cueing to modulate VWM; involuntary color-based attention facilitates VWM independently of task, whereas shape-based facilitation is task-dependent, i.e., confined only to the shape judgment task, presumably reflecting different attention-guiding potencies of the two features.

Feature-based attentional selection affects the perceived duration of a stimulus having two superposed patterns

The perceived duration of a visual event is highly related to stimulus attributes. It is well known that a moving stimulus appears to last longer than a static one does. Previous studies have demonstrated that the time dilation in a moving stimulus can be influenced by perceived motion, rather than by mere physical motion, and that a faster motion appears to last longer than a slower one does. However, whether a top-down attentional set for the feature value can modulate the time dilation in a moving stimulus when two different visual patterns coexist within the same region of the visual field is still unknown. To test this, in Experiment 1, we presented a moving and a static random-dot pattern simultaneously within the same region, and instructed the observer to attend to one of these two patterns. The results demonstrate that perceived duration was longer when attention was directed to the moving, rather than static pattern, although both patterns physically coexisted at the same time and place and for the same duration. In Experiment 2, slow and/or fast moving patterns were presented at the same time and place, and again, feature-based attentional selection affected the perceived duration of the identical physical display. These results suggest that attention to a moving stimulus is an essential factor that determines the time dilation in a moving stimulus. This study revealed that feature-based attention, as opposed to location-based attention, plays an important role in motion-induced time dilation.

A two-level hierarchical framework of visual short-term memory

Over the last couple of decades, a vast amount of research has been dedicated to understanding the nature and the architecture of visual short-term memory (VSTM), the mechanism by which currently relevant visual information is maintained. According to discrete-capacity models, VSTM is constrained by a limited number of discrete representations held simultaneously. In contrast, shared-resource models regard VSTM as limited in resources, which can be distributed flexibly between varying numbers of representations; and a new interference model posits that capacity is limited by interference among items. In this article, we begin by reviewing benchmark findings regarding the debate over VSTM limitations, focusing on whether VSTM storage is all-or-none and on whether object complexity affects capacity. After that, we put forward a hybrid framework of VSTM architecture, arguing that this system is composed of a two-level hierarchy of memory stores, each containing a different set of representations: (1) perceptual memory, a resourcelike level containing analog automatically formed representations of visual stimuli in varying degrees of activation, and (2) visual working memory, in which a subset of three to four items from perceptual memory are bound to conceptual representations and to their locations, thus conveying discrete (digital/symbolic) information which appears quantized. While perceptual memory has a large capacity and is relatively nonselective, visual working memory is restricted in the number of items that can be maintained simultaneously, and its content is regulated by a gating mechanism.

In search of the focus of attention in working memory: 13 years of the retro-cue effect

The concept of attention has a prominent place in cognitive psychology. Attention can be directed not only to perceptual information, but also to information in working memory (WM). Evidence for an internal focus of attention has come from the retro-cue effect: Performance in tests of visual WM is improved when attention is guided to the test-relevant contents of WM ahead of testing them. The retro-cue paradigm has served as a test bed to empirically investigate the functions and limits of the focus of attention in WM. In this article, we review the growing body of (behavioral) studies on the retro-cue effect. We evaluate the degrees of experimental support for six hypotheses about what causes the retro-cue effect: (1) Attention protects representations from decay, (2) attention prioritizes the selected WM contents for comparison with a probe display, (3) attended representations are strengthened in WM, (4) not-attended representations are removed from WM, (5) a retro-cue to the retrieval target provides a head start for its retrieval before decision making, and (6) attention protects the selected representation from perceptual interference. The extant evidence provides support for the last four of these hypotheses.

Feature-based attentional weighting and spreading in visual working memory

Attention can be directed at features and feature dimensions to facilitate perception. Here, we investigated whether feature-based-attention (FBA) can also dynamically weight feature-specific representations within multi-feature objects held in visual working memory (VWM). Across three experiments, participants retained coloured arrows in working memory and, during the delay, were cued to either the colour or the orientation dimension. We show that directing attention towards a feature dimension (1) improves the performance in the cued feature dimension at the expense of the uncued dimension, (2) is more efficient if directed to the same rather than to different dimensions for different objects, and (3) at least for colour, automatically spreads to the colour representation of non-attended objects in VWM. We conclude that FBA also continues to operate on VWM representations (with similar principles that govern FBA in the perceptual domain) and challenge the classical view that VWM representations are stored solely as integrated objects.

Retro-dimension-cue benefit in visual working memory

In visual working memory (VWM) tasks, participants’ performance can be improved by a retro-object-cue. However, previous studies have not investigated whether participants’ performance can also be improved by a retro-dimension-cue. Three experiments investigated this issue. We used a recall task with a retro-dimension-cue in all experiments. In Experiment 1, we found benefits from retro-dimension-cues compared to neutral cues. This retro-dimension-cue benefit is reflected in an increased probability of reporting the target, but not in the probability of reporting the non-target, as well as increased precision with which this item is remembered. Experiment 2 replicated the retro-dimension-cue benefit and showed that the length of the blank interval after the cue disappeared did not influence recall performance. Experiment 3 replicated the results of Experiment 2 with a lower memory load. Our studies provide evidence that there is a robust retro-dimension-cue benefit in VWM. Participants can use internal attention to flexibly allocate cognitive resources to a particular dimension of memory representations. The results also support the feature-based storing hypothesis.

Different Cortical Mechanisms for Spatial vs. Feature-Based Attentional Selection in Visual Working Memory

The limited capacity of visual working memory (VWM) necessitates attentional mechanisms that selectively update and maintain only the most task-relevant content. Psychophysical experiments have shown that the retroactive selection of memory content can be based on visual properties such as location or shape, but the neural basis for such differential selection is unknown. For example, it is not known if there are different cortical modules specialized for spatial vs. feature-based mnemonic attention, in the same way that has been demonstrated for attention to perceptual input. Here, we used transcranial magnetic stimulation (TMS) to identify areas in human parietal and occipital cortex involved in the selection of objects from memory based on cues to their location (spatial information) or their shape (featural information). We found that TMS over the supramarginal gyrus (SMG) selectively facilitated spatial selection, whereas TMS over the lateral occipital cortex (LO) selectively enhanced feature-based selection for remembered objects in the contralateral visual field. Thus, different cortical regions are responsible for spatial vs. feature-based selection of working memory representations. Since the same regions are involved in terms of attention to external events, these new findings indicate overlapping mechanisms for attentional control over perceptual input and mnemonic representations.

The (Un)Clear Effects of Invalid Retro-Cues

Studies with the retro-cue paradigm have shown that validly cueing objects in visual working memory long after encoding can still benefit performance on subsequent change detection tasks. With regard to the effects of invalid cues, the literature is less clear. Some studies reported costs, others did not. We here revisit two recent studies that made interesting suggestions concerning invalid retro-cues: One study suggested that costs only occur for larger set sizes, and another study suggested that inclusion of invalid retro-cues diminishes the retro-cue benefit. New data from one experiment and a reanalysis of published data are provided to address these conclusions. The new data clearly show costs (and benefits) that were independent of set size, and the reanalysis suggests no influence of the inclusion of invalid retro-cues on the retro-cue benefit. Thus, previous interpretations may be taken with some caution at present.

Retrospective cues based on object features improve visual working memory performance in older adults

Research with younger adults has shown that retrospective cues can be used to orient top-down attention toward relevant items in working memory. We examined whether older adults could take advantage of these cues to improve memory performance. Younger and older adults were presented with visual arrays of five colored shapes; during maintenance, participants were presented either with an informative cue based on an object feature (here, object shape or color) that would be probed, or with an uninformative, neutral cue. Although older adults were less accurate overall, both age groups benefited from the presentation of an informative, feature-based cue relative to a neutral cue. Surprisingly, we also observed differences in the effectiveness of shape versus color cues and their effects upon post-cue memory load. These results suggest that older adults can use top-down attention to remove irrelevant items from visual working memory, provided that task-relevant features function as cues.