The neural mechanisms of percept-memory comparison in visual working memory

Selectively maintaining an object's feature in visual working memory: A comparison between highly discriminable and fine-grained features

Selectively maintaining information is an essential function of visual working memory (VWM). Recent VWM studies have mainly focused on selective maintenance of objects, leaving the mechanisms of selectively maintaining an object's feature in VWM unknown. Based on the interactive model of perception and VWM, we hypothesized that there are distinct selective maintenance mechanisms for objects containing fine-grained features versus objects containing highly discriminable features. To test this hypothesis, we first required participants to memorize a dual-feature object (colored simple shapes vs. colored polygons), and informed them about the target feature via a retro-cue. Then a visual search task was added to examine the fate of the irrelevant feature. The selective maintenance of an object's feature predicted that the irrelevant feature should be removed from the active state of VWM and should not capture attention when presented as a distractor in the visual search task. We found that irrelevant simple shapes impaired performance in the visual search task (Experiment 1). However, irrelevant polygons did not affect visual search performance (Experiment 2), and this could not be explained by decay of polygons (Experiment 3) or by polygons not capturing attention (Experiment 4). These findings suggest that VWM adopts dissociable mechanisms to selectively maintain an object's feature, depending on the feature's perceptual characteristics.

Behavioral and electrophysiological evidence for the flexible recruitment of feature- and object-based processing in visual working memory comparison

Previous research is inconclusive on when visual working memory (VWM) can be object-based or feature-based. Prior event-related potential (ERP) studies using change detection tasks have found that amplitudes of the N200-an ERP index of VWM comparison- are sensitive to changes in both relevant and irrelevant features, suggesting a bias toward object-based processing. To test whether VWM comparison processing can operate in a feature-based manner, we aimed to create circumstances that would support feature-based processing by: 1) using a strong task-relevance manipulation, and 2) repeating features within a display. Participants completed two blocks of a change detection task for four-item displays in which they were told to respond to color changes (task relevant) but not shape changes (task irrelevant). The first block contained only task-relevant changes to create a strong task-relevance manipulation. In the second block, both relevant and irrelevant changes were present. In both blocks, half of the arrays contained within-display feature repetitions (e.g. two items of the same color or shape). We found that during the second block, N200 amplitudes were sensitive to task-relevant but not irrelevant features regardless of repetition status, consistent with feature-based processing. However, analyses of behavioral data and N200 latencies suggested that object-based processing was occurring at some stages of VWM processing on task-irrelevant feature change trials. In particular, task-irrelevant changes may be processed after no task-relevant feature change is revealed. Overall, the results from the current study suggest that the VWM processing is flexible and can be either object- or feature-based.

Luminance Contrast Drives Interactions between Perception and Working Memory

Visual working memory (WM) enables the use of past sensory experience in guiding behavior. Yet, laboratory tasks commonly evaluate WM in a way that separates it from its sensory bottleneck. To understand how perception interacts with visual memory, we used a delayed shape recognition task to probe how WM may differ for stimuli that bias processing toward different visual pathways. Luminance compared with chromatic signals are more efficient in driving the processing of shapes and may thus also lead to better WM encoding, maintenance, and memory recognition. To evaluate this prediction, we conducted two experiments. In the first psychophysical experiment, we measured contrast thresholds for different WM loads. Luminance contrast was encoded into WM more efficiently than chromatic contrast, even when both sets of stimuli were equated for discriminability. In the second experiment, which also equated stimuli for discriminability, early sensory responses in the EEG that are specific to luminance pathways were modulated by WM load and thus likely reflect the neural substrate of the increased efficiency. Our results cannot be accounted for by simple saliency differences between luminance and color. Rather, they provide evidence for a direct connection between low-level perceptual mechanisms and WM by showing a crucial role of luminance for forming WM representations of shape.

Object-based encoding in visual working memory: A critical revisit

Visual working memory (VWM) is responsible for the temporal retention and manipulation of visual information. It has been suggested that VWM employs an object-based encoding (OBE) manner to extract highly discriminable information from visual perception: Whenever one feature dimension of the objects is selected for entry into VWM, the other task-irrelevant highly discriminable dimension is also extracted into VWM involuntarily. However, the task-irrelevant feature in OBE studies might reflect a high capacity fragile VWM (FVWM) trace that stores maskable sensory representations. To directly compare the VWM storage hypothesis and the FVWM storage hypothesis, we used a unique characteristic of FVWM that the representations in FVWM could be erased by backward masks presented at the original locations of the memory array. We required participants to memorise the orientations of three coloured bars while ignoring their colours, and presented backward masks during the VWM maintenance interval. In four experiments, we consistently observed that the OBE occurs regardless of the presentation of the backward masks, except when even the task-relevant features in VWM were significantly interrupted by immediate backward masks, suggesting that the task-irrelevant features of objects are stored in VWM rather than in FVWM.

Oscillatory brain activity and maintenance of verbal and visual working memory: A systematic review

Brain oscillations likely play a significant role in the storage of information in working memory (WM). Despite the wide popularity of the topic, current attempts to summarize the research in the field are narrative reviews. We address this gap by providing a descriptive systematic review, in which we investigated oscillatory correlates of maintenance of verbal and visual information in WM. The systematic approach enabled us to challenge some common views popularized by previous research. The identified literature (100 EEG/MEG studies) highlighted the importance of theta oscillations in verbal WM: frontal midline theta enhanced with load in most verbal studies, while more equivocal results have been obtained in visual studies. Increasing WM load affected alpha activity in most studies, but the direction of the effect was inconsistent: the ratio of studies that found alpha increase versus decrease with increasing load was 80/20% in the verbal WM domain and close to 60/40% in the visual domain. Alpha asymmetry (left < right) was a common finding in both verbal and visual WM studies. Beta and gamma activity studies yielded the least convincing data: a diversity in the spatial and frequency distribution of beta activity prevented us from making a coherent conclusion; gamma rhythm was virtually neglected in verbal WM studies with no systematic support for sustained gamma changes during the delay in EEG studies in general.

Event-based encoding of biological motion and location in visual working memory

We make use of discrete yet meaningful events to orient ourselves to the dynamic environment. Among these events, biological motion, referring to the movements of animate entities, is one of the most biologically salient. We usually encounter biological motions of multiple human beings taking place simultaneously at distinct locations. How we encode biological motions into visual working memory (VWM) to form a coherent experience of the external world and guide our social behaviour remains unclear. This study for the first time addressed the VWM encoding mechanism of biological motions and their corresponding locations. We tested an event-based encoding hypothesis for biological motion and location: When one element of an event is required to be memorised, the irrelevant element of an event will also be extracted into VWM. We presented participants with three biological motions at different locations and required them to memorise only the biological motions or their locations while ignoring the other dimension. We examined the event-based encoding by probing a distracting effect: If the event-based encoding took place, the change of irrelevant dimension in the probe would lead to a significant distraction and impair the performance of detecting target dimension. We found significant distracting effects, which lasted for 3 s but vanished at 6 s, regardless of the target dimension (biological motions vs. locations, Experiment 1) and the exposure time of memory array (1 s vs. 3 s, Experiment 2). These results together support an event-based encoding mechanism during VWM encoding of biological motions and their corresponding locations.

Object features reinstated from episodic memory guide attentional selection

When observers search for an object in the environment, they compare the incoming sensory information to the attentional template, a representation of the target in visual working memory (VWM). Previous studies have shown that visual search is more efficient when the attentional template is precise. We pursued the hypothesis that the attentional template in VWM is automatically complemented by features from long-term memory, possibly to increase its precision. At the beginning of the experiment, observers learned associations between shape and color. Then, we tested whether selecting one of these shapes was influenced by the previously associated color. To this end, we ran a saccadic selection task consisting of a memory and choice display. In the memory display, the target shape was presented at central fixation and participants were instructed to foveate this shape in the subsequent choice display. In the choice display, the target shape appeared together with a distractor shape at eccentric positions. Importantly, the target shape was colorless (gray) in the memory display so that only shape, but not color was loaded into VWM. However, saccades went more frequently to the target shape when it was shown in the learned color than when this color was shown in the distractor. Thus, the color of the target shape was reinstated from episodic memory to complement the attentional template in VWM.

Agent identity drives adaptive encoding of biological motion into working memory

To engage in normal social interactions, we have to encode human biological motions (BMs, e.g., walking and jumping), which is one of the most salient and biologically significant types of kinetic information encountered in everyday life, into working memory (WM). Critically, each BM in real life is produced by a distinct person, carrying a dynamic motion signature (i.e., identity). Whether agent identity influences the WM processing of BMs remains unknown. Here, we addressed this question by examining whether memorizing BMs with different identities promoted the WM processing of task-irrelevant clothing colors. Two opposing hypotheses were tested: (a) WM only stores the target action (element-based hypothesis) and (b) WM stores both action and irrelevant clothing color (event-based hypothesis), interpreting each BM as an event. We required participants to memorize actions that either performed by one agent or distinct agents, while ignoring clothing colors. Then we examined whether the irrelevant color was also stored in WM by probing a distracting effect: If the color was extracted into WM, the change of irrelevant color in the probe would lead to a significant distracting effect on action performance. We found that WM encoding of BMs was adaptive: Once the memorized actions had different identities, WM adopted an event-based encoding mode regardless of memory load and probe identity (Experiment 1, different-identity group of Experiment 2, and Experiment 3). However, WM used an element-based encoding mode when memorized-actions shared the same identity (same-identity group of Experiment 2) or were inverted (Experiment 4). Overall, these findings imply that agent identity information has a significant effect on the WM processing of BMs.

Age-related changes to oscillatory dynamics during maintenance and retrieval in a relational memory task

In aging, structural and/or functional brain changes may precede changes in cognitive performance. We previously showed that despite having hippocampal volumes similar to those of younger adults, older adults showed oscillatory changes during the encoding phase of a short-delay visuospatial memory task that required spatial relations among objects to be bound across time (Rondina et al., 2016). The present work provides a complementary set of analyses to examine age-related changes in oscillatory activity during maintenance and retrieval of those spatial relations in order to provide a comprehensive examination of the neural dynamics that support memory function in aging. Participants were presented with three study objects sequentially. Following a delay (maintenance phase), the objects were re-presented simultaneously and participants had to determine whether the relative spatial relations among the objects had been maintained (retrieval phase). Older adults had similar task accuracy, but slower response times, compared to younger adults. Both groups showed a decrease in theta (2-7Hz), alpha (9-14Hz), and beta (15-30Hz) power during the maintenance phase. During the retrieval phase, younger adults showed theta and beta power increases that predicted greater task accuracy, whereas older adults showed a widespread decrease in each of the three frequency ranges that predicted longer response latencies. Older adults also showed distinct patterns of behaviour-related activity depending on whether the analysis was time-locked to the onset of the stimulus or to the onset of the response during the test phase. These findings suggest that older adults may experience declines in relational binding and/or comparison processes that are reflected in oscillatory changes prior to structural decline.

[Event-related potentials of working memory impairment in children with primary nocturnal enuresis]

OBJECTIVE

To study the cognitive neural mechanism of working memory impairment in children with primary nocturnal enuresis using event-related potential (ERP).

METHODS

A total of 14 children with primary nocturnal enuresis were enrolled as enuresis group, and 14 normal children were enrolled as control group. The learning-recognition task test was applied, and the ERP components (P2, N2, and P3) at Fz lead while identifying old pictures (learned) and new ones (unlearned) were measured and compared between the two groups.

RESULTS

While identifying the old pictures, the enuresis group had a lower amplitude of P2 and N2 than the control group (P<0.05). There were no significant differences between the two groups in the latency of P2, N2, and P3 and the amplitude of P3. While identifying the new pictures, the enuresis group had a longer latency of P2 and a significantly lower amplitude of N2 than the control group (P<0.05). There were no significant differences between the two groups in the amplitude of P2 and P3 and the latency of N2 and P3.

CONCLUSIONS

Compared with normal children, the children with primary nocturnal enuresis have reduced abilities of classified information extraction, a prolonged reaction time, and reductions in memory capacity, memory consolidation, and conflict monitoring, which causes working memory impairment.

Change perception and change interference within and across feature dimensions

The ability to perceive a change in a visual object is reduced when that change is presented in competition with other changes which are task-irrelevant. We performed two experiments which investigate the basis of this change interference effect. We tested whether change interference occurs as a consequence of some form of attentional capture, or whether the interference occurs at a stage prior to attentional selection of the task-relevant change. A modified probe-detection task was used to explore this issue. Observers were required to report the presence/absence of a specified change-type (colour, shape) in the probe, in a context in which - on certain trials - irrelevant changes occur in non-probe items. There were two key variables in these experiments: the attentional state of the observer, and the dimensional congruence of changes in the probe and non-probe items. Change interference was strongest when the irrelevant changes were the same as those on the report dimension. However the interference pattern persisted even when observers did not know the report dimension at the time the changes occurred. These results seem to rule out attention as a factor. Our results fit best with an interpretation in which change interference produces feature-specific sensory noise which degrades the signal quality of the target change.

Electrophysiological Correlates of Change Detection during Delayed Matching Task: A Comparison of Different References

Detecting the changed information between memory representation and incoming sensory inputs is a fundamental cognitive ability. By offering the promise of excellent temporal resolution, event-related potential (ERP) technique has served as a primary tool for studying this process with reference of the linked mastoid (LM). However, given that LM may distort the ERP signals, it is still undetermined whether LM is the best reference choice. The goal of the current study was to systematically compare LM, reference electrode standardization technique (REST) and average reference (AR) for assessing the ERP correlates of change detection during a delayed matching task. Colored shapes were adopted as materials while both the task-relevant shape feature and -irrelevant color feature could be changed. The results of the ERP amplitude showed that both of the task-relevant and -conjunction feature changes elicited significantly more positive posterior P2 in REST and AR, but not in LM. Besides, significantly increased N270 was observed in task-relevant and -conjunction feature changes in both the REST and LM, but in the conjunction feature change in AR. Only the REST-obtained N270 revealed a significant increment in task-irrelevant feature change, which was compatible with the delayed behavioral performance. Statistical parametric scalp mapping (SPSM) results showed a left posterior distribution for AR, an anterior distribution for LM, and both the anterior and left posterior distributions for REST. These results indicate that different types of references may provide distinct cognitive interpretations. Interestingly, only the SPSM of REST was consistent with previous fMRI findings. Combined with the evidence of simulation studies and the current observations, we take the REST-based results as the objective one, and recommend using REST technology in the future ERP data analysis.

The role of central attention in retrieval from visual short-term memory

The role of central attention in visual short-term memory (VSTM) encoding and maintenance is well established, yet its role in retrieval has been largely unexplored. This study examined the involvement of central attention in retrieval from VSTM using a dual-task paradigm. Participants performed a color change-detection task. Set size varied between 1 and 3 items, and the memory sample was maintained for either a short or a long delay period. A secondary tone discrimination task was introduced at the end of the delay period, shortly before the appearance of a central probe, and occupied central attention while participants were searching within VSTM representations. Similarly to numerous previous studies, reaction time increased as a function of set size reflecting the occurrence of a capacity-limited memory search. When the color targets were maintained over a short delay, memory was searched for the most part without the involvement of central attention. However, with a longer delay period, the search relied entirely on the operation of central attention. Taken together, this study demonstrates that central attention is involved in retrieval from VSTM, but the extent of its involvement depends on the duration of the delay period. Future studies will determine whether the type of memory search (parallel or serial) carried out during retrieval depends on the nature of the attentional mechanism involved the task.

Object-based Encoding in Visual Working Memory: Evidence from Memory-driven Attentional Capture

Visual working memory (VWM) adopts a specific manner of object-based encoding (OBE) to extract perceptual information: Whenever one feature-dimension is selected for entry into VWM, the others are also extracted. Currently most studies revealing OBE probed an 'irrelevant-change distracting effect', where changes of irrelevant-features dramatically affected the performance of the target feature. However, the existence of irrelevant-feature change may affect participants' processing manner, leading to a false-positive result. The current study conducted a strict examination of OBE in VWM, by probing whether irrelevant-features guided the deployment of attention in visual search. The participants memorized an object's colour yet ignored shape and concurrently performed a visual-search task. They searched for a target line among distractor lines, each embedded within a different object. One object in the search display could match the shape, colour, or both dimensions of the memory item, but this object never contained the target line. Relative to a neutral baseline, where there was no match between the memory and search displays, search time was significantly prolonged in all match conditions, regardless of whether the memory item was displayed for 100 or 1000 ms. These results suggest that task-irrelevant shape was extracted into VWM, supporting OBE in VWM.

Coarse-to-fine construction for high-resolution representation in visual working memory

Background

This study explored whether the high-resolution representations created by visual working memory (VWM) are constructed in a coarse-to-fine or all-or-none manner. The coarse-to-fine hypothesis suggests that coarse information precedes detailed information in entering VWM and that its resolution increases along with the processing time of the memory array, whereas the all-or-none hypothesis claims that either both enter into VWM simultaneously, or neither does.

Methodology/Principal Findings

We tested the two hypotheses by asking participants to remember two or four complex objects. An ERP component, contralateral delay activity (CDA), was used as the neural marker. CDA is higher for four objects than for two objects when coarse information is primarily extracted; yet, this CDA difference vanishes when detailed information is encoded. Experiment 1 manipulated the comparison difficulty of the task under a 500-ms exposure time to determine a condition in which the detailed information was maintained. No CDA difference was found between two and four objects, even in an easy-comparison condition. Thus, Experiment 2 manipulated the memory array’s exposure time under the easy-comparison condition and found a significant CDA difference at 100 ms while replicating Experiment 1′s results at 500 ms. In Experiment 3, the 500-ms memory array was blurred to block the detailed information; this manipulation reestablished a significant CDA difference.

Conclusions/Significance

These findings suggest that the creation of high-resolution representations in VWM is a coarse-to-fine process.

Building blocks of visual working memory: objects or Boolean maps?

The nature of the building blocks of information in visual working memory (VWM) is a fundamental issue that has not been well resolved. Most researchers take objects as the building blocks, although this perspective has received criticism. The objects could be physically separated ones (strict object hypothesis) or hierarchical objects created from separated individuals (broad object hypothesis). Meanwhile, a newly proposed Boolean map theory for visual attention suggests that Boolean maps may be the building blocks of VWM (Boolean map hypothesis); this perspective could explain many critical findings of VWM. However, no previous study has examined these hypotheses. We explored this issue by focusing on a critical point on which they make distinct predictions. We asked participants to remember two distinct objects (2-object), three distinct objects (3-object), or three objects with repeated information (mixed-3-object, e.g., one red bar and two green bars, green bars could be represented as one hierarchical object) and adopted contralateral delay activity (CDA) to tap into the maintenance phase of VWM. The mixed-3-object condition could generate two Boolean maps, three objects, or three objects most of the time (hierarchical objects are created in certain trials, retaining two objects). Simple orientations (Experiment 1) and colors (Experiments 2 and 3) were used as stimuli. Although the CDA of the mixed-3-object condition was slightly lower than that of the 3-object condition, no significant difference was revealed between them. Both conditions displayed significantly higher CDAs than the 2-object condition. These findings support the broad object hypothesis. We further suggest that Boolean maps might be the unit for retrieval/comparison in VWM.