Tracking the mismatch information in visual short term memory: An event-related potential study

Behavioral and electrophysiological evidence for the flexible recruitment of feature- and object-based processing in visual working memory comparison

Previous research is inconclusive on when visual working memory (VWM) can be object-based or feature-based. Prior event-related potential (ERP) studies using change detection tasks have found that amplitudes of the N200-an ERP index of VWM comparison- are sensitive to changes in both relevant and irrelevant features, suggesting a bias toward object-based processing. To test whether VWM comparison processing can operate in a feature-based manner, we aimed to create circumstances that would support feature-based processing by: 1) using a strong task-relevance manipulation, and 2) repeating features within a display. Participants completed two blocks of a change detection task for four-item displays in which they were told to respond to color changes (task relevant) but not shape changes (task irrelevant). The first block contained only task-relevant changes to create a strong task-relevance manipulation. In the second block, both relevant and irrelevant changes were present. In both blocks, half of the arrays contained within-display feature repetitions (e.g. two items of the same color or shape). We found that during the second block, N200 amplitudes were sensitive to task-relevant but not irrelevant features regardless of repetition status, consistent with feature-based processing. However, analyses of behavioral data and N200 latencies suggested that object-based processing was occurring at some stages of VWM processing on task-irrelevant feature change trials. In particular, task-irrelevant changes may be processed after no task-relevant feature change is revealed. Overall, the results from the current study suggest that the VWM processing is flexible and can be either object- or feature-based.

Discriminative neural pathways for perception-cognition activity of color and face in the human brain

Human performance can be examined using a visual lens. The identification of psychophysical colors and emotional faces with perceptual visual pathways may remain invalid for simple detection tasks. In particular, how the visual dorsal and ventral processing streams handle discriminative visual perceptions and subsequent cognition activities are obscure. We explored these issues using stereoelectroencephalography recordings, which were obtained from patients with pharmacologically resistant epilepsy. Delayed match-to-sample paradigms were used for analyzing the processing of simple colors and complex emotional faces in the human brain. We showed that the angular-cuneus gyrus acts as a pioneer in discriminating the 2 features, and dorsal regions, including the middle frontal gyrus (MFG) and postcentral gyrus, as well as ventral regions, such as the middle temporal gyrus (MTG) and posterior superior temporal sulcus (pSTS), were involved in processing incongruent colors and faces. Critically, the beta and gamma band activities between the cuneus and MTG and between the cuneus and pSTS would tune a separate pathway of incongruency processing. In addition, posterior insular gyrus, fusiform, and MFG were found for attentional modulation of the 2 features via alpha band activities. These findings suggest the neural basis of the discriminative pathways of perception-cognition activities in the human brain.

Task Demands Differentially Affect Processing of Intrinsic and Extrinsic Object Features in Working Memory

Some argue that visual working memory operates on integrated object representations. Here, we contend that obligatory feature integration occurs with intrinsic but not extrinsic object features. Working memory for shapes and colors was assessed using a change-detection task with a central test probe, while recording event-related potentials (ERPs). Color was either an intrinsic surface feature of a shape or connected to the shape via a proximal but spatially disjunct extrinsic frame. There were two types of test: The direct test required memory for shape and color; the indirect test required only shape memory. Study-test changes of color were therefore either task-relevant or task-irrelevant. We assessed performance costs and event-related potential (ERP) effects arising from color changes. In the direct test, performance was poorer for extrinsic than intrinsic stimuli; task-relevant color changes elicited enhanced frontal negativity (N2, FN400) for both intrinsic and extrinsic stimuli. In the indirect test, performance costs and ERP effects associated with irrelevant color change were larger for intrinsic than extrinsic stimuli. This suggests intrinsic information is more readily integrated into the working-memory representation and evaluated against the test probe. Findings imply that feature integration is not obligatory under all conditions but influenced by stimulus-driven and task-related focus of attention.

Object-based encoding in visual working memory: A critical revisit

Visual working memory (VWM) is responsible for the temporal retention and manipulation of visual information. It has been suggested that VWM employs an object-based encoding (OBE) manner to extract highly discriminable information from visual perception: Whenever one feature dimension of the objects is selected for entry into VWM, the other task-irrelevant highly discriminable dimension is also extracted into VWM involuntarily. However, the task-irrelevant feature in OBE studies might reflect a high capacity fragile VWM (FVWM) trace that stores maskable sensory representations. To directly compare the VWM storage hypothesis and the FVWM storage hypothesis, we used a unique characteristic of FVWM that the representations in FVWM could be erased by backward masks presented at the original locations of the memory array. We required participants to memorise the orientations of three coloured bars while ignoring their colours, and presented backward masks during the VWM maintenance interval. In four experiments, we consistently observed that the OBE occurs regardless of the presentation of the backward masks, except when even the task-relevant features in VWM were significantly interrupted by immediate backward masks, suggesting that the task-irrelevant features of objects are stored in VWM rather than in FVWM.

The basis of report-difference superiority in delayed perceptual comparison tasks

A major role for visual short-term memory (VSTM) is to mediate perceptual comparisons of visual information across successive glances and brief temporal interruptions. Research that has focused on the comparison process has noted a marked tendency for performance to be better when participants are required to report a difference between the displays rather than report the absence of a difference (i.e. a sameness). We refer to this performance asymmetry as report-difference superiority (RDS). It has been suggested that RDS reflects the operation of a reflexive mechanism that generates a mismatch signal during the comparison of visual input with information maintained in VSTM. This bottom-up mechanism therefore gives evidence for the presence of a feature change but not for the absence of such a change; consequently, a sameness is harder to detect than a difference between two displays. We test this explanation, and determine whether by itself it is a sufficient explanation of the RDS. In a delayed comparison task we find the RDS effect is most prevalent when items retain the same display locations; however, the effect does persist even when compared item locations were scrambled across memory and test arrays. However, with a conjunction task this scrambling of locations was effective in wholly abolishing the RDS effect. We consider that the RDS effect is a consequence of local comparisons of features, as well as global statistical comparisons.

Event-based encoding of biological motion and location in visual working memory

We make use of discrete yet meaningful events to orient ourselves to the dynamic environment. Among these events, biological motion, referring to the movements of animate entities, is one of the most biologically salient. We usually encounter biological motions of multiple human beings taking place simultaneously at distinct locations. How we encode biological motions into visual working memory (VWM) to form a coherent experience of the external world and guide our social behaviour remains unclear. This study for the first time addressed the VWM encoding mechanism of biological motions and their corresponding locations. We tested an event-based encoding hypothesis for biological motion and location: When one element of an event is required to be memorised, the irrelevant element of an event will also be extracted into VWM. We presented participants with three biological motions at different locations and required them to memorise only the biological motions or their locations while ignoring the other dimension. We examined the event-based encoding by probing a distracting effect: If the event-based encoding took place, the change of irrelevant dimension in the probe would lead to a significant distraction and impair the performance of detecting target dimension. We found significant distracting effects, which lasted for 3 s but vanished at 6 s, regardless of the target dimension (biological motions vs. locations, Experiment 1) and the exposure time of memory array (1 s vs. 3 s, Experiment 2). These results together support an event-based encoding mechanism during VWM encoding of biological motions and their corresponding locations.

Agent identity drives adaptive encoding of biological motion into working memory

To engage in normal social interactions, we have to encode human biological motions (BMs, e.g., walking and jumping), which is one of the most salient and biologically significant types of kinetic information encountered in everyday life, into working memory (WM). Critically, each BM in real life is produced by a distinct person, carrying a dynamic motion signature (i.e., identity). Whether agent identity influences the WM processing of BMs remains unknown. Here, we addressed this question by examining whether memorizing BMs with different identities promoted the WM processing of task-irrelevant clothing colors. Two opposing hypotheses were tested: (a) WM only stores the target action (element-based hypothesis) and (b) WM stores both action and irrelevant clothing color (event-based hypothesis), interpreting each BM as an event. We required participants to memorize actions that either performed by one agent or distinct agents, while ignoring clothing colors. Then we examined whether the irrelevant color was also stored in WM by probing a distracting effect: If the color was extracted into WM, the change of irrelevant color in the probe would lead to a significant distracting effect on action performance. We found that WM encoding of BMs was adaptive: Once the memorized actions had different identities, WM adopted an event-based encoding mode regardless of memory load and probe identity (Experiment 1, different-identity group of Experiment 2, and Experiment 3). However, WM used an element-based encoding mode when memorized-actions shared the same identity (same-identity group of Experiment 2) or were inverted (Experiment 4). Overall, these findings imply that agent identity information has a significant effect on the WM processing of BMs.

Working Memory for Low-Level Visual Features: Sensory Mechanisms for Detecting the Mismatch between the Current Orientations and Those Stored in Memory

The amplitudes of the P100 and N150 early components of evoked potentials in the visual cortex have been analyzed on 33 volunteers with normal vision during matching between the current orientation and that stored in memory. An increase in the P100 response in the occipital and parietal cortical areas was identified as an informative indicator of mismatch between the current and stored-in-memory orientations. This effect was not found for more complex stimuli, namely, spatial patterns. The N150 component demonstrated a similar effect, but in contrast to P100 it was not stimulus specific. Thus, in the first 100 ms, a signal of mismatch between the current and stored-in-memory orientations arises in the early visual areas that represents a mechanism for early implicit response to changes in the basic characteristics of the visual space.

Feature-based information filtering in visual working memory is impaired in Parkinson's disease

Increasing attention has been given to working memory (WM) impairment in Parkinson's disease (PD) patients. Previous studies revealed that the space-orientated feature-based filtering (target and distractors in distinct locations) was impaired in PD patients. However, the object-orientated feature-based filtering (target and distractor information pertaining to one object) ability in PD patients remains unclear. In this study, we examined the object-orientated feature-based filtering ability of 14 PD patients and 14 healthy controls in a change detection task under EEG monitoring. Participants were asked to remember the colors of two different objects while ignoring their shapes. Critically, the irrelevant feature could be changed in the probe. A failure in complete feature-based filtering would lead to an "irrelevant-change distracting effect," where the change of the irrelevant feature would impair the performance of the target feature, and lead to an enhanced anterior N2. We found that the distracting effect was larger in PD patients than in the control group in terms of d'; however, the N2 amplitude evoked by the irrelevant change was smaller in PD patients than in the control group. These results suggested that the object-orientated feature-based filtering ability was impaired in PD, which might derive from the deficit of their executive control.

Object-based Encoding in Visual Working Memory: Evidence from Memory-driven Attentional Capture

Visual working memory (VWM) adopts a specific manner of object-based encoding (OBE) to extract perceptual information: Whenever one feature-dimension is selected for entry into VWM, the others are also extracted. Currently most studies revealing OBE probed an 'irrelevant-change distracting effect', where changes of irrelevant-features dramatically affected the performance of the target feature. However, the existence of irrelevant-feature change may affect participants' processing manner, leading to a false-positive result. The current study conducted a strict examination of OBE in VWM, by probing whether irrelevant-features guided the deployment of attention in visual search. The participants memorized an object's colour yet ignored shape and concurrently performed a visual-search task. They searched for a target line among distractor lines, each embedded within a different object. One object in the search display could match the shape, colour, or both dimensions of the memory item, but this object never contained the target line. Relative to a neutral baseline, where there was no match between the memory and search displays, search time was significantly prolonged in all match conditions, regardless of whether the memory item was displayed for 100 or 1000 ms. These results suggest that task-irrelevant shape was extracted into VWM, supporting OBE in VWM.

An Event Related Potentials Study of the Effects of Age, Load and Maintenance Duration on Working Memory Recognition

Age-related decline in cognitive capacities has been attributed to a generalized slowing of processing speed and a reduction in working memory (WM) capacity. Nevertheless, it is unclear how age affects visuospatial WM recognition and its underlying brain electrical activity. Whether age modulates the effects of memory load or information maintenance duration, which determine the limits of WM, remains also elusive. In this exploratory study, performance in a delayed match to sample task declined with age, particularly in conditions with high memory load. Event related potentials analysis revealed longer N2 and P300 latencies in old than in young adults during WM recognition, which may reflect slowing of stimulus evaluation and classification processes, respectively. Although there were no differences between groups in N2 or P300 amplitudes, the latter was more homogeneously distributed in old than in young adults, which may indicate an age-related increased reliance in frontal vs parietal resources during WM recognition. This was further supported by an age-related reduced posterior cingulate activation and increased superior frontal gyrus activation revealed through standardized low resolution electromagnetic tomography. Memory load and maintenance duration effects on brain activity were similar in both age groups. These behavioral and electrophysiological results add evidence in support of age-related decline in WM recognition theories, with a slowing of processing speed that may be limited to stimulus evaluation and categorization processes -with no effects on perceptual processes- and a posterior to anterior shift in the recruitment of neural resources.

Costs of storing colour and complex shape in visual working memory: Insights from pupil size and slow waves

We investigated the impact of perceptual processing demands on visual working memory of coloured complex random polygons during change detection. Processing load was assessed by pupil size (Exp. 1) and additionally slow wave potentials (Exp. 2). Task difficulty was manipulated by presenting different set sizes (1, 2, 4 items) and by making different features (colour, shape, or both) task-relevant. Memory performance in the colour condition was better than in the shape and both condition which did not differ. Pupil dilation and the posterior N1 increased with set size independent of type of feature. In contrast, slow waves and a posterior P2 component showed set size effects but only if shape was task-relevant. In the colour condition slow waves did not vary with set size. We suggest that pupil size and N1 indicates different states of attentional effort corresponding to the number of presented items. In contrast, slow waves reflect processes related to encoding and maintenance strategies. The observation that their potentials vary with the type of feature (simple colour versus complex shape) indicates that perceptual complexity already influences encoding and storage and not only comparison of targets with memory entries at the moment of testing.

Coarse-to-fine construction for high-resolution representation in visual working memory

Background

This study explored whether the high-resolution representations created by visual working memory (VWM) are constructed in a coarse-to-fine or all-or-none manner. The coarse-to-fine hypothesis suggests that coarse information precedes detailed information in entering VWM and that its resolution increases along with the processing time of the memory array, whereas the all-or-none hypothesis claims that either both enter into VWM simultaneously, or neither does.

Methodology/Principal Findings

We tested the two hypotheses by asking participants to remember two or four complex objects. An ERP component, contralateral delay activity (CDA), was used as the neural marker. CDA is higher for four objects than for two objects when coarse information is primarily extracted; yet, this CDA difference vanishes when detailed information is encoded. Experiment 1 manipulated the comparison difficulty of the task under a 500-ms exposure time to determine a condition in which the detailed information was maintained. No CDA difference was found between two and four objects, even in an easy-comparison condition. Thus, Experiment 2 manipulated the memory array’s exposure time under the easy-comparison condition and found a significant CDA difference at 100 ms while replicating Experiment 1′s results at 500 ms. In Experiment 3, the 500-ms memory array was blurred to block the detailed information; this manipulation reestablished a significant CDA difference.

Conclusions/Significance

These findings suggest that the creation of high-resolution representations in VWM is a coarse-to-fine process.

The time course of psychological stress as revealed by event-related potentials

Psychological stress is common in everyday life and is believed to affect emotion, cognition and health. Previous brain imaging studies have been able to identify the brain regions involved in the stress response. However, our understanding of the temporal neurological response to psychological stress is limited. The present work aims to investigate the time course of psychological stress induced by a mental arithmetic task, utilizing event-related potentials (ERPs). The elicitation of stress was verified by self-reports of stress and increases in salivary cortisol levels. The subjective and physiological data showed that the stress-elicitation paradigm successfully induced a mild-to-moderate level of psychological stress. The electrophysiological data showed that the amplitude of occipital N1 was more negative in the control task than in the stress task, and the latency of frontal P2 was shorter in the stress task than in the control task. Our results provide electrophysiological evidence that psychological stress occurs primarily at the early stage of cognitive processing.

The neural mechanisms of percept-memory comparison in visual working memory

Researchers have revealed that comparing the perceptual input with the representations stored in visual working memory initiates a rapid attention-shift, which is predominantly triggered by the relevant-feature change. The comprehension of the change contents further necessitates a follow-up comparison that contrasts all the object features regardless of the task relevancy. However, whether such a distinct stage exists and how the process is carried on need further verification. We explored this issue by investigating the underlying neural mechanisms of the percept-memory comparison. By recording EEG, we found that both the task-relevant and -irrelevant feature changes elicited significantly more negative anterior N2 waves (230-340ms) rooting in the anterior cingulate cortex (ACC), and meanwhile activated the frontal theta (5-8Hz, 250-550ms). These results suggest that a distinct comparison stage does exist, which is supported by the anterior N2, ACC and frontal theta.

Visual working memory capacity does not modulate the feature-based information filtering in visual working memory

Background

The limited capacity of visual working memory (VWM) requires us to select the task relevant information and filter out the irrelevant information efficiently. Previous studies showed that the individual differences in VWM capacity dramatically influenced the way we filtered out the distracters displayed in distinct spatial-locations: low-capacity individuals were poorer at filtering them out than the high-capacity ones. However, when the target and distracting information pertain to the same object (i.e., multiple-featured object), whether the VWM capacity modulates the feature-based filtering remains unknown.

Methodology/Principal Findings

We explored this issue mainly based on one of our recent studies, in which we asked the participants to remember three colors of colored-shapes or colored-landolt-Cs while using two types of task irrelevant information. We found that the irrelevant high-discriminable information could not be filtered out during the extraction of VWM but the irrelevant fine-grained information could be. We added 8 extra participants to the original 16 participants and then split the overall 24 participants into low- and high-VWM capacity groups. We found that regardless of the VWM capacity, the irrelevant high-discriminable information was selected into VWM, whereas the irrelevant fine-grained information was filtered out. The latter finding was further corroborated in a second experiment in which the participants were required to remember one colored-landolt-C and a more strict control was exerted over the VWM capacity.

Conclusions/Significance

We conclude that VWM capacity did not modulate the feature-based filtering in VWM.