Investigating the effects of perceptual complexity versus conceptual meaning on the object benefit in visual working memory

Previous research has demonstrated greater visual working memory (VWM) performance for real-world objects compared with simple features. Greater amplitudes of the contralateral delay activity (CDA)-a sustained event-related potential measured during the delay period of a VWM task-have also been noted for meaningful stimuli, despite being thought of as a neural marker of a fixed working memory capacity. The current study aimed to elucidate the factors underlying improved memory performance for real-world objects by isolating the relative contributions of perceptual complexity (i.e., number of visual features) and conceptual meaning (i.e., availability of semantic, meaningful features). Participants (N = 22) performed a lateralized VWM task to test their memory of intact real-world objects, scrambled real-world objects and colours. The CDA was measured during both encoding and WM retention intervals (600-1000 ms and 1300-1700 ms poststimulus onset, respectively), and behavioural performance was estimated by using d' (memory strength in a two-alternative forced choice task). Behavioural results revealed significantly better performance within-subjects for real-world objects relative to scrambled objects and colours, with no difference between colours and scrambled objects. The amplitude of the CDA was also largest for intact real-world objects, with no difference in magnitude for scrambled objects and colours, during working memory maintenance. However, during memory encoding, both the colours and intact real-world objects had significantly greater amplitudes than scrambled objects and were comparable in magnitude. Overall, findings suggest that conceptual meaning (semantics) supports the memory benefit for real-world objects.

The role of visual working memory in capacity-limited cross-modal ensemble coding

Ensemble coding refers to the brain's ability to rapidly extract summary statistics, such as average size and average cost, from a large set of visual stimuli. Although ensemble coding is thought to circumvent a capacity limit of visual working memory, we recently observed a VWM-like capacity limit in an ensemble task where observers extracted the average sweetness of groups of food pictures (i.e., they could only integrate information from four out of six available items), thus suggesting the involvement of VWM in this novel form of cross-modal ensemble coding. Therefore, across two experiments we investigated if this cross-modal ensemble capacity limit could be explained by individual differences in VWM processing. To test this, observers performed both an ensemble task and a VWM task, and we determined 1) how much information they integrated into their ensemble percepts, and 2) how much information they remembered from those displays. Interestingly, we found that individual differences in VWM capacity did not explain differences in performance on the ensemble coding task (i.e., high-capacity individuals did not have significantly higher "ensemble abilities" than low-capacity individuals). While our data cannot definitively state whether or not VWM is necessary to perform the ensemble task, we conclude that it is certainly not sufficient to support this cognitive process. We speculate that the capacity limit may be explained by 1) a bottleneck at the perceptual stage (i.e., a failure to process multiple visual features across multiple items, as there are no singular features that convey taste), or 2) the interaction of multiple cognitive systems (e.g., VWM, gustatory working memory, long term memory). Our results highlight the importance of examining ensemble perception across multiple sensory and cognitive domains to provide a clearer picture of the mechanisms underlying everyday behavior.

Effects of eye and hand movement on cross-modal memory

In this study, we determined whether information in eye-hand/hand-eye cross-modal memory is maintained by the input modality used for encoding, the output modality used for testing, or both. In experiments, two categories of effect were examined: facilitatory, produced by rehearsal work with eyes or hands corresponding to the movement of the stimulus to be memorized after its presentation, and interference, formed through the performance of a noncorresponding movement. The results indicated that both the eye and hand facilitated the eye-hand cross-modal memory tasks (Experiment 1A), confirming that both serve a rehearsal function. Subsequently, we conducted an interference effect experiment (Experiment 1B) using the same memory task as that used in Experiment 1A and found that neither modality produced interference effects. This result indicates that information was preserved via output-modality-specific representations when the eye-interference task interfered with the information retention of input-modality-specific representations and via input-modality-specific representations when the hand-interference task interfered with the information retention of output-modality-specific representations. We observed the same facilitation and disappearance of interference effects for the hand-eye cross-modal memory task (Experiments 2A and 2B). In the eye-hand/hand-eye cross-modal memory tasks, the effects of eye and hand rehearsals were found to be comparable, which indicated that the two types of representations functioned together during encoding and testing. From the disappearance of interference effects, the possibility that modality-specific representations have functional aspects arises.

Sensory Delay Activity: More than an Electrophysiological Index of Working Memory Load

Sustained contralateral delay activity emerges in the retention period of working memory (WM) tasks and has been commonly interpreted as an electrophysiological index of the number of items held in a discrete-capacity WM resource. More recent findings indicate that these visual and tactile components are sensitive to various cognitive operations beyond the storage of discrete items in WM. In this Perspective, we present recent evidence from unisensory and multisensory visual and tactile WM tasks suggesting that, in addition to memory load, sensory delay activity may also be indicative of attentional and executive processes, as well as reflecting the flexible, rather than discrete, allocation of a continuous WM resource. Together, these findings challenge the traditional model of the functional significance of the contralateral delay activity as a pure measure of item load, and suggest that it may also reflect executive, attentional, and perceptual mechanisms operating in hierarchically organized WM systems.

What makes somatosensory short-term memory maintenance effective? An EEG study comparing contralateral delay activity between sighted participants and participants who are blind

Somatosensory short-term memory is essential for object recognition, sensorimotor learning, and, especially, Braille reading for people who are blind. This study examined how visual sensory deprivation and a compensatory focus on somatosensory information influences memory processes in this domain. We measured slow cortical negativity developing during short-term tactile memory maintenance (tactile contralateral delay activity, tCDA) in frontal and somatosensory areas while a sample of 24 sighted participants and 22 participants who are blind completed a tactile change-detection task where varying loads of Braille pin patterns served as stimuli. Auditory cues, appearing at varying latencies between sample arrays, could be used to reduce memory demands during maintenance. Participants who are blind (trained Braille readers) outperformed sighted participants behaviorally. In addition, while task-related frontal activation featured in both groups, participants who are blind uniquely showed higher tCDA amplitudes specifically over somatosensory areas. The site specificity of this component's functional relevance in short-term memory maintenance was further supported by somatosensory tCDA amplitudes first correlating across the whole sample with behavioral performance, and secondly showing sensitivity to varying memory load. The results substantiate sensory recruitment models and provide new insights into the effects of visual sensory deprivation on tactile processing. Between-group differences in the interplay between frontal and somatosensory areas during somatosensory maintenance also suggest that efficient maintenance of complex tactile stimuli in short-term memory is primarily facilitated by lateralized activity in somatosensory cortex.

Multisensory Rather than Unisensory Representations Contribute to Statistical Context Learning in Tactile Search

Using a combination of behavioral and EEG measures in a tactile odd-one-out search task with collocated visual items, we investigated the mechanisms underlying facilitation of search by repeated (vs. nonrepeated) spatial distractor-target configurations ("contextual cueing") when either the tactile (same-modality) or the visual array (different-modality) context was predictive of the location of the tactile singleton target. Importantly, in both conditions, the stimulation was multisensory, consisting of tactile plus visual items, although the target was singled out in the tactile modality and so the visual items were task-irrelevant. We found that when the predictive context was tactile, facilitation of search RTs by repeated configurations was accompanied by, and correlated with, enhanced lateralized ERP markers of pre-attentive (N1, N2) and, respectively focal-attentional processing (contralateral delay activity) not only over central ("somatosensory"), but also posterior ("visual") electrode sites, although the ERP effects were less marked over visual cortex. A similar pattern-of facilitated RTs and enhanced lateralized (N2 and contralateral delay activity) ERP components-was found when the predictive context was visual, although the ERP effects were less marked over somatosensory cortex. These findings indicate that both somatosensory and visual cortical regions contribute to the more efficient processing of the tactile target in repeated stimulus arrays, although their involvement is differentially weighted depending on the sensory modality that contains the predictive information.

Attention effects in working memory that are asymmetric across sensory modalities

A key unanswered question about working memory is the nature of interference between items. At one extreme of existing theories, interference occurs between any two items because of a general capacity limit. At another extreme, interference depends on the similarity between particular features of different items. We examine this question in three experiments by presenting two sets of items on each trial, comprising tones or colors, with three levels of similarity between the two sets: cross-modal, unimodal with different marking features (two different musical instruments or shapes), and unimodal with the same marking feature. Another question is the extent to which the entry of presented items into working memory is obligatory or optional, which we examined by requiring retention of the first, the second, or both sets of stimuli for a recognition test shortly after the presentation of the two sets. The combination of the set similarity and attention manipulations allows us to draw conclusions about the nature of working-memory storage. The findings were not entirely in accord with any pre-existing theory. The effects of feature similarity were present in both modalities but more pronounced for sounds, whereas the detrimental effects of attention to both sets for retention occurred only for visual stimuli. Based on the findings we suggest a new, hybrid conception of working memory storage.

The tactile Eriksen flanker effect: A time course analysis

In the present paper, we explore the temporal dynamics of the flanker effect in the tactile modality and compare it with findings in the visual modality. We created a tactile version of the flanker task in which we presented groups of dots oriented either vertically or horizontally to a participant's fingertips. The target stimulus was presented to the middle finger, while the flanker stimuli were presented to the index and the ring fingers. Distributional analyses of the latency data show a tactile flanker effect: an overall facilitation for congruent target-flanker trials relative to incongruent trials, and a higher effect size for slower correct responses (higher quantiles) than for faster correct responses (lower quantiles). While this congruency effect may resemble the one typically reported in the visual modality, there are differences between these two modalities as well, mainly related to the relative speed of error and correct responses. These differences suggest that some of the mechanisms responsible for the flanker effect differ across sensory modalities.

Encoding, storage, and response preparation-Distinct EEG correlates of stimulus and action representations in working memory

Working memory (WM) allows for the active storage of stimulus- and higher level representations, such as action plans. This electroencephalography (EEG) study investigated the specific electrophysiological correlates dissociating action-related from stimulus-related representations in WM using three different experimental conditions based on the same stimulus material. In the experiment, a random sequence of single numbers (from 1 to 6) was presented and participants had to indicate whether the current number (N0 condition), the preceding number (N-1 condition), or the sum of the current and the preceding number (S-1 condition) was odd or even. Accordingly, participants had to store a stimulus representation in S-1 and an action representation in N-1 until the onset of the next stimulus. In the EEG, the storage of stimulus representations (S-1) was reflected by a fronto-central slow wave indicating the rehearsal of information that was required for the response in the following trial. In contrast, the storage of action representations (N-1) went along with a posterior positive slow wave, suggesting that the action plan was actively stored in WM until the presentation of the next stimulus. Crucially, preparing for the next response in N-1 was associated with increased contralateral mu/beta suppression, predicting the response time in the given trial. Our findings, thus, show that the WM processes for stimulus- and action representations can be clearly dissociated from each other with a distinct sequence of EEG correlates for encoding, storage, and response preparation.

Shifts of Spatial Attention in Visual and Tactile Working Memory are Controlled by Independent Modality-Specific Mechanisms

The question whether the attentional control of working memory (WM) is shared across sensory modalities remains controversial. Here, we investigated whether attention shifts in visual and tactile WM are regulated independently. Participants memorized visual and tactile targets in a first memory sample set (S1) before encoding targets in a second sample set (S2). Importantly, visual or tactile S2 targets could appear on the same side as the corresponding S1 targets, or on opposite sides, thus, requiring shifts of spatial attention in visual or tactile WM. The activation of WM representations in modality-specific visual and somatosensory areas was tracked by recording visual and tactile contralateral delay activity (CDA/tCDA). CDA/tCDA components emerged contralateral to the side of visual or tactile S1 targets, and reversed polarity when S2 targets in the same modality appeared on the opposite side. Critically, the visual CDA was unaffected by the presence versus absence of concurrent attention shifts in tactile WM, and the tactile CDA remained insensitive to visual attention shifts. Visual and tactile WM performance was also not modulated by attention shifts in the other modality. These results show that the dynamic control of visual and tactile WM activation processes operates in an independent modality-specific fashion.

Retrospective Selection in Visual and Tactile Working Memory Is Mediated by Shared Control Mechanisms

Selective attention regulates the activation of working memory (WM) representations. Retro-cues, presented after memory sample stimuli have been stored, modulate these activation states by triggering shifts of attention to task-relevant samples. Here, we investigated whether the control of such attention shifts is modality-specific or shared across sensory modalities. Participants memorized bilateral tactile and visual sample stimuli before an auditory retro-cue indicated which visual and tactile stimuli had to be retained. Critically, these cued samples were located on the same side or opposite sides, thus requiring spatially congruent or incongruent attention shifts in tactile and visual WM. To track the attentional selection of retro-cued samples, tactile and visual contralateral delay activities (tCDA and CDA components) were measured. Clear evidence for spatial synergy effects from attention shifts in visual WM on concurrent shifts in tactile WM were observed: Tactile WM performance was impaired, and tCDA components triggered by retro-cues were strongly attenuated on opposite-sides relative to same-side trials. These spatial congruency effects were eliminated when cued attention shifts in tactile WM occurred in the absence of simultaneous shifts within visual WM. Results show that, in contrast to other modality-specific aspects of WM control, concurrent attentional selection processes within tactile and visual WM are mediated by shared supramodal control processes.

The Sources of Dual-task Costs in Multisensory Working Memory Tasks

We investigated the sources of dual-task costs arising in multisensory working memory (WM) tasks, where stimuli from different modalities have to be simultaneously maintained. Performance decrements relative to unimodal single-task baselines have been attributed to a modality-unspecific central WM store, but such costs could also reflect increased demands on central executive processes involved in dual-task coordination. To compare these hypotheses, we asked participants to maintain two, three, or four visual items. Unimodal trials, where only this visual task was performed, and bimodal trials, where a concurrent tactile WM task required the additional maintenance of two tactile items, were randomly intermixed. We measured the visual and tactile contralateral delay activity (CDA/tCDA components) as markers of WM maintenance in visual and somatosensory areas. There were reliable dual-task costs, as visual CDA components were reduced in size and visual WM accuracy was impaired on bimodal relative to unimodal trials. However, these costs did not depend on visual load, which caused identical CDA modulations in unimodal and bimodal trials, suggesting that memorizing tactile items did not reduce the number of visual items that could be maintained. Visual load did not also affect tCDA amplitudes. These findings indicate that bimodal dual-task costs do not result from a competition between multisensory items for shared storage capacity. Instead, these costs reflect generic limitations of executive control mechanisms that coordinate multiple cognitive processes in dual tasks. Our results support hierarchical models of WM, where distributed maintenance processes with modality-specific capacity limitations are controlled by a central executive mechanism.

Common and Unique Neural Systems Underlying the Working Memory Maintenance of Emotional vs. Bodily Reactions to Affective Stimuli: The Moderating Role of Trait Emotional Awareness

Many leading theories suggest that the neural processes underlying the experience of one's own emotional reactions partially overlap with those underlying bodily perception (i.e., interoception, somatosensation, and proprioception). However, the goal-directed maintenance of one's own emotions in working memory (EWM) has not yet been compared to WM maintenance of one's own bodily reactions (BWM). In this study, we contrasted WM maintenance of emotional vs. bodily reactions to affective stimuli in 26 healthy individuals while they underwent functional magnetic resonance imaging. Specifically, we examined the a priori hypothesis that individual differences in trait emotional awareness (tEA) would lead to greater differences between these two WM conditions within medial prefrontal cortex (MPFC). We observed that MPFC activation during EWM (relative to BWM) was positively associated with tEA. Whole-brain analyses otherwise suggested considerable similarity in the neural activation patterns associated with EWM and BWM. In conjunction with previous literature, our findings not only support a central role of body state representation/maintenance in EWM, but also suggest greater engagement of MPFC-mediated conceptualization processes during EWM in those with higher tEA.

Dual Task Effects on Visual Attention Capacity in Normal Aging

Older adults show higher dual task performance decrements than younger adults. While this is assumed to be related to attentional capacity reductions, the precise affected functions are not specified. Such specification is, however, possible based on the "theory of visual attention" (TVA) which allows for modeling of distinct attentional capacity parameters. Furthermore, it is unclear whether older adults show qualitatively different attentional effects or whether they show the same effects as younger adults experience under more challenging conditions. By varying the complexity of the secondary task, it is possible to address this question. In our study, participants performed a verbal whole report of briefly presented letter arrays. TVA-based fitting of report performance delivered parameters of visual threshold t0, processing speed C, and visual short-term memory (VSTM) storage capacity K. Furthermore, participants performed a concurrent motor task consisting of continuous tapping of a (simple or complex) sequence. Both TVA and tapping tasks were performed under single and dual task conditions. Two groups of 30 younger adults each performed either the simple or complex tapping, and a group of 30 older adults performed the simple tapping condition. In older participants, VSTM storage capacity declined under dual task conditions. While no such effect was found in younger subjects performing the simple tapping sequence under dual task conditions, the younger group performing the complex tapping task under dual task conditions also showed a significant VSTM capacity reduction. Generally, no significant effect on other TVA parameters or on tapping accuracy was found. Comparable goodness-of-fit measures were obtained for the TVA modeling data in single and dual tasks, indicating that tasks were executed in a qualitatively similar, continuous manner, although quantitatively less efficiently under dual- compared to single-task conditions. Taken together, our results show that the age-specific effects of motor-cognitive dual task interference are reflected by a stronger decline of VSTM storage capacity. They support an interpretation of VSTM as central attentional capacity, which is shared across visual uptake and concurrent motor performance. Capacity limits are reached earlier, and already under lower motor task complexity, in older compared to younger adults.